<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alexander Hirner on Medium]]></title>
        <description><![CDATA[Stories by Alexander Hirner on Medium]]></description>
        <link>https://medium.com/@alex_hirner?source=rss-48c5c152b8d0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*h0THqayDgGmxwWH6.jpeg</url>
            <title>Stories by Alexander Hirner on Medium</title>
            <link>https://medium.com/@alex_hirner?source=rss-48c5c152b8d0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 15:50:15 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alex_hirner/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Few-shot Object Detection in Practice]]></title>
            <link>https://medium.com/moonvision/few-shot-object-detection-in-practice-4f8fa98cba57?source=rss-48c5c152b8d0------2</link>
            <guid isPermaLink="false">https://medium.com/p/4f8fa98cba57</guid>
            <category><![CDATA[active-learning]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[few-shot-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Alexander Hirner]]></dc:creator>
            <pubDate>Fri, 12 Apr 2019 12:04:28 GMT</pubDate>
            <atom:updated>2019-04-12T12:04:28.245Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/959/0*DQhsAam-KCSDDBdX" /><figcaption><em>Fig. 1. Object detection example for </em><a href="https://www.dishtracker.at/?lang=en"><em>dishtracker</em></a><em> and the highest resolution activation maps for background and 10 foreground classes.</em></figcaption></figure><p>Object detection is vital to automate manual tasks, such as checking the completeness of objects and the exact types of its parts. In contrast to segmentation, objects are located and classified as discrete instances. This is achieved by decoding regression and activation maps after a cascade of convolutions. You can read more about state-of-the art in object detection in <a href="https://arxiv.org/abs/1809.03193v1">this survey</a>.</p><p>However, contemporary issues in object detection are often studied in isolation. In production use cases though, multiple constraints must be solved at once. In this post, we describe the combination of techniques that we’ve developed over time that meet many of these constraints.</p><h3>Problem</h3><p>As with any machine learning task, the amount of training data is limited. As we will review below, there are many approaches to conquer low-data scenarios, each with its own remaining problems.</p><p>First, raw or weakly supervised data might be available plentifully (think of social media photos or passively recorded video). Such data can be mined by iterative methods: from few ground-truth samples you can generate new training samples and start retraining from the expanded set (<a href="https://arxiv.org/abs/1706.08249">fully supervised</a> or <a href="http://openaccess.thecvf.com/content_ECCV_2018/papers/Peng_Tang_Weakly_Supervised_Region_ECCV_2018_paper.pdf">weakly supervised</a>). The remaining problem is to achieve <strong>competitive accuracy</strong> with this approach, especially regarding positions and sizes of the bounding boxes. These so called <a href="https://drive.google.com/a/moonvision.io/open?id=14k2Zy-4NylBKQWKD_Wak45Ltmsojgmu8xOJGjGzw6oc">IoU metrics</a> are often underreported in benchmarks.</p><p>Secondly, if your target domain is similar to a richly annotated source domain, you can transfer detection know-how from that source domain. Concretely, localizing objects benefits greatly from such techniques and can be done with different but often complicated means (<a href="https://arxiv.org/abs/1803.01529">domain transfer with regularization</a>, <a href="https://arxiv.org/abs/1903.09372v1">meta-learning to transfer</a>). Of course, this technique fails if there is no <strong>properly annotated source domain</strong>. More exotic approaches try to localize and classify by searching for <a href="https://arxiv.org/abs/1801.03329">similarity in latent space</a>. This however has a high computational cost per class and is limited to single instances or fragile bounding shapes based on thresholding operations.</p><p>Another problem, often neglected in works that focuses on low-data regimes, are <strong>unbalanced classes</strong>. This means that there are many more examples of some classes than for others (think Catahoula Leopard Dog vs. German Shepherd). The problem cannot be fully solved by simply <a href="https://imbalanced-learn.readthedocs.io/en/stable/user_guide.html">over- or undersampling</a>, because rare items co-occur with frequent items in the same image.</p><p>A fourth problem is that of <strong>extensibility</strong>. If you finally have a well trained model for a set of classes, it’s hard to add new ones, even if it’s just a special case to a common category.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/742/1*KKwhfxnOmfBW-B0R1HIw8w.png" /><figcaption><em>Fig. 2from left to right: Instance ID, trained super class (text color), detailed label in text, visualization of ROI-align features (leftmost is actual instance, right are its nearest neighbors). If the 10x10 feature map has a light stripe it’s of the same instance as on the left. Each 10x10 feature is reduced to three channels (RGB) by PCA. </em>The model was trained on 10 super classes out of 76 highly unbalanced classes.</figcaption></figure><p>Our experiments in Summer 2018 showed that <a href="https://arxiv.org/abs/1703.06870">ROI-align based descriptors</a> eradicate features between different instances of the same class. The visualization in Fig. 1 shows that ROI-align features from an SSD model are insensitive to differences between instances of the same super class (similar color). Thus intra-class variance cannot be inferred from such features without <a href="https://arxiv.org/abs/1612.08242">full retraining on the annotated hierarchy</a> or the risk of catastrophic forgetting. Taken to the extreme, every new example can come from <a href="https://arxiv.org/abs/1811.08581">entirely unknown classes</a>.</p><h3>Solution</h3><p>We address the remaining problems mentioned above by a pipeline that combines <em>object mining,</em> <em>fast annotation</em> and <em>few-shot</em> classification.</p><h4>Object mining</h4><p>Our domains are often new-to-the-world. Thus the most generic way to acquire raw data is <strong>video</strong>. We use a pretrained model to mine objects if there is a similar one. On an entirely new domain, hand-engineered detectors or motion patterns can be used instead. In one case, we processed 600GB of videos and produced over 12.000 object candidates automatically to bootstrap that process. Unlike seeding some ground truth examples only in the beginning, we ask experts on demand for a few examples before iterating that process. This provides a way to add details that don’t exist anywhere yet and allows to evaluate models with high rigor. Thanks to our recommendation system, this annotation process is also fast.</p><h4>Few-show detection</h4><p>As seen above, fast object detectors don’t cope well with unbalanced or evolving data. Once we possess the initial training set in the form of shapes and labels, we automatically divide the hierarchy into groups that have enough in common to be determined at first sight and those which differ in subtle differences. Instances of the latter kind are then classified by additional layers at higher resolution.</p><p>These layers have the objective to transform instances with the same visual attributes to a low dimensional vector within a tight volume in euclidean space or onto the patch of a hypersphere. Conversely, instances with different attributes occupy a distant volume [Metric learning paper].</p><p>With such a metric, you can easily classify unseen instances by nearest-neighbor search, build classifiers with guaranteed convergence like SVMs or determine the novelty of incoming examples. The exact way to create these embedding layers warrants another post. However, the full body of know-how regarding unbalanced classification can be utilized rather easily.</p><p><strong>As a result, the combination of smart object mining and few-shot detection has the following properties:</strong></p><ul><li>Effective use of <em>transfer learning</em>, starting from unlabelled video</li><li>Robust against class <em>imbalance</em> and <em>overfitting</em></li><li>Tractable for <em>realtime</em> use</li><li><em>Ability to update</em> classes in near realtime</li><li>Trace back a classification result to a single instance in the training data for better model <em>introspection</em></li></ul><h3>Applications and Outlook</h3><p>The problem of detecting a quickly rotating set classes was used in many applications. In the case of counting dishes, meals and some items appear only rarely and change daily. Here, having just a few (as low as 1) examples for training is paramount to automate the checkout process. Similarly, for our automated checkout solution at <a href="https://www.derbrutkasten.com/moonvision-hotel-sacher/">Sacher</a>, we were able to accommodate over 50 items from just 2h of raw video data. Finally, many industrial assembly tasks require checking an ever changing catalogue of items.</p><p>The approach outlined above allows us to focus computation on the details that matter whilst learning continuously. Moreover, the pipeline is very flexible, so that we were able to integrate ever more advanced techniques like generative adversarial training (GAN) easily. <strong>Many more improvements are on the way.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4f8fa98cba57" width="1" height="1" alt=""><hr><p><a href="https://medium.com/moonvision/few-shot-object-detection-in-practice-4f8fa98cba57">Few-shot Object Detection in Practice</a> was originally published in <a href="https://medium.com/moonvision">Moonvision</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[One Year at MoonVision]]></title>
            <link>https://medium.com/@alex_hirner/one-year-at-moonvision-5bcfa76c9ea6?source=rss-48c5c152b8d0------2</link>
            <guid isPermaLink="false">https://medium.com/p/5bcfa76c9ea6</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[quality-assurance]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Alexander Hirner]]></dc:creator>
            <pubDate>Tue, 01 Jan 2019 23:16:10 GMT</pubDate>
            <atom:updated>2019-01-01T23:42:07.781Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ypNG16TpViW8bQUyzDjDsA.png" /></figure><p>2018 was an ambitious year for <a href="https://moonvision.io/">MoonVision</a>. We accomplished a lot on getting the word out, developed exciting technology and got new people working with us on next level machine vision.</p><p>For those interested in our journey, I’d like to share personal highlights and a small glimpse into 2019.</p><h3>Outreach</h3><p>Early last year, Austria’s Chamber of Commerce did a great job organizing the <a href="https://www.derbrutkasten.com/aermel-hoch-und-hands-bei-der-applied-artificial-intelligence-conference-2018/"><strong>AAIC</strong></a>. In addition to discussing the merits and use case of clustering, I held a workshop on “<em>Beyond Image Classification: How Detection, Segmentation and Instance Segmentation work</em>”. The goal was to educate experts from various domains on modern computer vision methods, specifically how they correspond to an automation problem. At this point I also like to thank enliteAI, who <a href="https://www.enlite.ai/works/ailandscapeaustria">summarized the landscape</a> of technology providers very well!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e1UlXFNrrHj_M8LpeBAOmQ.jpeg" /><figcaption>Panel Discussion on Clustering at AAIC (WKO 2018)</figcaption></figure><p>This year’s <a href="https://eccv2018.org/"><strong>ECCV in Munich</strong></a> was one of the most interesting conferences. It was helpful to discuss details with authors who’s work we build on. Not less important were discussions with tech scouts from OEMs and larger companies. At this point it was already clear that the demand for self-driving car peaked, which helped to get a decent influx of interested talent as well.</p><p>At the <a href="https://www.messe-stuttgart.de/vision/en/"><strong>VISION 2018 in Stuttgart</strong></a> we not only got to know new customers and partners, especially on image acquisition hardware. Overall, the VISION is truly unique in bringing together the world’s most important suppliers into one spot. We are already looking forward to the next one.</p><p>Overall, we always learned something, be it in the DACH region, Warsaw and even <a href="https://www.linkedin.com/feed/update/urn:li:activity:6480363961228230656">Silicon Valley</a>. Our outreach was crucial to get exposure and the assurance of tackling computer vision problems the right way, for example by solving issues that are usually neglected in academic research or competing approaches.</p><h3>Technology</h3><p>In order to solve sophisticated inspection tasks, you need three ingredients:</p><ul><li>Transfer know-how from experts to machine vision</li><li>Continuously learn from that input</li><li>Integration with machines and software on the shop floor</li></ul><p>At one point, we were asked to reconcile an end-to-end system with best the practices from quality assurance. This poses an interesting challenge, because now it’s not only about “<em>machine-based assurance</em>” (which can be measured by some established metrics) but “<em>process assurance</em>” which includes input from multiple experts. Thus, we have developed a rigorous data management approach that can map compliance metrics to an evolving multi-user history.</p><p>Alongside, we achieved new highs in terms of data efficiency. At the extreme, this means detecting a new type of object with only one example. We achieve that by breaking detection problems down into a decision tree, thereby combining the best of deep learning and program composition. To implement such algorithms, we use cutting-edge tools like <a href="https://pytorch.org/"><strong>PyTorch</strong></a>. For example, we maintain a <a href="https://github.com/ahirner/vision/tree/layers">branch of torchvision</a> which enables compiled ROI-align operations (useful for composing different inference tasks) with the latest version of PyTorch. I’m very excited to contribute more to sophisticated and upcoming <a href="https://pytorch.org/ecosystem">tools</a>. For those interested in state of the art tools, I recommend to watch the very informative <a href="https://www.youtube.com/watch?v=KJAnSyB6mME&amp;list=PLnu7sDRFQNk8mSzF4JgzAAfK1CLizYS_K">talks of PyTorch’s 1.0 release conference</a>!</p><p>The end of the year was marked with integrating into production systems. This means two things. First, models developed in the cloud have to run at the edge on constrained hardware. We therefore extended our software stack with offline inference and platform specific optimizations. For certain hardware, this gave us close to 20x performance gains. Second, the number of supported image acquisition sources needed to grow. Therefore, we now support the <a href="https://www.emva.org/standards-technology/genicam/"><strong>GeniCam</strong></a> standard. First released in 2006, the standard enables us configuration and image acquisition from many industrial cameras like <strong>Basler</strong>, <strong>Lucid</strong>, <strong>FLIR</strong>, <strong>Matrix Vision</strong> and so on.</p><h3>People</h3><p>We are to happy to have filled the most critical roles in data engineering, data science and full stack software development this year. Also, I think it’s fair to call us the most diverse team within a big radius, which I’m truly grateful to work with. It took a good amount of time and experimentation, but now I think the combined skill set makes perfect sense.</p><p>As the team and projects grew, so did our engineering stack. We increased test coverage by leaps and bounds, adopted git-flow as a solid foundation for versioning and steadily increased the depth of our build pipeline. One important credo at MoonVision is learning from each other. Hence we encourage team members to present their insights. For example, our lead data engineer initated <em>containerized deployment and scaling</em>, which makes us more productive and is worth sharing internally and at <a href="https://www.pydays.at/speakers/">PyDays.at</a>!</p><p>Moreover, we keep push limits in other ways too and seek an extraordinary challenge from time to time. Therefore, we participated in a couple of hackathons with a good degree of success. That’s definitely a thing we’ll continue in 2019.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*mZesMeT31pzzrWT4AMuFhA.jpeg" /><figcaption>MoonVision devs with Phoenix Contact sponsors at <a href="https://www.elektrotechnik.vogel.de/index.cfm?pid=15935">www.automation-hackathon.de</a> (Vogel 2018)</figcaption></figure><p>Last but not least, I’d like to welcome <a href="https://github.com/CarlosUziel">Carlos</a> as our newest research engineer!</p><h3>Outlook</h3><p>In 2019 we’ll continue our mission of automating visual inspection.</p><p>One key factor in enabling that at scale is data efficiency. There are fundamentally two categories to improve data efficiency: transferring know-how from differentiable programs that solved a similar problem and using efficient priors for detection algorithms that are specific to a given task (or meta-priors as Bengio coined them, see <a href="https://arxiv.org/abs/1812.05069">Tschannen et al. 2018</a> for a recent overview on unsupervised learning).</p><p>The other key factor is improving interface between human experts and machines to generate ground truth. Users will benefit as we release features that further automate the processing steps between a raw video feed and high-fidelity training data like advanced clustering, active learning and immediate feedback in the UI.</p><p>To execute on that, we are looking for the right people to join in 2019, which leads me to the final <strong>Plug</strong>: If you are a <em>senior computer vision researcher </em>or <em>app engineer</em>, hit me up!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5bcfa76c9ea6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Blockchain Hype and Beyond]]></title>
            <link>https://medium.com/@alex_hirner/blockchain-hype-and-beyond-18e31150aac9?source=rss-48c5c152b8d0------2</link>
            <guid isPermaLink="false">https://medium.com/p/18e31150aac9</guid>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[ethereum]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[market-data]]></category>
            <dc:creator><![CDATA[Alexander Hirner]]></dc:creator>
            <pubDate>Wed, 31 May 2017 09:44:37 GMT</pubDate>
            <atom:updated>2017-05-31T09:44:37.522Z</atom:updated>
            <content:encoded><![CDATA[<p>Today, the majority of consultants skim clients by perpetuating that <em>#blockchain</em> is what is isn’t: an omnipresent, DDoS resistant source of truthful data. Hence, everyone thinks they need <em>“the blockchain”</em> and hence, prototypes are conceived to become the new disruptive platform for X. However, that aim is only met in theory.</p><h3>Dominik Schiener on Twitter</h3><p>We honestly need to start calling out these consultants disguised as &quot;Blockchain specialists&quot;. They add absolute no value to the space.</p><p>First, DApps don’t scale on current public blockchains. The mantra of ridding <em>“the middleman” </em>obscures the fact, that <strong>a variable number of competitive, anonymous and slow miners become the middlemen instead</strong>. As an alternative to public blockchains, the PoC is delivered as a database with distributed administrators and some audit capabilities.</p><p>Second, the proposed solutions hardly take into account how valuable data should be shared without regret. Slowly but surely, such <strong>data-sharing tensions</strong> became the true riddle for <a href="https://www.r3cev.com/s/corda-introductory-whitepaper-final.pdf">R3</a> which in turn implemented that<em> “data is only shared with those parties which are required to see it”. </em>The same problem exists for public blockchains once pseudonyms don’t cut it: You can’t record and process your uber on a self-policing p2p platform if you don’t tell them where to go. If no ride data would be exploitable by stakeholders, how could a decentralized uber ever become a functioning market? There are solutions that maximize pseudonymity guarantees, but eventually, trusted computation on data is needed.</p><p>Those <strong>unmet expectations</strong> led to a few abandoned PoCs that I know of, probably some more and a stream of pivoting blockchain startups. The essence is, that many are looking at the wrong problem. It’s important to solve the scalability issues and we’ll get there. Also, these shared databases (“DLTs”) are becoming <a href="https://www.bigchaindb.com/">more</a> <a href="https://deepmind.com/blog/trust-confidence-verifiable-data-audit/">clever</a>. Even more important though is, why do users share which data, with whom and without regrets on future platforms?</p><h3>Alexander Hirner on Twitter</h3><p>What people think they need: blockchain. What they need: open standards, differential privacy, secure multiparty computation. #consulting</p><p>Roughly two years ago, I realized that <strong>chaining blocks doesn’t solve what is a precondition for most of the hyped use cases</strong>: utilize aggregate data while rewarding contributors in a fair manner. At first, finding the ideal middle ground for a data-access, contribution and reward policies has little to do with technology. However, guaranteeing such a middle ground becomes more and more feasible with technologies like <a href="http://privacytools.seas.harvard.edu/files/privacytools/files/pedagogical-document-dp_0.pdf">differential privacy</a>, <a href="https://research.googleblog.com/2017/04/federated-learning-collaborative.html">federated learning</a> and <a href="https://www.microsoft.com/en-us/research/blog/microsoft-researchers-enable-secure-data-exchange-cloud/">homomorphic encryption</a>. The thoughtful application of those technologies will truly pave the way for cloud technology of the future. <strong>Eventually, intermediaries who use them will win consumers and industry over and make everyone better off.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/900/1*7HDx3P76nncR7fzuS2XB5g.png" /></figure><p>In this vein, I’m looking forward to keep on doing what <strong>Ethereum Vienna </strong>is best at: educating people about such cutting-edge technology. Recently, we’ve had a great talk by Johann Höchtl about the possibilities of homomorphic encryption. It’s one way to have trusted computation on aggregate data, because the processor only sees data in its encrypted form. You can check out the <a href="https://www.slideshare.net/jhoechtl/homomorphic-encryption-on-blockchain-principles">slides</a>, <a href="https://www.meetup.com/Ethereum-Vienna/events/238922694/">the meetup page</a> and might keep one thing in mind: let a <em>blockchain mindset</em> inspire you but figure out data-sharing incentives first for building really substantial PoCs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18e31150aac9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transfer learning with PyTorch]]></title>
            <link>https://medium.com/data-science/transfer-learning-with-pytorch-72a052297c51?source=rss-48c5c152b8d0------2</link>
            <guid isPermaLink="false">https://medium.com/p/72a052297c51</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[cnn]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[benchmark]]></category>
            <dc:creator><![CDATA[Alexander Hirner]]></dc:creator>
            <pubDate>Tue, 30 May 2017 21:09:27 GMT</pubDate>
            <atom:updated>2017-10-04T16:09:15.806Z</atom:updated>
            <content:encoded><![CDATA[<p>The future of machine learning is so bright, because incentives are aligned across the board: big players eagerly <em>open source tools</em> and invest in <em>faster hardware</em> to escape their ad-based business models. Tinkerers discover <em>niche applications</em> unheard of before. Data becomes more fungible for <em>private, public, scientific, leisure and adverse use</em>. I could talk for hours on the perfect storm ahead (and maybe some contrarian thoughts), but lets stick to a practical thing: how to<strong> </strong>utilize the recent uptick of available <strong>pre-trained machine learning models</strong>.</p><h4><strong>Transfer learning</strong></h4><p>Tasks that are related, require a small set of <em>underlying abilities to discriminate unseen data</em>. For example, no matter if you play the guitar or a piano, you will have a better time picking out chords than someone not used to play. In machine learning terms, that means you can copy someone else’s training effort and adapt it to rapidly classify pictures of <a href="https://www.theverge.com/tldr/2017/5/14/15639784/hbo-silicon-valley-not-hotdog-app-download">hotdogs and no hotdogs</a> or let it <a href="https://github.com/ahirner/generating-reviews-discovering-sentiment">generate rather novel product reviews</a>.</p><p><a href="http://sebastianruder.com/transfer-learning/">Sebastian Ruder</a> nicely recapped what compounding gains to expect from model reuse.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pj6I0mkWQFeNyMf0JYYPeA.png" /></figure><p>In particular, transfer learning improves on all dimensions that determine the technical aspects of a machine learning project:</p><ul><li><strong>Human efficiency</strong>. You need experts if you want to squeeze out the last ounce of signal, make models interpretable, tractable and robust. Thanks to academic research, architectures bubble up that are battle tested across related tasks.</li><li><strong>Compute efficiency</strong>. A state of the art paper usually trains around two weeks on clusters of 2 to 8 GPUs. Although, there is no limit really. With transfer learning, you can save on twiddling internal parameters by doing that only partially or less often.</li><li><strong>Data efficiency</strong>. If someone else trained on large datasets (that he or she doesn’t even need to disclose), less domain specific data is needed in most cases. In fact, it’s incredible how you can <a href="https://github.com/lizeng614/SqueezeNet-Neural-Style-Pytorch">repurpose a model</a> contained in less than a 5MB download.</li></ul><h4>Landmark Architectures</h4><p>Transfer learning is widely used to better distinguish specific image categories. <a href="https://arxiv.org/abs/1605.07678">Canziani et al. (2016)</a> compared the compute efficiency for prime architectures when competing on the imagenet dataset.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*-9ft10_jV7gBqfeX9N2vMA.png" /></figure><p>Specifically, you can further divide compute costs into<em> training time, inference time and memory needs</em>, depending on your device constraints. For a recent sidegig, I needed to dig deeper, given very specific constraints.</p><p>First, new data comes in often. Second, those images are also <em>potentially proprietary</em>. Thus, <em>retraining must happen reliably</em> on a local mid-tier GPU without pampering by external experts. From a users’s perspective, retraining is reliable if it gives consistent results in a consistent amount of time, just as if you’d click on a printing job to be done. Therefore, the baseline benchmark will use a simple <em>optimization method, which </em>favours convergence over data efficiency. Third, each prediction needs to happen at near real-time for a batch of around four images. Therefore, we care about <em>inference times</em>. Finally, we care about the exact class of an image (or part of an image) as an input to business decisions. Thus we take the <em>top-1 accuracy</em> into consideration.</p><p>Now, to bring down costs for experts to virtually nil with an evolving dataset and fit compute times to hard constraints, lets look into the model zoo of one of the most coveted tools of the trade: PyTorch.</p><h4>Results</h4><p>There are a few reasons why PyTorch gets its swag nowadays.</p><h3>Pytorch Debugger on Twitter</h3><p>TensorFlow</p><p>Fair enough: the framework is reasonably complete, succinct, defines the entire computation graph dynamically in code and is easy to debug. Six archetypes can be loaded with a single line of code from the torchvision package: AlexNet, DenseNets, Inception, SqueezeNet and VGG. To get an overview how they came about, I recommend <a href="https://adeshpande3.github.io/adeshpande3.github.io/The-9-Deep-Learning-Papers-You-Need-To-Know-About.html">this read</a>.</p><p>Unfortunately, the API does <a href="https://github.com/pytorch/vision/issues/173">not allow</a> to load pre-trained models with a custom number of classes sans reimplementing final layers manually. Therefore, I wrote a function that solves that in a principled way. It merges pre-trained parameters so that they won’t interfere with custom output layers. Thus, we can systematically compare the performance metrics we care about across all available architectures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*-6h7MHq1wsmoqXYRKCJvvw.png" /><figcaption>SGD with same momentum and learning rate for each model and 15 epochs on 2xK80</figcaption></figure><p>The models in the chart were retrained on final layers only (shallow), for the entire set of parameters (deep) or from its initialized state (from scratch). In all runs, the dual K80 GPUs ran on about 75%.</p><p>As the data shows, SqueezeNet 1.1 delivers on its promise to be a compute efficient architecture. Because the divide into retrained and static layers is somewhat arbitrary though, a conclusion based solely on shallowly retrained models would be too far fetched. For example, the final classifier of VGG13 has 8194 parameters, whereas ResNet34’s final layer is narrower with 1026 parameters. Therefore, only a hyperparameter search on learning strategies would make comparisons for the given objective truly valid. However, SqueezeNet learned incredibly fast shallow and deeply for the low number of classes in this dataset (binary to be precise).</p><h4>Conclusion</h4><p>One thing to notice is, how deeply retraining yield less accuracy for the same amount of training time than shallow retraining. That happened to me with other models in other tasks, too. My theory on this is, that wringing deeper convolutions from their previous local optima happens slowly on average (small error gradients) and on a less smooth manifold after random initialization. Thus, parameters end up juxtaposed in a transitioning phase. Such an intermittent mess should be especially true if unseen features differ in scale compared to the original data, which is the case for today’s toy dataset: sometimes there are ants up close and sometimes you observe an entire swarm.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*wUhPh6CEtQ3EPhf9hLxxuw.png" /></figure><p>So in fact you’d probably be well advised choose higher learning rates the more layers you tune, or at some point use rather <a href="https://github.com/msracver/Deformable-ConvNets">scale invariant networks</a>.</p><p>For the purpose of assessing compute efficiency composed of inference and retraining times, this is a pretty good start. If you want to increase retraining efficiency further, you can skip <a href="http://pytorch.org/docs/torchvision/transforms.html#transforms-on-pil-image">data augmentation</a> and cache results from untrained layers. Of course, conclusions will differ with the number of classes, other factors mentioned before and the particular constraints. That’s why you want to arrive on your own. I <a href="https://github.com/ahirner/pytorch-retraining">published the raw metrics and accompanying code</a> on github to do so.</p><p>Happy re-training!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=72a052297c51" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/transfer-learning-with-pytorch-72a052297c51">Transfer learning with PyTorch</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>