<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lilit Yolyan on Medium]]></title>
        <description><![CDATA[Stories by Lilit Yolyan on Medium]]></description>
        <link>https://medium.com/@lilityolyan?source=rss-b8a574f7365d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*3mdn-6-d0jRC6uwzM7z6uw.jpeg</url>
            <title>Stories by Lilit Yolyan on Medium</title>
            <link>https://medium.com/@lilityolyan?source=rss-b8a574f7365d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 07:07:42 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lilityolyan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Eight Years In, and the Door Looks Nothing Like It Did]]></title>
            <link>https://lilityolyan.medium.com/eight-years-in-and-the-door-looks-nothing-like-it-did-f2e9045e4fbb?source=rss-b8a574f7365d------2</link>
            <guid isPermaLink="false">https://medium.com/p/f2e9045e4fbb</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Lilit Yolyan]]></dc:creator>
            <pubDate>Sun, 28 Jun 2026 19:27:59 GMT</pubDate>
            <atom:updated>2026-06-30T22:53:45.545Z</atom:updated>
            <content:encoded><![CDATA[<h3>What I’ve watched change in machine learning since I started — in the field, in the work, and in how hard it became for my students to get in</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GGUSI5Ff68cKKg_9" /><figcaption>Photo by <a href="https://unsplash.com/@heyquilia?utm_source=medium&amp;utm_medium=referral">Quilia</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I haven’t written here in a while. The last post on this blog was about implementing computer vision papers, and looking at the date on it, that was a different version of the field — and, in some ways, a different version of me.</p><p>I started in machine learning around eight years ago. For most of that time, I had my head down: writing code, debugging training runs, and shipping models. But I also spent a chunk of those years teaching at Yerevan State University — a course called <em>Deep Learning and Neural Networks</em> — and teaching does something useful to your sense of time. Every year a new cohort walks in with roughly the same hopes, and every year you watch them try to walk out into the same field — except the field isn’t the same. It moved. Standing in front of a classroom, you feel that movement more sharply than you do at your desk.</p><p>So this is a post I’ve wanted to write for a while: what the field looked like when I started, what it looks like now, and how the path <em>in</em> has changed for the people coming after me.</p><h3>When I started, the craft was building things from scratch</h3><p>When I came into the field, deep learning had already had its “ImageNet moment,” and the energy was enormous. If you worked in computer vision — which I did — your world was convolutional networks. You knew your ResNets and your VGGs by heart. You argued about whether batch norm was helping or hiding a problem. GANs were the exciting, slightly unstable new thing everyone wanted to try. Reinforcement learning had just beaten humans at Go — and half the internet didn’t expect the field to change as fast as it did.</p><p>But the thing I remember most isn’t any single architecture. It’s that the <em>unit of work</em> was building.</p><p>To do machine learning well, you were expected to implement things yourself. You read a paper, and then you sat down and reproduced it — wiring up the data loader, getting the loss to actually go down, fighting the small bugs that quietly ruin a model without ever throwing an error. I wrote a whole post once about what I learned doing exactly that, because the act of reimplementation <em>was</em> the education. Kaggle was where you proved you could squeeze signal out of messy data. Feature engineering still mattered. Knowing your math — backprop, optimization, a bit of linear algebra you could actually use — was the price of entry, not a nice-to-have.</p><p>It was a craftsman’s field. Slow, hands-on, occasionally maddening, and deeply satisfying when the curve finally bent the right way.</p><h3>Then the ground shifted under all of us</h3><p>The pivot, if I have to name one, was the move from convolutional networks to transformers, and then from transformers to <em>foundation models</em> — large pretrained models that you adapt rather than build.</p><p>This changed the daily work more than people outside the field realize. Increasingly, the question stopped being “how do I build a model for this problem?” and became “how do I get the most out of a model that already exists?” The center of gravity moved from training to adapting: fine-tuning, prompting, retrieval, evaluation, and stitching models into systems that actually hold up in production. Generative models — large language models, diffusion models for images — went from research curiosities to the default tools. Multimodal systems that handle text, images, and audio together stopped being exotic.</p><p>I want to be careful here, because there’s a lazy version of this story that says “the old skills don’t matter anymore.” That’s not true, and I’d push back on it. The people who understand <em>why</em> a model behaves the way it does — who can read the loss, reason about the data, debug the thing when it quietly breaks — are still the ones you want in the room. What changed is that those skills are now necessary but no longer sufficient. The surface area of the job got much bigger.</p><p>And here’s the part that’s easy to miss from inside a stable job: the <em>entry point</em> into the field changed even more than the field did.</p><h3>What I saw from the front of the classroom</h3><p>When I started teaching, the story I could honestly tell students was simple. Learn the fundamentals, build a couple of solid projects, do well in a few competitions, and the door would open. It usually did.</p><p>Year by year, that story got harder to tell with a straight face.</p><p>Part of it was sheer numbers. Every year more people arrived — more graduates, more bootcamp finishers, more career-switchers — all holding a certificate that said “machine learning” on it. The supply of people who could <em>say</em> they did ML grew much faster than the supply of people who could actually do it under pressure, and employers noticed. A certificate stopped being a signal. Increasingly, the only thing that cut through was evidence: a real project, deployed, on a real system, that you could explain end to end. Portfolios over credentials.</p><p>The other part was harder to explain to a hopeful 22-year-old: the field was automating its own entry-level rung. The tasks that used to be how a junior earned their first paycheck — writing routine code, labeling data, building a simple baseline model — are exactly the tasks that AI tooling now does in seconds. The bottom rung of the ladder got sawed off, while the demand for experienced people who can design and run systems kept climbing.</p><p>If you look at the market in 2026, you can see this split in the numbers. Demand for AI and ML talent is genuinely high — by some counts it outstrips supply several times over, and salaries reflect that. But entry-level postings are a tiny sliver of those openings. The growth is real and the door is narrow <em>at the same time</em>, and those two facts are not a contradiction. They’re the whole story. The market didn’t shrink; it shifted up the experience curve. That’s a brutal thing to navigate when you’re standing at the bottom of it, and I felt it on behalf of my students every single year.</p><h3>So what should someone starting today actually have?</h3><p>This is the question I get asked most, so let me be concrete. Here’s how I’d contrast it with what I needed when I started.</p><p><strong>Then:</strong> build models from scratch. <strong>Now:</strong> still learn to do this — it’s how you actually understand models — but also learn to <em>build on top of</em> foundation models. Know how to take a pretrained model and make it useful: fine-tuning, prompting, retrieval-augmented generation, and the orchestration frameworks that connect models into something larger. This is where a lot of the day-to-day work now lives.</p><p><strong>Then:</strong> getting the model to train was the goal. <strong>Now:</strong> in plenty of areas you still train models from the ground up — but the bar moved to getting them into production <em>and keeping them honest</em>. The skills that are quietly most valuable are the unglamorous ones — evaluation, monitoring, handling data drift, versioning, serving at scale. Anyone can get a demo working. The job is everything after the demo.</p><p><strong>Then:</strong> a strong GitHub and a couple of Kaggle medals. <strong>Now:</strong> one genuinely deployed project beats ten notebooks. Show something that runs, that a stranger can use, that you can defend in detail. The signal employers are hunting for is “this person has shipped a real thing,” because that’s the rung the market is hiring at.</p><p><strong>Constant, then and now:</strong> the fundamentals. Python you can rely on, math you actually use, and — most of all — the habit of understanding <em>why</em> something works instead of just that it does. Tools change every eighteen months. The reasoning underneath them doesn’t. That’s the part worth investing in, because it’s the part that doesn’t expire.</p><p>If anything, the fundamentals matter <em>more</em> now, not less. With so many AI tools writing code for you, coding by hand no longer feels like the most important skill — it’s becoming something you supervise rather than something you do keystroke by keystroke. But that’s exactly why the foundations are worth having: they’re what keep you in control of the work. They let you look at what a tool produced and actually judge it — is it right, where is it cutting corners, when should you override it. The point isn’t to stop using these tools; they’re genuinely useful, and refusing them would be silly. The point is to never hand them your judgment. If you don’t understand what’s underneath, you’re not directing the work — you’re just trusting it blindly, and hoping.</p><p>If I were advising a student today, I’d tell them the truth I wish I’d been able to tell my last cohorts: the door is narrower, but it isn’t closed. It rewards different things now — evidence over credentials, shipped work over coursework, judgment over raw enthusiasm. Aim at the part of the market that’s actually growing, build one real thing all the way through, and learn to explain it like you mean it.</p><h3>Why I’m writing again</h3><p>Eight years is long enough to watch a field reinvent its tools, its day-to-day, and its front door. It’s also long enough to get a little quiet — to stop writing because the ground keeps moving and it’s hard to know where to stand.</p><p>But that’s exactly why I wanted to come back to this blog. The pace of change is the reason to write, not the reason to wait. So I’m planning to post more — about what I’m building, what I’m learning, and what I’m still figuring out. Some of it will be technical. Some of it, like this, will just be me thinking out loud about a field I’ve now spent a third of my life inside.</p><p>If you’re at the start of that path, I hope this was useful. It’s harder than it was when I began. It’s also more interesting than it has ever been.</p><p>Most likely, in a while I’ll look back at this post and think it’s already out of date. But these are my honest thoughts right now — in a field that moves so fast that sometimes the changes seem to arrive quicker than we can see them, let alone integrate them into our lives.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f2e9045e4fbb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Inference Optimization for Convolutional Neural Networks]]></title>
            <link>https://medium.com/data-science/inference-optimization-for-convolutional-neural-networks-e63b51b0b519?source=rss-b8a574f7365d------2</link>
            <guid isPermaLink="false">https://medium.com/p/e63b51b0b519</guid>
            <category><![CDATA[model-inference]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[model-quantization]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[inference-optimization]]></category>
            <dc:creator><![CDATA[Lilit Yolyan]]></dc:creator>
            <pubDate>Thu, 19 May 2022 18:00:56 GMT</pubDate>
            <atom:updated>2026-06-17T20:51:16.764Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Quantization and fusion for faster inference</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dj0uJny68QgvKwmn" /><figcaption>Photo by <a href="https://unsplash.com/es/@bradencollum?utm_source=medium&amp;utm_medium=referral">Braden Collum</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Modern deep learning research mainly focuses on developing and improving optimized solutions for problems such as object detection, segmentation, and self-supervised learning. This is helping many businesses and startups start using more accessible AI solutions for their products. However, when it comes to real-world applications, especially when models are developed for edge devices, we face many limitations such as large model size and low latency.</p><p>There are many methods to make AI models accessible to mobile and other edge devices. One option is to use small models designed for mobile devices (such as <a href="https://arxiv.org/abs/1704.04861">MobileNet</a> and <a href="https://github.com/pytorch/ios-demo-app/tree/master/ObjectDetection">Yolo for mobile</a> devices). Other methods include optimization at the inference level. The latter include methods such as model pruning, quantization, module fusion, etc. In this blog post, we will look at quantization and fusion methods for convolutional neural networks. We are going to use PyTorch’s quantization module and compare the size and latency of models with and without quantization.</p><p>Blog overview:</p><ol><li>What is quantization?</li><li>Quantization techniques</li><li>What is module fusion?</li><li>Application and comparison in PyTorch</li></ol><h4><strong>What Is Quantization?</strong></h4><p>Quantization is a simple technique to speed up deep learning models at the inference stage. It is a method of compressing information. Model parameters are stored in floating point numbers, and model operations are calculated using these floating point numbers. Floating points have high precision, however, they can be very memory intensive and computationally expensive. Quantization converts 32-bit floating point numbers to 8-bit integers. It performs some or all of the operations on 8-bit integers, which can reduce the model size and memory requirements by a factor of 4.</p><p>However, there is a cost to that. In order to reduce the size of the model and improve the execution time, we will sacrifice some precision. So there will be a trade-off between model accuracy and size/latency.</p><p>To do quantization, we need a mapping function that maps floating point numbers to integers. The most common and easiest way to do this is a linear transformation.</p><p>Q(r) = round(1/S + Z), where</p><ul><li>r is the input,</li><li>S is a scaling factor that is the ratio of the input range to the quantized output range.<br>The easiest way to find the scaling factor is to use the minimum and maximum values of the input and output ranges. However, PyTorch has other methods such as MSE minimization, entropy minimization, etc. The process of finding the scaling factor is called calibration.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/120/0*y0X_eYxHFBqf6A0A" /></figure><ul><li>Z is a zero point, this parameter helps to correctly map the zero point from the input to the output space.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/127/1*dK_CgZ8LOrXky5kNuBl7Vg.png" /></figure><p>PyTorch has an Observer module that can be used to calibrate a model. It collects statistics on the input values and calculates the S and Z parameters. The quantization parameters can also be calculated by tensor or by channel. In the first case, the Observer will take the entire tensor and extract statistics from it, in the second case, the S and Z parameters are calculated for each channel separately.</p><h4><strong>Quantization Techniques</strong></h4><p>In PyTorch there are 3 different ways to implement quantization:</p><ol><li><strong>Dynamic Quantization</strong><br>Model weights are quantized, and activations are quantized dynamically during inference. This can result in higher accuracy, as calibration is implemented for each input. This technique works on LSTM, GRU, RNN</li><li><strong>Post-Training Static Quantization</strong><br>In this scenario model weights and activations are pre-quantized. Calibration is implemented on validation data. This is faster than dynamic quantizations, however, it may need re-calibration from time to time to stay robust.</li><li><strong>Quantization-aware Training (QAT) </strong><br>These techniques aim to improve post-training quantization. It adds quantization error in the training loss. S and Z parameters can be learned during the training.</li></ol><p>In the continuation of this post, we will use Post-training quantization for the convolutional neural network.</p><h4><strong>What is module fusion?</strong></h4><p>Before moving on to the PyTorch codes, let’s understand one more technique, which is called fusion. Fusion aims to merge multiple layers into one, which can save inference time as well as reduce memory access. It sends the combined sequence of operations to a low-level library that can compute it in one go without an intermediate representation back to PyTorch. However, this technique also comes at a cost. Now the model is hard to debug when the layers are merged. Fusion only works for the following layer group: [Conv, Relu], [Conv, BatchNorm], [Conv, BatchNorm, Relu], [Linear, Relu].</p><h4><strong>Application and comparison in PyTorch</strong></h4><pre># Import packages <br>from torch import nn<br>from torchsummary import summary<br>import torch<br>import os</pre><p>First, let’s create a simple convolutional neural network.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cc8a9608d8473c4511ef3849649f52fc/href">https://medium.com/media/cc8a9608d8473c4511ef3849649f52fc/href</a></iframe><p>As you can see from the code snippet above, we have created a small convolutional network with two convolutional layers followed by relu and maxpooling. In the end, we have two fully connected layers and a Softmax output.</p><pre>n = Net().cuda()<br>summary(n, (3, 224, 224))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*te0JxhtUdN2QjpP_zY8-nw.png" /></figure><p>The model summary shows that we have about 70 million parameters and estimated model size of 294MB.</p><p>To create a quantized version of the same model, we will create 2 new attributes to quantize and dequantize the model. Next, during the forward pass, we will quantize the network input and dequantize before softmax.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/899031b89f8eb649e9ba18ee7fffe484/href">https://medium.com/media/899031b89f8eb649e9ba18ee7fffe484/href</a></iframe><p>To run the quantized model for the eval() setting we need to define the configuration. Torch has 2 backends for quantization: ‘fbgemm’ for x86 CPUs with AVX2 support or higher and ‘’qnnpack’’ for ARM CPUs(mobile/embedded device).</p><p>Next, we need to prepare the model using <em>torch.quantization.prepare</em>. This runs an observer method that collects statistics for input. And <em>torch.quantization.convert</em> converts from an observer state to a quantized state.</p><pre># Define original and quantized models and prepae for evaluation <br>net = Net()<br>net.eval()<br>net_quant = NetQuant()<br>net_quant.eval()</pre><pre># Prepare model quantization and convert to quantized version<br>net_quant.qconfig = torch.quantization.get_default_qconfig(&quot;fbgemm&quot;)<br>torch.backends.quantized.engine = &quot;fbgemm&quot;<br>net_quant = torch.quantization.prepare(net_quant.cpu(), inplace=False)<br>net_quant = torch.quantization.convert(net_quant, inplace=False)</pre><p><strong>Check model size</strong></p><pre># Check model size<br>def print_model_size(mdl):<br>    torch.save(mdl.state_dict(), &quot;tmp.pt&quot;)<br>    size = round(os.path.getsize(&quot;tmp.pt&quot;)/1e6)<br>    os.remove(&#39;tmp.pt&#39;)<br>    return size</pre><pre>net_size = print_model_size(net)<br>quant_size = print_model_size(net_quant)</pre><pre>print(f&#39;Size whitout quantization: {net_size} MB \n Size whit quantization: {quant_size} MB&#39;)<br>print(f&#39;Size ratio: {round(net_size/quant_size, 2)}&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/336/1*dvjRkT5rVBxu6Tjb31yHHg.png" /></figure><p>Now, we got the model which has 4 times smaller size than the original one.</p><p><strong>Check Model Latency</strong></p><pre># input for the model<br>inpp = torch.rand(32, 3, 224, 224)<br># compare the performance<br>print(&quot;Floating point FP32&quot;)<br>%timeit net(inpp)</pre><pre>print(&quot;Quantized INT8&quot;)<br>%timeit net_quant(inpp)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/905/1*hSaP5eBfCvtk0gTWlM3Bww.png" /></figure><p>The original model on CPU runs at 162 ms, while the quantized model is about 1.7 times faster.</p><p><strong>Fusion</strong></p><p>Next, let’s implement fusion for more optimization. As we can see in our model structure, there are only 3 ways to fuse layers:</p><pre># Perpare blocks for the fusion</pre><pre>moduls_to_fuse =  [[&#39;conv1&#39;, &#39;relu1&#39;], <br>                   [&#39;conv2&#39;, &#39;relu2&#39;], <br>                   [&#39;fc1&#39;, &#39;relu3&#39;]]</pre><pre>net_quant_fused = torch.quantization.fuse_modules(net_quant, moduls_to_fuse)</pre><pre>net_fused = torch.quantization.fuse_modules(net, moduls_to_fuse)</pre><p>Now let’s check the latency again:</p><pre>print(&quot;Fused and quantized model latency&quot;)<br>%timeit net_quant_fused(inpp)</pre><pre>print(&quot;Fused model latency&quot;)<br>%timeit net_fused(inpp)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*a3-AO3RV_QHP3hjlsMkBeg.png" /></figure><p>As we can see, we saved some run time by using fusion with or without quantization. However, this did not affect accuracy like quantization did.</p><h4>Conclusion</h4><p>Optimizing models for inference is not an easy task. PyTorch makes it much easier to use these optimization techniques. However, always remember to carefully check the accuracy of the model before and after quantization to ensure that your model is not only fast but also accurate.</p><p><strong>Find all the codes on GitHub: </strong><a href="https://github.com/LilitYolyan/inference_optimization_cnn">https://github.com/LilitYolyan/inference_optimization_cnn</a></p><p><strong>Check references:</strong></p><ul><li><a href="https://pytorch.org/blog/quantization-in-practice/">Practical Quantization in PyTorch</a></li><li><a href="https://pytorch.org/docs/stable/quantization.html">Quantization - PyTorch 2.9 documentation</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e63b51b0b519" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/inference-optimization-for-convolutional-neural-networks-e63b51b0b519">Inference Optimization for Convolutional Neural Networks</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What I Learned While Implementing Computer Vision Papers]]></title>
            <link>https://medium.com/data-science/what-i-learned-while-implementing-computer-vision-papers-42f4fd730a97?source=rss-b8a574f7365d------2</link>
            <guid isPermaLink="false">https://medium.com/p/42f4fd730a97</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[learning-to-code]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Lilit Yolyan]]></dc:creator>
            <pubDate>Mon, 07 Mar 2022 18:48:38 GMT</pubDate>
            <atom:updated>2022-03-08T05:10:02.434Z</atom:updated>
            <content:encoded><![CDATA[<h4>An effective way to practice multiple skills simultaneously</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bLY_BmctTFyPBcaP" /><figcaption>Photo by <a href="https://unsplash.com/@emilep?utm_source=medium&amp;utm_medium=referral">Emile Perron</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>As computer vision engineers, <a href="https://github.com/AnnaManasyan">a friend of mine</a> and I were thinking of a new way to improve multiple skills simultaneously which should be effective and time-saving. We came up with the idea of implementing computer vision papers on GitHub. In this short article, I want to talk about all the interesting details that we faced during the process, all the positive and negative findings that we ran into.</p><p>Positive sides first.</p><h4><strong>Computer vision skills</strong></h4><p>To implement a paper you need to understand every detail of it. You may need to read some sections more than 10 times. Sometimes, to understand a small concept, you may need to read many other reference papers. As a result, you get T-shaped knowledge: broad knowledge of the topic you are trying to implement and a deeper knowledge of that particular model and technique.</p><h4><strong>Programming skills</strong></h4><p>Here comes the responsibility in front of the community. When you’re working on an open-source project, you feel the responsibility to make it more understandable for everyone else. To write a small chunk of code you start to think longer and longer to find the optimal, understandable, useful, and effective way.</p><h4><strong>Other skills that you didn’t think you will improve</strong></h4><p>First, it seems you are working in two directions: deep learning knowledge and programming skills. However, it turns out there are a lot of things you need to learn, to make it happen.</p><ul><li>Project management skills to create a workflow for the whole process: create and prioritize tasks, assign to each other, etc.</li><li>Communication skills: communicate with collaborators, help each other, review codes.</li><li>Ownership and accountability: take ownership and be accountable to make it more effective to work with a team, be responsible for your tasks and the whole project.</li></ul><p>There is much more to learn and improve, and you will find out it during the process itself.</p><h4><strong>Collaboration with the whole world</strong></h4><p>The great advantage of GitHub and other platforms is it helps you to share your work and get the help of others. It’s an amazing feeling when people around the world start to work with you without knowing you personally and your skills for one purpose: community contribution.</p><h4><strong>Comunity reactions and interest</strong></h4><p>Community contribution is an amazing opportunity for each of us. Getting feedback from people who want to use your repository is very valuable. Many questions, requests, and open issues will have a huge input on your work.</p><p>Now let’s dive into the disadvantages we ran into.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*s9Hbq9ukUAxRcZLM" /><figcaption>Photo by <a href="https://unsplash.com/@elisa_ventur?utm_source=medium&amp;utm_medium=referral">Elisa Ventur</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><h4><strong>Paper structure and content</strong></h4><p>Many papers are hard to implement because of the acceptable way the authors write them. Sometimes, to understand a small concept, you may need to go up and down million times to find a key. Many papers don’t even contain the experimentation part: how they do the experiments, the elements of setting up experiments, and some general elements of model and technique description. It’s on you to assume many details or go back to other similar papers, references to understand those details.</p><h4><strong>Computational issues</strong></h4><p>Nowadays many computer vision papers from the best research labs look like a puzzle that gathers and combines different concepts from other papers. One of their main unfair advantages is the computational capacity they have. They can use an immense amount of data that is not open access, and huge computational capacity which we can’t afford. That is why it’s sometimes impossible to repeat their experiments and check their results.</p><h4><strong>Conclusion</strong></h4><p>Despite some challenges you may face while working on implementation, it is worth doing it because of the skills you’ll improve and the fun you’ll get from the process. Especially when you see support and help from the community you definitely understand why you’re doing that. I recommend everyone to make this practice part of their learning process. Find something you like and want to learn and start working on it. If you need help, community will be there for you.</p><p>GitHub Repositories: <a href="https://github.com/LilitYolyan">https://github.com/LilitYolyan</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=42f4fd730a97" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/what-i-learned-while-implementing-computer-vision-papers-42f4fd730a97">What I Learned While Implementing Computer Vision Papers</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Self-Supervised Learning for Anomaly Detection in Python: Part 2]]></title>
            <link>https://medium.com/data-science/self-supervised-learning-for-anomaly-detection-in-python-part-2-5c918b12a1bc?source=rss-b8a574f7365d------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c918b12a1bc</guid>
            <category><![CDATA[self-supervised-learning]]></category>
            <category><![CDATA[sota]]></category>
            <category><![CDATA[cutpaste]]></category>
            <category><![CDATA[kernel-density-estimation]]></category>
            <category><![CDATA[anomaly-detection]]></category>
            <dc:creator><![CDATA[Lilit Yolyan]]></dc:creator>
            <pubDate>Thu, 06 Jan 2022 13:20:46 GMT</pubDate>
            <atom:updated>2022-01-06T13:20:46.287Z</atom:updated>
            <content:encoded><![CDATA[<h4>CutPaste: self-supervised learning as an improvement for Kernel Density Estimation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xK8sGpycrpwzHkxg" /><figcaption>Photo by <a href="https://unsplash.com/@randyfath?utm_source=medium&amp;utm_medium=referral">Randy Fath</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><a href="https://towardsdatascience.com/review-on-self-supervised-contrastive-learning-93171f695140">Self-supervised learning</a> is one of the most popular fields in modern deep-learning research. As Yann Lecun likes to say <a href="https://ai.facebook.com/blog/self-supervised-learning-the-dark-matter-of-intelligence/">self-supervised learning is the dark matter of intelligence</a> and the way to create common sense in AI systems. The ideas and techniques of this paradigm attract many researchers to try and enlarge the application of self-supervised learning into new research fields. Of course, anomaly detection is not an exception.</p><p><a href="https://towardsdatascience.com/kernel-density-estimation-for-anomaly-detection-in-python-part-1-452c5d4c32ec"><strong>In Part 1 of this article</strong></a>, we discussed the definition of anomaly detection and a technique called Kernel Density Estimation. KDE is one of the most popular and simple ways to find abnormal data points in a dataset. It works well on tabular data. However, we are interested in implementation for images. Visual inspection is a very popular application where the focus is to find defects or outliers in objects we are interested in.</p><p><em>Well, how can we conduct KDE on images without losing too much information and by reducing the computational cost?</em></p><p>The answer may be obvious for many people: find a good backbone that can provide us with good representations of images. Then we can use the representations to conduct KDE.</p><p>The best way to finetune the backbone for a specific use case is self-supervised learning. Recently many researchers around the world work on combining self-supervised learning techniques with classical anomaly detection techniques. Moreover, researchers came up with brand new transformation approaches to improve pretext tasks for self-supervised learning for anomaly detection.</p><p>In this part of the article, we are going to discuss a new self-supervised technique from Google called CutPaste[1] and see how a new creative way of data transformation can improve the training results.</p><p>The main components of the CutPaste are:</p><ol><li>New improved pretext tasks</li><li>Self-supervised model structure</li></ol><h4><strong>CutPaste and Scar-CutPaste</strong></h4><p>A very popular type of self-supervised pretext task is called Cutout. As the name implies, it randomly cuts out a small rectangular patch from an image. This works very well in many self-supervised settings and as data augmentation. And it seems to be a good simulation for anomaly detection use-case.</p><p>Researchers from the paper came up with a new variation of this technique called CatPaste which copies a small part from an image and replaces it somewhere else. It creates local irregularities, which seem to perfectly align with the anomaly detection problem setting.</p><p>The other variation of this pretext task is called CutPaste Scar which is the improvement of the original Scar Cutout technique. In the original Scar Cutout, we take a long-thin rectangular patch from an image and replace it with a random color. In this case, we use a scar-like rectangular patch from an image and again insert it randomly somewhere else in the image.</p><p><a href="https://github.com/LilitYolyan/CutPaste/blob/main/cutpaste.py">Follow the GitHub link to see CutPaste and Scar-CutPaste implementation in PyTorch.</a></p><p>Seems like a minor change in the pretext task definition. However, experiment results show that it improves the results of final anomaly detection. The image below shows some examples of CutPaste and Scar-CutPaste. As we can notice, in some cases it may look like a real defect in the object.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/936/0*DydynHSdcvNZtFPm" /><figcaption>Image source from paper [1]. See references</figcaption></figure><p>Is that mean that created defects from CutPaste are a good simulation for original defects? To answer that question the authors showed t-SNE plots of the representations of CutPaste, Scar-CutPaste, normal and abnormal samples. The image below shows that in most cases pretext tasks don’t overlap with anomalies thus we can say that it’s not a good mimic for real anomalies. However, the training results show that despite the distribution differences these transformations help to create better representations for anomaly detection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/916/0*wArW87rWkti_DzIR" /><figcaption>t-SNE plot of Normal, Abnormal, CutPaste, and Scar-CutPaste samples. Image source from paper [1]. See references</figcaption></figure><h4><strong>Self-supervised model structure</strong></h4><p>To create a model originally ResNet-18 was used. Next, the MLP projection head was added followed by the last linear layer which outputs the representations.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5bd22b404a98ee8d66bc9dfc0616dcc4/href">https://medium.com/media/5bd22b404a98ee8d66bc9dfc0616dcc4/href</a></iframe><p>Two types of experiments were conducted: binary classification and 3-way classification.</p><p>In the first case, the model uses the original image and randomly transforms it with CutPaste or CutPaste-Scar. Then, the original image and transformed images become two different classes and we conduct binary classification on top of it.</p><p>Next, is the 3-way classification technique where instead of using Cutpaste and CutPaste-Scar randomly, we use both as separate classes and we add normal class as the third one. In the end, we have classification with 3 classes.</p><p>In this example, our CutPaste object can return the transformed images based on classification type:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e1faf633cf33a50018f9c14c1b6a1c49/href">https://medium.com/media/e1faf633cf33a50018f9c14c1b6a1c49/href</a></iframe><p>The loss is a cross-entropy where CP(.) is a CutPaste transformation of an image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/0*nwGh91y4M9YGtsVb" /></figure><h3>References</h3><p>[1]Li, Chun-Liang &amp; Sohn, Kihyuk &amp; Yoon, Jinsung &amp; Pfister, Tomas. (2021). <a href="https://arxiv.org/abs/2104.04015">CutPaste: Self-Supervised Learning for Anomaly Detection and Localization.</a></p><h4>Follow the link for <a href="https://github.com/LilitYolyan/CutPaste">Codes from Github</a></h4><h4>Find Part 1 of the article here: <a href="https://towardsdatascience.com/kernel-density-estimation-for-anomaly-detection-in-python-part-1-452c5d4c32ec">Kernel Density Estimation for Anomaly Detection in Python</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c918b12a1bc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/self-supervised-learning-for-anomaly-detection-in-python-part-2-5c918b12a1bc">Self-Supervised Learning for Anomaly Detection in Python: Part 2</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Kernel Density Estimation for Anomaly Detection in Python: Part 1]]></title>
            <link>https://medium.com/data-science/kernel-density-estimation-for-anomaly-detection-in-python-part-1-452c5d4c32ec?source=rss-b8a574f7365d------2</link>
            <guid isPermaLink="false">https://medium.com/p/452c5d4c32ec</guid>
            <category><![CDATA[cutpaste]]></category>
            <category><![CDATA[anomaly-detection]]></category>
            <category><![CDATA[self-supervised-learning]]></category>
            <category><![CDATA[kernel-density-estimation]]></category>
            <category><![CDATA[gaussian-density]]></category>
            <dc:creator><![CDATA[Lilit Yolyan]]></dc:creator>
            <pubDate>Tue, 04 Jan 2022 15:44:49 GMT</pubDate>
            <atom:updated>2022-01-04T15:44:49.798Z</atom:updated>
            <content:encoded><![CDATA[<h4>Combining classic approaches with deep learning for better representations</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Nkrkm4JUUWC8Pbkt" /><figcaption>Photo by <a href="https://unsplash.com/@mullyadii?utm_source=medium&amp;utm_medium=referral">Mulyadi</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Fraud detection in the financial datasets, rare event detection in network traffic, visual image inspection for buildings and road monitoring, and defect detection in production lines: these are very common problems where machine learning techniques for Anomaly Detection can be helpful.</p><blockquote>In short, anomaly detection is a field of research that aims to find abnormal observations in datasets.</blockquote><p>For this problem settings, we assume that we have two types of data points: <strong>normal </strong>and <strong>abnormal</strong>. Mostly, it’s a highly unbalanced dataset, because abnormal data points accrue less than normal ones. In different scenarios, the word abnormal can have different meanings. For example, it can be called <strong>novelty </strong>when we got a data point that differs from every other type of abnormalities we had before, or we can call it an <strong>outlier </strong>when it’s a very rare but known cause [1].</p><p>Despite the definition and use-case, we can generalize all these types of problems into anomaly detection.</p><p>PCA, SVM, or Kernel Density Estimation are classical machine learning techniques used to find abnormal observations. Nowadays, many deep learning techniques are used for anomaly detection like GANs, Autoencoders, etc.</p><p>A very common approach for anomaly detection in images is <em>one-class classification with a combination of self-supervised learning</em>. It’s called one-class classification because we use only normal data to fit a model. Roughly saying, we force the model to learn good representations of normal data points. This may help to see minor differences later when we use a test set with combination of normal and abnormal data points. These techniques become state of the art in anomaly detection.</p><p>The first part of this article will cover the classic approach for detecting abnormal data using Kernel Density Estimation, next we will dive deeper into self-supervised techniques used to improve KDE. For more practical description we will dive deeper into CutPaste [2] a recent paper from Google, which combines a novel self-supervised technique with KDE.</p><h4><strong>Kernel Density Estimation</strong></h4><p>Kernel Density Estimation (KDE) is an unsupervised learning technique that helps to estimate the PDF of a random variable in a non-parametric way. It’s related to a histogram but with a data smoothing technique.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/542/0*zTOzin8kKywY0dOi" /><figcaption>Histogram and KDE visualizations: <a href="https://scikit-learn.org/stable/modules/density.html">Image source</a></figcaption></figure><p>As we can see in the example above, we use the same data to plot histograms. However, shifting bins in histograms can result in huge differences in visualization. To smoothen the distribution different kernels can be used. In the example above, Tophat and Gaussian Kernels were used. The output is a smooth density estimate. So, how to get this distribution?</p><p>The generalized formula for KDE is as follow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/249/0*vAm7sZfXn08EVkx7" /></figure><p>Where<em> K</em> is a kernel and<em> h</em> is a bandwidth parameter that is responsible for smoothness. If we choose a higher number for<em> h</em>, we will get smoother distribution.<em> y</em> is a given estimate and <em>xi </em>is a point from the sample dataset.</p><p>As was mentioned above K is a kernel where we have multiple options like Gaussian, Tophat, Epanechnikov, etc. For anomaly detection we are going to use Gaussian Kernel Estimation, where we calculate the density using the following formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/228/0*UqXA9A9o1eA8gbGj" /></figure><p>Following to CutPaste paper[2], we calculate anomaly scores using the formula above.</p><p>The following steps describe the process of anomaly detection using Gaussian Density Estimation:</p><p>Step 1: Fit normal data points from train split to GDE</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/fcaa49cc15281c2df69d30f1ab8858e8/href">https://medium.com/media/fcaa49cc15281c2df69d30f1ab8858e8/href</a></iframe><p>Step 2: Calculate anomaly scores from the test dataset</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d90183753286036a0a1840aca69337c6/href">https://medium.com/media/d90183753286036a0a1840aca69337c6/href</a></iframe><p>Step 3: If anomaly scores are smaller than a predefined threshold, you have an abnormal data point, otherwise, you don’t have an anomaly</p><h4><strong>ROCAUC metric for anomaly detection</strong></h4><p>In the benchmark, ROCAUC is used to compare different models. MVTec dataset is the most common benchmark dataset for anomaly detection.</p><p>To calculate the accuracy of a model, the<strong> Receiver Operating Characteristic (ROC) curve</strong> is plotted and Area Under Curve (AUC) is calculated. To calculate ROC and AUC we use our test data with abnormal and normal examples, and if we get higher results on AUC, we can detect defects better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*tz93CYq3Z9jg8QG5TOxzgA.jpeg" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*H1aiMkRMy1k1nUEiMuA8kg.jpeg" /><figcaption>Examples of ROC curve after training the model: Image from author</figcaption></figure><p>To calculate ROC curve and AUC <em>sklearn </em>package can be used:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cc974a8022b454058a175d3af085cb83/href">https://medium.com/media/cc974a8022b454058a175d3af085cb83/href</a></iframe><h4><strong>Self-supervised setting</strong></h4><p>We saw how to implement GDA for Anomaly Detection. However, it’s very hard to get good results on images. If we have good representations retrieved from an image we could get better results on GDE. To get that, in the second part of the article we create a self-supervised model with unique pretrained tasks which can improve the results of GDE.</p><h4>References</h4><p>[1] Ruff, Lukas &amp; Kauffmann, Jacob &amp; Vandermeulen, Robert &amp; Montavon, Gregoire &amp; Samek, Wojciech &amp; Kloft, Marius &amp; Dietterich, Thomas &amp; Müller, Klaus-Robert. (2021).<a href="https://arxiv.org/abs/2009.11732"> A Unifying Review of Deep and Shallow Anomaly Detection</a>. Proceedings of the IEEE. PP. 1–40. 10.1109/JPROC.2021.3052449.</p><p>[2]Li, Chun-Liang &amp; Sohn, Kihyuk &amp; Yoon, Jinsung &amp; Pfister, Tomas. (2021). <a href="https://arxiv.org/abs/2104.04015">CutPaste: Self-Supervised Learning for Anomaly Detection and Localization.</a></p><h4><a href="https://github.com/LilitYolyan/CutPaste">Codes from Github</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=452c5d4c32ec" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/kernel-density-estimation-for-anomaly-detection-in-python-part-1-452c5d4c32ec">Kernel Density Estimation for Anomaly Detection in Python: Part 1</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Review on Few Shot Object Detection]]></title>
            <link>https://medium.com/data-science/review-on-few-shot-object-detection-185965e0e6a2?source=rss-b8a574f7365d------2</link>
            <guid isPermaLink="false">https://medium.com/p/185965e0e6a2</guid>
            <category><![CDATA[paper-review]]></category>
            <category><![CDATA[few-shot-object-detection]]></category>
            <category><![CDATA[object-detection]]></category>
            <category><![CDATA[fsod]]></category>
            <category><![CDATA[few-shot-learning]]></category>
            <dc:creator><![CDATA[Lilit Yolyan]]></dc:creator>
            <pubDate>Sun, 28 Nov 2021 07:03:43 GMT</pubDate>
            <atom:updated>2021-12-02T13:49:46.070Z</atom:updated>
            <content:encoded><![CDATA[<h3>Review on Few-Shot Object Detection</h3><h4>Introduction and overview of few-shot object detection</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sBdj9cNfygbAtPMO" /><figcaption>Photo by <a href="https://unsplash.com/@glenncarstenspeters?utm_source=medium&amp;utm_medium=referral">Glenn Carstens-Peters</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Deep learning solutions for classification and object detection are state of the art in computer vision and that’s not news anymore. Despite the high accuracy and speed of recent SOTA algorithms, there is one big issue: for a good-performing solution, we need a huge amount of data. In addition, the data must be annotated, which requires a lot of manual work. That was the reason for the development of several new paradigms like <a href="https://towardsdatascience.com/review-on-self-supervised-contrastive-learning-93171f695140">self-supervised learning</a> and few-shot learning.</p><p>Recent progress in the few-shot classification helped to significantly improve the performance of “learn to learn” problem in classification, however few-shot object detection (FSOD) has large potential to grow and improve. <br>Large amount of research has been done around this topic. Yet, a huge performance gap exists compared to classic object detection or few-shot classification.</p><p>Let’s now dive into FSOD more profoundly and understand:</p><ol><li>Problem definition of few-shot object detection</li><li>Benchmark 3 recent SOTA algorithms</li><li>Review and compare the 3 above-mentioned papers.</li></ol><h4><strong>Few Shot Object Detection</strong></h4><p>Few-shot object detection aims to generalize on novel objects using limited supervision and annotated samples. Let (S1, … Sn) be a set of support classes and Q be a query image with multiple instances and backgrounds. For the given (S1, … Sn) and Q models aim to detect and localize all objects from support sets found in Q. During the training most FSOD applications divide classes into two non-overlapping parts: base and novel classes. Training dataset includes base classes to train the baseline model. Then, model is finetuned where a combined dataset of base and novel classes is used. The last stage includes testing on a dataset composed of only novel classes</p><p>This is the most popular problem definition for few-shot object detection. It may slightly differ from paper to paper. However the main idea is common for every research: create a model which can find objects on new, never-seen classes in a class-agnostic way.</p><h4>Benchmark</h4><p>Two popular few shot object detection tasks are used for benchmark: MS-COCO on 10-shot and MS-COCO on 30-shot. Let’s look at the top 3 models for each of these tasks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*eOC5hCBLP8f4NvEhewJDgg.png" /><figcaption>Benchmark of 3 models for two different tasks. Graph by author</figcaption></figure><p>Depending on the task these 3 algorithms outperform others, however, there is a huge gap in accuracy between classic object detection tasks and few-shot object detection. Let’s dive into each of these to understand the structures and differences of each of these.</p><h4><a href="https://arxiv.org/pdf/2108.09017v1.pdf"><strong>DeFRCN: Decoupled Faster R-CNN for Few-Shot Object Detection</strong></a></h4><p>As the name of the paper implies the model modifies Faster R-CNN for few-shot object detection. Faster R-CNN consists of 3 blocks: “shared convolutional backbone for extracting generalized features, Region Proposal Network (RPN) for generating class-agnostic proposals, and a task-specific RCNN head for performing class-relevant classification and localization” [1].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/687/0*09MENQQ3gdndp4Nm" /><figcaption>Image 1: Faster R-CNN and Decoupled Faster R_CNN Architectures. Image source from paper [1] (see references)</figcaption></figure><p>To modify Faster R-CNN to work on few-shot settings authors are trying to solve two problems:</p><ol><li><em>The problem of multi-task learning</em>: R-CNN head of model is responsible for classification, in other words, what to look at, whilst RPN head aims to understand where to look, it solves localization problem. “First head needs translation invariant features whereas the localization head needs translation covariant features”[1]. The joint optimization of these two heads in the case of FSOD can lead to worse results when we have individual small tasks.</li><li><em>The problem of shared backbone</em>. As we can see in the image above, Faster R-CNN has one shared backbone for 2 heads. It works very well in object detection, but in a few shot settings, there can be a decrease in accuracy when fine-tuning for novel classes. Foreground-background confusion can arise, which means background in base training can become foreground in the novel fine-tuning phase. That is why gradients from RPN cause overfitting of shared backbone and model can’t converge.</li></ol><p>To solve these problems, the authors suggest changing the model by adding two modules: <strong>Gradient Decoupled Layers (GDL)</strong> and <strong>Prototypical Calibration Block (PCB)</strong> to improve object detection in few-shot settings. Let’s look at each of these modules closely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rrz_DBD0VmFOzEUh" /><figcaption>Image 2: DeFRCN Architecture. Image source from paper [1] (see references)</figcaption></figure><p>As we can see in Image 2 two Gradient Decoupled Layers (GDL)are placed after the backbone. “GDL does an affine transformation, which is parameterized by learnable channel-wise weights. During the forward propagation, the features from the shared backbone are transformed into different feature spaces through Aᵣₚₙ and Aᵣ𝒸ₙₙ”[1].</p><p>Constant λ ∈ [0, 1] is defined, which helps to set the decoupling degree of backbone, RPN, and RCNN during backpropagation. The decoupling degree can be adjusted by λᵣₚₙ and λᵣ𝒸ₙₙ which roughly speaking decides how much gradients will affect on preceding layer. To stop updates from RPN or RCNN we can set λ = 0. Elsewise, we can scale gradients by setting λ larger than 0. In other words, λᵣₚₙ and λᵣ𝒸ₙₙ decide the individual contribution of each RPN and RCNN on a shared backbone. This solves the second problem of shared backbone.</p><p>The next input of the paper is the Prototypical Calibration Block. “Classification needs translation invariant features whereas localization needs translation covariant features. Thus the localization branch may force the backbone to gradually learn translation covariant properties, which potentially downgrades the performance of the classifier. PCB consists of a strong classifier from ImageNet pre-trained model, a RoIAlign layer, and a prototype bank”[1].</p><p>First, it calculated prototypes from the support set. Let’s define a few-shot problem with M-way K-shot setting. PCB extracts feature maps from support set images. After aligning these feature maps with ground-truth boxes using RoIAlign it produces MK instance representations. Then it shrinks the representations to a prototype bank.</p><p>For object proposals from fine-tuned few-shot detector PCB first performs RoIAling on the predicted box and generates object features. Then it calculates cosine similarity sᵢᶜᵒˢ between this object feature and support set prototypes. This similarity is used as a score for the predicted category. To calculate the final classification score model performs weighted aggregation between the score given from PCB ( sᵢᶜᵒˢ) and the score from the fine-tuned few-shot detector (sᵢ ).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/228/1*Ok6-kHggHkAioqbUdGVZKw.gif" /></figure><p>There are no shared “parameters between the few-shot detector and PCB module so that the PCB can not only preserve the quality of the classification-aimed translation invariance feature but also better decouple the classification task and regression task within the RCNN. Furthermore, since the PCB module is offline without any further training, it can be plug-and-play and easily equipped to any other architecture to build stronger few-shot detectors”[1].</p><h4><a href="https://arxiv.org/pdf/2102.12152v3.pdf"><strong>Dual-Awareness Attention for Few-Shot Object Detection</strong></a></h4><p>Authors of this paper address two other problems:</p><ol><li><em>Problem with quality of support features</em>: during FSOD we have limited support information about novel objects, thus we need high-quality features for better results. To solve this problem authors are trying to undermine the influence of noise.</li><li><em>Problem with correlation: </em>Again with a limited number of examples it’s hard to get high correlation between support and query sets. Here, the goal is to improve object-wise correlations.</li></ol><p>The authors present a novel mechanism called Dual-Awareness Attention (DAnA) which combines two new modules called Background Attenuation (BA) and Cross-Image Spatial Attention (CISA).</p><p>To undermine the noise influence <strong>background attenuation block (BA) </strong>was suggested. The following image shows the structure of this module. First, it reshapes the support feature map and transforms it by linear learnable matrix Wₑ.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/0*CQKwbF1zDGMY3vUi" /><figcaption>Image 3: Background Attenuation Block. Image source from paper [2] (see references)</figcaption></figure><p>The attention map is calculated using each pixel of the support feature map and Wₑ. Next, they combine the attention map with support features which gives the most important features. In the end, leaky relu is used for a softer attention strategy. This transformation helps to get more discriminative support features.</p><p>The next block presented in the paper is called <strong>Cross-Image Spatial Attention Block (CISA)</strong>. The goal of this block is to help the model to focus on the most representative parts of the objects to determine intra-class similarity.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/0*PkobTvwi1SFEsgV0" /><figcaption>Image 4: Cross-Image Spatial Attention Block. Image source from paper [2] (see references)</figcaption></figure><p>“The core idea of Cross-Image Spatial Attention (CISA) is to adaptively transform each support feature map into query position-aware (QPA) support vectors that represent specific information of a support image”[2].</p><p>Let denote Z as support feature map processed in BA block and X as a query feature map. CISA transforms X and Z into the query and key embeddings 𝑸 and 𝓚 using W𝓺 and Wₖ weight matrices.</p><p>Then it calculates similarity scores between query and support:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/386/0*Mu9E9GMtR3hJ358l" /></figure><p>Where µ𝑸 and µ𝓚 are the averaged embedding values over all pixels.</p><p>For the next step, self-attention is added as authors assume that attention should be based not only on query-support correlations but also on the support image itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/361/0*YZy2BnlpB_Yu05Yj" /></figure><p>𝛽 is a constant coefficient.</p><p>The final transformation is a calculation of query position-aware vectors by multiplying the results described above with position-aware support feature map Z. The result is ready to be passed to object detection.</p><p>DAnA can be combined with object detection frameworks. For that reason, CISA output can be combined with a query feature map and sent to the modules such as region proposal network. The authors experimented on Faster-RCNN and RetinaNet.</p><h4><a href="https://arxiv.org/pdf/2103.11731v3.pdf"><strong>Meta-DETR: Image-Level Few-Shot Object Detection with Inter-Class Correlation Exploitation</strong></a></h4><p>Most few-shot object detection frameworks combine meta-learning techniques with object detection models. Meta-DETR does the same. Most approaches are based on Faster R-CNN or similar object detection algorithms. Despite the achievements of these models researchers of this paper see two major issues which they address and try to improve with their solution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/0*8Dn36GaVG9j5YB99" /><figcaption>Image 5: Issue of poor classification between similar classes. Image source from paper [3] (see references)</figcaption></figure><ol><li><em>The problem of region proposals</em>: This may work well on a large number of images however under a few-shot setup we have only a limited number of examples per class. Moreover, we try to generalize on novel classes, which makes it harder to get high-quality region proposals.</li><li><em>The problem of poorly defined meta-learning tasks.</em> Each support class is treated independently which causes the problem of well distinguishing similar classes like bikes and motorcycles, cows and sheep, etc (Image 5).</li></ol><p>Image 6 describes the whole Meta-DETR algorithm. First, a feature extractor with shared weights is used for query and support images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rTTrWNjKZi_Jppmg" /><figcaption>Image 6: Meta-DETR algorithm architecture. Image source from paper [3] (see references)</figcaption></figure><p>To solve the problem with high correlation between similar classes, the authors suggest a new module called <strong>Correlational Aggregation Module (CAM).</strong> It aggregates query features with support classes for class-agnostic prediction. The main difference from other methods is that it can aggregate multiple support classes simultaneously, which helps to capture inter-class correlation and reduce misclassification. CAM first matches the query features with a set of support classes. Then it maps the set of support classes to a set of pre-defined task encodings that differentiate these support classes in a class agnostic manner.</p><p>CAM outputs support-aggregated query features which then become the input of a transformer-based model for object detection. Recently proposed Deformable DETR was used for object detection. DETR uses a transformer technique for one stage object detection, hence there are no region proposals in the algorithm. This solves the next issue presented in the paper: poor quality region proposals.</p><p>Hungarian Loss was used for the model, the same as for Deformable DETR. Additionally, cosine similarity cross-entropy is used after CAM to classify class prototypes.</p><p>The training procedure for the algorithm is the same as described above. First, the base dataset is used for full training, then novel classes are used with base classes for fine-tuning.</p><h4>Conclusion</h4><p>The authors of each paper reviewed here are trying to solve few-shot object detection problem with a new creative approach. As was mentioned above, there is a huge gap between the accuracies of classic object detection and FSOD. However, this paradigm has huge potential and hopefully one day we can have FSOD algorithms as effective as classical object detection.</p><h4>References</h4><p>[1] Qiao, L., Zhao, Y., Li, Z., Qiu, X., Wu, J., &amp; Zhang, C. <a href="https://arxiv.org/pdf/2108.09017v1.pdf">DeFRCN: Decoupled Faster R-CNN for Few-Shot Object Detection</a> (2021).| <a href="https://github.com/er-muyue/DeFRCN">Github</a></p><p>[2]Chen, T., Liu, Y., Su, H., Chang, Y., Lin, Y., Yeh, J., &amp; Hsu, W.H. <a href="https://arxiv.org/pdf/2102.12152v3.pdf">Dual-Awareness Attention for Few-Shot Object Detection</a> (2021).| <a href="https://github.com/Tung-I/Dual-awareness-Attention-for-Few-shot-Object-Detection">Github</a></p><p>[3] Zhang, G., Luo, Z., Cui, K., Lu, S., <a href="https://arxiv.org/pdf/2103.11731v3.pdf">Meta-DETR: Image-Level Few-Shot Object Detection with Inter-Class Correlation Exploitation</a> (2021).| <a href="https://github.com/ZhangGongjie/Meta-DETR">Github</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=185965e0e6a2" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/review-on-few-shot-object-detection-185965e0e6a2">Review on Few Shot Object Detection</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Review on Self-Supervised Contrastive Learning]]></title>
            <link>https://medium.com/data-science/review-on-self-supervised-contrastive-learning-93171f695140?source=rss-b8a574f7365d------2</link>
            <guid isPermaLink="false">https://medium.com/p/93171f695140</guid>
            <category><![CDATA[simclr]]></category>
            <category><![CDATA[swav]]></category>
            <category><![CDATA[self-supervised-learning]]></category>
            <category><![CDATA[contrastive-learning]]></category>
            <category><![CDATA[paper-review]]></category>
            <dc:creator><![CDATA[Lilit Yolyan]]></dc:creator>
            <pubDate>Tue, 27 Apr 2021 18:19:54 GMT</pubDate>
            <atom:updated>2021-04-30T12:54:50.811Z</atom:updated>
            <content:encoded><![CDATA[<h4>Brief introduction and overview of self-supervised contrastive learning</h4><p>For the past few years, contrastive learning and self-supervised techniques became a hot topic in computer vision. Many researchers from different AI research labs are working on creating new architectures and exploring the effectiveness of these techniques.</p><p>In this article, we are going to discuss the advantages and disadvantages of self-supervised contrastive learning, overview the recent advancements, and then review the latest two most popular algorithms: SwAV and SimCLR. Let’s dive deeper into it!</p><p>Why is the idea of self-supervised learning so attractive for many professionals? As data scientists, we almost always face the problem of scarcity of labeled, clean data. In order to conduct object detection or segmentation, we need an immense amount of labeled data. Sometimes it’s expensive to annotate that much data, and in some cases, we can’t even get enough data from the specific domain. That’s where self-supervised techniques come in handy!</p><h4><strong>Self-supervised Learning and Contrastive Learning</strong></h4><p>Self-supervised learning is a subset of unsupervised learning. Unlike supervised learning, it doesn’t require any labeled data. Instead, it creates self-defined pseudo labels as supervision and learns representations, which are then used in downstream tasks.</p><p>Self-supervised learning is used mostly in two directions: GANs and contrastive learning. Contrastive learning aims to group similar samples closer and diverse samples far from each other.</p><p>The main motivation for contrastive learning comes from human learning patterns. Humans recognize objects without remembering all the little details. For example, it’s easy for us to look at an image and find a table in it based on the color, shape, and certain other features. Roughly speaking, we create some kind of representations in our minds, and then we use them to recognize new objects. The main goal of self-supervised learning and contrastive learning are respectively to create and generalize these representations.</p><p>Self-supervised tasks are called pretext tasks and they aim to automatically generate pseudo labels. There are a lot of different ways to automatically create tasks, like</p><ol><li>Color augmentation</li><li>Image rotation and cropping</li><li>Other geometric transformations.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/285/0*hR41WqNAeyEZENxj" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/0*TCDmN5tEJDFEwhbJ" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/283/0*2hMAGvVTiUCBkDDH" /><figcaption>Examples of a. Geometric transformation, b. Color transformation, c.Jigsaw puzzle: Image source [1]</figcaption></figure><p>There are a lot of new methods and techniques to conduct self-supervised contrastive learning. The three most important components of these techniques are the definition of pretext tasks, backbone, and contrastive loss. Many pieces of research were conducted to find the best combination of these components.</p><p>Two of the most popular works in this domain are:</p><ol><li>SimCLR [3]: an algorithm created by Google Research</li><li>SwAV [2]: the recent state of the art (SoTA) algorithm published by Facebook Research</li></ol><h4><strong>A Simple Framework for Contrastive Learning of Visual Representations (SimCLR)</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/826/0*UNHJt_PyIEfOmQQx" /><figcaption>Image source: <a href="https://amitness.com/2020/03/illustrated-simclr/">https://amitness.com/2020/03/illustrated-simclr/</a></figcaption></figure><p>To create the architecture of SimCLR Google research team has done immense work by comparing different components of self-supervised learning algorithms and in the end, they found the best architecture which at the time of publication of the paper was SoTA in contrastive learning. Their architecture consists of four major components:</p><ol><li>Data Augmentation</li><li>Base Encoder</li><li>Projection Head</li><li>Contrastive Loss</li></ol><p>For creating pretext tasks SimCLR uses data augmentation. Authors have tried many different augmentations compositions and got their best results with random cropping, resizing, color distortions, and Gaussian blur. They show that as augmentations get more complicated the model starts to behave better.</p><p>The second Component is the base encoder. They evaluated many different backbones and as a result, they use ResNet with different depths. Their research shows that deeper networks significantly improve training results.</p><p>The next experiment was the selection of a projection head. Architectures with projection heads showed much better results on downstream tasks than the ones without projection heads. Many experiments were carried out to select the final structure of the model. As a result, the Danse-Relu-Dense structure was chosen as the projection head.</p><p>Last but not least: loss function. Normalized Temperature-scaled Cross-Entropy loss (NT-Xnet) was used as a contrastive loss, which aims to pull close representations and push away different representations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/0*LQd6QKiswgcsiTda" /><figcaption>(1) Normalized Temperature-scaled Cross-Entropy loss (NT-Xnet)</figcaption></figure><p>First, the similarity between one positive and all other negative representations is calculated. Cosine similarity is used as a distance metric. Next, the loss function for a positive pair of examples (i, j) is calculated with Negative Cross-Entropy loss (1). The final loss is computed across all positive pairs.</p><p>As mentioned above, a large number of experiments were carried out, and in the end, the research team presented the main results that led to their success: which is <strong>scaling up significantly improves performance</strong>. Scaling means using a larger batch size (for the model training batch size varies from 256 to 8192), larger backbone, and training for longer epochs (SimCLR can continue its improvement even after 800 epochs of training).</p><h4><strong>Swapping Assignments between multiple Views (SwAV)</strong></h4><p>The recent SoTA in self-supervised learning is SwAV presented by Facebook Research. Like SimCLR, SwAV architecture was also created based on experimenting with different components of self-supervised learning techniques. However, their success is based on two major changes they made in architecture: augmentation type and clustering instead of projection head. Here is the final structure of their architecture:</p><ol><li>Multy-crop augmentation</li><li>Base encoder</li><li>Clustering</li><li>Loss with “Swapping” prediction problem.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LFsdzRzfw5bggD6y" /><figcaption>SwAV architecture: Image source [2]</figcaption></figure><p>For the data augmentation SwAV again uses geometric and color transformations, however, the authors present a new unique and effective technique for pretext task creation. The technique is called Multi-Crop augmentation. This type of augmentation creates multiple views of the same images without increasing computational cost. After augmentation, we get two variations of standard high resolution (224X224) images and also many sampled low resolution (96x96) views of the same image. These views are cropped and color augmented from the main sample image. The use of low-resolution views helps to get better samples of images without increasing computation too much.</p><p>For the base encoder again ResNets with different depths were used.</p><p>One of the major differences of this model is the third component: cluster assignments. The main problem of clustering algorithms is that they work offline which means we need to feed the entire dataset to the algorithm to get assignments. That can’t work in a self-supervised context. SwAV authors treat clustering as an online task and feed clustering algorithms with mini-batch data. The Sinkhorn Knopp algorithm was used for clustering.</p><p>To calculate the loss SwAV authors use the swapping problem technique.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/0*x3Ldq3y6YDEgKfpw" /><figcaption>(2) Swapped loss</figcaption></figure><p>Here, l(z,q) denotes the cross-entropy where q is called code (the output of clustering) and z is the softmax of the dot product between the image features and all the prototypes. Codes can be considered as ground truths. The assumption is that the representations of two different views of the same image should contain similar information, thus it should be possible to predict one view’s code from another view’s representation.</p><h4><strong>Benchmark results</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/298/0*kknAxd5jVeKZaNd7" /><figcaption>Benchmark results of self-supervised models: Image source [2]</figcaption></figure><h4><strong>Conclusion</strong></h4><p>The Benchmark results on downstream tasks show that self-supervised techniques can’t surpass supervised learning techniques for now, but yet these are powerful methods that can be used in different domains where data annotations scarcity or difficulty to label data are challenging.</p><h4>Literature</h4><p>[1] <a href="https://arxiv.org/pdf/2011.00362.pdf">https://arxiv.org/pdf/2011.00362.pdf</a></p><p>[2] <a href="https://arxiv.org/pdf/2006.09882.pdf">https://arxiv.org/pdf/2006.09882.pdf</a></p><p>[3] <a href="https://arxiv.org/pdf/2002.05709.pdf">https://arxiv.org/pdf/2002.05709.pdf</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93171f695140" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/review-on-self-supervised-contrastive-learning-93171f695140">Review on Self-Supervised Contrastive Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>