<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dhruv Matani on Medium]]></title>
        <description><![CDATA[Stories by Dhruv Matani on Medium]]></description>
        <link>https://medium.com/@dhruvbird?source=rss-63f5d5495279------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*FYD0sd3nwpCGl88e7tNYUQ@2x.jpeg</url>
            <title>Stories by Dhruv Matani on Medium</title>
            <link>https://medium.com/@dhruvbird?source=rss-63f5d5495279------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 06:28:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dhruvbird/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Interpreting Weight Regularization In Machine Learning]]></title>
            <link>https://medium.com/data-science/interpreting-weight-regularization-in-machine-learning-99f2677f7ef5?source=rss-63f5d5495279------2</link>
            <guid isPermaLink="false">https://medium.com/p/99f2677f7ef5</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[regularization]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[l1-and-l2-regularization]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Dhruv Matani]]></dc:creator>
            <pubDate>Fri, 23 Aug 2024 23:55:25 GMT</pubDate>
            <atom:updated>2024-08-23T23:55:25.208Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why do L1 and L2 regularization result in model sparsity and weight shrinkage? What about L3 regularization? Keep reading to find out more!</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p3mFUtIvvfNqRqJQ" /><figcaption>Photo by <a href="https://unsplash.com/@dkoi?utm_source=medium&amp;utm_medium=referral">D koi</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Co-authored with <a href="https://medium.com/@brocolishbroxoli">Naresh Singh</a>.</p><h3>Introduction</h3><p>After reading this article, you’ll be very well equipped with the tools and reasoning capability to think about the effects of any Lk regularization term and decide if it applies to your situation.</p><h4>What is regularization in machine learning?</h4><p>Let’s look at some definitions on the internet and generalize based on those.</p><ol><li>Regularization is a set of methods for reducing overfitting in machine learning models. Typically, regularization trades a marginal decrease in training accuracy for an increase in generalizability. (<a href="https://www.ibm.com/topics/regularization">IBM</a>)</li><li>Regularization makes models stable across different subsets of the data. It reduces the sensitivity of model outputs to minor changes in the training set. (<a href="https://www.geeksforgeeks.org/regularization-in-machine-learning/">geeksforgeeks</a>)</li><li>Regularization in machine learning serves as a method to forestall a model from overfitting. (<a href="https://www.simplilearn.com/tutorials/machine-learning-tutorial/regularization-in-machine-learning">simplilearn</a>)</li></ol><p>In general, regularization is a technique to prevent the model from overfitting and to allow the model to generalize its predictions on unseen data. Let’s look at the role of weight regularization in particular.</p><h4>Why use weight regularization?</h4><p>One could employ many forms of regularization while training a machine learning model. Weight regularization is one such technique, which is the focus of this article. Weight regularization means applying some constraints on the learnable weights of your machine learning model so that they allow the model to generalize to unseen inputs.</p><p>Weight regularization improves the performance of neural networks by penalizing the weight matrices of nodes. This penalty discourages the model from having large parameter (weight) values. It helps control the model’s ability to fit the noise in the training data. Typically, the biases in the machine learning model are not subject to regularization.</p><h4>How is regularization implemented in deep neural networks?</h4><p>Typically, a regularization loss is added to the model’s loss during training. It allows us to control the model’s weights during training. The formula looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/0*G09CJZlFDtlU8Mqq" /></figure><p><em>Figure-1: Total loss as a sum of the model loss and regularization loss. k is a floating point value and indicates the regularization norm. Alpha is the weighting factor for the regularization loss.</em></p><p>Typical values of k used in practice are 1 and 2. These are called the L1 and L2 regularization schemes.</p><p>But why do we use just these two values for the most part, when in fact there are infinitely many values of k one could use? Let’s answer this question with an interpretation of the L1 and L2 regularization schemes.</p><h3>Interpretation of different weight regularization types</h3><p>The two most common types of regularization used for machine learning models are L1 and L2 regularization. We will start with these two, and continue to discuss some unusual regularization types such as L0.5 and L3 regularization. We will take a look at the gradients of the regularization losses and plot them to intuitively understand how they affect the model weights.</p><h4>L1 regularization</h4><p>L1 regularization adds the average of the absolute value of the weights together as the regularization loss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/0*E-3zOsD8HUdZtzth" /></figure><p><em>Figure-2: L1 regularization loss and its partial derivative with respect to each weight Wi.</em></p><p>It has the effect of adjusting the weights by a constant (in this case alpha times the learning rate) in the direction that minimizes the loss. Figure 3 shows a graphical representation of the function and its derivative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*NGQ5-iz4BIbOz-Aq" /></figure><p><em>Figure-3: The blue line is |w| and the red line is the derivative of |w|.</em></p><p>You can see that the derivative of the L1 norm is a constant (depending on the sign of w), which means that the gradient of this function only depends on the sign of w and not its magnitude. The gradient of the L1 norm is not defined at w=0.</p><p>It means that the weights are moved towards zero by a constant value at each step during backpropagation. Throughout training, it has the effect of driving the weights to converge at zero. That is why the L1 regularization makes a model sparse (i.e. some of the weights become 0). It might cause a problem in some cases if it ends up making a model too sparse. The L2 regularization does not have this side-effect. Let’s discuss it in the next section.</p><h4>L2 regularization</h4><p>L2 regularization adds the average of the square of the absolute value of the weights together as the regularization loss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/524/0*aoPs8F32JFrh3dtq" /></figure><p><em>Figure-4: L2 regularization loss and its partial derivative with respect to each weight Wi.</em></p><p>It has the effect of adjusting each weight by a multiple of the weight itself in the direction that minimizes the loss. Figure 5 shows a graphical representation of the function and its derivative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*wM3TtQgWjrJ2ahUY" /></figure><p><em>Figure-5: The blue line is pow(|w|, 2) and the red line is the derivative of pow(|w|, 2).</em></p><p>You can see that the derivative of the L2 norm is just the sign-adjusted square root of the norm itself. The gradient of the L2 norm depends on both the sign and magnitude of the weight.</p><p>This means that at every gradient update step, the weights will be adjusted toward zero by an amount that is proportional to the weight’s value. Over time, this has the effect of drawing the weights toward zero, but never exactly zero, since subtracting a constant factor of a value from the value itself never makes the result exactly zero unless it is zero to begin with. The L2 norm is commonly used for weight decay during machine learning model training.</p><p>Let’s consider L0.5 regularization next.</p><h4>L0.5 regularization</h4><p>L0.5 regularization adds the average of the square root of the absolute value of the weights together as the regularization loss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/0*yPan8fxoc8uK2AuO" /></figure><p><em>Figure-6: L0.5 regularization loss and its partial derivative with respect to each weight Wi.</em></p><p>This has the effect of adjusting each weight by a multiple (in this case alpha times the learning rate) of the inverse square root of the weight itself in the direction that minimizes the loss. Figure 7 shows a graph of the function and its derivative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*VemXTNSFaozguJHu" /></figure><p><em>Figure-7: The blue line is pow(|w|, 0.5) and the red line is the derivative of pow(|w|, 0.5).</em></p><p>You can see that the derivative of the L0.5 norm is a discontinuous function, which peaks at the positive values of w close to 0 and it reaches negative infinity for the negative values of w close to 0. Further, we can draw the following conclusions from the graph:</p><ol><li>As |w| tends to 0, the magnitude of the gradient tends to infinity. During backpropagation, these values of w will quickly swing to past 0 because large gradients will cause a large change in the value of w. In other words, negative w will become positive and vice-versa. This cycle of flip flops will continue to repeat itself.</li><li>As |w| increases, the magnitude of the gradient decreases. These values of w are stable because of small gradients. However, with each backpropagation step, the value of w will be drawn closer to 0.</li></ol><p>This is hardly what one would want from a weight regularization routine, so it’s safe to say that L0.5 isn’t a great weight regularizer. Let’s consider L3 regularization next.</p><h4>L3 regularization</h4><p>L3 regularization adds the average of the cube of the absolute value of the weights together as the regularization loss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/0*MYTHxif8NbG7tSDd" /></figure><p><em>Figure-8: L3 regularization loss and its partial derivative with respect to each weight Wi.</em></p><p>This has the effect of adjusting each weight by a multiple (in this case alpha times the learning rate) of the square of the weight itself in the direction that minimizes the loss.</p><p>Graphically, this is what the function and its derivative look like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*RIITUNu9juPjjj6r" /></figure><p><em>Figure-9: The blue line is pow(|w|, 3) and the red line is the derivative of pow(|w|, 3).</em></p><p>To really understand what’s going on here, we need to zoom in to the chart around the w=0 point.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Apf9lBSGk9tI0Wep" /></figure><p><em>Figure-10: The blue line is pow(|w|, 3) and the red line is the derivative of pow(|w|, 3), zoomed in at small values of w around 0.0.</em></p><p>You can see that the derivative of the L3 norm is a continuous and differentiable function (despite the presence of |w| in the derivative), which has a large magnitude at large values of w and a small magnitude for small values of w.</p><p>Interestingly, the gradient is very close to zero for very small values of w around the 0.0 mark.</p><p>The interpretation of the gradient for L3 is interesting.</p><ol><li>For large values of w, the magnitude of the gradient is large. During backpropagation, these values will be pushed towards 0.</li><li>Once the weight w reaches an inflection point (close to 0.0), the gradient almost vanishes, and the weights will stop getting updated.</li></ol><p>The effect is that it will drive the weights with large magnitudes close to 0, but not exactly 0.</p><p>Let’s consider higher norms to see how this plays out in the limiting case.</p><h4>Beyond L3 regularization</h4><p>To understand what happens for Linfinity, we need to see what happens in the case of the L10 regularization case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XCb4xD-LriRW4nqk" /></figure><p><em>Figure-11: The blue line is pow(|w|, 10) and the red line is the derivative of pow(|w|, 10), zoomed in at small values of w around 0.0.</em></p><p>One can see that the gradients for values of |w| &lt; 0.5 are extremely small, which means that regularization won’t be effective for those values of w.</p><h4>Exercise</h4><p>Based on everything we saw above, L1 and L2 regularization are fairly practical based on what you want to achieve. As an exercise, try to reason about the behavior of the L1.5 regularization, whose chart is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TyclirXp19rbAnrk" /></figure><p><em>Figure-12: The blue line is pow(|w|, 1.5) and the red line is the derivative of pow(|w|, 1.5).</em></p><h3>Conclusion</h3><p>We took a visual and intuitive look at the L1 and L2 (and in general Lk) regularization terms to understand why L1 regularization results in sparse model weights and L2 regularization results in model weights close to 0. Framing the solution as inspecting the resulting gradients is extremely valuable during this exercise.</p><p>We explored L0.5, L3, and L10 regularization terms and graphically, and you (the reader) reasoned about regularization terms between L1 and L2 regularization, and developed an intuitive understanding of what implications it would have on a model’s weights.</p><p>We hope that this article has added to your toolbox of tricks you can use when considering regularization strategies during model training to fine-tuning.</p><p>All the charts in this article were created using the online desmos graphing calculator. <a href="https://www.desmos.com/calculator/hp5bjnh8ul">Here is a link</a> to the functions used in case you wish to play with them.</p><p>All the images were created by the author(s) unless otherwise mentioned.</p><h3>References</h3><p>We found the following articles useful while researching the topic, and we hope that you find them useful too!</p><ol><li><a href="https://stats.stackexchange.com/questions/269298/why-do-we-only-see-l-1-and-l-2-regularization-but-not-other-norms">Stackexchange discussion</a></li><li><a href="https://towardsdatascience.com/courage-to-learn-ml-demystifying-l1-l2-regularization-part-3-ee27cd4b557a">TDS: Demystifying L1 &amp; L2 Regularization (part 3)</a></li><li><a href="https://www.linkedin.com/pulse/intuitive-visual-explanation-differences-between-l1-l2-xiaoli-chen/">Visual explanation of L1 and L2 regularization</a></li><li>Deep Learning by Ian Goodfellow</li><li>An introduction to statistical learning by Gareth James</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=99f2677f7ef5" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/interpreting-weight-regularization-in-machine-learning-99f2677f7ef5">Interpreting Weight Regularization In Machine Learning</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Challenges of Detecting AI-Generated Text]]></title>
            <link>https://medium.com/data-science/challenges-of-detecting-ai-generated-text-6d85bf779448?source=rss-63f5d5495279------2</link>
            <guid isPermaLink="false">https://medium.com/p/6d85bf779448</guid>
            <category><![CDATA[language-model]]></category>
            <category><![CDATA[perplexity]]></category>
            <category><![CDATA[getting-started]]></category>
            <category><![CDATA[gptzero]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Dhruv Matani]]></dc:creator>
            <pubDate>Wed, 27 Sep 2023 04:35:18 GMT</pubDate>
            <atom:updated>2023-09-27T04:35:18.860Z</atom:updated>
            <content:encoded><![CDATA[<h4>We’ll take an in-depth look at the challenges of detecting AI-generated text, and the effectiveness of the techniques used in practice.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lWfXGsaUElKfgFG2" /><figcaption>Photo by <a href="https://unsplash.com/@houcinencibphotography?utm_source=medium&amp;utm_medium=referral">Houcine Ncib</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Co-authored with <a href="https://medium.com/@brocolishbroxoli">Naresh Singh</a>.</p><h3>Table of contents</h3><ol><li><a href="#86be">Introduction</a></li><li><a href="#64d4">Building an intuition for text source detection</a></li><li><a href="#542b">What is the perplexity of a language model?</a></li><li><a href="#dc6a">Computing the perplexity of a language model’s prediction</a></li><li><a href="#f256">Detecting AI-generated text</a></li><li><a href="#b56a">Misinformation</a></li><li><a href="#a778">What’s next?</a></li><li><a href="#8446">Conclusion</a></li></ol><h3>Introduction</h3><p>AI-assisted technologies for writing articles or posts are everywhere now! ChatGPT has unlocked numerous applications of language-based AI, and the use of AI for any kind of content generation has gone through the roof.</p><p>However, in school assignments such as creative writing, students are expected to create their own content. However, due to the popularity and effectiveness of AI for such tasks, they might be tempted to use it. In such cases, it is important for teachers to have access to reliable and trustworthy tools to detect AI-generated content.</p><p>This article aims to provide an intuition as well as the technical specification of building such a tool. It is intended for the readers who want to understand how AI detection works intuitively as well as the technical audience who wants to build such a tool.</p><p>Let’s jump straight in!</p><h3>Building an intuition for text source detection</h3><p>At a high level, we’re trying to answer the question, <em>“How likely is it that an AI language model such as GPT-3 has generated all or part of this text?”</em></p><p>If you step back, you will realize this is a typical daily scenario. For example, how likely would it be for your mother to say the following sentence to you?</p><ol><li>Dear, please go to bed before 8:00 pm.</li><li>Dear, please go to bed after 11:00 pm.</li></ol><p>We would guess that the former is much more likely than the latter because you have built an understanding of the world around you and have a sense of which events are more likely to occur.</p><p>This is exactly how a language model works. Language models learn something about the world around them, specifically language. They learn to predict the next token or word given an incomplete sentence.</p><p>In the example above, if you’re told that your mother is saying something, and what has been said so far is <em>“Dear, please go to bed”</em>, then the most likely continuation of the sentence is going to be “before 8:00 pm”, and not “after 11:00 pm”. In technical terms, we say that you’d be more <em>perplexed</em> to hear the 2nd sentence as opposed to the 1st one.</p><p>Let’s dig deeper into what perplexity means in the context of a language mode.</p><h3>What is the perplexity of a language model?</h3><p>According to <a href="https://www.dictionary.com/browse/perplexity">dictionary.com</a>, perplexity is defined as</p><blockquote>the state of being perplexed; confusion; uncertainty.</blockquote><p>In the real world, if you encounter a situation that you don’t expect, you will be more perplexed than if you encounter a situation that you expect. For example, when driving on a road, if you see a traffic light, then you’re less likely to be perplexed as opposed to if you see a goat crossing the street.</p><p>Similarly, for a language model that is trying to predict the next word in a sentence, we say that the model will perplex us if it completes the sentence using a word that we don’t expect v/s if it uses a word that we expect. Some examples.</p><p><strong>Sentences with a low perplexity would look like the following</strong></p><ol><li>It’s a sunny day outside.</li><li>I’m sorry I missed the flight and was unable to reach the national park in time.</li></ol><p><strong>Sentences with a high perplexity would look like the following</strong></p><ol><li>It’s a bread day outside.</li><li>I’m convenient missed light outside and could not reach national park.</li></ol><p>Next, let’s look at how we can compute the perplexity of a prediction made by the language model.</p><h3>Computing the perplexity of a language model’s prediction</h3><p>The perplexity of a language model is related to the probability of being able to unsurprisingly predict the next token (word) of a sentence.</p><p>Suppose we train a language model with a vocabulary of 6600 tokens and run a single prediction step to have the model predict the next token in a sentence. Let’s assume that the likelihood of picking this token is 5/6600 (i.e., this token was not very likely). Its perplexity is the inverse of the probability which is 6600/5 = 1320, which suggests that <em>we are highly perplexed</em> by this suggestion. If the likelihood of picking this token was 6000/6600 instead, then the perplexity would be 6600/6000 = 1.1 which suggests that we are just slightly perplexed by this suggestion.</p><p>Hence, the perplexity of our model on a more likely prediction is lower than the perplexity of our model on a less likely prediction.</p><p>The perplexity of predicting all tokens in a sentence “x” is formally defined as the Nth root of the inverse of the product of the token probabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/0*didIcoYhriOI75OG" /></figure><p>However, to ensure numerical stability, we can define it in terms of the log function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/0*t4yW3Er0YKKwFedu" /></figure><p>Which is e (2.71828) to the power of the average negative log-likelihood of the predicted token being the ground truth token.</p><h4>Training and validation perplexity</h4><p>The model’s training and validation perplexity can be computed directly from the batch or epoch loss.</p><h4>Prediction perplexity</h4><p>One can’t compute the prediction perplexity since it requires having a set of ground truth labels for each prediction.</p><h4>PyTorch code to compute perplexity</h4><p>Let’s assume that the variable <em>probs</em> is a <em>torch.Tensor</em> of shape <em>(sequence_length,)</em>, which contains the probability of the ground-truth token being predicted by the language model at that position in the sequence.</p><p>The per-token perplexity can be computed using this code.</p><pre>token_perplexity = (probs.log() * -1.0).exp()<br>print(f&quot;Token Perplexity: {token_perplexity}&quot;)</pre><p>The sample perplexity can be computed using this code.</p><pre># The perplexity is e^(average NLL).<br>sentence_perplexity = (probs.log() * -1.0).mean().exp().item()<br>print(f&quot;Sentence Perplexity: {sentence_perplexity:.2f}&quot;)</pre><p>Next, let’s look at the code that computes this per-token probability given a sentence.</p><pre>def get_pointwise_loss(self, inputs: List[str], tok):<br>    self.model.eval()<br>    all_probs = []<br>    with torch.inference_mode():<br>        for input in inputs:<br>            ids_list: List[int] = tok.encode(input).ids<br>            # ids has shape (1, len(ids_list))<br>            ids: Torch.Tensor = torch.tensor(ids_list, device=self.device).unsqueeze(0)<br><br>            # probs below is the probability that the token at that location<br>            # completes the sentence (in ids) so far.<br>            y = self.model(ids)<br>            criterion = nn.CrossEntropyLoss(reduction=&#39;none&#39;, ignore_index=0)<br><br>            # Compute the loss starting from the 2nd token in the model&#39;s output.<br>            loss = criterion(y[:,:,:-1], ids[:,1:])<br><br>            # To compute the probability of each token, we need to compute the<br>            # negative of the log loss and exponentiate it.<br>            loss = loss * -1.0<br><br>            # Set the probabilities that we are not interested in to -inf.<br>            # This is done to make the softmax set these values to 0.0<br>            loss[loss == 0.0] = float(&quot;-inf&quot;)<br><br>            # probs holds the probability of each token&#39;s prediction<br>            # starting from the 2nd token since we don&#39;t want to include<br>            # the probability of the model predicting the beginning of<br>            # a sentence given no existing sentence context.<br>            #<br>            # To compute perplexity, we should probably ignore the first<br>            # handful of predictions of the model since there&#39;s insufficient<br>            # context. We don’t do that here, though.<br>            probs = loss.exp()<br>            all_probs.append(probs)<br>        #<br>    #<br>    return all_probs<br>#</pre><p>Now that we know something about how language models function, and how we can compute the per-token as well as per-sentence perplexity, let’s try to tie it all together and take a look at how one can leverage this information to build a tool that can detect if some text was AI-generated.</p><h3>Detecting AI-generated text</h3><p>We have all the ingredients we need to check if a piece of text is AI-generated. Here’s everything we need:</p><ol><li>The text (sentence or paragraph) we wish to check.</li><li>The tokenized version of this text, tokenized using the tokenizer that was used to tokenize the training dataset for this model.</li><li>The trained language model.</li></ol><p>Using 1, 2, and 3 above, we can compute the following:</p><ol><li>Per-token probability as predicted by the model.</li><li>Per-token perplexity using the per-token probability.</li><li>Total perplexity for the entire sentence.</li><li>The perplexity of the model on the training dataset.</li></ol><p>To check if a text is AI-generated, we need to compare the sentence perplexity with the model’s perplexity scaled by a fudge-factor, alpha. If the sentence perplexity is more than the model’s perplexity with scaling, then it’s probably human-written text (i.e. not AI-generated). Otherwise, it’s probably AI-generated. The reason for this is that we expect the model to not be perplexed by text it would generate itself, so if it encounters some text that it itself would not generate, then there’s reason to believe that the text isn’t AI-generated. If the perplexity of the sentence is less than or equal to the model’s training perplexity with scaling, then it’s likely that it was generated using this language model, but we can’t be very sure. This is because it’s possible for a human to have written that text, and it just happens to be something that the model could also have generated. After all, the model was trained on a lot of human-written text so in some sense, the model represents an “average human’s writing”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I-Kw2ouWUiEoXNAx" /></figure><p><em>ppx(x)</em> in the formula above means the perplexity of the input “x”.</p><p>Next, let’s take a look at examples of human-written v/s AI-generated text.</p><h4>Examples of AI-generated v/s human written text</h4><p>We’ve written some Python code that colors each token in a sentence based on its perplexity relative to the model’s perplexity. The first token is always coloured black if we don’t consider its perplexity. Tokens that have a perplexity that is less than or equal to the model’s perplexity with scaling are coloured red, indicating that they may be AI-generated, whereas the tokens with higher perplexity are coloured green, indicating that they were definitely not AI-generated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*6ORGZ_vBlbw1YA1z" /></figure><p>The numbers in the square brackets before the sentence indicate the perplexity of the sentence as computed using the language model. Note that some words are part red and part blue. This is due to the fact that we used a subword tokenizer.</p><p>Here’s the code that generates the HTML above.</p><pre>def get_html_for_token_perplexity(tok, sentence, tok_ppx, model_ppx):<br>    tokens = tok.encode(sentence).tokens<br>    ids = tok.encode(sentence).ids<br>    cleaned_tokens = []<br>    for word in tokens:<br>        m = list(map(ord, word))<br>        m = list(map(lambda x: x if x != 288 else ord(&#39; &#39;), m))<br>        m = list(map(chr, m))<br>        m = &#39;&#39;.join(m)<br>        cleaned_tokens.append(m)<br>    #<br>    html = [<br>        f&quot;&lt;span&gt;{cleaned_tokens[0]}&lt;/span&gt;&quot;,<br>    ]<br>    for ct, ppx in zip(cleaned_tokens[1:], tok_ppx):<br>        color = &quot;black&quot;<br>        if ppx.item() &gt;= 0:<br>            if ppx.item() &lt;= model_ppx * 1.1:<br>                color = &quot;red&quot;<br>            else:<br>                color = &quot;green&quot;<br>            #<br>        #<br>        html.append(f&quot;&lt;span style=&#39;color:{color};&#39;&gt;{ct}&lt;/span&gt;&quot;)<br>    #<br>    return &quot;&quot;.join(html)<br>#</pre><p>As we can see from the examples above, if a model detects some text as human-generated, it’s definitely human-generated, but if it detects the text as AI-generated, there’s a chance that it’s not AI-generated. So why does this happen? Let’s take a look next!</p><h4>False positives</h4><p>Our language model is trained on a LOT of text written by humans. It’s generally hard to detect if something was written (digitally) by a specific person. The model’s inputs for training comprise many, many different styles of writing, likely written by a large number of people. This causes the model to learn many different writing styles and content. It’s very likely that your writing style very closely matches the writing style of some text the model was trained on. This is the result of false positives and why the model can’t be sure that some text is AI-generated. However, the model can be sure that some text was human-generated.</p><p><strong>OpenAI:</strong> OpenAI recently announced that it would discontinue its tools for detecting AI-generated text, citing a low accuracy rate (Source: <a href="https://tech.hindustantimes.com/tech/news/openai-kills-off-its-own-ai-text-detection-tool-shocking-reason-behind-it-71690364760759.html">Hindustan Times</a>).</p><blockquote>The original version of the AI classifier tool had certain limitations and inaccuracies from the outset. Users were required to input at least 1,000 characters of text manually, which OpenAI then analyzed to classify as either AI or human-written. Unfortunately, the tool’s performance fell short, as it properly identified only 26 percent of AI-generated content and mistakenly labeled human-written text as AI about 9 percent of the time.</blockquote><p>Here’s the <a href="https://openai.com/blog/new-ai-classifier-for-indicating-ai-written-text">blog post from OpenAI</a>. It seems like they used a different approach compared to the one mentioned in this article.</p><blockquote>Our classifier is a language model fine-tuned on a dataset of pairs of human-written text and AI-written text on the same topic. We collected this dataset from a variety of sources that we believe to be written by humans, such as the pretraining data and human demonstrations on prompts submitted to InstructGPT. We divided each text into a prompt and a response. On these prompts, we generated responses from a variety of different language models trained by us and other organizations. For our web app, we adjust the confidence threshold to keep the false positive rate low; in other words, we only mark text as likely AI-written if the classifier is very confident.</blockquote><p><strong>GPTZero:</strong> Another popular AI-generated text detection tool is <a href="https://gptzero.me/">GPTZero</a>. It seems like GPTZero uses <a href="https://ecoagi.ai/topics/ChatGPT/high-perplexity-score-gpt-zero">perplexity and burstiness</a> to detect AI-generated text. <em>“Burstiness refers to the phenomenon where certain words or phrases appear in bursts within a text. In other words if a word appears once in a text, it’s likely to appear again in close proximity”</em> (<a href="https://ecoagi.ai/topics/ChatGPT/high-perplexity-score-gpt-zero">source</a>).</p><p>GPTZero claims to have a very high success rate. According to the <a href="https://gptzero.me/faq">GPTZero FAQ</a>, <em>“At a threshold of 0.88, 85% of AI documents are classified as AI, and 99% of human documents are classified as human.”</em></p><h4>The generality of this approach</h4><p>The approach mentioned in this article doesn’t generalize well. What we mean by this is that if you have 3 language models, for example, GPT3, GPT3.5, and GPT4, then you must run the input text through all the 3 models and check perplexity on all of them to see if the text was generated by any one of them. This is because each model generates text slightly differently, and they all need to independently evaluate text to see if any of them may have generated the text.</p><p>With the proliferation of large language models in the world as of August 2023, it seems unlikely that one can check any piece of text as having originated from any of the language models in the world.</p><p>In fact, new models are being trained every day, and trying to keep up with this rapid progress seems hard at best.</p><p>The example below shows the result of asking our model to predict if the sentences generated by ChatGPT are AI-generated or not. As you can see, the results are mixed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*D8c_I4jJzNy5urcs" /><figcaption>The sentences in the purple box are correctly identified as AI-generated by our model, whereas the rest are incorrectly identified as human written.</figcaption></figure><p>There are many reasons why this may happen.</p><ol><li><strong>Train corpus size:</strong> Our model is trained on very little text, whereas ChatGPT was trained on terabytes of text.</li><li><strong>Data distribution:</strong> Our model is trained on a different data distribution as compared to ChatGPT.</li><li><strong>Fine-tuning:</strong> Our model is just a GPT model, whereas ChatGPT was fine-tuned for chat-like responses, making it generate text in a slightly different tone. If you had a model that generates legal text or medical advice, then our model would perform poorly on text generated by those models as well.</li><li><strong>Model size:</strong> Our model is very small (less than 100M parameters compared to &gt; 200B parameters for ChatGPT-like models).</li></ol><p>It’s clear that we need a better approach if we hope to provide a reasonably high-quality result to check if any text is AI-generated.</p><p>Next, let’s take a look at some misinformation about this topic circulating around the internet.</p><h3>Misinformation</h3><p>Some articles interpret perplexity incorrectly. For example, if you search google.com for <a href="https://www.google.com/search?q=does+human+written+content+have+high+or+low+perplexity%3F">“does human written content have high or low perplexity?”</a>, you’ll get the <a href="https://www.linkedin.com/pulse/how-write-content-using-chatgpt-outsmarts-ai-tools-99-dasun-sucharith/">following result</a> in the first position.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*oJJVEAmZmJ1grvT4" /></figure><p>This is incorrect since human-written content typically has higher perplexity compared with AI-generated content.</p><p>Let’s take a look at techniques that researchers in this area are exploring to do better than where we are.</p><h3>What’s next?</h3><p>We’ve established that detecting AI-generated text is a hard problem, and that success rates are so low that they aren’t any better than guessing. Let’s look at what state of the art techniques researchers in this area are exploring to get a better handle on things.</p><p><strong>Watermarking:</strong> <a href="https://arstechnica.com/ai/2023/07/openai-google-will-watermark-ai-generated-content-to-hinder-deepfakes-misinfo/">OpenAI and Google have promised to watermark AI generated text</a> so that it’s possible to identify it programmatically.</p><p>The technical details of how this watermark might work are unclear and neither company has disclosed any details related to it.</p><p>Even if OpenAI and Google adopt a watermarking technique, we can’t be certain that every language model that is deployed out there will have a watermark included. It would still be possible for people to deploy their own models to generate text and put it out there in the wild. Even if companies decide to watermark generated text, it’s not clear if this will be a standard or if every company will have their own proprietary strategy and potentially paid tool to check if any text was generated using their AI-based text-generation tools. If it’s an open standard there’s a chance that people will be able to work around it unless it’s something like a cryptographic cipher that requires a ton of computation to undo. If it’s not an open standard, then people will be at the mercy of these companies to provide open and free access to the tools and APIs needed to perform these checks. There’s also the question of how effective these will be in the long run since it may even be possible to train models to ingest the AI-generated text with a watermark, and return AI-generated text without a watermark.</p><p><a href="https://www.nytimes.com/interactive/2023/02/17/business/ai-text-detection.html">This article</a> discusses a possible technique for adding a watermark for AI-generated text, and mentions the significant challenges with the approach.</p><p><strong>Personalization:</strong> In our opinion, this problem of detecting AI-generated text is going to remain challenging in the near term. We believe that strategies will need to get more intrusive and personalized to be more effective. For example, instead of asking if some text is AI-generated it may be more reasonable to ask if this text was written by a specific person. However, this would require the system to have access to large amounts of text written by that specific person. Additionally, the problem becomes more complex if something was written by more than one person, such as this article.</p><p>Let’s look at the impact such a personalized system for detecting human-written text would have on educators and students.</p><p>If such a solution existed, <strong>educators</strong> would be more inclined to hand out individual assignments to students instead of group assignments. This would also require every student to first provide a large amount of text that they themselves have written. This could mean spending several hours in-person at a university before enrolling for classes. Surely, this would have a negative effect on the ability to teach students the importance of working together as a team to accomplish a common goal.</p><p>On the other hand, having access to AI-based text generation could free students in some cases to focus on the actual problem at hand such as performing the research or literature study instead of spending time writing out their learnings in a polished manner. One can imagine that students will end up spending more of their time learning concepts and techniques in math or science class as opposed to writing about it. That part can be taken care of by the AI.</p><h3>Conclusion</h3><p>In this article, we built intuition around how one can detect AI-generated text. The main metric we can use is the perplexity of the generated text. We saw some PyTorch code to check if a given text may be AI-generated using the perplexity of that text. We also saw some of the drawbacks of this approach, including the possibility of false positives. We hope this helps you understand and appreciate the nitty-gritty details behind detecting AI-generated text.</p><p>This is a constantly evolving space and researchers are trying hard to figure out a way to detect AI-generated text with higher accuracy. The impact of this technology to our lives promises to be significant, and in many ways unknown.</p><p>While we discussed the techniques for detecting AI-generated text, we assumed that the entire text is either human-written or AI-generated. In practice, text tends to be partially human-written and partially AI-generated, and this complicates things considerably.</p><p>If you want to read about additional approaches to detect AI-generated text, such as the use of the burstiness metric, you can read about them <a href="https://towardsdatascience.com/how-do-we-know-if-a-text-is-ai-generated-82e710ea7b51">here</a>.</p><p>All the images in this article (except for the first one) were created by the author(s).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6d85bf779448" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/challenges-of-detecting-ai-generated-text-6d85bf779448">Challenges of Detecting AI-Generated Text</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Language Models for Sentence Completion]]></title>
            <link>https://medium.com/data-science/language-models-for-sentence-completion-6a5298a85e43?source=rss-63f5d5495279------2</link>
            <guid isPermaLink="false">https://medium.com/p/6a5298a85e43</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[next-word-prediction]]></category>
            <category><![CDATA[language-model]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <dc:creator><![CDATA[Dhruv Matani]]></dc:creator>
            <pubDate>Fri, 15 Sep 2023 06:54:32 GMT</pubDate>
            <atom:updated>2023-09-15T06:54:32.783Z</atom:updated>
            <content:encoded><![CDATA[<h4>A practical application of a language model that picks the most likely candidate word that extends an English sentence by a single word</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*GCVKIOPDnMM1XI5Z" /><figcaption>Photo by <a href="https://unsplash.com/@brett_jordan?utm_source=medium&amp;utm_medium=referral">Brett Jordan</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Co-authored with <a href="https://medium.com/@brocolishbroxoli">Naresh Singh</a>.</p><h3>Table of contents</h3><p><a href="#6d2d"><strong>Introduction</strong></a><br><a href="#93ee"><strong>Problem Statement</strong></a><br><a href="#7fa3"><strong>Brainstorming a solution</strong></a></p><ul><li><a href="#9890">Algorithms and Data Structures</a></li><li><a href="#d107">NLP (Natural Language Processing)</a></li><li><a href="#e846">Deep Learning (Neural Networks)</a></li></ul><p><a href="#2ecd"><strong>An LSTM model</strong></a></p><ul><li><a href="#4a4c">Tokenization</a></li><li><a href="#75ba">The PyTorch Model</a></li><li><a href="#7ad8">Using the model to prune invalid suggestions</a></li><li><a href="#1049">Computing the next word probability</a></li></ul><p><a href="#9241"><strong>A Transformer model</strong></a></p><p><a href="#597e"><strong>Conclusion</strong></a></p><h3>Introduction</h3><p>Language models such as GPT have become very popular recently and are being used for a variety of text generation tasks, such as in ChatGPT or other conversational AI systems. These language models are huge, often exceeding tens of billions of parameters, and need a lot of computing resources and money to run.</p><p>In the context of English language models, these massive models are <a href="https://www.reddit.com/r/MachineLearning/comments/ib4rth/d_why_does_models_like_gpt3_or_bert_dont_have/">over-parameterized</a> since they use the model’s parameters to memorize and learn aspects of our world instead of just modeling the English language. We can likely use a much smaller model if we have an application that requires the model to understand just the language and its constructs.</p><p>The complete code for running inference on the trained model can be found <a href="https://github.com/dhruvbird/ml-notebooks/blob/main/next_word_probability/inference-next-word-probability.ipynb">in this notebook</a>.</p><h3>Problem Statement</h3><p>Let’s assume we’re building a swipe keyboard system that tries to predict the word you type in next on your mobile phone. Based on the pattern traced by the swipe pattern, there are many possibilities for the user’s intended word. However, many of these possible words aren’t actual words in English and can be eliminated. Even after this initial pruning and elimination step, many candidates remain, and we need to pick one as a suggestion for the user.</p><p>To further prune this list of candidates, we can use a deep-learning-based language model that looks at the provided context and tells us which candidate is most likely to complete the sentence.</p><p>For example, if the user has typed the sentence <em>“I’ve scheduled this”</em> and then swipes a pattern as shown below</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/0*kntJcWP0dbvnbuNG" /></figure><p>Then, some possible English language words that the user could have meant are:</p><ol><li>messing</li><li>meeting</li></ol><p>However, if we think about it, it’s probably more likely that the user meant “meeting” and not “messing” because of the word “scheduled” in the earlier part of the sentence.</p><p>Given everything we know so far, what options do we have for doing this pruning programmatically? Let’s brainstorm some solutions in the section below.</p><h3>Brainstorming a solution</h3><h4>Algorithms and Data Structures</h4><p>Using first principles, it seems reasonable to start with a corpus of data, find pairs of words that come together, and train a <a href="https://en.wikipedia.org/wiki/Markov_model">Markov model</a> that predicts the probability of the pair occurring in a sentence. You’ll notice two significant issues with this approach.</p><ol><li><strong>Space utilization</strong>: There are anywhere <a href="https://www.merriam-webster.com/help/faq-how-many-english-words">between 250k to 1 million words in the English language</a>, which don’t include the numerous proper nouns that are constantly growing in volume. Hence, any traditional software solution modeling the probability of a pair of words occurring together must maintain a lookup table with 250k*250k = 62.5 billion word pairs, which is somewhat excessive. It seems likely that many pairs don’t occur very often and can be pruned. Even after pruning, there are a lot of pairs to worry about.</li><li><strong>Completeness</strong>: Encoding the probability of a pair of words doesn’t do justice to the problem at hand. For example, the earlier sentence context is completely lost when you’re looking at just the most recent pair of words. In the sentence <em>“How is your day coming”</em> if you want to check the word after “coming”, you’d have a lot of pairs starting with “coming”. This misses the entire sentence context before that word. One can imagine using word triplets, etc.… but this exacerbates the problem of space utilization mentioned above.</li></ol><p>Let’s shift our focus to a solution that leverages the nature of the English language and see if that can help us here.</p><h4>NLP (Natural Language Processing)</h4><p>Historically, the area of <a href="https://en.wikipedia.org/wiki/Natural_language_processing">NLP (natural language processing)</a> involved understanding the <a href="https://www.englishclub.com/grammar/parts-of-speech.php">parts of speech (POS)</a> of a sentence and using that information to perform such pruning and prediction decisions. One can imagine using a POS tag associated with each word to determine if the following word in a sentence is valid.</p><p>However, the process of computing the parts of speech for a sentence is a complex process in itself, and requires specialized understanding of language as evidenced in <a href="https://www.nltk.org/book/ch05.html">this page on NLTK’s parts of speech tagging</a>.</p><p>Next, let’s take a look at a deep-learning-based approach that requires a lot more tagged data, but not as much language expertise to build.</p><h4>Deep Learning (Neural Networks)</h4><p>The area of NLP has been upended by the advent of deep learning. With the invention of LSTM and Transformer based language models, the solution more often than not involves throwing some high-quality data at a model and training it to predict the next word.</p><p>In essence, this is what the GPT model is doing. GPT (Generative Pre-Trained Transformer) models are trained to predict the next word (token) given a prefix of a sentence.</p><p>Given the sentence prefix <em>“It is such a wonderful”</em>, it’s likely for the model to provide the following as high-probability predictions for the word following the sentence.</p><ol><li>day</li><li>experience</li><li>world</li><li>life</li></ol><p>It’s also likely that the following words will have a lower probability of completing the sentence prefix.</p><ol><li>red</li><li>mouse</li><li>line</li></ol><p>The <a href="https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)">Transformer model architecture</a> is at the heart of systems such as ChatGPT. However, for the more restricted use case of learning English language semantics, we can use a cheaper-to-run model architecture such as an <a href="https://en.wikipedia.org/wiki/Long_short-term_memory">LSTM (long short-term memory)</a> model.</p><h3>An LSTM model</h3><p>Let’s build a simple LSTM model and train it to predict the next token given a prefix of tokens. Now, you might ask what a token is.</p><h4>Tokenization</h4><p>Typically for language models, a token can mean</p><ol><li>A single character (or a single byte)</li><li>An entire word in the target language</li><li>Something in between 1 and 2. This is usually called a sub-word</li></ol><p>Mapping a single character (or byte) to a token is very restrictive since we’re overloading that token to hold a lot of context about where it occurs. This is because the character “c” for example, occurs in many different words, and to predict the next character after we see the character “c” requires us to really look hard at the leading context.</p><p>Mapping a single word to a token is also problematic since English itself has anywhere between 250k and 1 million words. In addition, what happens when a new word is added to the language? Do we need to go back and re-train the entire model to account for this new word?</p><p>Sub-word tokenization is considered the industry standard in the year 2023. It assigns substrings of bytes frequently occurring together to unique tokens. Typically, language models have anywhere from a few thousand (say 4,000) to tens of thousands (say 60,000) of unique tokens. The algorithm to determine what constitutes a token is determined by the<a href="https://huggingface.co/learn/nlp-course/chapter6/5?fw=pt"> BPE (Byte pair encoding) algorith</a>m.</p><p>To choose the number of unique tokens in our vocabulary (called the vocabulary size), we need to be mindful of a few things:</p><ol><li>If we choose too few tokens, we’re back in the regime of a token per character, and it’s hard for the model to learn anything useful.</li><li>If we choose too many tokens, we end up in a situation where the model’s embedding tables over-shadow the rest of the model’s weight and it becomes hard to deploy the model in a constrained environment. The size of the embedding table will depend on the number of dimensions we use for each token. It’s not uncommon to use a size of 256, 512, 786, etc… If we use a token embedding dimension of 512, and we have 100k tokens, we end up with an embedding table that uses 200MiB in memory.</li></ol><p>Hence, we need to strike a balance when choosing the vocabulary size. In this example, we pick 6600 tokens and train our tokenizer with a vocabulary size of 6600. Next, let’s take a look at the model definition itself.</p><h4>The PyTorch Model</h4><p>The model itself is pretty straightforward. We have the following layers:</p><ol><li>Token Embedding (vocab size=6600, embedding dim=512), for a total size of about 15MiB (assuming 4 byte float32 as the embedding table’s data type)</li><li>LSTM (num layers=1, hidden dimension=786) for a total size of about 16MiB</li><li>Multi-Layer Perceptron (786 to 3144 to 6600 dimensions) for a total size of about 93MiB</li></ol><p>The complete model has about 31M trainable parameters for a total size of about 120MiB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*aIaX1HtNih4GRtlG" /></figure><p>Here’s the PyTorch code for the model.</p><pre>class WordPredictionLSTMModel(nn.Module):<br>    def __init__(self, num_embed, embed_dim, pad_idx, lstm_hidden_dim, lstm_num_layers, output_dim, dropout):<br>        super().__init__()<br>        self.vocab_size = num_embed<br>        self.embed = nn.Embedding(num_embed, embed_dim, pad_idx)<br>        self.lstm = nn.LSTM(embed_dim, lstm_hidden_dim, lstm_num_layers, batch_first=True, dropout=dropout)<br>        self.fc = nn.Sequential(<br>            nn.Linear(lstm_hidden_dim, lstm_hidden_dim * 4),<br>            nn.LayerNorm(lstm_hidden_dim * 4),<br>            nn.LeakyReLU(),<br>            nn.Dropout(p=dropout),<br><br>            nn.Linear(lstm_hidden_dim * 4, output_dim),<br>        )<br>    #<br>    <br>    def forward(self, x):<br>        x = self.embed(x)<br>        x, _ = self.lstm(x)<br>        x = self.fc(x)<br>        x = x.permute(0, 2, 1)<br>        return x<br>    #<br>#</pre><p>Here’s the model summary using torchinfo.</p><p>LSTM Model Summary</p><pre>=================================================================<br>Layer (type:depth-idx) Param #<br>=================================================================<br>WordPredictionLSTMModel - <br>├─Embedding: 1–1 3,379,200<br>├─LSTM: 1–2 4,087,200<br>├─Sequential: 1–3 - <br>│ └─Linear: 2–1 2,474,328<br>│ └─LayerNorm: 2–2 6,288<br>│ └─LeakyReLU: 2–3 - <br>│ └─Dropout: 2–4 - <br>│ └─Linear: 2–5 20,757,000<br>=================================================================<br>Total params: 30,704,016<br>Trainable params: 30,704,016<br>Non-trainable params: 0<br>=================================================================</pre><p><strong>Interpreting the accuracy</strong>: After training this model on 12M English language sentences for about 8 hours on a P100 GPU, we achieved a loss of 4.03, a top-1 accuracy of 29% and a top-5 accuracy of 49%. This means that 29% of the time, the model was able to correctly predict the next token, and 49% of the time, the next token in the training set was one of the top 5 predictions by the model.</p><p><strong>What should our success metric be?</strong> While the top-1 and top-5 accuracy numbers for our model aren’t impressive, they aren’t as important for our problem. Our candidate words are a small set of possible words that fit the swipe pattern. What we want from our model is to be able to select an ideal candidate to complete the sentence such that it is syntactically and semantically coherent. Since our model learns the <em>nature</em> of language through the training data, we expect it to assign a higher probability to coherent sentences. For example, if we have the sentence <em>“The baseball player”</em> and possible completion candidates (“ran”, “swam”, “hid”), then the word “ran” is a better follow-up word than the other two. So, if our model predicts the word <em>ran</em> with a higher probability than the rest, it works for us.</p><p><strong>Interpreting the loss</strong>: A loss of 4.03 means that the negative log-likelihood of the prediction is 4.03, which means that the probability of predicting the next token correctly is e^-4.03 = 0.0178 or 1/56. A randomly initialized model typically has a loss of about 8.8 which is -log_e(1/6600), since the model randomly predicts 1/6600 tokens (6600 being the vocabulary size). While a loss of 4.03 may not seem great, it’s important to remember that the trained model is about 120x better than an untrained (or randomly initialized) model.</p><p>Next, let’s take a look at how we can use this model to improve suggestions from our swipe keyboard.</p><h4>Using the model to prune invalid suggestions</h4><p>Let’s take a look at a real example. Suppose we have a partial sentence <em>“I think”</em>, and the user makes the swipe pattern shown in blue below, starting at “o”, going between the letters “c” and “v”, and ending between the letters “e” and “v”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/0*ZxYzvXUsJCyNIack" /></figure><p>Some possible words that could be represented by this swipe pattern are</p><ol><li>Over</li><li>Oct (short for October)</li><li>Ice</li><li>I’ve (with the apostrophe implied)</li></ol><p>Of these suggestions, the most likely one is probably going to be <em>“I’ve”</em>. Let’s feed these suggestions into our model and see what it spits out.</p><pre>[I think] [I&#39;ve] = 0.00087<br>[I think] [over] = 0.00051<br>[I think] [ice] = 0.00001<br>[I think] [Oct] = 0.00000</pre><p>The value after the = sign is the probability of the word being a valid completion of the sentence prefix. In this case, we see that the word “I’ve” has been assigned the highest probability. Hence, it is the most likely word to follow the sentence prefix “I think”.</p><p>The next question you might have is how we can compute these next-word probabilities. Let’s take a look.</p><h4>Computing the next word probability</h4><p>To compute the probability that a word is a valid completion of a sentence prefix, we run the model in eval (inference) mode and feed in the tokenized sentence prefix. We also tokenize the word after adding a whitespace prefix to the word. This is done because the HuggingFace pre-tokenizer splits words with spaces at the beginning of the word, so we want to make sure that our inputs are consistent with the tokenization strategy used by HuggingFace Tokenizers.</p><p>Let’s assume that the candidate word is made up of 3 tokens T0, T1, and T2.</p><ol><li>We first run the model with the original tokenized sentence prefix. For the last token, we check the probability of predicting token T0. We add this to the “probs” list.</li><li>Next, we run a prediction on the prefix+T0 and check the probability of token T1. We add this probability to the “probs” list.</li><li>Next, we run a prediction on the prefix+T0+T1 and check the probability of token T2. We add this probability to the “probs” list.</li></ol><p>The “probs” list contains the individual probabilities of generating the tokens T0, T1, and T2 in sequence. Since these tokens correspond to the tokenization of the candidate word, we can multiply these probabilities to get the combined probability of the candidate being a completion of the sentence prefix.</p><p>The code for computing the completion probabilities is shown below.</p><pre> def get_completion_probability(self, input, completion, tok):<br>      self.model.eval()<br>      ids = tok.encode(input).ids<br>      ids = torch.tensor(ids, device=self.device).unsqueeze(0)<br>      completion_ids = torch.tensor(tok.encode(completion).ids, device=self.device).unsqueeze(0)<br>      probs = []<br>      for i in range(completion_ids.size(1)):<br>          y = self.model(ids)<br>          y = y[0,:,-1].softmax(dim=0)<br>          # prob is the probability of this completion.<br>          prob = y[completion_ids[0,i]]<br>          probs.append(prob)<br>          ids = torch.cat([ids, completion_ids[:,i:i+1]], dim=1)<br>      #<br>      return torch.tensor(probs)<br>  #</pre><p>We can see some more examples below.</p><pre>[That ice-cream looks] [really] = 0.00709<br>[That ice-cream looks] [delicious] = 0.00264<br>[That ice-cream looks] [absolutely] = 0.00122<br>[That ice-cream looks] [real] = 0.00031<br>[That ice-cream looks] [fish] = 0.00004<br>[That ice-cream looks] [paper] = 0.00001<br>[That ice-cream looks] [atrocious] = 0.00000<br><br>[Since we&#39;re heading] [toward] = 0.01052<br>[Since we&#39;re heading] [away] = 0.00344<br>[Since we&#39;re heading] [against] = 0.00035<br>[Since we&#39;re heading] [both] = 0.00009<br>[Since we&#39;re heading] [death] = 0.00000<br>[Since we&#39;re heading] [bubble] = 0.00000<br>[Since we&#39;re heading] [birth] = 0.00000<br><br>[Did I make] [a] = 0.22704<br>[Did I make] [the] = 0.06622<br>[Did I make] [good] = 0.00190<br>[Did I make] [food] = 0.00020<br>[Did I make] [color] = 0.00007<br>[Did I make] [house] = 0.00006<br>[Did I make] [colour] = 0.00002<br>[Did I make] [pencil] = 0.00001<br>[Did I make] [flower] = 0.00000<br><br>[We want a candidate] [with] = 0.03209<br>[We want a candidate] [that] = 0.02145<br>[We want a candidate] [experience] = 0.00097<br>[We want a candidate] [which] = 0.00094<br>[We want a candidate] [more] = 0.00010<br>[We want a candidate] [less] = 0.00007<br>[We want a candidate] [school] = 0.00003<br><br>[This is the definitive guide to the] [the] = 0.00089<br>[This is the definitive guide to the] [complete] = 0.00047<br>[This is the definitive guide to the] [sentence] = 0.00006<br>[This is the definitive guide to the] [rapper] = 0.00001<br>[This is the definitive guide to the] [illustrated] = 0.00001<br>[This is the definitive guide to the] [extravagant] = 0.00000<br>[This is the definitive guide to the] [wrapper] = 0.00000<br>[This is the definitive guide to the] [miniscule] = 0.00000<br><br>[Please can you] [check] = 0.00502<br>[Please can you] [confirm] = 0.00488<br>[Please can you] [cease] = 0.00002<br>[Please can you] [cradle] = 0.00000<br>[Please can you] [laptop] = 0.00000<br>[Please can you] [envelope] = 0.00000<br>[Please can you] [options] = 0.00000<br>[Please can you] [cordon] = 0.00000<br>[Please can you] [corolla] = 0.00000<br><br>[I think] [I&#39;ve] = 0.00087<br>[I think] [over] = 0.00051<br>[I think] [ice] = 0.00001<br>[I think] [Oct] = 0.00000<br><br>[Please] [can] = 0.00428<br>[Please] [cab] = 0.00000<br><br>[I&#39;ve scheduled this] [meeting] = 0.00077<br>[I&#39;ve scheduled this] [messing] = 0.00000</pre><p>These examples show the probability of the word completing the sentence before it. The candidates are sorted in decreasing order of probability.</p><p>Since Transformers are slowly replacing LSTM and RNN models for sequence-based tasks, let’s take a look at what a Transformer model for the same objective would look like.</p><h3>A Transformer model</h3><p>Transformer-based models are a very popular architecture for training language models to predict the next word in a sentence. The specific technique we’ll use is the causal attention mechanism. We’ll train just the <a href="https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoder.html#torch.nn.TransformerEncoder">transformer encoder layer in PyTorch</a> using causal attention. Causal attention means we’ll allow every token in the sequence to only look at the tokens before it. This resembles the information that a unidirectional LSTM layer uses when trained only in the forward direction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CB7TzptLF3rkPu2t" /></figure><p>The Transformer model we’ll see here is based directly on the <a href="https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoder.html#torch.nn.TransformerEncoder">nn.TransformerEncoder</a> and <a href="https://pytorch.org/docs/stable/generated/torch.nn.TransformerEncoderLayer.html#torch.nn.TransformerEncoderLayer">nn.TransformerEncoderLayer</a> in PyTorch.</p><pre>import math<br><br>def generate_src_mask(sz, device):<br>    return torch.triu(torch.full((sz, sz), True, device=device), diagonal=1)<br>#<br><br>class PositionalEmbedding(nn.Module):<br>    def __init__(self, sequence_length, embed_dim):<br>        super().__init__()<br>        self.sqrt_embed_dim = math.sqrt(embed_dim)<br>        self.pos_embed = nn.Parameter(torch.empty((1, sequence_length, embed_dim)))<br>        nn.init.uniform_(self.pos_embed, -1.0, 1.0)<br>    #<br>    <br>    def forward(self, x):<br>        return x * self.sqrt_embed_dim + self.pos_embed[:,:x.size(1)]<br>    #<br>#<br><br>class WordPredictionTransformerModel(nn.Module):<br>    def __init__(self, sequence_length, num_embed, embed_dim, pad_idx, num_heads, num_layers, output_dim, dropout, norm_first, activation):<br>        super().__init__()<br>        self.vocab_size = num_embed<br>        self.sequence_length = sequence_length<br>        self.embed_dim = embed_dim<br>        self.sqrt_embed_dim = math.sqrt(embed_dim)<br>        self.embed = nn.Sequential(<br>            nn.Embedding(num_embed, embed_dim, pad_idx),<br>            PositionalEmbedding(sequence_length, embed_dim),<br>            nn.LayerNorm(embed_dim),<br>            nn.Dropout(p=0.1),<br>        )<br><br>        encoder_layer = nn.TransformerEncoderLayer(<br>            d_model=embed_dim, nhead=num_heads, dropout=dropout, batch_first=True, norm_first=norm_first, activation=activation,<br>        )<br>        self.encoder = nn.TransformerEncoder(encoder_layer, num_layers=num_layers)<br>        self.fc = nn.Sequential(<br>            nn.Linear(embed_dim, embed_dim * 4),<br>            nn.LayerNorm(embed_dim * 4),<br>            nn.LeakyReLU(),<br>            nn.Dropout(p=dropout),<br><br>            nn.Linear(embed_dim * 4, output_dim),<br>        )<br>    #<br>    <br>    def forward(self, x):<br>        src_attention_mask = generate_src_mask(x.size(1), x.device)<br>        x = self.embed(x)<br>        x = self.encoder(x, is_causal=True, mask=src_attention_mask)<br>        x = self.fc(x)<br>        x = x.permute(0, 2, 1)<br>        return x<br>    #<br>#</pre><p>We can plug this model in place of the LSTM model that we used before since it’s API is compatible. This model takes longer to train for the same amount of training data and has comparable performance.</p><p>Transformer models are better for long sequences. In our case, we have sequences of length 256. Most of the context needed to perform next-word completion tends to be local, so we don’t really need the power of Transformers here.</p><h3>Conclusion</h3><p>We saw how we can solve very practical NLP problems using deep learning techniques based on LSTM (RNN) and Transformer models. Not every language task requires the use of models with billions of parameters. Specialized applications that require modeling language itself, and not memorizing large volumes of information can be handled using much smaller models that can be deployed easily and more efficiently than the massive language models that we are used to seeing these days.</p><p>All the image(s) except for the first one were created by the author(s).</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6a5298a85e43" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/language-models-for-sentence-completion-6a5298a85e43">Language Models for Sentence Completion</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Tensor Quantization: The Untold Story]]></title>
            <link>https://medium.com/data-science/tensor-quantization-the-untold-story-d798c30e7646?source=rss-63f5d5495279------2</link>
            <guid isPermaLink="false">https://medium.com/p/d798c30e7646</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[normalization]]></category>
            <category><![CDATA[quantization]]></category>
            <category><![CDATA[model-quantization]]></category>
            <category><![CDATA[zero-point]]></category>
            <dc:creator><![CDATA[Dhruv Matani]]></dc:creator>
            <pubDate>Fri, 08 Sep 2023 05:20:41 GMT</pubDate>
            <atom:updated>2023-09-08T05:20:41.426Z</atom:updated>
            <content:encoded><![CDATA[<h4>A close look at the implementation details of quantization in machine learning frameworks</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/0*nKEzyyqQukfSm6Ft" /></figure><p>Co-authored with <a href="https://medium.com/@brocolishbroxoli">Naresh Singh</a>.</p><h3>Table Of Contents</h3><ul><li><a href="#1d38">Introduction</a></li><li><a href="#038c">What do the terms scale and zero-point mean for quantization?</a></li><li><a href="#b2df">Types of Quantization Schemes</a></li><li><a href="#0dd2">Quantization Scale and Zero Point examples</a></li><li><a href="#3046">Quantization and Activation Normalization</a></li><li><a href="#8bf9">Conclusion</a></li><li><a href="#390b">References</a></li></ul><h3>Introduction</h3><p>Throughout the rest of this article, we will try to answer the following questions with concrete examples.</p><ol><li>What do the terms scale and zero-point mean for quantization?</li><li>What are the different types of quantization schemes?</li><li>How to compute scale and zero-point for the different quantization schemes</li><li>Why is zero-point important for quantization?</li><li>How do normalization techniques benefit quantization</li></ol><h3>What do the terms scale and zero-point mean for quantization?</h3><p><strong>Scale:</strong> When quantizing a floating point range, one would typically represent a floating point range [Fmin..Fmax] in the quantized range [Qmin..Qmax]. In this case, the scale is the ratio of the floating point range and the quantized range.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/0*ucxfYyT2ZUCiPz5g" /></figure><p>We will see an example of how to compute it later.</p><p><strong>Zero-point:</strong> The zero-point for quantization is the representation of floating point 0.0 in the quantized range. Specifically, the zero-point is a quantized value, and it represents the floating point value 0.0 for all practical purposes. We shall see how it’s computed with examples later, along with why such a representation is of practical interest to us.</p><p>Next, let’s take a look at the main quantization schemes used in practice and familiarize ourselves with how they are similar and how they differ.</p><h3>Types of Quantization Schemes</h3><p>When considering the types of quantization available for use during model compression, there are 2 main types to pick from</p><ol><li><strong>Symmetric quantization</strong>: In this case, the zero-point is zero — i.e. 0.0 of the floating point range is the same as 0 in the quantized range. Typically, this is more efficient to compute at runtime but may result in lower accuracy if the floating point range is unequally distributed around the floating point 0.0.</li><li><strong>Affine (or asymmetric) quantization</strong>: This is the one that has a zero-point that is non-zero in value.</li></ol><p>But before we jump into the details, let’s try to define what zero-point means.</p><h3>Quantization Scale and Zero Point examples</h3><p>Let’s start with a very simple example and build it up.</p><h4>Example-1: Symmetric uint8 quantization</h4><p>Let’s say we wish to map the floating point range [0.0 .. 1000.0] to the quantized range [0 .. 255]. The range [0 .. 255] is the set of values that can fit in an unsigned 8-bit integer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/568/0*1exMKZMh14ZrRc5p" /></figure><p>To perform this transformation, we want to rescale the floating point range so that the following is true:</p><p>Floating point 0.0 = Quantized 0</p><p>Floating point 1000.0 = Quantized 255</p><p>This is called symmetric quantization because the floating point 0.0 is quantized 0.</p><p>Hence, we define a scale, which is equal to</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/404/0*NH-CXHbYEug52O9P" /></figure><p>Where,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/293/0*YWVRZrkS_TiADvlS" /></figure><p>In this case, scale = 3.9215</p><p>To convert from a floating point value to a quantized value, we can simply divide the floating point value by the scale. For example, the floating point value 500.0 corresponds to the quantized value</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/653/0*e4Ocq96D0_dJnC7B" /></figure><p>In this simple example, the 0.0 of the floating point range maps exactly to the 0 in the quantized range. This is called symmetric quantization. Let’s see what happens when this is not the case.</p><h4>Example-2: Affine uint8 quantization</h4><p>Let’s say we wish to map the floating point range [-20.0 .. 1000.0] to the quantized range [0 .. 255].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/578/0*lep5OKOJmHf33LU5" /></figure><p>In this case, we have a different scaling factor since our <em>xmin</em> is different.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/0*T7hV0dAfFOBSu0yy" /></figure><p>Let’s see what the floating point number 0.0 is represented by in the quantized range if we apply the scaling factor to 0.0</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/0*aAEL5MIEDTeoiYlI" /></figure><p>Well, this doesn’t quite seem right since, according to the diagram above, we would have expected the floating point value -20.0 to map to the quantized value 0.</p><p>This is where the concept of zero-point comes in. <strong>The zero-point acts as a bias for shifting the scaled floating point value and corresponds to the value in the quantized range that represents the floating point value 0.0.</strong> In our case, the zero point is the negative of the scaled floating point representation of -20.0, which is -(-5) = 5. The zero point is always the negative of the representation of the minimum floating point value since the minimum will always be negative or zero. We’ll find out more about why this is the case in the section that explains example 4.</p><p>Whenever we quantize a value, we will always add the zero-point to this scaled value to get the actual quantized value in the valid quantization range. In case we wish to quantize the value -20.0, we compute it as the scaled value of -20.0 plus the zero-point, which is -5 + 5 = 0. Hence, quantized(-20.0, scale=4, zp=5) = 0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/511/0*RKKmFBuKbWYoMhru" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/588/0*YErZ5YAeMMa4lnYF" /></figure><h4>Example-3: Affine int8 quantization</h4><p>What happens if our quantized range is a signed 8-bit integer instead of an unsigned 8-bit integer? Well, the range is now [-128 .. 127].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/556/0*zRyGYYs0XqM6yTjC" /></figure><p>In this case, -20.0 in the float range maps to -128 in the quantized range, and 1000.0 in the float range maps to 127 in the quantized range.</p><p>The way we calculate zero point is that we compute it as if the quantized range is [0 .. 255] and then offset it with -128, so the zero point in the new range is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/0*65jbaHMKmk-Y6kXQ" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/575/0*BM2tNJsxxQ5sYumf" /></figure><p>Hence, the zero-point for the new range is -123.</p><p>So far, we’ve looked at examples where the floating point range includes the value 0.0. In the next set of examples, we’ll take a look at what happens when the floating point range doesn’t include the value 0.0</p><h4>The importance of 0.0</h4><p>Why is it important for the floating point value 0.0 to be represented in the floating point range?</p><p>When using a padded convolution, we expect the border pixels to be padded using the value 0.0 in the most common case. Hence, it’s important for 0.0 to be represented in the floating point range. Similarly, if the value X is going to be used for padding in your network, you need to make sure that the value X is represented in the floating point range and that quantization is aware of this.</p><h4>Example-4: The untold story — skewed floating point range</h4><p>Now, let’s take a look at what happens if 0.0 isn’t part of the floating point range.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/562/0*Fiz_qTJYIiLfLwEA" /></figure><p>In this example, we’re trying to quantize the floating point range [40.0 .. 1000.0] into the quantized range [0 .. 255].</p><p>Since we can’t represent the value 0.0 in the floating point range, we need to extend the lower limit of the range to 0.0.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/549/0*oDWX6xFkKWnGmEKu" /></figure><p>We can see that some part of the quantized range is wasted. To determine how much, let’s compute the quantized value that the floating point value 40.0 maps to.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/811/0*_bdVimG64b0tJe-h" /></figure><p>Hence, we’re wasting the range [0 .. 9] in the quantized range, which is about 3.92% of the range. This could significantly affect the model’s accuracy post-quantization.</p><p>This skewing is necessary if we wish to make sure that the value 0.0 in the floating point range can be represented in the quantized range.</p><p>Another reason for including the value 0.0 in the floating point range is that efficiently comparing a quantized value to check if it’s 0.0 in the floating point range is very valuable. Think of operators such as ReLU, which clip all values below 0.0 in the floating point range to 0.0.</p><p>It is important for us to be able to <strong>represent the zero-point using the same data type</strong> (signed or unsigned int8) <strong>as the quantized values</strong>. This enables us to perform these comparisons quickly and efficiently.</p><p>Next, let’s take a look at how activation normalization helps with model quantization. We’ll specifically focus on how the standardization of the activation values allows us to use the entire quantized range effectively.</p><h3>Quantization and Activation Normalization</h3><p>Batch/Layer Normalization changes the activation tensor to have zero mean and unit variance per channel or per layer.</p><p>Suppose we have an input tensor with a floating point range of [2000.0 .. 4000.0]. This is what the quantized range would look like.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/551/0*NEuv8GroWSu-jB26" /></figure><p>We observe that half of the quantized range [-127 .. -1] is unused. This is problematic since we’re quantizing the entire floating point range using just 7 of the available 8 bits. This will undoubtedly result in a higher quantization error and reduced model accuracy. To address this let’s apply layer normalization to the activation tensor.</p><p>After applying layer normalization to an activation tensor, the activation tensor will have a floating point range of [-2.0 .. 2.0]. This can be represented in the signed int8 range [-128 .. 127]. To ensure symmetry of the distribution, we restrict the quantized range to be [-127 .. 127].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/530/0*EdLdtkgYPwwQ7IGK" /></figure><p>Hence, normalization avoids holes or unused parts in the quantized range.</p><h3>Conclusion</h3><p>We saw what affine (asymmetric) and symmetric quantization are and how they differ. We also learned about what scale and zero-point mean and how to compute them for both these types of quantization schemes.</p><p>Next, we saw the need to include float 0.0 in the floating point range and why and how this is done in practice. This results in a downside, namely wasted space in the quantized range.</p><p>Lastly, we saw how normalization helps quantization by bringing the activations in a fixed range and avoiding wasted space in the quantized range. In fact, the 0 mean based normalization can help convert affine quantization to symmetric quantization, and that can speed things up during inference.</p><p>All images in this post are created by the author(s).</p><h3>References</h3><ol><li><a href="https://github.com/EfficientDL/book/raw/main/book/%5BEDL%5D%20Chapter%202%20-%20Compression%20Techniques.pdf">Efficient Deep Learning Book, Chapter-2: Introduction to Compression Techniques</a></li><li><a href="https://huggingface.co/docs/optimum/concept_guides/quantization">Hugging Face: Quantization</a></li><li><a href="https://docs.nvidia.com/deeplearning/tensorrt/tensorflow-quantization-toolkit/docs/docs/intro_to_quantization.html">TensorRT: Quantization</a></li><li><a href="https://intellabs.github.io/distiller/algo_quantization.html">Neural Network Distiller: Quantization</a></li><li><a href="https://leimao.github.io/article/Neural-Networks-Quantization/">Lei Mao: Quantization</a></li><li><a href="https://zeux.io/2010/12/14/quantizing-floats/">Quantizing floats</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d798c30e7646" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/tensor-quantization-the-untold-story-d798c30e7646">Tensor Quantization: The Untold Story</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Efficient Image Segmentation Using PyTorch: Part 4]]></title>
            <link>https://medium.com/data-science/efficient-image-segmentation-using-pytorch-part-4-6c86da083432?source=rss-63f5d5495279------2</link>
            <guid isPermaLink="false">https://medium.com/p/6c86da083432</guid>
            <category><![CDATA[pytorch-image-segment]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[deep-dives]]></category>
            <category><![CDATA[vision-transformer]]></category>
            <dc:creator><![CDATA[Dhruv Matani]]></dc:creator>
            <pubDate>Tue, 27 Jun 2023 06:13:30 GMT</pubDate>
            <atom:updated>2023-06-27T06:13:30.879Z</atom:updated>
            <content:encoded><![CDATA[<h4>A Vision Transformer-based model</h4><p>In this 4-part series, we’ll implement image segmentation step by step from scratch using deep learning techniques in PyTorch. This part will focus on implementing a Vision Transformer based model for image segmentation.</p><p>Co-authored with <a href="https://medium.com/@brocolishbroxoli">Naresh Singh</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/696/0*c3rF_3Zpfv6CCUxq" /><figcaption>Figure 1: Result of running image segmentation using a vision transformer model architecture. From top to bottom, input images, ground truth segmentation masks, and predicted segmentation masks. Source: Author(s)</figcaption></figure><h3>Article outline</h3><p>In this article, we will visit the <a href="https://ai.googleblog.com/2017/08/transformer-novel-neural-network.html">Transformer architecture</a> which has taken the world of deep learning by storm. The transformer is a multimodal architecture which can model different modalities such as language, vision, and audio.</p><p>In this article, we will</p><ol><li>Learn about the transformer architecture and the key concepts involved</li><li>Understand the vision transformer architecture</li><li>Introduce a vision transformer model that is written from scratch so that you can appreciate all the building blocks and moving parts</li><li>Follow an input tensor fed into this model and inspect how it changes shape</li><li>Use this model to perform image segmentation on the Oxford IIIT Pet dataset</li><li>Observe the results of this segmentation task</li><li>Briefly introduce the SegFormer, a state of the art vision transformer for semantic segmentation</li></ol><p>Throughout this article, we will reference code and results from this <a href="https://github.com/dhruvbird/ml-notebooks/blob/main/pets_segmentation/vision-transformer-for-oxford-iiit-pet-segmentatio.ipynb">notebook</a> for model training. If you wish to reproduce the results, you’ll need a GPU to ensure that the first notebook completes running in a reasonable amount of time.</p><h3>Articles in this series</h3><p>This series is for readers at all experience levels with deep learning. If you want to learn about the practice of deep learning and vision AI along with some solid theory and hands-on experience, you’ve come to the right place! This is expected to be a 4-part series with the following articles:</p><ol><li><a href="https://medium.com/p/89e8297a0923/">Concepts and Ideas</a></li><li><a href="https://medium.com/p/bed68cadd7c7/">A CNN-based model</a></li><li><a href="https://medium.com/p/3534cf04fb89/">Depthwise separable convolutions</a></li><li><strong>A Vision Transformer-based model (this article)</strong></li></ol><p>Let’s start our journey into vision transformers with an introduction and intuitive understanding of the transformer architecture.</p><h3>The Transformer Architecture</h3><p>We can think of the transformer architecture as a composition of interleaving layers of <em>communication</em> and <em>computation</em>. This idea is depicted visually in figure 2. The transformer has N processing units (N is 3 in Figure 2), each of which is responsible for processing a 1/N fraction of the input. For those processing units to produce meaningful results, each of them need to have a <em>global view</em> of the input. Hence, the system repeatedly communicates information about the data in every processing unit to every other processing unit; shown using the red, green, and blue arrows going from every processing unit to every other processing unit. This is followed by some computation based on this information. After sufficient repetitions of this process, the model is able to produce desired results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bpWZaQdbzM4lT7dx" /><figcaption>Figure 2: Interleaved communication and computation in transformers. The image shows just 2 layers of communication and computation. In practice, there are many more such layers. Source: Author(s).</figcaption></figure><p>It’s worth noting most online resources typically discuss both the encoder and the decoder of the transformer as presented in the paper titled “<a href="https://arxiv.org/abs/1706.03762">Attention is all you need</a>.” However, in this article, we will describe just the encoder part of the transformer.</p><p>Let’s take a closer look at what constitutes communication and computation in transformers.</p><h4>Communication in transformers: Attention</h4><p>In transformers, communication is implemented by a layer known as the attention layer. In PyTorch, this is called <a href="https://pytorch.org/docs/stable/generated/torch.nn.MultiheadAttention.html">MultiHeadAttention</a>. We’ll get to the reason for that name in a bit.</p><p>The documentation says:</p><p><em>“Allows the model to jointly attend to information from different representation subspaces as described in the paper: </em><a href="https://arxiv.org/abs/1706.03762"><em>Attention is all you need</em></a><em>.”</em></p><p>The attention mechanism consumes an input tensor <em>x</em> of shape (Batch, Length, Features), and it produces a similarly shaped tensor <em>y</em> such that the features for each input are updated based on which other inputs in the same instance the tensor is paying attention to. Hence, the features of each tensor of length “Features” in the instance of size “Length” are updated based on every other tensor. This is where the quadratic cost of the attention mechanism comes in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/433/0*9cWryzyUlC6x9m5x" /><figcaption>Figure 3: Attention of the word “it ” shown relative to the other words in the sentence. We can see that “it “ is paying attention to the words “animal “, “too “, and “tire(d) ” in the same sentence. Source: Generated using <a href="https://colab.research.google.com/github/tensorflow/tensor2tensor/blob/master/tensor2tensor/notebooks/hello_t2t.ipynb">this colab</a>.</figcaption></figure><p>In the context of a vision transformer, the input to the transformer is an image. Let’s assume this to be a 128 x 128 (width, height) image. We chunk it into multiple smaller patches of size (16 x 16). For a 128 x 128 image, we get 64 patches (Length), 8 patches in each row, and 8 rows of patches.</p><p>Each one of these 64 patches of size 16 x 16 pixels is considered to be a separate input to the transformer model. Without getting too deep into the details, it should be sufficient to think of this process as being driven by 64 different processing units, each of which is processing a single 16x16 image patch.</p><p>In each round, the attention mechanism in each processing unit is responsible for looking at the image patch it is responsible for and querying each one of the other remaining 63 processing units to ask them for any information that may be relevant and useful to help it effectively process its own image patch.</p><p>The communication step via attention is followed by computation, which we will look at next.</p><h4>Computation in transformers: Multi Layer Perceptron</h4><p>Computation in transformers is nothing but a MultiLayerPerceptron (MLP) unit. This unit is composed of 2 Linear layers, with a GeLU non-linearity in between. One can consider using other non-linearities as well. This unit first projects the input to 4x the size and reprojects it back to 1x, which is the same as the input size.</p><p>In the code we’ll see in our notebook, this class is called MultiLayerPerceptron. The code is shown below.</p><pre>class MultiLayerPerceptron(nn.Sequential):<br>    def __init__(self, embed_size, dropout):<br>        super().__init__(<br>            nn.Linear(embed_size, embed_size * 4),<br>            nn.GELU(),<br>            nn.Linear(embed_size * 4, embed_size),<br>            nn.Dropout(p=dropout),<br>        )<br>    # end def<br># end class</pre><p>Now that we understand the high level working of the transformer architecture, let’s focus our attention on the vision transformer since we’re going to be performing image segmentation.</p><h3>The Vision Transformer</h3><p>The vision transformer was first introduced by the paper titled “<a href="https://arxiv.org/abs/2010.11929">An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale</a>”. The paper discusses how the authors apply the vanilla transformer architecture to the problem of image classification. This is done by splitting the image into patches of size 16x16, and treating each patch as an input token to the model. The transformer encoder model is fed these input tokens, and is asked to predict a class for the input image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/0*cuLnz_JK5Afenyrf" /><figcaption>Figure 4: Source: <a href="https://arxiv.org/pdf/2010.11929.pdf">Transformers for image recognition at scale</a>.</figcaption></figure><p>In our case, we are interested in image segmentation. We can consider it to be a pixel-level classification task because we intend to predict a target class per pixel..</p><p>We make a small but important change to the vanilla vision transformer and replace the MLP head for classification by an MLP head for pixel level classification. We have a single linear layer in the output that is shared by every patch whose segmentation mask is predicted by the vision transformer. This shared linear layer predicts a segmentation mask for every patch that was sent as input to the model.</p><p>In the case of the vision transformer, a patch of size 16x16 is considered to be equivalent to a single input token at a specific time step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JbCsJbSNF9j_wPVF" /><figcaption>Figure 5: The end to end working of the vision transformer for image segmentation. Image generated using this <a href="https://github.com/dhruvbird/ml-notebooks/blob/main/pets_segmentation/Vision%20Transformer%20Article%20Images.ipynb">notebook</a>. Source: Author(s).</figcaption></figure><h4>Building an intuition for tensor dimensions in vision transformers</h4><p>When working with deep CNNs, the tensor dimensions we used for the most part was (N, C H, W), where the letters stand for the following:</p><ul><li>N: Batch size</li><li>C: Number of channels</li><li>H: Height</li><li>W: Width</li></ul><p>You can see that this format is geared toward 2d image processing, since it smells of features that are very specific to images.</p><p>With transformers on the other hand, things become a lot more generic and domain agnostic. What we’ll see below applies to vision, text, NLP, audio or other problems where input data can be represented as a sequence. It is worth noting that there’s little vision specific bias in the representation of tensors as they flow through our vision transformer.</p><p>When working with transformers and attention in general, we expect the tensors to have the following shape: (B, T, C), where the letters stand for the following:</p><ul><li>B: Batch size (same as that for CNNs)</li><li>T: Time dimension or sequence length. This dimension is also sometimes called L. In the case of vision transformers, each image patch corresponds to this dimension. If we have 16 image patches, then the value of the T dimension will be 16</li><li>C: The channel or embedding size dimension. This dimension is also sometimes called E. When processing images, each patch of size 3x16x16 (Channel, Width, Height) is mapped via a patch embedding layer to an embedding of size C. We’ll see how this is done later.</li></ul><p>Let’s dive into how the input image tensor gets mutated and processed along its way to predicting the segmentation mask.</p><h4>The journey of a tensor in a vision transformer</h4><p>In deep CNNs, the journey of a tensor looks something like this (in a UNet, SegNet, or other CNN based architecture).</p><p>The input tensor is typically of shape (1, 3, 128, 128). This tensor goes through a series of convolution and max-pooling operations where its spatial dimensions are reduced and channel dimensions are increased, typically by a factor of 2 each. This is called the feature encoder. After this, we do the reverse operation where we increase the spatial dimensions and reduce the channel dimensions. This is called the feature decoder. After the decoding process, we get a tensor of shape (1, 64, 128, 128). This is then projected into the number of output channels C that we desire as (1, C, 128, 128) using a 1x1 pointwise convolution without bias.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*01xjpWZkVkBlfcjT" /><figcaption>Figure 6: Typical progression of tensor shapes through a deep CNN used for image segmentation. Source: Author(s).</figcaption></figure><p>With vision transformers, the flow is much more complex. Let’s take a look at an image below and then try to understand how the tensor transforms shapes at every step along the way.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ruzy639FhSDbfNCQ" /><figcaption>Figure 7: Typical progression of tensor shapes through a vision transformer for image segmentation. Source: Author(s).</figcaption></figure><p>Let’s look at each step in more detail and see how it updates the shape of the tensor flowing through the vision transformer. To understand this better, let’s take concrete values for our tensor dimensions.</p><ol><li><strong>Batch Normalization:</strong> The input and output tensors have shape (1, 3, 128, 128). The shape is unchanged, but the values are normalized to zero mean and unit variance.</li><li><strong>Image to patches:</strong> The input tensor of shape (1, 3, 128, 128) is converted into a stacked patch of 16x16 images. The output tensor has shape (1, 64, 768).</li><li><strong>Patch embedding:</strong> The patch embedding layer maps the 768 input channels to 512 embedding channels (for this example). The output tensor is of shape (1, 64, 512). The patch embedding layer is basically just an nn.Linear layer in PyTorch.</li><li><strong>Position embedding:</strong> The position embedding layer doesn’t have an input tensor, but effectively contributes a learnable parameter (trainable tensor in PyTorch) o f the same shape as the patch embedding. This is of shape (1, 64, 512).</li><li><strong>Add:</strong> The patch and position embeddings are added together piecewise to produce the input to our vision transformer encoder. This tensor is of shape (1, 64, 512). You’ll notice that the main workhorse of the vision transformer, i.e. the encoder basically leaves this tensor shape unchanged.</li><li><strong>Transformer encoder:</strong> The input tensor of shape (1, 64, 512) flows through multiple transformer encoder blocks, each of which have multiple attention heads (communication) followed by an MLP layer (computation). The tensor shape remains unchanged as (1, 64, 512).</li><li><strong>Linear output projection:</strong> If we assume that we want to segment each image into 10 classes, then we will need each patch of size 16x16 to have 10 channels. The nn.Linear layer for output projection will now convert the 512 embedding channels to 16x16x10 = 2560 output channels, and this tensor will look like (1, 64, 2560). In the diagram above C’ = 10. Ideally, this would be a multi-layer perceptron, since <em>“</em><a href="https://en.wikipedia.org/wiki/Multilayer_perceptron"><em>MLPs are universal function approximators</em></a><em>”</em>, but we use a single linear layer since this is an educational exercise</li><li><strong>Patch to image:</strong> This layer converts the 64 patches encoded as a (1, 64, 2560) tensor back into something that looks like a segmentation mask. This can be 10 single channel images, or in this case a single 10 channel image, with each channel being the segmentation mask for one of the 10 classes. The output tensor is of shape (1, 10, 128, 128).</li></ol><p>That’s it — we’ve successfully segmented an input image using a vision transformer! Next, let’s take a look at an experiment along with some results.</p><h4>Vision transformers in action</h4><p>This <a href="https://github.com/dhruvbird/ml-notebooks/blob/main/pets_segmentation/vision-transformer-for-oxford-iiit-pet-segmentatio.ipynb">notebook</a> contains all the code for this section.</p><p>As far as the code and class structure is concerned, it closely mimics the block diagram above. Most of the concepts mentioned above have a 1:1 correspondence to class names in this <a href="https://github.com/dhruvbird/ml-notebooks/blob/main/pets_segmentation/vision-transformer-for-oxford-iiit-pet-segmentatio.ipynb">notebook</a>.</p><p>There are some concepts related to the attention layers that are critical hyperparameters for our model. We didn’t mention anything about the details of the multi-head attention earlier since we mentioned that it’s out of scope for the purposes of this article. We highly recommend reading the reference material mentioned above before proceeding if you don’t have a basic understanding of the attention mechanism in transformers.</p><p>We used the following model parameters for the vision transformer for segmentation.</p><ol><li>768 embedding dimensions for the PatchEmbedding layer</li><li>12 Transformer encoder blocks</li><li>8 attention heads in each transformer encoder block</li><li>20% dropout in multi-head attention and MLP</li></ol><p>This configuration can be seen in the VisionTransformerArgs Python dataclass.</p><pre>@dataclass<br>class VisionTransformerArgs:<br>    &quot;&quot;&quot;Arguments to the VisionTransformerForSegmentation.&quot;&quot;&quot;<br>    image_size: int = 128<br>    patch_size: int = 16<br>    in_channels: int = 3<br>    out_channels: int = 3<br>    embed_size: int = 768<br>    num_blocks: int = 12<br>    num_heads: int = 8<br>    dropout: float = 0.2<br># end class</pre><p>A similar configuration as <a href="https://medium.com/p/bed68cadd7c7/">before</a> was used during model training and validation. The configuration is specified below.</p><ol><li>The <em>random horizontal flip</em> and <em>colour jitter</em> data augmentations are applied to the training set to prevent overfitting</li><li>The images are resized to 128x128 pixels in a non-aspect preserving resize operation</li><li>No input normalization is applied to the images — instead a <a href="https://towardsdatascience.com/replace-manual-normalization-with-batch-normalization-in-vision-ai-models-e7782e82193c">batch normalization layer is used as the first layer of the model</a></li><li>The model is trained for 50 epochs using the Adam optimizer with a LR of 0.0004 and a StepLR scheduler that decays the learning rate by 0.8x every 12 epochs</li><li>The cross-entropy loss function is used to classify a pixel as belonging to a pet, the background, or a pet border</li></ol><p>The model has 86.28M parameters and achieved a validation accuracy of 85.89% after 50 training epochs. This is less than the 88.28% accuracy achieved by deep CNN model after 20 training epochs. This could be due to a few factors that need to be validated experimentally.</p><ol><li>The last output projection layer is a single nn.Linear and not a multi-layer perceptron</li><li>The 16x16 patch size is too large to capture more fine grained detail</li><li>Not enough training epochs</li><li>Not enough training data — it’s known that transformer models need a lot more data to train effectively compared to deep CNN models</li><li>The learning rate is too low</li></ol><p>We plotted a gif showing how the model is learning to predict the segmentation masks for 21 images in the validation set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/0*lcQNZVJ7F3UVqWuz" /><figcaption>Figure 8: A gif showing the progression of segmentation masks predicted by the vision transformer for image segmentation model. Source: Author(s).</figcaption></figure><p>We notice something interesting in the early training epochs. The predicted segmentation masks have some strange blocking artifacts. The only reason we could think of for this is because we’re breaking down the image into patches of size 16x16 and after very few training epochs, the model hasn’t learned anything useful beyond some very coarse grained information regarding whether this 16x16 patch is generally covered by a pet or by background pixels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/0*DxR-tXo4kmKxx9ND" /><figcaption>Figure 9: The blocking artifacts seen in the predicted segmentation masks when using the vision transformer for image segmentation. Source: Author(s).</figcaption></figure><p>Now that we have seen a basic vision transformer in action, let’s turn our attention to a state of the art vision transformer for segmentation tasks.</p><h4>SegFormer: Semantic segmentation with transformers</h4><p>The SegFormer architecture was proposed in <a href="https://arxiv.org/pdf/2105.15203.pdf">this paper</a> in 2021. The transformer we saw above is a simpler version of the SegFormer architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Xt4yS8unp-aRi9so" /><figcaption>Figure 10: The SegFormer architecture. Source: <a href="https://arxiv.org/pdf/2105.15203.pdf">SegFormer paper (2021)</a>.</figcaption></figure><p>Most notably, the SegFormer:</p><ol><li>Generates 4 sets of images with patches of size 4x4, 8x8, 16x16, and 32x32 instead of a single patched image with patches of size 16x16</li><li>Uses 4 transformer encoder blocks instead of just 1. This feels like a model ensemble</li><li>Uses convolutions in the pre and post phases of self-attention</li><li>Doesn’t use positional embeddings</li><li>Each transformer block processes images at spatial resolution H/4 x W/4, H/8 x W/8, H/16 x W/16, and H/32, W/32</li><li>Similarly, the channels increase when the spatial dimensions reduce. This feels similar to deep CNNs</li><li>Predictions at multiple spatial dimensions are upsampled and then merged together in the decoder</li><li>An MLP combines all these predictions to provide a final prediction</li><li>The final prediction is at spatial dimension H/4, W/4 and not at H, W.</li></ol><h3>Conclusion</h3><p>In part-4 of this series, we were introduced to the transformer architecture and vision transformers in particular. We developed an intuitive understanding of how vision transformers work, and the basic building block involved in the communication and computation phases of vision transformers. We saw the unique patch based approach adopted by vision transformers for predicting segmentation masks and then combining the predictions together.</p><p>We reviewed an experiment that shows vision transformers in action, and were able to compare results with deep CNN approaches. While our vision transformer is not state of the art, it was able to achieve pretty decent results. We provided a glimpse into state-of-the-art approaches such as SegFormer.</p><p>It should be clear by now that transformers have a lot more moving parts and are more complex compared to deep CNN-based approaches. From a raw FLOPs point of view, transformers hold the promise of being more efficient. In transformers, the only real layer that is computationally heavy is nn.Linear. This is implemented using optimized matrix multiplication on most architectures. Due to this architectural simplicity, transformers hold the promise of being easier to optimize and speed up compared to deep CNN-based approaches.</p><p>Congratulations on making it this far! We’re glad you enjoyed reading this series on efficient image segmentation in PyTorch. If you have questions or comments, please feel free to leave them in the comments section.</p><h3>Further reading</h3><p>The details of the attention mechanism are out of scope for this article. Besides, there are numerous high quality resources that you can refer to to understand the attention mechanism in great detail. Here are some that we highly recommend.</p><ol><li><a href="http://jalammar.github.io/illustrated-transformer/">The Illustrated Transformer</a></li><li><a href="https://www.youtube.com/watch?v=kCc8FmEb1nY">NanoGPT from scratch using PyTorch</a></li></ol><p>We’ll provide links to articles that provide more details on vision transformers below.</p><ol><li><a href="https://towardsdatascience.com/implementing-visualttransformer-in-pytorch-184f9f16f632">Implementing Vision Transformer (ViT) in PyTorch</a>: This article details the implementation of a vision transformer for image classification in PyTorch. Notably, their implementation uses <a href="https://einops.rocks/">einops</a>, which we avoid, since this is an education-focused exercise (we recommend learning and using einops for code readability though). We instead use native PyTorch operators for permuting and rearranging tensor dimensions. Additionally, there are a few places where the author uses Conv2d instead of Linear layers. We wanted to build an implementation of vision transformers without the use of convolutional layers entirely.</li><li><a href="https://theaisummer.com/vision-transformer/">Vision Transformer: AI Summer</a></li><li><a href="https://towardsdatascience.com/implementing-segformer-in-pytorch-8f4705e2ed0e">Implementing SegFormer in PyTorch</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6c86da083432" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/efficient-image-segmentation-using-pytorch-part-4-6c86da083432">Efficient Image Segmentation Using PyTorch: Part 4</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Efficient Image Segmentation Using PyTorch: Part 3]]></title>
            <link>https://medium.com/data-science/efficient-image-segmentation-using-pytorch-part-3-3534cf04fb89?source=rss-63f5d5495279------2</link>
            <guid isPermaLink="false">https://medium.com/p/3534cf04fb89</guid>
            <category><![CDATA[depthwise-convolution]]></category>
            <category><![CDATA[pytorch-image-segment]]></category>
            <category><![CDATA[cnn]]></category>
            <category><![CDATA[segnet]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[Dhruv Matani]]></dc:creator>
            <pubDate>Tue, 27 Jun 2023 06:12:37 GMT</pubDate>
            <atom:updated>2023-06-27T06:12:37.867Z</atom:updated>
            <content:encoded><![CDATA[<h4>Depthwise separable convolutions</h4><p>In this 4-part series, we’ll implement image segmentation step by step from scratch using deep learning techniques in PyTorch. This part will focus on optimizing our CNN baseline model using depthwise separable convolutions to reduce the number of trainable parameters, making the model deployable on mobile and other edge devices.</p><p>Co-authored with <a href="https://medium.com/@brocolishbroxoli">Naresh Singh</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*Q4ZM8UwFIrJKMbPJp6mlJg.png" /><figcaption>Figure 1: Result of running image segmentation using a CNN with depth-wise separable convolutions instead of regular convolutions. From top to bottom, input images, ground truth segmentation masks, and predicted segmentation masks. Source: Author(s)</figcaption></figure><h3>Article outline</h3><p>In this article, we will augment the <a href="https://en.wikipedia.org/wiki/Convolutional_neural_network">Convolutional Neural Network (CNN)</a> we <a href="https://medium.com/p/bed68cadd7c7/">built earlier</a> to reduce the number of learnable parameters in our network. The task of identifying pet pixels (pixels belonging to cats, dogs, hamsters, etc…) in an input image remains unchanged. Our network of choice will remain <a href="https://arxiv.org/abs/1511.00561">SegNet</a>, and the only change we’ll make is to replace our convolutional layers with depth-wise-separable-convolutions (DSC). Before we do this, we will dive into the theory and practice of depth-wise-separable-convolutions, and appreciate the idea behind the technique.</p><p>Throughout this article, we will reference code and results from this <a href="https://github.com/dhruvbird/ml-notebooks/blob/main/pets_segmentation/oxford-iiit-pets-segmentation-using-pytorch-segnet-and-depth-wise-separable-convs.ipynb">notebook</a> for model training, and this <a href="https://github.com/dhruvbird/ml-notebooks/blob/main/pets_segmentation/types%20of%20convolutions.ipynb">notebook</a> for a primer on DSC. If you wish to reproduce the results, you’ll need a GPU to ensure that the first notebook completes running in a reasonable amount of time. The second notebook can be run on a regular CPU.</p><h3>Articles in this series</h3><p>This series is for readers at all experience levels with deep learning. If you want to learn about the practice of deep learning and vision AI along with some solid theory and hands-on experience, you’ve come to the right place! This is expected to be a 4-part series with the following articles:</p><ol><li><a href="https://medium.com/p/89e8297a0923/">Concepts and Ideas</a></li><li><a href="https://medium.com/p/bed68cadd7c7/">A CNN-based model</a></li><li><strong>Depthwise separable convolutions (this article)</strong></li><li><a href="https://medium.com/p/6c86da083432/">A Vision Transformer-based model</a></li></ol><h3>Introduction</h3><p>Let’s start this discussion with a closer look at the convolutions from the perspective of model size and computation cost. The number of trainable parameters is a good indication of the size of a model and the number of the tensor operations reflects the model complexity or computation cost. Consider that we have a convolution layer with n filters with size dₖ x dₖ. Further assume that this layer processes input with shape <em>m x h x w, </em>where <em>m</em> is the number of input channels, and <em>h</em> and <em>w</em> are height and width dimensions respectively. In this case, the convolution layer will produce an output with shape <em>n x h x w</em> as shown in Figure 2. We are assuming that the convolution uses <em>stride=1</em>. Let’s go ahead and evaluate this setup in terms of trainable parameters and computation cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hZMdN-RcJVCG_efD" /><figcaption>Figure 2: Regular convolutional filters applied to input to produce output. Assume stride=1 and padding=dₖ-2. Source: <a href="https://efficientdlbook.com/">Efficient Deep Learning Book</a></figcaption></figure><p><strong>Evaluation Of Trainable Parameters:</strong> We have <em>n</em> filters, each of which has <em>m x dₖ x dₖ</em> learnable parameters. This results in a total of <em>n x m x dₖ x dₖ</em> learnable parameters. Bias terms are ignored to simplify this discussion. Let’s look at the PyTorch code below to validate our understanding.</p><pre>import torch<br>from torch import nn<br>def num_parameters(m):<br>return sum([p.numel() for p in m.parameters()])<br>dk, m, n = 3, 16, 32<br>print(f&quot;Expected number of parameters: {m * dk * dk * n}&quot;)<br>conv1 = nn.Conv2d(in_channels=m, out_channels=n, kernel_size=dk, bias=False)<br>print(f&quot;Actual number of parameters: {num_parameters(conv1)}&quot;)</pre><p>Prints the following.</p><pre>Expected number of parameters: 4608<br>Actual number of parameters: 4608</pre><p>Now, let’s evaluate the computation costs of convolution.</p><p><strong>Evaluation Of Computational Cost:</strong> A single convolutional filter of shape <em>m x dₖ x dₖ</em> when run with a <em>stride=1</em> and a <em>padding=dₖ-2</em> on an input with size <em>h x w</em> will apply the convolutional filter <em>h x w</em> times, once for each image section with size <em>dₖ x dₖ</em> amounting to a total of <em>h x w</em> sections. It results in a cost of <em>m x dₖ x dₖ x h x w</em> per filter or output channel. Since we wish to compute n output channels, the total cost will be <em>m x dₖ x dₖ x h x n</em>. Let’s go ahead and validate this using the torchinfo PyTorch package.</p><pre>from torchinfo import summary<br>h, w = 128, 128<br>print(f&quot;Expected total multiplies: {m * dk * dk * h * w * n}&quot;)<br>summary(conv1, input_size=(1, m, h, w))</pre><p>Will print the following.</p><pre>Expected total multiplies: 75497472<br><br><br>==========================================================================================<br>Layer (type:depth-idx)                   Output Shape              Param #<br>==========================================================================================<br>Conv2d                                   [1, 32, 128, 128]         4,608<br>==========================================================================================<br>Total params: 4,608<br>Trainable params: 4,608<br>Non-trainable params: 0<br>Total mult-adds (M): 75.50<br>==========================================================================================<br>Input size (MB): 1.05<br>Forward/backward pass size (MB): 4.19<br>Params size (MB): 0.02<br>Estimated Total Size (MB): 5.26<br>==========================================================================================</pre><p>If we ignore the implementation details of a convolution layer for a moment, we would realize that, on a high level, a convolution layer just transforms a <em>m x h x w</em> input into a <em>n x h x w</em> output. The transformation is achieved through trainable filters which progressively learn features as they <em>see</em> inputs. The question that follows is: Is it possible to achieve this transformation using fewer learnable parameters and simultaneously ensuring minimum compromise in the learning capabilities of the layer? Depthwise Separable Convolutions were proposed to answer this exact question. Let’s understand them in detail and learn how they stack up on our evaluation metrics.</p><h3>Depthwise Separable Convolution</h3><p>The concept of Depthwise Separable Convolutions (DSC) was first proposed by Laurent Sifre in their PhD thesis titled <a href="https://www.di.ens.fr/data/publications/papers/phd_sifre.pdf">Rigid-Motion Scattering For Image Classification</a>. Since then, they have been used successfully in various popular deep convolutional networks such as <a href="https://arxiv.org/abs/1610.02357">XceptionNet</a> and <a href="https://towardsdatascience.com/review-mobilenetv1-depthwise-separable-convolution-light-weight-model-a382df364b69">MobileNet</a>.</p><p>The main difference between a regular convolution, and a DSC is that a DSC is composed of 2 convolutions as described below:</p><ol><li>A <strong>depthwise grouped convolution</strong>, where the number of input channels m is equal to the number of output channels such that each output channel is affected only by a single input channel. In PyTorch, this is called a “grouped” convolution. You can read more about grouped convolutions in PyTorch <a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html">here</a>.</li><li>A <strong>pointwise convolution</strong> (filter size=1), which operates like a regular convolution such that each of the n filters operates on all m input channels to produce a single output value.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g1cvC1PNtX5TCuwc" /><figcaption>Figure 3: Depthwise Separable Convolution filters applied to input to produce output. Assume <em>stride=1</em> and <em>padding=dₖ-2</em>. Source: <a href="https://efficientdlbook.com/">Efficient Deep Learning Book</a></figcaption></figure><p>Let’s perform the same exercise that we did for regular convolutions for DSCs and compute the number of trainable parameters and computations.</p><p><strong>Evaluation Of Trainable Parameters:</strong> The “grouped” convolutions have m filters, each of which has <em>dₖ x dₖ</em> learnable parameters which produces m output channels. This results in a total of <em>m x dₖ x dₖ</em> learnable parameters. The pointwise convolution has n filters of size <em>m x 1 x 1</em> which adds up to <em>n x m x 1 x 1</em> learnable parameters. Let’s look at the PyTorch code below to validate our understanding.</p><pre>class DepthwiseSeparableConv(nn.Sequential):<br>    def __init__(self, chin, chout, dk):<br>        super().__init__(<br>            # Depthwise convolution<br>            nn.Conv2d(chin, chin, kernel_size=dk, stride=1, padding=dk-2, bias=False, groups=chin),<br>            # Pointwise convolution<br>            nn.Conv2d(chin, chout, kernel_size=1, bias=False),<br>        )<br><br>conv2 = DepthwiseSeparableConv(chin=m, chout=n, dk=dk)<br>print(f&quot;Expected number of parameters: {m * dk * dk + m * 1 * 1 * n}&quot;)<br>print(f&quot;Actual number of parameters: {num_parameters(conv2)}&quot;)</pre><p>Which will print.</p><pre>Expected number of parameters: 656<br>Actual number of parameters: 656</pre><p>We can see that the DSC version has roughly <em>7x</em> less parameters. Next, let’s focus our attention on the computation costs for a DSC layer.</p><p><strong>Evaluation Of Computational Cost:</strong> Let’s assume our input has spatial dimensions <em>m x h x w</em>. In the grouped convolution segment of DSC, we have <strong>m</strong> filters, each with size <em>dₖ x dₖ</em>. A filter is applied to its corresponding input channel resulting in the segment cost of <em>m x dₖ x dₖ x h x w</em>. For the pointwise convolution, we apply <strong>n</strong> filters of size <em>m x 1 x 1</em><strong> </strong>to produce <strong>n </strong>output channels. This results in the segment cost of <em>n x m x 1 x 1 x h x w</em>. We need to add up the costs of the grouped and pointwise operations to compute the total cost. Let’s go ahead and validate this using the <a href="https://pypi.org/project/torchinfo/">torchinfo</a> PyTorch package.</p><pre>print(f&quot;Expected total multiplies: {m * dk * dk * h * w + m * 1 * 1 * h * w * n}&quot;)<br>s2 = summary(conv2, input_size=(1, m, h, w))<br>print(f&quot;Actual multiplies: {s2.total_mult_adds}&quot;)<br>print(s2)</pre><p>Which will print.</p><pre>Expected total multiplies: 10747904<br>Actual multiplies: 10747904<br>==========================================================================================<br>Layer (type:depth-idx)                   Output Shape              Param #<br>==========================================================================================<br>DepthwiseSeparableConv                   [1, 32, 128, 128]         --<br>├─Conv2d: 1-1                            [1, 16, 128, 128]         144<br>├─Conv2d: 1-2                            [1, 32, 128, 128]         512<br>==========================================================================================<br>Total params: 656<br>Trainable params: 656<br>Non-trainable params: 0<br>Total mult-adds (M): 10.75<br>==========================================================================================<br>Input size (MB): 1.05<br>Forward/backward pass size (MB): 6.29<br>Params size (MB): 0.00<br>Estimated Total Size (MB): 7.34<br>==========================================================================================</pre><p>Let’s compare the sizes and costs of both the convolutions for a few examples to gain some intuition.</p><h4>Size and Cost comparison for regular and depthwise separable convolutions</h4><p>To compare the size and cost of regular and depthwise separable convolution, we will assume an input size of <em>128 x 128</em> to the network, a kernel size of <em>3 x 3</em>, and a network that progressively halves the spatial dimensions and doubles the number of channel dimensions. We assume a single 2d-conv layer at every step, but in practice, there could be more.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QDXpcqsv7euq_t-F" /><figcaption>Figure 4: Comparing the number of trainable parameters (size) and multi-adds (cost) of regular and depthwise separable convolutions. We also show the ratio of the size and cost for the 2 types of convolutions. Source: Author(s).</figcaption></figure><p>You can see that on average both the size and computational cost of DSC is about 11% to 12% of the cost of regular convolutions for the configuration mentioned above.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*94FsN28xUjEf4-Hd" /><figcaption>Figure 5: Relative size and cost of regular v/s DSC. Source: Author(s).</figcaption></figure><p>Now that we have developed a good understanding of the types of convolutions and their relative costs, you must be wondering if there’s any downside of using DSCs. Everything we’ve seen so far seems to suggest that they are better in every way! Well, we haven’t yet considered an important aspect which is the impact they have on the accuracy of our model. Let’s dive into it via an experiment below.</p><h3>SegNet Using Depthwise Separable Convolutions</h3><p><a href="https://github.com/dhruvbird/ml-notebooks/blob/main/pets_segmentation/oxford-iiit-pets-segmentation-using-pytorch-segnet-and-depth-wise-separable-convs.ipynb">This notebook</a> contains all the code for this section.</p><p>We will adapt our SegNet model from the <a href="https://medium.com/p/bed68cadd7c7/">previous post</a> and replace all the regular convolutional layers with a DSC layer. Once we do this, we notice that the number of parameters in our notebook drops from 15.27M to 1.75M parameters, which is a reduction of 88.5%! This is inline with our earlier estimates of an 11% to 12% reduction in the number of trainable parameters of the network.</p><p>A similar configuration as <a href="https://medium.com/p/bed68cadd7c7/">before</a> was used during model training and validation. The configuration is specified below.</p><ol><li>The <em>random horizontal flip</em> and <em>colour jitter</em> data augmentations are applied to the training set to prevent overfitting</li><li>The images are resized to 128x128 pixels in a non-aspect preserving resize operation</li><li>No input normalization is applied to the images — instead a <a href="https://towardsdatascience.com/replace-manual-normalization-with-batch-normalization-in-vision-ai-models-e7782e82193c">batch normalization layer is used as the first layer of the model</a></li><li>The model is trained for 20 epochs using the Adam optimizer with a LR of 0.001 and no LR scheduler</li><li>The cross-entropy loss function is used to classify a pixel as belonging to a pet, the background, or a pet border</li></ol><p>The model achieved a validation accuracy of 86.96% after 20 training epochs. This is less than the 88.28% accuracy achieved by the model using regular convolutions for the same number of training epochs. We have determined experimentally that training for more epochs improves the accuracy of both models, so 20 epochs is definitely not the end of the training cycle. We stop at 20 epochs for the purposes of this article for demonstration purposes.</p><p>We plotted a gif showing how the model is learning to predict the segmentation masks for 21 images in the validation set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/0*E7Qphfwo2gIGpWDc" /><figcaption>Figure 6: A gif showing how the SegNet model with DSC is learning to predict segmentation masks for 21 images in the validation set. Source: Author(s)</figcaption></figure><p>Now that we have seen how the model progresses through the training cycle, let’s compare the train cycles of models with regular convolutions and DSC.</p><h4>Accuracy Comparisons</h4><p>We found it useful to look at the training cycles of the models using regular convolutions and DSC. The main difference we noticed is in the early phases (epochs) of training, after which both models settled roughly into the same prediction flow. In fact after training both models for 100 epochs, we noticed that the accuracy of the model with DSC is just about 1% less than the model with regular convolutions. This is inline with our observations from just 20 epochs of training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/0*-FP-stW37_Wx4BaE" /><figcaption>Figure 7: A gif showing the progression of segmentation masks predicted by the SegNet model using regular convolutions v/s DSC. Source: Author(s).</figcaption></figure><p>You would have noticed that both models get the predictions roughly right after just 6 training epochs — i.e. one can visually see that the models are predicting something useful. Most of the hard work of training the model is then above ensuring that the borders of the predicted masks are as tight as possible and as close to the actual pets in the image as possible. This means that while one can expect a lesser absolute increase in accuracy in the later training epochs, the impact of this on the quality of predictions is much more. We’ve noticed that a single digit of accuracy improvement at higher absolute accuracy values (going from 89% to 90%) results in significant qualitative improvements to the predictions.</p><h4>Comparison with a UNet model</h4><p>We ran an experiment that changed a lot of hyperparameters with a focus on improving the overall accuracy to get a sense of how far this setting is from close to optimal. Here’s the configuration of that experiment.</p><ol><li>Image size: 128 x 128 — same as the experiments so far</li><li>Train epochs: 100 — current experiments trained for 20 epochs</li><li>Augmentations: A lot more augmentations such as image rotation, channel dropping, random block removal. We used Albumentations instead of torchvision transforms. Albumentations automatically transforms segmentation masks for us</li><li>LR Scheduler: A StepLR scheduler was used with a decay of 0.8x every 25 train epochs</li><li>Loss function: We tried 4 different loss functions: Cross Entropy, Focal, Dice, Weighted Cross Entropy. Dice performed worst whereas the rest were pretty much comparable to each other. In fact the difference in best accuracy between the rest after 100 epochs was in the 4th digit after the decimal (assuming the accuracy is a number between 0.0 and 1.0)</li><li>Convolution type: Regular</li><li>Model type: UNet — current experiments used a SegNet model</li></ol><p>We achieved a best validation accuracy of 91.3% for the setting above. We noticed that the image size significantly impacts the best validation accuracy. For example, when we changed the image size to 256 x 256, the best validation accuracy went up to 93.0%. However, training took much longer, and used more memory, which meant that we had to reduce the batch size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*V27KB5cxTKQTS9Fv" /><figcaption>Figure 8: Result of training a UNet model for 100 train epochs with the hyperparameters mentioned above. Source: Author(s).</figcaption></figure><p>You can see that the predictions are much smoother and crisper compared to the ones we have been seeing so far.</p><h3>Conclusion</h3><p>In part-3 of this series, we learned about depth wise separable convolutions (DSC) as a technique to reduce model size and training/inference cost without a significant loss in validation accuracy. We learned about the size/cost tradeoff to expect between regular and DSC for a specific setting.</p><p>We showed how to adapt the SegNet model to use DSC in PyTorch. This technique can be applied to any deep CNN. In fact we can selectively replace some of the convolutional layers with DSC — i.e. we don’t need to necessarily replace all of them. Choosing which layers to replace will depend on the balance you wish to strike between model size/runtime-cost and prediction accuracy. This decision will depend on your specific use case and deployment setup.</p><p>While this article trained models for 20 epochs, we explained that this is insufficient for production workloads, and provided a glimpse into what one can expect if one trains the model for more epochs. In addition, we provided an introduction to some of the hyperparameters that one can tune during model training. While this list is by no means comprehensive, it should allow you to appreciate the complexity and decision making needed to train an image segmentation model for production workloads.</p><p>In the <a href="https://medium.com/p/6c86da083432/">next part of this series</a>, we’ll take a look at Vision Transformers, and how we can use this model architecture to perform image segmentation for the pets segmentation task.</p><h3>References and further reading</h3><ul><li><a href="https://github.com/EfficientDL/book/raw/main/book/%5BEDL%5D%20Chapter%204%20-%20Efficient%20Architectures.pdf">Efficient Deep Learning Book Chapter 04 — Efficient Architectures</a></li><li><a href="https://towardsdatascience.com/a-basic-introduction-to-separable-convolutions-b99ec3102728">A Basic Introduction to Separable Convolutions</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3534cf04fb89" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/efficient-image-segmentation-using-pytorch-part-3-3534cf04fb89">Efficient Image Segmentation Using PyTorch: Part 3</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Efficient Image Segmentation Using PyTorch: Part 2]]></title>
            <link>https://medium.com/data-science/efficient-image-segmentation-using-pytorch-part-2-bed68cadd7c7?source=rss-63f5d5495279------2</link>
            <guid isPermaLink="false">https://medium.com/p/bed68cadd7c7</guid>
            <category><![CDATA[pytorch-image-segment]]></category>
            <category><![CDATA[convolutional-network]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[cnn]]></category>
            <dc:creator><![CDATA[Dhruv Matani]]></dc:creator>
            <pubDate>Tue, 27 Jun 2023 06:11:46 GMT</pubDate>
            <atom:updated>2023-06-27T06:11:46.933Z</atom:updated>
            <content:encoded><![CDATA[<h4>A CNN-based model</h4><p>This is the second part of the 4-part series to implement image segmentation step by step from scratch using deep learning techniques in PyTorch. This part will focus on implementing a baseline image segmentation Convolutional Neural Network (CNN) model.</p><p>Co-authored with <a href="https://medium.com/@brocolishbroxoli">Naresh Singh</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/659/1*ZC7cxgypMANduMyG_zOIGw.png" /><figcaption>Figure 1: Result of running image segmentation using a CNN. In order from top to bottom, input images, ground truth segmentation masks, predicted segmentation masks. Source: Author(s)</figcaption></figure><h3>Article outline</h3><p>In this article, we will implement a <a href="https://en.wikipedia.org/wiki/Convolutional_neural_network">Convolutional Neural Network (CNN)</a> based architecture called <a href="https://arxiv.org/abs/1511.00561">SegNet</a> which will assign each pixel in an input image to a corresponding pet such as a cat or a dog. The pixels which don’t belong to any pet will be classified as background pixels. We will build and train this model on the <a href="https://www.robots.ox.ac.uk/~vgg/data/pets/">Oxford Pets dataset</a> using <a href="https://pytorch.org/">PyTorch</a> to develop a sense of what it takes to deliver a successful image segmentation task. The model building process will be hands-on where we will discuss in detail the role of each layer in our model. The article will contain plenty of references to research papers and articles for further learning.</p><p>Throughout this article, we will reference the code and results from this <a href="https://github.com/dhruvbird/ml-notebooks/blob/main/pets_segmentation/oxford-iiit-pets-segmentation-using-pytorch-segnet-and-depth-wise-separable-convs.ipynb">notebook</a>. If you wish to reproduce the results, you’ll need a GPU to ensure that the notebook completes running in a reasonable amount of time.</p><h3>Articles in this series</h3><p>This series is for readers at all experience levels with deep learning. If you want to learn about the practice of deep learning and vision AI along with some solid theory and hands-on experience, you’ve come to the right place! This is expected to be a 4-part series with the following articles:</p><ol><li><a href="https://medium.com/p/89e8297a0923/">Concepts and Ideas</a></li><li><strong>A CNN-based model (this article)</strong></li><li><a href="https://medium.com/p/3534cf04fb89/">Depthwise separable convolutions</a></li><li><a href="https://medium.com/p/6c86da083432/">A Vision Transformer-based model</a></li></ol><p>Let’s start this discussion with a short introduction to the Convolution layers and a few other layers that are typically used together as a convolution block.</p><h3>Conv-BatchNorm-ReLU and Max Pooling/Unpooling</h3><p>A convolution, batch-normalization, ReLU block is the holy trinity of vision AI. You’ll see it used frequently with CNN-based vision AI models. Each of these terms stands for a distinct layer implemented in PyTorch. The convolution layer is responsible for performing a cross-correlation operation of learned filters on the input tensor. Batch Normalization centers the elements in the batch to zero mean and unit variance, and ReLU is a non-linear activation function that keeps just the positive values in the input.</p><p>A typical CNN progressively reduces the input spatial dimensions as layers are stacked. The motivation behind the reduction of spatial dimensions is discussed in the next section. This reduction is achieved by pooling the neighboring values using a simple function such as max or average. We will discuss this further in the Max-Pooling section. In classification problems, the stack of Conv-BN-ReLU-Pool blocks is followed by a classification head which predicts the probability that input belongs to one of the target classes. Some sets of problems such as Semantic Segmentation require per-pixel prediction. For such cases, a stack of upsampling blocks are appended after the downsampling blocks to project their output to the required spatial dimension. The upsampling blocks are nothing but Conv-BN-ReLU-Unpool blocks which replace the pooling layer with an un-pooling layer. We will talk more about un-pooling in the Max-Pooling section.</p><p>Now, let’s further elaborate on the motivation behind convolution layers.</p><h4>Convolution</h4><p>Convolutions are the basic building blocks of vision AI models. They are used heavily in computer vision and have historically been used to implement vision transformations such as:</p><ol><li>Edge detection</li><li>Image blurring and sharpening</li><li>Embossing</li><li>Intensification</li></ol><p>A convolution operation is an elementwise multiplication and aggregation of two matrices. An example convolution operation is shown in Figure 2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/704/0*9-7pcgAWstNxekPT" /><figcaption>Figure 2: An illustration of the convolution operation. Source: Author(s)</figcaption></figure><p>In a deep learning context, convolution is carried out between an <em>n-dimensional</em> parameter matrix called a filter or a kernel over a larger-sized input. This is achieved by sliding the filter over the input and applying convolution to the corresponding section. The extent of the slide is configured using a stride parameter. A stride of one means the kernel slides over by one step to operate on the next section. As opposed to the traditional approaches where a fixed filter is used, deep learning learns the filter from the data using backpropagation.</p><p><strong>So how do convolutions assist in deep learning?</strong></p><p>In deep learning, a convolution layer is used to detect visual features. A typical CNN model contains a stack of such layers. The bottom layers in the stack detect simple features such as lines and edges. As we move up in the stack, the layers detect increasingly complex features. Middle layers in the stack detect combinations of lines and edges and the top layers detect complex shapes such as a car, a face or an airplane. Figure 3 shows visually the output of top and bottom layers for a trained model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TSlsgbgCw7Ddq8ux" /><figcaption>Figure 3: What convolutional filters learn to identify. Source: <a href="http://web.eecs.umich.edu/~honglak/icml09-ConvolutionalDeepBeliefNetworks.pdf">Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations</a></figcaption></figure><p>A convolution layer has a set of learnable filters that act on small regions in the input to produce a representative output value for each region. For example, a 3x3 filter operates over a 3x3 size region and produces a value representative of the region. The repeated application of a filter over input regions produces an output which becomes the input to the next layer in the stack. Intuitively, the layers higher up get to “see” a larger region of the input. For example, a 3x3 filter in the second convolution layer operates on the output of the first convolution layer where each cell contains information about the 3x3 sized region in the input. If we assume a convolution operation with stride=1, then the filter in the second layer will “see’’ the 5x5 sized region of the original input. This is called the <a href="https://theaisummer.com/receptive-field/">receptive field</a> of the convolution. The repeated application of convolutional layers progressively reduces the spatial dimensions of the input image and increases the field of vision of the filters which enables them to “see” complex shapes. Figure 4 shows the processing of a 1-D input by a convolution network. An element in the output layer is a representative of a relatively larger input chunk.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/675/0*LKQOxTXGm5lTJtoc" /><figcaption>Figure 4: Receptive field of a 1d convolution with kernel size=3, applied 3 times. Assume stride=1 and no padding. After the 3rd successive application of the convolutional kernel, a single pixel is able to see 7 pixels in the original input image. Source: Author(s)</figcaption></figure><p>Once a convolutional layer can detect these objects and is able to generate their representations, we can use these representations for image classification, image segmentation, and object detection and localization. Broadly speaking, CNNs adhere to the following general principles:</p><ol><li>A Convolution layer either keeps the number of output channels © intact or doubles them.</li><li>It keeps the spatial dimensions intact using a stride=1 or reduces them to a half using stride=2.</li><li>It’s common to pool the output of a convolution block to change the spatial dimensions of an image.</li></ol><p>A convolution layer applies the kernel independently to each input. This could cause its output to vary for different inputs. A Batch Normalization layer typically follows a convolution layer to address this problem. Let’s understand its role in detail in the next section.</p><h4>Batch Normalization</h4><p>Batch Normalization layer normalizes the channel values in the batch input to have a zero mean and a unit variance. This normalization is performed independently for each channel in the batch to ensure that the channel values for the inputs have the same distribution. Batch Normalization has the following benefits:</p><ol><li>It stabilizes the training process by preventing the gradients from becoming too small.</li><li>It achieves faster convergence on our tasks.</li></ol><p>If all we had was a stack of convolution layers, it would essentially be equivalent to a single convolution layer network because of the cascading effect of linear transformations. In other words, a sequence of linear transformations can be replaced with a single linear transformation which has the same effect. Intuitively, if we multiply a vector with a constant <em>k₁</em> followed by multiplication with another constant <em>k₂</em>, it is equivalent to a single multiplication by a constant <em>k₁k₂</em>. Hence, for the networks to be realistically deep, they must have a non-linearity to prevent their <em>collapse</em>. We will discuss ReLU in the next section which is frequently used as a non-linearity.</p><h4>ReLU</h4><p>ReLU is a simple non-linear activation function which clips the lowest input values to be greater than or equal to 0. It also helps with the vanishing gradients problem limiting the outputs to be greater than or equal to 0. The ReLU layer is typically followed by a pooling layer to shrink the spatial dimensions in the downscaling subnetwork or an un-pooling layer to bump the spatial dimensions in the upscaling subnetwork. The details are provided in the next section.</p><h4>Pooling</h4><p>A pooling layer is used to shrink the spatial dimensions of our inputs. Pooling with <em>stride=2</em> will transform an input with spatial dimensions (H, W) to (H/2, W/2). <a href="https://pytorch.org/docs/stable/generated/torch.nn.MaxPool2d.html">Max-pooling</a> is the most commonly used pooling technique in deep CNNs. It projects the maximum value in a grid of (say) 2x2 onto the output. Then, we slide the 2x2 pooling window to the next section based on the stride similar to convolutions. Doing this repeatedly with a <em>stride=2</em> results in an output that is half the height and half the width of the input. Another commonly used pooling layer is the <a href="https://pytorch.org/docs/stable/generated/torch.nn.AvgPool2d.html#torch.nn.AvgPool2d">average-pooling</a> layer, which computes the average instead of the max.</p><p>The reverse of a pooling layer is called an un-pooling layer. It takes an (H, W) dimension input and converts it into a (2H, 2W) dimension output for <em>stride=2</em>. A necessary ingredient of this transformation is selecting the location in the 2x2 section of the output to project the input value. To do this, we need a <a href="https://pytorch.org/docs/stable/generated/torch.nn.MaxUnpool2d.html#torch.nn.MaxUnpool2d">max-unpooling</a>-index-map which tells us the target locations in the output section. This unpooling-map is produced by a previous max-pooling operation. Figure 5 shows examples of pooling and un-pooling operations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/0*ZFlYSQX-8bTujUT4" /><figcaption>Figure 5: Max pooling and un-pooling. Source: <a href="https://www.researchgate.net/publication/306081538_DeepPainter_Painter_Classification_Using_Deep_Convolutional_Autoencoders">DeepPainter: Painter Classification Using Deep Convolutional Autoencoders</a></figcaption></figure><p>We can consider max-pooling as a type of non-linear activation function. However, it’s <a href="https://ai.stackexchange.com/questions/17937/is-a-non-linear-activation-function-needed-if-we-perform-max-pooling-after-the-c">reported</a> that using it to replace a non-linearity such as ReLU <a href="https://arxiv.org/pdf/1606.02228.pdf">affects the network’s performance</a>. In contrast, average pooling can not be considered as a nonlinear function since it uses all its inputs to produce an output that is a linear combination of its inputs.</p><p>This covers all the basic building blocks of deep CNNs. Now, let’s put them together to create a model. The model we have chosen for this exercise is called a SegNet. We’ll discuss it next.</p><h3>SegNet: A CNN-based model</h3><p><a href="https://arxiv.org/abs/1511.00561">SegNet</a> is a deep CNN model based on the fundamental blocks that we have discussed in this article. It has two distinct sections. The bottom section, also called an encoder, down-samples the input to generate features representative of the input. The top decoder section up-samples the features to create per-pixel classification. Each section is composed of a sequence of Conv-BN-ReLU blocks. These blocks also incorporate pooling or un-pooling layers in downsampling and upsampling paths respectively. Figure 6 shows the arrangement of the layers in more detail. SegNet uses the pooling indices from the max-pooling operation in the encoder to determine which values to copy over during the max-unpooling operation in the decoder. While each element of an activation tensor is 4-bytes (32-bits), an offset within a 2x2 square can be stored using just 2-bits. This is more efficient in terms of memory used since these activations (or indices in the case of SegNet) need to be stored while the model runs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4ch5izpkkQaJuGjH" /><figcaption>Figure 6: The SegNet model architecture for image segmentation. Source: SegNet: <a href="https://arxiv.org/pdf/1511.00561.pdf">A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation</a></figcaption></figure><p><a href="https://github.com/dhruvbird/ml-notebooks/blob/main/pets_segmentation/oxford-iiit-pets-segmentation-using-pytorch-segnet-and-depth-wise-separable-convs.ipynb">This notebook</a> contains all the code for this section.</p><p>This model has 15.27M trainable parameters.</p><p>The following configuration was used during model training and validation.</p><ol><li>The <em>random horizontal flip</em> and <em>color jitter</em> data augmentations are applied to the training set to prevent overfitting</li><li>The images are resized to 128x128 pixels in a non-aspect preserving resize operation</li><li>No input normalization is applied to the images; instead a <a href="https://towardsdatascience.com/replace-manual-normalization-with-batch-normalization-in-vision-ai-models-e7782e82193c">batch normalization layer is used as the first layer of the model</a></li><li>The model is trained for 20 epochs using the Adam optimizer with a LR of 0.001 and a StepLR Scheduler that decays the learning rate by 0.7 every 7 epochs</li><li>The cross-entropy loss function is used to classify a pixel as belonging to a pet, the background, or a pet border</li></ol><p>The model achieved a validation accuracy of 88.28% after 20 training epochs.</p><p>We plotted a gif showing how the model is learning to predict the segmentation masks for 21 images in the validation set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/712/0*NPAnhNGXXCnP3W_B" /><figcaption>Figure 6: A gif showing how the SegNet model is learning to predict segmentation masks for 21 images in the validation set. Source: Author(s)</figcaption></figure><p>The definitions of all the validation metrics are described in <a href="https://medium.com/p/89e8297a0923/">Part-1 of this series</a>.</p><p>If you’d like to see a fully-convolutional model for segmenting pet images implemented using Tensorflow, please see Chapter-4: Efficient Architectures of the <a href="https://efficientdlbook.com/#table-of-contents">Efficient Deep Learning Book</a>.</p><h4>Observations from model learning</h4><p>Based on the development of the predictions that the trained model makes after every epoch, we can observe the following.</p><ol><li>The model is able to learn enough to make the output look in the right ballpark of the pet in the image even as early as 1 training epoch</li><li>The border pixels are harder to segment since we’re using an unweighted loss function that treats each success (or failure) equally, so getting the border pixels wrong doesn’t cost the model much in terms of the loss. We would encourage you to investigate this and check what strategies you could try to fix this issue. Try using <a href="https://towardsdatascience.com/focal-loss-a-better-alternative-for-cross-entropy-1d073d92d075">Focal Loss</a> and see how it performs</li><li>The model seems to be learning even after 20 training epochs. This suggests that we could improve validation accuracy if we trained the model longer</li><li>Some of the ground-truth labels themselves are hard to figure out — for example, the mask of the dog in the middle row, last column has a lot of unknown pixels in the area where the body of the dog is occluded by plants. This is very hard for the model to figure out, so one should always expect loss in accuracy for such examples. However, this doesn’t mean that the model isn’t doing well. One should always spot check the predictions to develop a sense of the model’s behavior in addition to looking at the overall validation metrics.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/828/0*6KfiVzgGqga_XOry" /><figcaption>Figure 7: An example of the ground truth segmentation mask containing a lot of unknown pixels. This is a very hard input for any ML model. Source: Author(s)</figcaption></figure><h3>Conclusion</h3><p>In part-2 of this series, we learned about the basic building blocks of deep CNNs for vision AI. We saw how to implement the SegNet model from scratch in PyTorch, and we visualized how the model trained on successive epochs performs on 21 validation images. This should help you appreciate how quickly models can learn enough to make the output look somewhere in the right ballpark. In this case, we can see segmentation masks that roughly resemble the actual segmentation mask as early as the first training epoch!</p><p>In the <a href="https://medium.com/p/3534cf04fb89/">next part of this series</a>, we’ll take a look at how we can optimize our model for on-device inference and reduce the number of trainable parameters (and hence model size) while keeping the validation accuracy roughly the same.</p><h3>Further Reading</h3><p>Read more about convolutions here:</p><ol><li>The course titled “<a href="https://pjreddie.com/courses/computer-vision/">Ancient Secrets of computer vision</a>” at The University of Washington taught by Joseph Redmon has an excellent set of videos on Convolutions (especially chapters 4, 5, and 13), which we highly recommend watching</li><li><a href="https://arxiv.org/abs/1603.07285">A guide to convolution arithmetic for deep learning</a> (highly recommended)</li><li><a href="https://towardsdatascience.com/computer-vision-convolution-basics-2d0ae3b79346">https://towardsdatascience.com/computer-vision-convolution-basics-2d0ae3b79346</a></li><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.Conv2d.html#torch.nn.Conv2d">The Conv2d layer in PyTorch (documentation)</a></li><li><a href="https://www.kdnuggets.com/2016/11/intuitive-explanation-convolutional-neural-networks.html/3">What do convolutions learn?</a></li><li><a href="https://ezyang.github.io/convolution-visualizer/index.html">Convolution visualizer</a></li></ol><p>Read more about batch normalization here:</p><ol><li><a href="https://en.wikipedia.org/wiki/Batch_normalization">Batch normalization: Wikipedia</a></li><li><a href="https://machinelearningmastery.com/batch-normalization-for-training-of-deep-neural-networks/">Batch normalization: Machine learning mastery</a></li><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html">BatchNorm2d layer in PyTorch here</a>.</li></ol><p>Read more about activation functions and ReLU here:</p><ol><li><a href="https://machinelearningmastery.com/rectified-linear-activation-function-for-deep-learning-neural-networks/">ReLU: Machine learning mastery</a></li><li><a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)">ReLU: Wikipedia</a></li><li><a href="https://www.quora.com/Why-is-ReLU-used-so-much-in-neural-networks">ReLU: Quora</a></li><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.ReLU.html">ReLU API in PyTorch</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bed68cadd7c7" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/efficient-image-segmentation-using-pytorch-part-2-bed68cadd7c7">Efficient Image Segmentation Using PyTorch: Part 2</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Efficient Image Segmentation Using PyTorch: Part 1]]></title>
            <link>https://medium.com/data-science/efficient-image-segmentation-using-pytorch-part-1-89e8297a0923?source=rss-63f5d5495279------2</link>
            <guid isPermaLink="false">https://medium.com/p/89e8297a0923</guid>
            <category><![CDATA[image-segmentation]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[pytorch-image-segment]]></category>
            <dc:creator><![CDATA[Dhruv Matani]]></dc:creator>
            <pubDate>Tue, 27 Jun 2023 06:10:37 GMT</pubDate>
            <atom:updated>2023-06-27T06:10:37.483Z</atom:updated>
            <content:encoded><![CDATA[<h4>Concepts and Ideas</h4><p>In this 4-part series, we’ll implement image segmentation step by step from scratch using deep learning techniques in PyTorch. We’ll start the series with the basic concepts and ideas needed for image segmentation in this article.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/475/1*NuH_7BtF6-SVD8iD6JIGeA.png" /><figcaption>Figure 1: Pet images and their segmentation masks (Source: <a href="https://www.robots.ox.ac.uk/~vgg/data/pets/">The Oxford-IIIT Pet Dataset</a>)</figcaption></figure><p><strong>Co-authored with </strong><a href="https://medium.com/@brocolishbroxoli"><strong>Naresh Singh</strong></a></p><p><a href="https://en.wikipedia.org/wiki/Image_segmentation">Image Segmentation</a> is a technique to isolate pixels belonging to specific objects in an image. Isolation of object pixels opens doors to interesting applications. For example, in Figure 1, the images on the right are the masks corresponding to the pet images on the left where the yellow pixels belong to the pet. Once the pixels are identified, we could easily make the pet bigger or change the image backgrounds. This technique is widely used in face filter features in several social media applications.</p><p>Our goal at the end of this article series is to give the reader a sense of all the steps it takes to build a vision AI model and run experiments with different settings using PyTorch.</p><h3>Articles in this series</h3><p>This series is for readers at all experience levels with deep learning. If you want to learn about the practice of deep learning and vision AI along with some solid theory and hands-on experience, you’ve come to the right place! This is expected to be a 4-part series with the following articles:</p><ol><li><strong>Concepts and Ideas (this article)</strong></li><li><a href="https://medium.com/p/bed68cadd7c7/">A CNN-based model</a></li><li><a href="https://medium.com/p/3534cf04fb89/">Depthwise separable convolutions</a></li><li><a href="https://medium.com/p/6c86da083432/">A Vision Transformer-based model</a></li></ol><h3>Introduction to Image Segmentation</h3><p><a href="https://en.wikipedia.org/wiki/Image_segmentation">Image segmentation</a> partitions or segments an image into regions that correspond to objects, backgrounds, and boundaries. Take a look at Figure 2 which shows a city scene. It marks regions corresponding to cars, motorcycles, trees, buildings, sidewalks, and other interesting objects with different color masks. These regions are identified through image segmentation techniques.</p><p>Historically, we have used <a href="https://www.v7labs.com/blog/image-segmentation-guide#h4">specialized image processing tools and pipelines</a> to break down an image into regions. However, due to the incredible growth of visual data in the last two decades, deep learning has emerged as a go-to solution to the image segmentation task. It significantly reduces the reliance on an expert to build a domain-specific image segmentation strategy as was done in the past. A deep learning practitioner can train an image segmentation model if enough training data is available for the task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/1*8TVBNJFjaQC6dsJY_88WCA.jpeg" /><figcaption>Figure 2: A segmented scene from the <a href="https://aev-autonomous-driving-dataset.s3.eu-central-1.amazonaws.com/LICENSE.txt">a2d2 dataset (CC BY-ND 4.0)</a></figcaption></figure><h4>What are the applications of image segmentation?</h4><p>Image Segmentation has applications in diverse fields such as communication, agriculture, transportation, healthcare, and more. Moreover, its applications are growing with the growth of visual data. Here are a few examples:</p><ul><li>In <strong>self-driving cars</strong>, a deep learning model constantly processes the video feed from the car cameras to segment the scene into objects such as cars, pedestrians, and traffic lights which is essential for the car to operate safely.</li><li>In <strong>medical imaging</strong>, image segmentation assists doctors to identify areas in the medical scans that correspond to tumors, lesions, and other abnormalities.</li><li>In <strong>Zoom video calls</strong>, it is used to preserve an individual’s privacy by replacing the background with virtual scenes.</li><li>In <strong>agriculture</strong>, the information about weed and crop regions identified using image segmentation is used to <a href="https://www.sciencedirect.com/science/article/pii/S2214317323000112">maintain healthy crop yields</a>.</li></ul><p>You can read more details about the practical applications of image segmentation on this <a href="https://www.v7labs.com/blog/image-segmentation-guide#h6">page by v7labs</a>.</p><h4>What are the different types of image segmentation tasks?</h4><p>There are many different types of image segmentation tasks, each with its advantages and disadvantages. The 2 most common types of image segmentation tasks are:</p><ul><li><strong>Class or Semantic segmentation</strong>: Class Segmentation assigns a semantic class such as <em>background</em>, <em>road</em>, <em>car,</em> or <em>person </em>to each image pixel. If the image has 2 cars in it, then the pixels corresponding to both cars will be tagged as car pixels. It is often used for tasks such as autonomous driving and <a href="https://ps.is.mpg.de/research_fields/semantic-scene-understanding">scene understanding</a>.</li><li><strong>Object or Instance segmentation</strong>: Object segmentation identifies objects and assigns a mask to each unique object in an image. If the image has 2 cars in it, then the pixels corresponding to each car will be identified as belonging to separate objects. Object segmentation is often used for tracking individual objects, such as a self-driving car programmed to follow a specific car ahead of it</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/705/1*7mQrNov8Fd7OnPcOJ8ipyQ.png" /><figcaption>Figure 3: Object and class segmentations (Source: <a href="https://cocodataset.org/#home">MS Coco — Creative Commons Attribution license</a>)</figcaption></figure><p>In this series, we’ll focus on class segmentation.</p><h4>Decisions needed to implement efficient image segmentation</h4><p>Efficiently training your model for speed and accuracy involves making numerous important decisions during the lifecycle of your project. This includes (but is not limited to):</p><ol><li>Choice of your deep learning framework</li><li>Choosing a good model architecture</li><li>Selecting an effective loss function that optimizes the aspect you care about</li><li>Avoiding overfitting and underfitting</li><li>Evaluating the model’s accuracy</li></ol><p>In the rest of this article, we’ll dive deeper into each of the aspects mentioned above and provide numerous links to articles that discuss each topic in a lot more detail that can be covered here.</p><h3>PyTorch for efficient image segmentation</h3><h4>What is PyTorch?</h4><p><em>“</em><a href="https://pytorch.org/"><em>PyTorch</em></a><em> is an open source deep learning framework built to be flexible and modular for research, with the stability and support needed for production deployment. PyTorch provides a Python package for high-level features like tensor computation (like NumPy) with strong GPU acceleration and TorchScript for an easy transition between eager mode and graph mode. With the latest release of PyTorch, the framework provides graph-based execution, distributed training, mobile deployment, and quantization.”</em> (source: <a href="https://ai.facebook.com/tools/pytorch/">Meta AI page on PyTorch</a>)</p><p>PyTorch is written in Python and C++, which makes it easy to use and learn as well as efficient to run. It supports a wide range of hardware platforms, including (server and mobile) CPUs, GPUs, and TPUs.</p><h4>Why is PyTorch a good choice for image segmentation?</h4><p>PyTorch is a popular choice for deep learning research and development, as it provides a flexible and powerful environment for creating and training neural networks. It is a great choice of framework for implementing deep learning-based image segmentation due to the following features:</p><ul><li><strong>Flexibility</strong>: PyTorch is a flexible framework that allows you to create and train neural networks in a variety of ways. You can use pre-trained models, or you can create your own from scratch very easily</li><li><strong>Backend support</strong>: PyTorch supports multiple backends such as GPU/TPU hardware</li><li><strong>Domain libraries</strong>: PyTorch has a rich set of domain libraries that make working with specific data verticals very easy. For example, for vision (image/video) related AI, PyTorch provides a library called <a href="https://pytorch.org/vision/main/index.html">torchvision</a> that we’ll use extensively throughout this series</li><li><strong>Ease of use and community adoption</strong>: PyTorch is an easy-to-use framework that is well-documented and has a large <a href="https://dev-discuss.pytorch.org/">community</a> of users and developers. Many researchers use PyTorch for their experiments, and the results in their published papers have an implementation of the model in PyTorch freely available</li></ul><h3>Choice of dataset</h3><p>We’re going to use the <a href="https://www.robots.ox.ac.uk/~vgg/data/pets/">Oxford IIIT Pet dataset</a> (licensed under CC BY-SA 4.0) for class segmentation. This dataset has 3680 images in the training set, and each image has a segmentation trimap associated with it. The trimap is one of 3 pixels classes:</p><ol><li>Pet</li><li>Background</li><li>Border</li></ol><p>We choose this dataset since it is sufficiently diverse to provide us with a non-trivial class segmentation task. Additionally, it’s not so complex that we end up spending time on things like dealing with class imbalance, etc… and lose track of the main problem we want to learn about and solve; namely class segmentation.</p><p>Other popular datasets used for image segmentation tasks include:</p><ol><li><a href="http://host.robots.ox.ac.uk/pascal/VOC/">Pascal VOC (visual object classes)</a></li><li><a href="https://cocodataset.org/#home">MS Coco</a></li><li><a href="https://www.cityscapes-dataset.com/examples/">Cityscapes</a></li></ol><h3>Efficient image segmentation with PyTorch</h3><p>In this series, we’ll train multiple models for class segmentation from scratch. There are many considerations to account for when building and training a model from scratch. Below, we will look at some of the key decisions that you need to make when doing so.</p><h4>Choosing the right model for your task</h4><p>There are many factors to consider when choosing the right deep-learning model for image segmentation. Some of the most important factors include:</p><ul><li><strong>The type of image segmentation task</strong>: There are two main types of image segmentation tasks: class (semantic) segmentation and object (instance) segmentation. Since we’re focusing on the simpler class segmentation problem, we shall consider modeling our problem accordingly.</li><li><strong>Size and complexity of the dataset</strong>: The size and complexity of the dataset will affect the complexity of the model that we need to use. For example, if we are working with images with a small spatial dimension, we may use a simpler (or shallower) model, such as a fully convolutional network (FCN). If we are working with a large and complex dataset, we may use a more complex (or deeper) model such as a U-Net.</li><li><strong>Availability of pre-trained models</strong>: There are many pre-trained models available for image segmentation. These models can be used as a starting point for our own model or we can use them directly. However, if we use a pre-trained model, we may be constrained by the spatial dimensions of the input image to the model. In this series, we shall focus on training a model from scratch.</li><li><strong>Computational resources available</strong>: Deep learning models can be computationally expensive to train. If we have limited computational resources, we may need to choose simpler models or more efficient model architectures.</li></ul><p>In this series, we are going to work with the Oxford IIIT Pet dataset since it’s big enough for us to be able to train a medium size model and require the use of a GPU. We would highly recommend creating an account on <a href="https://www.kaggle.com/">kaggle.com</a> or use <a href="https://colab.research.google.com/">Google Colab</a>’s free GPU for running the notebooks and code referenced in this series.</p><h4>Model architectures</h4><p>Here are some of the most popular deep learning model architectures for image segmentation:</p><ul><li><a href="https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/"><strong>U-Net</strong></a>: The U-Net is a convolutional neural network that is commonly used for image segmentation tasks. It uses skip connections, which can help train the network faster and result in better overall accuracy. If you have to choose, U-Net is always an <strong>excellent default choice</strong>!</li><li><a href="https://towardsdatascience.com/review-fcn-semantic-segmentation-eb8c9b50d2d1"><strong>FCN</strong></a>: The Fully Convolutional Network (FCN) is a fully convolutional network, but it is <a href="https://stackoverflow.com/questions/50239795/intuition-behind-u-net-vs-fcn-for-semantic-segmentation">not as deep as the U-Net</a>. The lack of depth is mainly due to the fact that at higher network depths, the accuracy drops. This makes it faster to train, but it may not be as accurate as the U-Net.</li><li><a href="https://arxiv.org/abs/1511.00561"><strong>SegNet</strong></a>: SegNet is a popular model architecture similar to U-Net, and uses lesser activation memory than U-Net. We shall use SegNet in this series.</li><li><a href="https://arxiv.org/abs/2010.11929"><strong>Vision Transformer (ViT)</strong></a>: Vision Transformers have recently gained popularity due to their simple structure and applicability of the attention mechanism to text, vision, and other domains. Vision Transformers can be more efficient (compared to CNNs) for both training and inference, but historically have needed more data to train compared to convolutional neural networks. We shall also use ViT in this series.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lBODM2MmQwVjwro2" /><figcaption>Figure 4: The U-Net model architecture. Source: <a href="https://lmb.informatik.uni-freiburg.de/people/ronneber/u-net/">The University of Freiburg, original author(s) of U-Net</a>.</figcaption></figure><p>These are just a few of the many deep learning models that can be used for image segmentation. The best model for your specific task will depend on the factors mentioned earlier, on the specific task, and your own experiments.</p><h4>Choosing the right loss function</h4><p>The choice of loss function for image segmentation tasks is an important one, as it can have a significant impact on the performance of the model. There are many different loss functions available, each with its own advantages and disadvantages. The most popular loss functions for image segmentation are:</p><ul><li><a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html"><strong>Cross-entropy loss</strong></a>: Cross-entropy loss is a measure of the difference between the predicted probability distribution and the ground truth probability distribution</li><li><strong>IoU loss</strong>: IoU loss measures the amount of overlap between the predicted mask and ground-truth mask per class. IoU loss penalizes cases where either the prediction or recall would suffer. IoU as defined is not differentiable, so we need to slightly tweak it to use it as a loss function</li><li><a href="https://torchmetrics.readthedocs.io/en/stable/classification/dice.html"><strong>Dice loss</strong></a>: Dice loss is also a measure of the overlap between the predicted mask and the ground truth mask.</li><li><a href="https://arxiv.org/abs/1706.05721"><strong>Tversky loss</strong></a>: Tversky loss is proposed as a robust loss function that can be used to handle imbalanced datasets.</li><li><a href="https://pytorch.org/vision/main/generated/torchvision.ops.sigmoid_focal_loss.html#torchvision.ops.sigmoid_focal_loss"><strong>Focal loss</strong></a>: Focal loss is designed to focus on hard examples, which are examples that are difficult to classify. This can be helpful for improving the performance of the model on challenging datasets.</li></ul><p>The best loss function for a particular task will depend on the specific requirements of the task. For example, if accuracy is more important, then IoU loss or Dice loss may be better choices. If the task is imbalanced, then Tversky loss or Focal loss may be good choices. The specific loss function used may impact the rate of convergence of your model when training it.</p><p>The loss function is a hyperparameter of your model, and using a different loss based on the results we see can allow us to reduce the loss faster and improve the model’s accuracy.</p><p><strong>Default</strong>: In this series, we shall use cross entropy loss, since it’s always a good <em>default</em> to choose when the results are not known.</p><p>You can use the following resources to learn more about loss functions.</p><ol><li><a href="https://neptune.ai/blog/pytorch-loss-functions">PyTorch Loss Functions: The Ultimate Guide</a></li><li><a href="https://pytorch.org/vision/main/ops.html#losses">Torchvision — Losses</a></li><li><a href="https://torchmetrics.readthedocs.io/en/stable/pages/overview.html#metrics-and-differentiability">Torchmetrics</a></li></ol><p>Let’s take a detailed look at the IoU Loss we define below as a robust alternative to the Cross Entropy Loss for segmentation tasks.</p><h4>The Custom IoU Loss</h4><p><a href="https://towardsdatascience.com/intersection-over-union-iou-calculation-for-evaluating-an-image-segmentation-model-8b22e2e84686">IoU</a> is defined as intersection over union. For image segmentation tasks, we can compute this by computing (for each class), the intersection of pixels in that class as predicted by the model and in the ground truth segmentation mask.</p><p>For example, if we have 2 classes:</p><ol><li>Background</li><li>Person</li></ol><p>Then we can determine which pixels were classified as a person, and compare that with the ground truth pixels for a person, and compute the IoU for the person class. Similarly, we can compute the IoU for the background class.</p><p>Once we have these class-specific IoU metrics, we can choose to average them unweighted or weigh them before averaging them to account for any sort of class imbalance as we saw in the example earlier.</p><p>The IoU metric as defined requires us to compute hard labels for each metric. This requires the use of the argmax() function, which isn’t differentiable, so we can’t use this metric as a loss function. Hence, instead of using hard labels, we apply softmax() and use the predicted probabilities as soft labels to compute the IoU metric. This results in a differentiable metric that we can then compute the loss from. Hence, sometimes, the IoU metric is also known as the soft-IoU-metric when used in the context of a loss function.</p><p>If we have a metric (M) that takes values between 0.0 and 1.0, we can compute the loss (L) as:</p><p><em>L = 1 — M</em></p><p>However, here’s another trick one can use to convert a metric into a loss if your metric has the value between 0.0 and 1.0. Compute:</p><p><em>L = -log(M)</em></p><p>I.e. compute the negative log of the metric. This is meaningfully different from the previous formulation, and you can read about it <a href="https://towardsdatascience.com/why-we-care-about-the-log-loss-50c00c8e777c">here</a> and <a href="https://towardsdatascience.com/intuition-behind-log-loss-score-4e0c9979680a">here</a>. Basically, it results in better learning for your model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/0*iBKuiZ5Z1jZPkvt3" /><figcaption>Figure 6: Comparing the loss resulting from 1-P(x) with -log(P(x)). Source: Author(s).</figcaption></figure><p>Using IoU as our loss also brings the loss function closer to capturing what we really care about. There are pros and cons of using an evaluation metric as the loss function. If you’re interested in exploring this space more, you can start with <a href="https://stats.stackexchange.com/questions/577556/why-not-use-evaluation-metrics-as-the-loss-function">this discussion on stackexchange</a>.</p><h4>Data Augmentation</h4><p>To train your model efficiently and effectively for good accuracy, one needs to be mindful of the amount and kind of training data used to train the model. The choice of training data used will significantly impact the final model’s accuracy, so if there’s one thing you wish to take away from this article series then this should be it!</p><p>Typically, we’d split our data into 3 parts with the parts being roughly in the proportions mentioned below.</p><ol><li>Training (80%)</li><li>Validation (10%)</li><li>Test (10%)</li></ol><p>You’d train your model on the training set, evaluate accuracy on the validation set, and repeat the process till you’re happy with the reported metrics. Only then would you evaluate the model on the test set, and then report the numbers. This is done to prevent any sort of bias from creeping into your model’s architecture and hyperparameters used during training and evaluation. In general, the more you tweak your setup based on the outcomes you see with the test data, the less reliable your results will get. Hence, we must limit our decision making to only the results we see on the training and validation datasets.</p><p>In this series, we shall not use a test dataset. Instead, we’ll use our test dataset as the validation dataset, and apply <a href="https://en.wikipedia.org/wiki/Data_augmentation">data augmentation</a> on the test dataset so that we’re always validating our models on data that’s slightly different. This kind of prevents us from overfitting our decisions on the validation dataset. This is a bit of a hack, and we’re doing this just for expediency and as a short-cut. For production model development, you should try to stick with the standard recipe mentioned above.</p><p>The dataset we’re going to use in this series has 3680 images in the training set. While this may seem like a large number of images, we want to make sure that our model doesn’t overfit on these images since we’ll be training the model for numerous epochs.</p><p>In a single training epoch, we train the model on the entire training dataset, and we’d typically train models in production for 60 or more epochs. In this series, we shall train the model only for 20 epochs for faster iteration times. To <strong>prevent overfitting</strong>, we’ll employ a technique called <a href="https://efficientdlbook.com/#table-of-contents">data augmentation</a> that is used to generate new input data from existing input data. The basic idea behind data augmentation for image inputs is that if you change the image slightly, it feels like a new image to the model, but one can reason about whether the expected outputs would be the same. Here are some examples of data augmentations that we’ll apply in this series.</p><ol><li><a href="https://pytorch.org/vision/main/generated/torchvision.transforms.RandomHorizontalFlip.html">Random horizontal flip</a></li><li><a href="https://pytorch.org/vision/main/generated/torchvision.transforms.ColorJitter.html">Random Color jitter</a></li></ol><p>While we’re going to use the Torchvision library for applying the data augmentations above, we’d encourage you to evaluate the Albumentations data augmentation library for vision tasks as well. Both libraries have a rich set of transformations available for use with image data. We personally continue to use Torchvision simply because it’s what we started with. <a href="https://albumentations.ai/">Albumentations</a> supports richer primitives for data augmentation that can make changes to the input image as well as the ground truth labels or masks at the same time. For example, if you were to resize or flip an image, you’d need to make the same change to the ground truth segmentation mask. Albumentations can do this for you out of the box.</p><p>Broadly speaking, both libraries support transformations that are applied to the image either at the pixel-level or change the spatial dimensions of the image. The pixel-level transforms are called color transforms by torchvision, and the spatial transforms are called Geometric transforms by torchvision.</p><p>Below, we shall see some examples of both pixel-level as well as geometric transforms applied by the Torchvision and Albumentations libraries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*mnvOOM13PLafo0og" /><figcaption>Figure 7: Examples of pixel level data augmentations applied to images using Albumentations. Source: <a href="https://albumentations.ai/">Albumentations</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/0*PVpt9Vp9fK7hOQxW" /><figcaption>Figure 8: Examples of data augmentations applied to images using Torchvision transforms. Source: Author(s) (<a href="https://www.kaggle.com/code/dhruv4930/starter-for-oxford-iiit-pet-using-torchvision/notebook?scriptVersionId=129839935">notebook</a>)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/0*q0HtWciPr8HeAYd6" /><figcaption>Figure 9: Examples of spatial level transforms applied using Albumentations. Source: Author(s) (<a href="https://www.kaggle.com/code/dhruv4930/starter-for-oxford-iiit-pet-using-torchvision/notebook?scriptVersionId=129839935">notebook</a>)</figcaption></figure><h4>Evaluating your model’s performance</h4><p>When evaluating your model’s performance, you’d want to know how it performs on a metric that is representative of the quality of the model’s performance on real world data. For example, for the image segmentation task, we’d want to know how accurately a model is able to predict the correct class for a pixel. Hence, we say that <a href="https://torchmetrics.readthedocs.io/en/stable/classification/accuracy.html">Pixel Accuracy</a> is the validation metric for this model.</p><p>You could use your evaluation metric as the loss function (why not optimize what you really care about!) except that this <a href="https://jonathan-sands.com/metric/loss/2021/05/13/Metric-vs-Loss.html#what-is-a-loss-function">may not always be possible</a>.</p><p>In addition to <a href="https://torchmetrics.readthedocs.io/en/stable/classification/accuracy.html">Accuracy</a>, we’ll also track the IoU metric (also called <a href="https://torchmetrics.readthedocs.io/en/stable/classification/jaccard_index.html">Jaccard Index</a>), and the Custom IoU metric we defined above.</p><p>To read more about various accuracy metrics applicable to image segmentation tasks, please see:</p><ul><li><a href="https://www.kaggle.com/code/yassinealouini/all-the-segmentation-metrics">All the segmentation metrics — Kaggle</a></li><li><a href="https://towardsdatascience.com/how-accurate-is-image-segmentation-dd448f896388">How to evaluate image segmentation models</a></li><li><a href="https://towardsdatascience.com/evaluating-image-segmentation-models-1e9bb89a001b">Evaluating image segmentation models</a></li></ul><h4>The downside of using pixel accuracy as a performance metric</h4><p>While the accuracy metric may be a good default choice to measure the performance of image segmentation tasks, it does have its own drawbacks, which may be significant based on your specific situation.</p><p>For example, consider an image segmentation task to identify a person’s eyes in a picture, and mark those pixels accordingly. The model will hence classify each pixels as either one of:</p><ol><li>Background</li><li>Eye</li></ol><p>Assume that there’s just 1 person in each image, and 98% of the pixels don’t correspond to an eye. In this case, the model can simply learn to predict every pixel as being a background pixel and achieve 98% pixel accuracy on the segmentation task. Wow!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/1*GFVF_jA6ePrTMWrT0nUaGQ.png" /><figcaption>Figure 10: An image of a person’s face and the corresponding segmentation mask for their eyes. You can see that the eyes occupy a very small fraction of the overall image. Source: Adapted from <a href="https://unsplash.com/photos/iFgRcqHznqg">Unsplash</a></figcaption></figure><p>In such cases, using the IoU or Dice metric may be a much better idea, since IoU would capture how much of the prediction was correct, and wouldn’t necessarily be biased by the region that each class or category occupies in the original image. You could even consider using the IoU or Dice coefficient per class as a metric. This may better capture the performance of your model for the task at hand.</p><p>When considering pixel accuracy alone, the <a href="https://developers.google.com/machine-learning/crash-course/classification/precision-and-recall">precision and recall</a> of the object we’re looking to compute the segmentation mask for (eyes in the example above) can capture the details we’re looking for.</p><p>Now that we have covered a large part of the theoretical underpinnings of image segmentation, let’s take a detour into considerations related to inference and deployment of image segmentation for real-world workloads.</p><h4>Model size and inference latency</h4><p>Last but not least, we’d want to ensure that our model has a reasonable number of parameters but not too many, since we want a small and efficient model. We shall look into this aspect in greater detail in a future post related to reducing model size using <a href="https://efficientdlbook.com/#table-of-contents">efficient model architectures</a>.</p><p>As far as inference latency is concerned, what matters is the number of mathematical operations (mult-adds) our model executes. Both the model size and mult-adds can be displayed using the <a href="https://pypi.org/project/torchinfo/">torchinfo</a> package. While mult-adds is a great proxy for determining the model’s latency, there can be a large variation in latency across various backends. The only real way to determine the performance of your model on a specific backend or device is to profile and benchmark it on that specific device with the set of inputs you expect to see in production settings.</p><pre>from torchinfo import summary<br>model = nn.Linear(1000, 500)<br>summary(<br>  model,<br>  input_size=(1, 1000),<br>  col_names=[&quot;kernel_size&quot;, &quot;output_size&quot;, &quot;num_params&quot;, &quot;mult_adds&quot;],<br>  col_width=15,<br>)</pre><p>Output:</p><pre>====================================================================================================<br>Layer (type:depth-idx)                   Kernel Shape    Output Shape    Param #         Mult-Adds<br>====================================================================================================<br>Linear                                   --              [1, 500]        500,500         500,500<br>====================================================================================================<br>Total params: 500,500<br>Trainable params: 500,500<br>Non-trainable params: 0<br>Total mult-adds (M): 0.50<br>====================================================================================================<br>Input size (MB): 0.00<br>Forward/backward pass size (MB): 0.00<br>Params size (MB): 2.00<br>Estimated Total Size (MB): 2.01<br>====================================================================================================</pre><h3>Further Reading</h3><p>The following articles provide additional information regarding the basics of image segmentation. If you’re the kind of person who likes reading different perspectives on the same subject, please consider reading them.</p><ol><li><a href="https://encord.com/blog/image-segmentation-for-computer-vision-best-practice-guide/">Guide to Image Segmentation in Computer Vision: Best Practices</a></li><li><a href="https://www.v7labs.com/blog/image-segmentation-guide#h5">An Introduction to Image Segmentation: Deep Learning vs. Traditional [+Examples]</a></li><li><a href="https://datagen.tech/guides/image-annotation/image-segmentation/">Image Segmentation: The Basics and 5 Key Techniques</a></li></ol><p>If you wish to get your hands dirty with the Oxford IIIT Pet dataset and use torchvision and Albumentations to perform image augmentations, we have provided a <a href="https://www.kaggle.com/code/dhruv4930/starter-for-oxford-iiit-pet-using-torchvision/notebook?scriptVersionId=129839935">starter notebook on Kaggle</a> that you can clone and play around with. Many of the images in this article were generated by that notebook!</p><h3>Article recap</h3><p>Here’s a quick recap of what we discussed so far.</p><ul><li>Image segmentation is a technique to partition an image into multiple segments (source: <a href="https://en.wikipedia.org/wiki/Image_segmentation">Wikipedia</a>)</li><li>There are two main types of image segmentation tasks: class (semantic) segmentation and object (instance) segmentation. Class segmentation assigns each pixel in an image to a semantic class. Object segmentation identifies each individual object in an image and assigns a mask to each unique object</li><li>We shall use PyTorch as the deep learning framework and the Oxford IIIT Pet dataset in this series of efficient image segmentation</li><li>There are many factors to consider when choosing the right deep learning model for image segmentation, including (but not limited to) the type of image segmentation task, the size and complexity of the dataset, the availability of pre-trained models, and the computational resources available. Some of the most popular deep learning model architectures for image segmentation include U-Net, FCN, SegNet, and Vision Transformer (ViT)</li><li>The choice of loss function for image segmentation tasks is an important one, as it can have a significant impact on the performance of the model and on training efficiency. For image segmentation tasks, we can use cross-entropy loss, IoU Loss, Dice loss, or Focal loss (among others)</li><li>Data augmentation is a valuable technique that is used to prevent overfitting as well as deal with insufficient training data</li><li>Evaluating your model’s performance is important for the task at hand and one must choose this metric carefully</li><li>Model size and inference latency are vital metrics to consider when developing a model, especially if you intend it to be used in real-time applications such as face segmentation or background noise removal</li></ul><p>In the <a href="https://medium.com/p/bed68cadd7c7/">next post</a>, we shall look at a Convolutional Neural Network (CNN) built from scratch using PyTorch to perform image segmentation on the Oxford IIIT Pet dataset.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=89e8297a0923" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/efficient-image-segmentation-using-pytorch-part-1-89e8297a0923">Efficient Image Segmentation Using PyTorch: Part 1</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NT-Xent (Normalized Temperature-Scaled Cross-Entropy) Loss Explained and Implemented in PyTorch]]></title>
            <link>https://medium.com/data-science/nt-xent-normalized-temperature-scaled-cross-entropy-loss-explained-and-implemented-in-pytorch-cc081f69848?source=rss-63f5d5495279------2</link>
            <guid isPermaLink="false">https://medium.com/p/cc081f69848</guid>
            <category><![CDATA[loss-function]]></category>
            <category><![CDATA[self-supervised-learning]]></category>
            <category><![CDATA[binary-cross-entropy]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[cross-entropy]]></category>
            <dc:creator><![CDATA[Dhruv Matani]]></dc:creator>
            <pubDate>Tue, 13 Jun 2023 18:46:41 GMT</pubDate>
            <atom:updated>2023-06-13T18:46:41.382Z</atom:updated>
            <content:encoded><![CDATA[<h4>An intuitive explanation of the NT-Xent loss with a step-by-step explanation of the operation and our implementation in PyTorch</h4><p>Co-authored with <a href="https://medium.com/u/1e659a80cffd">Naresh Singh</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/867/1*ocLhrfF9CpucKneGbiVfxw.png" /><figcaption>Formula for NT-Xent loss. Source: <a href="https://paperswithcode.com/method/nt-xent">Papers with code</a> (CC-BY-SA)</figcaption></figure><h3>Introduction</h3><p>Recent advances in <a href="https://neptune.ai/blog/self-supervised-learning">self-supervised learning</a> and <a href="https://towardsdatascience.com/understanding-contrastive-learning-d5b19fd96607">contrastive learning</a> have excited researchers and practitioners in Machine Learning (ML) to explore this space with renewed interest.</p><p>In particular, the <a href="https://arxiv.org/pdf/2002.05709.pdf">SimCLR</a> paper that presents a simple framework for contrastive learning of visual representations has gained a lot of attention in the self-supervised and contrastive learning space.</p><p>The central idea behind the paper is very simple — allow the model to learn if a pair of images were derived from the same or different initial image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*JTj6w4ybgwM8UT9VvUEjgA.png" /><figcaption>Figure 1: The high-level idea behind SimCLR. Source: <a href="https://arxiv.org/pdf/2002.05709.pdf">SimCLR paper</a></figcaption></figure><p>The SimCLR approach encodes each input image <strong><em>i</em></strong> as a feature vector <strong><em>zi</em></strong>. There are 2 cases to consider:</p><ol><li><strong>Positive Pairs</strong>: The same image is augmented using a different set of augmentations, and the resulting feature vectors <strong><em>zi</em></strong> and <strong><em>zj</em></strong> are compared. These feature vectors are forced to be similar by the loss function.</li><li><strong>Negative Pairs</strong>: Different images are augmented using a different set of augmentations, and the resulting feature vectors <strong><em>zi</em></strong> and <strong><em>zk</em></strong> are compared. These feature vectors are forced to be dissimilar by the loss function.</li></ol><p>The rest of this article will focus on explaining and understanding this loss function, and its efficient implementation using PyTorch.</p><h3>The NT-Xent Loss</h3><p>At a high level, the contrastive learning model is fed 2N images, originating from N underlying images. Each of the N underlying images is augmented using a random set of image augmentations to produce 2 augmented images. This is how we end up with 2N images in a single train batch fed to the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/969/0*6mw1wxsJ-HCmL7hK" /><figcaption>Figure 2: A batch of 6 images in a single training batch for contrastive learning. The number below each image is the index of that image in the input batch when fed into a contrastive learning model. Image Source: <a href="https://www.robots.ox.ac.uk/~vgg/data/pets/">Oxford Visual Geometry Group</a> (CC-SA).</figcaption></figure><p>In the following sections, we will dive deep into the following aspects of the NT-Xent loss.</p><ol><li><a href="#8cb8">The effect of temperature on SoftMax and Sigmoid</a></li><li><a href="#40d2">A simple and intuitive interpretation of the NT-Xent loss</a></li><li><a href="#29d4">A step-by-step implementation of NT-Xent in PyTorch</a></li><li><a href="#f004">Motivating the need for a multi-label loss function (NT-BXent)</a></li><li><a href="#242d">A step-by-step implementation of NT-BXent in PyTorch</a></li></ol><p>All the code for steps 2–5 can be found in <a href="https://github.com/dhruvbird/ml-notebooks/blob/main/nt-xent-loss/NT-Xent%20Loss.ipynb">this notebook</a>. The code for step-1 can be found in <a href="https://github.com/dhruvbird/ml-notebooks/blob/main/nt-xent-loss/SoftMax%20and%20Sigmoid%20with%20temperature.ipynb">this notebook</a>.</p><h4>The effect of temperature on SoftMax and Sigmoid</h4><p>To understand all the moving parts of the contrastive loss function we’ll be studying in this article, we need to first understand the effect of temperature on the SoftMax and Sigmoid activation functions.</p><p>Typically, temperature scaling is applied to the input to SoftMax or Sigmoid to either smooth out or accentuate the output of those activation functions. The input logits are divided by the temperature before passing into the activation functions. You can find all the code for this section in <a href="https://github.com/dhruvbird/ml-notebooks/blob/main/nt-xent-loss/SoftMax%20and%20Sigmoid%20with%20temperature.ipynb">this notebook</a>.</p><p><strong>SoftMax</strong>: For SoftMax, a high temperature reduces the variance in the output distribution which results in softening of the labels. A low temperature increases the variance in the output distribution and makes the maximum value stand out over the other values. See the charts below for the effect of temperature on SoftMax when fed with the input tensor [0.1081, 0.4376, 0.7697, 0.1929, 0.3626, 2.8451].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bXxtQ57ZoLNduhom2Fp2ew.png" /><figcaption>Figure 3: Effect of temperature on SoftMax. Source: Author(s)</figcaption></figure><p><strong>Sigmoid</strong>: For Sigmoid, a high-temperature results in an output distribution that is pulled towards 0.0, whereas a low temperature stretches the inputs to higher values, stretching the outputs to be closer to either 0.0 or 1.0 depending on the unsigned magnitude of the input.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9qshoOkmDvRw-yqzvGwkSw.png" /><figcaption>Figure 4: Effect of temperature on Sigmoid. Source: Author(s)</figcaption></figure><p>Now that we understand the effect of various temperature values on the SoftMax and Sigmoid functions, let’s see how this applies to our understanding of the NT-Xent loss.</p><h4>Interpreting the NT-Xent loss</h4><p>The NT-Xent loss is understood by understanding the individual terms in the name of this loss.</p><ol><li>Normalized: Cosine similarity produces a normalized score in the range [-1.0 to +1.0]</li><li>Temperature-scaled: The all-pairs cosine similarity is scaled by a temperature before computing the cross-entropy loss</li><li>Cross-entropy loss: The underlying loss is a multi-class (single-label) cross-entropy loss</li></ol><p>As mentioned above, we assume that for a batch of size 2N, the feature vectors at the following indices represent positive pairs (0, 1), (2, 3), (4, 5), (6, 7), … and the rest of the combinations represent negative pairs. This is an important factor to keep in mind throughout the interpretation of the NT-Xent loss as it relates to SimCLR.</p><p>Now that we understand what the terms mean in the context of the NT-Xent loss, let’s take a look at the mechanical steps needed to compute the NT-Xent loss on a batch of feature vectors.</p><ol><li>The all-pairs Cosine Similarity score is computed for each of the 2N vectors produced by the SimCLR model. This results in (2N)² similarity scores represented as a 2N x 2N matrix</li><li>Comparison results between the same value (i, i) are discarded (since a distribution is perfectly similar to itself and can’t possibly allow the model to learn anything useful)</li><li>Each value (cosine similarity) is scaled by a temperature parameter 𝜏 (which is a hyper-parameter)</li><li>Cross-entropy loss is applied to each row of the resulting matrix above. The following paragraph explains more in detail</li><li>Typically, the mean of these losses (one loss per element in a batch) is used for backpropagation</li></ol><p>The way that the cross-entropy loss is used here is semantically slightly different from how it’s used in standard classification tasks. In classification tasks, a final “classification head” is trained to produce a one-hot-probability vector for each input, and we compute the cross-entropy loss on that one-hot-probability vector since we’re effectively computing the difference between 2 distributions. <a href="https://www.youtube.com/watch?v=Pwgpl9mKars">This video</a> explains the concept of cross-entropy loss beautifully. In the NT-Xent loss, there isn’t a 1:1 correspondence between a trainable layer and the output distribution. Instead, a feature vector is computed for each input, and we then compute the cosine similarity between every pair of feature vectors. The trick here is that since each image is similar to exactly 1 other image in the input batch (positive pair) (if we ignore the similarity of a feature vector with itself), we can consider this to be a classification-like setting where the probability distribution of the similarity probability between images represents a classification task where one of them will be close to 1.0 and the rest will be close to 0.0.</p><p>Now that we have a solid overall understanding of the NT-Xent loss, we should be in great shape to implement these ideas in PyTorch. Let’s get going!</p><h4>Implementation of NT-Xent loss in PyTorch</h4><p>All the code in this section can be found in <a href="https://github.com/dhruvbird/ml-notebooks/blob/main/nt-xent-loss/NT-Xent%20Loss.ipynb">this notebook</a>.</p><p><strong>Code Reuse</strong>: Many <a href="https://github.com/KevinMusgrave/pytorch-metric-learning/blob/master/src/pytorch_metric_learning/losses/ntxent_loss.py">implementations</a> of the NT-Xent loss seen online implement all the operations from scratch. Furthermore, some of them implement the loss function inefficiently, preferring to use <a href="https://stackoverflow.com/questions/62793043/tensorflow-implementation-of-nt-xent-contrastive-loss-function">for loops instead of GPU parallelism</a>. Instead, we will use a different approach. We’ll implement this loss in terms of the standard cross-entropy loss that PyTorch already provides. To do this, we need to massage the predictions and ground-truth labels in a format that cross_entropy can accept. Let’s see how to do this below.</p><p><strong>Predictions Tensor</strong>: First, we need to create a PyTorch tensor that will represent the output from our contrastive learning model. Let’s assume that our batch size is 8 (2N=8), and our feature vectors have 2 dimensions (2 values). We’ll call our input variable <em>“x”</em>.</p><pre>x = torch.randn(8, 2)</pre><p><strong>Cosine Similarity</strong>: Next, we’ll compute the all-pairs cosine similarity between every feature vector in this batch and store the result in the variable named <em>“xcs”</em>. If the line below seems confusing, please read the details on <a href="https://medium.com/@dhruvbird/all-pairs-cosine-similarity-in-pytorch-867e722c8572">this page</a>. This is the “normalize” step.</p><pre>xcs = F.cosine_similarity(x[None,:,:], x[:,None,:], dim=-1)</pre><p>As mentioned above, we need to ignore the self-similarity score of every feature vector since it doesn’t contribute to the model’s learning and will be an unnecessary nuisance later on when we want to compute the cross-entropy loss. For this purpose, we’ll define a variable <em>“eye”</em> which is a matrix with the elements on the principal diagonal having a value of 1.0 and the rest being 0.0. We can create such a matrix using the following command.</p><pre>eye = torch.eye(8)</pre><p>Now let’s convert this into a boolean matrix so that we can index into the <em>“xcs”</em> variable using this mask matrix.</p><pre>eye = eye.bool()</pre><p>Let’s clone the tensor <em>“xcs”</em> into a tensor named <em>“y”</em> so that we can reference the “xcs” tensor later.</p><pre>y = xcs.clone()</pre><p>Now, we will set the values along the principal diagonal of the all-pairs cosine similarity matrix to <em>-inf</em> so that when we compute the softmax on each row, this value will contribute nothing.</p><pre>y[eye] = float(&quot;-inf&quot;)</pre><p>The tensor <em>“y”</em> scaled by a temperature parameter will be one of the inputs (predictions) to the<a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.cross_entropy.html#torch.nn.functional.cross_entropy"> cross-entropy loss API in PyTorch</a>. Next, we need to compute the ground-truth labels (target) that we need to feed to the cross-entropy loss API.</p><p><strong>Ground Truth labels (Target tensor)</strong>: For the example we are using (2N=8), this is what the ground-truth tensor should look like.</p><blockquote>tensor([1, 0, 3, 2, 5, 4, 7, 6])</blockquote><p>That’s because the following index pairs in the tensor <em>“y”</em> contain positive pairs.</p><blockquote>(0, 1), (1, 0)</blockquote><blockquote>(2, 3), (3, 2)</blockquote><blockquote>(4, 5), (5, 4)</blockquote><blockquote>(6, 7), (7, 6)</blockquote><p>To interpret the index pairs above, we look at a single example. The pair (4, 5) means that column 5 at row 4 is supposed to be set to 1.0 (positive pair), which is what the tensor above is also saying. Great!</p><p>To create the tensor above, we can use the following PyTorch code, which stores the ground-truth labels in the variable <em>“target”</em>.</p><pre>target = torch.arange(8)<br>target[0::2] += 1<br>target[1::2] -= 1</pre><p><strong>cross-entropy Loss</strong>: We have all the ingredients we need to compute our loss! The only thing that remains to be done is to call the cross_entropy API in PyTorch.</p><pre>loss = F.cross_entropy(y / temperature, target, reduction=&quot;mean&quot;)</pre><p>The variable “loss” now contains the computed NT-Xent loss. Let’s wrap all the code in a single python function below.</p><pre>def nt_xent_loss(x, temperature):<br>  assert len(x.size()) == 2<br><br>  # Cosine similarity<br>  xcs = F.cosine_similarity(x[None,:,:], x[:,None,:], dim=-1)<br>  xcs[torch.eye(x.size(0)).bool()] = float(&quot;-inf&quot;)<br><br>  # Ground truth labels<br>  target = torch.arange(8)<br>  target[0::2] += 1<br>  target[1::2] -= 1<br><br>  # Standard cross-entropy loss<br>  return F.cross_entropy(xcs / temperature, target, reduction=&quot;mean&quot;)</pre><p>The code above works as long as each feature vector has exactly one positive pair in the batch when training our contrastive learning model. Let’s take a look at how to handle multiple positive pairs in a contrastive learning task.</p><h4>A multi-label loss for contrastive learning: NT-BXent</h4><p>In the SimCLR paper, every image <strong><em>i</em></strong> has exactly 1 similar pair at index <strong><em>j</em></strong>. This makes cross-entropy loss a perfect choice for the task since it resembles a multi-class problem. Instead, if we have M &gt; 2 augmentations of the same image fed into the contrastive learning model’s single training batch, then each batch would have image M-1 similar pairs for image <strong><em>i</em></strong>. This task would resemble a multi-label problem.</p><p>The obvious choice would be to replace <em>cross-entropy loss</em> with <em>binary cross-entropy loss</em>. Hence the name NT-BXent loss, which stands for Normalized Temperature-scaled Binary cross-entropy Loss.</p><p>The formulation below shows the loss <strong><em>Li</em></strong> for the element <strong><em>i</em></strong>. The σ in the formula below stands for the <a href="https://en.wikipedia.org/wiki/Sigmoid_function">Sigmoid function</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/958/0*eh_WA6AvteTizYns" /><figcaption>Figure 5: Formulation for the NT-BXent loss. Image source: Author(s) of this article</figcaption></figure><p>To avoid the class imbalance problem, we weigh the positive and negative pairs by the inverse of the number of positive and negative pairs in our mini-batch. The final loss in the mini-batch used for backpropagation will be the mean of the losses of each sample in our mini-batch.</p><p>Next, let’s focus our attention on our implementation of the NT-BXent loss in PyTorch.</p><h4>Implementation of NT-BXent loss in PyTorch</h4><p>All the code in this section can be found in <a href="https://github.com/dhruvbird/ml-notebooks/blob/main/nt-xent-loss/NT-Xent%20Loss.ipynb">this notebook</a>.</p><p><strong>Code Reuse</strong>: Similar to our implementation of the NT-Xent loss, we shall re-use the Binary Cross-entropy (BCE) loss method provided by PyTorch. The setup of our ground-truth labels will be similar to that of a multi-label classification problem where BCE loss is used.</p><p><strong>Predictions Tensor</strong>: We’ll use the same (8, 2) predictions tensor as we used for the implementation of the NT-Xent loss.</p><pre>x = torch.randn(8, 2)</pre><p><strong>Cosine Similarity</strong>: Since the input tensor <strong><em>x</em></strong> is same, the all-pairs cosine similarity tensor <strong><em>xcs</em></strong> will also be the same. Please see <a href="https://medium.com/@dhruvbird/all-pairs-cosine-similarity-in-pytorch-867e722c8572">this page</a> for a detailed explanation of what the line below does.</p><pre>xcs = F.cosine_similarity(x[None,:,:], x[:,None,:], dim=-1)</pre><p>To ensure that the loss from the element at position <strong><em>(i, i)</em></strong> is <strong><em>0</em></strong>, we’ll need to perform some gymnastics to have our <strong><em>xcs</em></strong> tensor contain a value <strong><em>1</em></strong> at every index <strong><em>(i, i)</em></strong> after Sigmoid is applied to it. Since we’ll be using BCE Loss, we will mark the self-similarity score of every feature vector with the value <strong><em>infinity</em></strong> in tensor <strong><em>xcs</em></strong>. That’s because applying the sigmoid function on the <strong><em>xcs</em></strong> tensor, will convert infinity to the value <strong><em>1</em></strong>, and we will set up our ground-truth labels so that every position <strong><em>(i, i)</em></strong> in the ground-truth labels has the value <strong><em>1</em></strong>.</p><p>Let’s create a masking tensor that has the value <strong><em>True</em></strong> along the principal diagonal (<strong><em>xcs</em></strong> has self-similarity scores along the principal diagonal), and <strong><em>False</em></strong> everywhere else.</p><pre>eye = torch.eye(8).bool()</pre><p>Let’s clone the tensor <em>“xcs”</em> into a tensor named <em>“y”</em> so that we can reference the <em>“xcs”</em> tensor later.</p><pre>y = xcs.clone()</pre><p>Now, we will set the values along the principal diagonal of the all-pairs cosine similarity matrix to <em>infinity</em> so that when we compute the sigmoid on each row, we get 1 in these positions.</p><pre>y[eye] = float(&quot;inf&quot;)</pre><p>The tensor <em>“y”</em> scaled by a temperature parameter will be one of the inputs (predictions) to the <a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.binary_cross_entropy.html">BCE loss API</a> in PyTorch. Next, we need to compute the ground-truth labels (target) that we need to feed to the BCE loss API.</p><p><strong>Ground Truth labels (Target tensor)</strong>: We will expect the user to pass to us the pair of all (x, y) index pairs which contain positive examples. This is a departure for what we did for the NT-Xent loss, since the positive pairs were implicit, whereas here, the positive pairs are explicit.</p><p>In addition to the locations provided by the user, we will set all the diagonal elements as positive pairs as explained above. We will use the PyTorch tensor indexing API to pluck out all the elements at those locations and set them to 1, whereas the rest are initialized to 0.</p><pre>target = torch.zeros(8, 8)<br>pos_indices = torch.tensor([<br>  (0, 0), (0, 2), (0, 4),<br>  (1, 4), (1, 6), (1, 1),<br>  (2, 3),<br>  (3, 7),<br>  (4, 3),<br>  (7, 6),<br>])<br># Add indexes of the principal diagonal as positive indexes.<br># This will be useful since we will use the BCELoss in PyTorch,<br># which will expect a value for the elements on the principal<br># diagonal as well.<br>pos_indices = torch.cat([pos_indices, torch.arange(8).reshape(8, 1).expand(-1, 2)], dim=0)<br># Set the values in the target vector to 1.<br>target[pos_indices[:,0], pos_indices[:,1]] = 1</pre><p><strong>Binary cross-entropy (BCE) Loss</strong>: Unlike the NT-Xent loss, we can’t simply call the torch.nn.functional.binary_cross_entropy_function, since we want to weigh the positive and negative loss based on how many positive and negative pairs the element at index i has in the current mini-batch.</p><p>The first step though is to compute the element-wise BCE loss.</p><pre>temperature = 0.1<br>loss = F.binary_cross_entropy((y / temperature).sigmoid(), target, reduction=&quot;none&quot;)</pre><p>We’ll create a binary mask of positive and negative pairs and then create 2 tensors, loss_pos and loss_neg that contain only those elements from the computed loss that correspond to the positive and negative pairs.</p><pre>target_pos = target.bool()<br>target_neg = ~target_pos<br># loss_pos and loss_neg below contain non-zero values only for those elements<br># that are positive pairs and negative pairs respectively.<br>loss_pos = torch.zeros(x.size(0), x.size(0)).masked_scatter(target_pos, loss[target_pos])<br>loss_neg = torch.zeros(x.size(0), x.size(0)).masked_scatter(target_neg, loss[target_neg])</pre><p>Next, we’ll sum up the positive and negative pair loss (separately) corresponding to each element i in our mini-batch.</p><pre># loss_pos and loss_neg now contain the sum of positive and negative pair losses<br># as computed relative to the i&#39;th input.<br>loss_pos = loss_pos.sum(dim=1)<br>loss_neg = loss_neg.sum(dim=1)</pre><p>To perform weighting, we need to track the number of positive and negative pairs corresponding to each element i in our mini-batch. Tensors <em>“num_pos”</em> and <em>“num_neg”</em> will store these values.</p><pre># num_pos and num_neg below contain the number of positive and negative pairs<br># computed relative to the i&#39;th input. In an actual setting, this number should<br># be the same for every input element, but we let it vary here for maximum<br># flexibility.<br>num_pos = target.sum(dim=1)<br>num_neg = target.size(0) - num_pos</pre><p>We have all the ingredients we need to compute our loss! The only thing that we need to do is weigh the positive and negative loss by the number of positive and negative pairs, and then average the loss across the mini-batch.</p><pre>def nt_bxent_loss(x, pos_indices, temperature):<br>    assert len(x.size()) == 2<br><br>    # Add indexes of the principal diagonal elements to pos_indices<br>    pos_indices = torch.cat([<br>        pos_indices,<br>        torch.arange(x.size(0)).reshape(x.size(0), 1).expand(-1, 2),<br>    ], dim=0)<br>    <br>    # Ground truth labels<br>    target = torch.zeros(x.size(0), x.size(0))<br>    target[pos_indices[:,0], pos_indices[:,1]] = 1.0<br><br>    # Cosine similarity<br>    xcs = F.cosine_similarity(x[None,:,:], x[:,None,:], dim=-1)<br>    # Set logit of diagonal element to &quot;inf&quot; signifying complete<br>    # correlation. sigmoid(inf) = 1.0 so this will work out nicely<br>    # when computing the Binary cross-entropy Loss.<br>    xcs[torch.eye(x.size(0)).bool()] = float(&quot;inf&quot;)<br><br>    # Standard binary cross-entropy loss. We use binary_cross_entropy() here and not<br>    # binary_cross_entropy_with_logits() because of<br>    # https://github.com/pytorch/pytorch/issues/102894<br>    # The method *_with_logits() uses the log-sum-exp-trick, which causes inf and -inf values<br>    # to result in a NaN result.<br>    loss = F.binary_cross_entropy((xcs / temperature).sigmoid(), target, reduction=&quot;none&quot;)<br>    <br>    target_pos = target.bool()<br>    target_neg = ~target_pos<br>    <br>    loss_pos = torch.zeros(x.size(0), x.size(0)).masked_scatter(target_pos, loss[target_pos])<br>    loss_neg = torch.zeros(x.size(0), x.size(0)).masked_scatter(target_neg, loss[target_neg])<br>    loss_pos = loss_pos.sum(dim=1)<br>    loss_neg = loss_neg.sum(dim=1)<br>    num_pos = target.sum(dim=1)<br>    num_neg = x.size(0) - num_pos<br><br>    return ((loss_pos / num_pos) + (loss_neg / num_neg)).mean()<br><br>pos_indices = torch.tensor([<br>    (0, 0), (0, 2), (0, 4),<br>    (1, 4), (1, 6), (1, 1),<br>    (2, 3),<br>    (3, 7),<br>    (4, 3),<br>    (7, 6),<br>])<br>for t in (0.01, 0.1, 1.0, 10.0, 20.0):<br>    print(f&quot;Temperature: {t:5.2f}, Loss: {nt_bxent_loss(x, pos_indices, temperature=t)}&quot;)</pre><p>Prints.</p><blockquote>Temperature: 0.01, Loss: 62.898780822753906</blockquote><blockquote>Temperature: 0.10, Loss: 4.851151943206787</blockquote><blockquote>Temperature: 1.00, Loss: 1.0727109909057617</blockquote><blockquote>Temperature: 10.00, Loss: 0.9827173948287964</blockquote><blockquote>Temperature: 20.00, Loss: 0.982099175453186</blockquote><h3>Conclusion</h3><p>Self-supervised learning is an upcoming field in deep learning and allows one to train models on unlabeled data. This technique lets us work around the requirement of labeled data at scale.</p><p>In this article, we learned about loss functions for contrastive learning. The first one, named NT-Xent loss, is used for learning on a single positive pair per input in a mini-batch. We introduced the NT-BXent loss which is used for learning on multiple (&gt; 1) positive pairs per input in a mini-batch. We learned to interpret them intuitively, building on our knowledge of cross-entropy loss and binary cross-entropy loss. Finally, we implemented them both efficiently in PyTorch.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cc081f69848" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/nt-xent-normalized-temperature-scaled-cross-entropy-loss-explained-and-implemented-in-pytorch-cc081f69848">NT-Xent (Normalized Temperature-Scaled Cross-Entropy) Loss Explained and Implemented in PyTorch</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[All Pairs Cosine Similarity in PyTorch]]></title>
            <link>https://medium.com/@dhruvbird/all-pairs-cosine-similarity-in-pytorch-867e722c8572?source=rss-63f5d5495279------2</link>
            <guid isPermaLink="false">https://medium.com/p/867e722c8572</guid>
            <category><![CDATA[similarity-metrics]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[efficiency]]></category>
            <category><![CDATA[cosine-similarity]]></category>
            <dc:creator><![CDATA[Dhruv Matani]]></dc:creator>
            <pubDate>Thu, 08 Jun 2023 05:31:13 GMT</pubDate>
            <atom:updated>2023-06-08T05:31:13.618Z</atom:updated>
            <content:encoded><![CDATA[<h4>PyTorch defines a cosine_similarity function to compute pairwise cosine similarity between pairs of vectors. However, there’s no method to compute the cosine similarity between every pair of vectors in a list of vectors. We’ll explore a very simple and efficient way to do this in PyTorch.</h4><p>Co-authored with<a href="https://medium.com/u/1e659a80cffd"> Naresh</a> and<a href="https://www.gaurav.ai/"> Gaurav</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*JK6p1tO9M9v4VpQqhDVkRQ.png" /><figcaption>Cosine similarity formula. Source: <a href="https://en.wikipedia.org/wiki/Cosine_similarity">Cosine Similarity (Wikipedia)</a></figcaption></figure><h3>Table of contents</h3><h4><a href="#a62d">Introduction</a></h4><ol><li><a href="#06d0">PyTorch API for Cosine Similarity</a></li><li><a href="#f7be">Where is cosine similarity typically used in Machine Learning (ML)?</a></li></ol><h4><a href="#3f42">Computing all pairs Cosine Similarity in PyTorch</a></h4><ol><li><a href="#7c68">Indexing a tensor with None</a></li><li><a href="#6ddf">Expanding a tensor using .expand(…)</a></li><li><a href="#5baf">All-pairs cosine similarity</a></li><li><a href="#ef9b">The last trick: Broadcasting</a></li><li><a href="#3430">A note on efficiency</a></li></ol><h4><a href="#f8f5">Conclusion</a></h4><h3>Introduction</h3><p>From Wikipedia,</p><blockquote><em>In data analysis, </em><strong><em>cosine similarity</em></strong><em> is a measure of similarity between two non-zero vectors defined in an inner product space. Cosine similarity is the cosine of the angle between the vectors; that is, it is the dot product of the vectors divided by the product of their lengths. It follows that the cosine similarity does not depend on the magnitudes of the vectors, but only on their angle. The cosine similarity always belongs to the interval [−1,1].</em></blockquote><h4>PyTorch API for Cosine Similarity</h4><p>torch.nn.functional.<a href="https://pytorch.org/docs/stable/generated/torch.nn.functional.cosine_similarity.html">cosine_similarity</a>(x1, x2, dim=1, eps=1e-8) -&gt; Tensor</p><p>This computes the pairwise cosine similarity between x1 and x2 along a specified dimension. I.e. if x1 and x2 have shape (10, 4, 5) each and we wish to compute the cosine similarity along the last dimension (with value 5), the result will have shape (10, 4).</p><p>For example,</p><pre>x, y = torch.randn(10, 4, 5), torch.randn(10, 4, 5)<br>print(F.cosine_similarity(x, y, dim=2).shape)</pre><p>Would print</p><blockquote>torch.Size([10, 4])</blockquote><p>This is because when we provide cosine_similarity a 3d tensor and ask it to run cosine similarity on the 3rd dimension (dimension index=2), it will collapse that index into a single value.</p><h4>Where is cosine similarity typically used in Machine Learning (ML)?</h4><p>From <a href="https://analyticsindiamag.com/cosine-similarity-in-machine-learning/">this page</a>,</p><blockquote><em>Cosine similarity is a measure of similarity between two data points in a plane. Cosine similarity is used as a metric in different machine learning algorithms like the KNN for determining the distance between the neighbors, in recommendation systems, it is used to recommend movies with the same similarities and for textual data, it is used to find the similarity of texts in the document. So in this article let us understand why cosine similarity is a popular metric for evaluation in various applications.</em></blockquote><p>The reason the author(s) got interested in this metric is because of its use in the <a href="https://arxiv.org/pdf/2002.05709.pdf">SimCLR</a> for the contractive learning of visual representations. When computing the NT-Xent (Normalized Temperature-scaled Cross Entropy) loss, the first step is to perform an <strong>all-pairs cosine similarity</strong> between all the result feature vectors produced by the model.</p><p>We won’t dive into the details of SimCLR or NT-Xent loss for the purposes of this article. Let’s assume that all-pairs cosine similarity between a set of N feature vectors is a necessary step to compute.</p><p>Next, we will take a detailed look at how we can mechanically compute all-pairs cosine similarity in PyTorch.</p><h3>Computing all pairs Cosine Similarity in PyTorch</h3><p>A quick internet search reveals that folks have been having a hard time finding a succinct and efficient way to perform an all-pairs cosine similarity operation.</p><ul><li><a href="https://stackoverflow.com/questions/69797614/indexing-a-tensor-with-none-in-pytorch">Indexing a tensor with None in PyTorch</a></li><li><a href="https://discuss.pytorch.org/t/pairwise-cosine-distance/30961">PyTorch Discuss: Pairwise cosine distance</a></li><li><a href="https://github.com/pytorch/pytorch/issues/48306">PyTorch Issues: example for pairwise distance matrix</a></li></ul><p>In fact, the problem is deemed to be so complex that there’s a metric dedicated to this subject on the <a href="https://torchmetrics.readthedocs.io/en/stable/pairwise/cosine_similarity.html">torchmetrics page</a>.</p><p>Fortunately, there’s a one-line solution to this problem (a variation of which is mentioned on <a href="https://github.com/pytorch/pytorch/issues/48306">this PyTorch GitHub issue</a>.</p><p>If “x” is the input tensor with 2 dimensions (batch, vector-length), then the one-line solution is:</p><blockquote>cosine_similarity(x[None,:,:], x[:,None,:], dim=-1)</blockquote><p>There’s a lot going on in there and it may not be obvious how it really works, so the rest of this post will focus on dissecting the various sub-parts of the solution by building it up from scratch.</p><p>Here’s a <a href="https://github.com/dhruvbird/ml-notebooks/blob/main/cosine_similarity/Cosine%20Similarity.ipynb">link to a notebook</a> implementing all the steps mentioned in this article.</p><p>In the following subsections, we’ll learn about the following as it applies to succinctly computing all-pairs cosine similarity.</p><ol><li>Indexing a tensor with “None”</li><li>Using tensor.expand() to expand tensors along singleton dimensions</li><li>Leveraging PyTorch broadcast semantics to implicity expand tensors along singleton dimensions</li></ol><h4>Indexing a tensor with None</h4><p>The first thing we need to understand is what happens when you index a PyTorch tensor using None. This<a href="https://stackoverflow.com/questions/69797614/indexing-a-tensor-with-none-in-pytorch"> stackoverflow topic</a> has the answer.</p><blockquote><em>Similar to NumPy you can insert a singleton dimension (“unsqueeze” a dimension) by indexing this dimension with None. In turn n[:, None] will have the effect of inserting a new dimension on dim=1. This is equivalent to n.unsqueeze(dim=1)</em></blockquote><pre>x = torch.randn(3)<br># Indexing with None does the same thing as unsqueezing the tensor<br># at that dimension. After this indexing operation, the tensors<br># x_row_dup and x_col_dup will have 1 additional dimension at<br># dimensions 0 and 1 respectively.<br>x_row_dup, x_col_dup = x[None,:], x[:,None]<br>print(x, x.shape)<br>print(x_row_dup, x_row_dup.shape)<br>print(x_col_dup, x_col_dup.shape)</pre><p>Prints the following.</p><pre>tensor([-1.2756, 1.1559, -0.0660]) torch.Size([3])<br>tensor([[-1.2756, 1.1559, -0.0660]]) torch.Size([1, 3])<br>tensor([[-1.2756],<br>        [ 1.1559],<br>        [-0.0660]]) torch.Size([3, 1])</pre><h4>Expanding a tensor using .expand(…)</h4><p>The PyTorch <a href="https://pytorch.org/docs/stable/generated/torch.Tensor.expand.html">expand(…)</a> API is used to expand the values of certain dimensions by repeating the values at those dimensions. Note that you can only expand a dimension if its value is 1. Let’s see an example below.</p><pre>x_row_dup, x_col_dup = x_row_dup.expand(3, 3), x_col_dup.expand(3, 3)<br>print(&quot;x stretched across rows&quot;)<br>print(&quot; - - - - - - - - - - - -&quot;)<br>print(x_row_dup, x_row_dup.shape)<br>print(&quot;&quot;)<br>print(&quot;x stretched across columns&quot;)<br>print(&quot; - - - - - - - - - - - - - &quot;)<br>print(x_col_dup, x_col_dup.shape)</pre><p>Prints the following.</p><pre>x stretched across rows<br>- - - - - - - - - - - -<br>tensor([[-1.2756, 1.1559, -0.0660],<br>        [-1.2756, 1.1559, -0.0660],<br>        [-1.2756, 1.1559, -0.0660]]) torch.Size([3, 3])<br><br>x stretched across columns<br>- - - - - - - - - - - - - <br>tensor([[-1.2756, -1.2756, -1.2756],<br>        [ 1.1559, 1.1559, 1.1559],<br>        [-0.0660, -0.0660, -0.0660]]) torch.Size([3, 3])</pre><h4>All-pairs cosine similarity</h4><p>Let’s assume our input tensor has 3 elements, namely (A, B, C). To compute the all-pairs cosine similarity, we will first expand this tensor along 3 rows and 3 columns.</p><p><strong>Unsqueeze</strong>: Our input tensor (A, B, C) has shape (3). We will first unsqueeze it along the 1st and 2nd dimensions to make it look like the following:</p><p>(A, B, C) squeezed along dimension 0 will look like ((A, B, C)) and have shape (1, 3).</p><p>(A, C, C) squeezed along dimension 1 will look like ((A), (B), ©) and have shape (3, 1).</p><p><strong>Expand</strong>: Now, we will expand these tensors along their singleton dimensions (dimensions with value 1) to make both the tensors square.</p><p>((A, B, C)) expanded to have shape (3, 3) will look like:</p><pre>((A, B, C),<br> (A, B, C),<br> (A, B, C))</pre><p>((A), (B), (C)) expanded to have shape (3, 3) will look like:</p><pre>((A, A, A),<br> (B, B ,B),<br> (C, C, C))</pre><p>If we perform the pairwise cosine similarity (which the PyTorch API can already do), then we get the all-pairs cosine similarity as shown in the figure below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NJbNjabBD8qOMqU1gMzpEQ.png" /><figcaption>Figure 1: All Pairs cosine similarity computed between 2 expanded versions of the input tensor (A, B, C). Source: Author(s)</figcaption></figure><p>That’s it! Here’s an example that shows it for the tensors we have been using so far.</p><pre># Add a dummy dimension at the end so that we can perform cosine<br># similarity on that last dimension.<br>x_row_dup = x_row_dup.reshape(3, 3, 1)<br>x_col_dup = x_col_dup.reshape(3, 3, 1)<br>x_cosine_similarity = F.cosine_similarity(x_row_dup, x_col_dup, dim=-1)<br>print(x_cosine_similarity)</pre><p>Will print.</p><pre>tensor([[ 1., -1., 1.],<br>        [-1., 1., -1.],<br>        [ 1., -1., 1.]])</pre><p>But wait! Why are all the values 1 or -1?! Turns out that the cosine similarity of 2 vectors of size 1 is always +1 or -1. That’s because a single element vector has an angle of 0 or 180 degree depending on whether they are pointing in the same or opposite directions.</p><p>Let’s try the same thing with a vector of size 2 instead of 1.</p><pre>x = torch.randn(3, 2)<br>x_row_dup, x_col_dup = x[None,:,:], x[:,None,:]<br>x_row_dup, x_col_dup = x_row_dup.expand(3, 3, 2), x_col_dup.expand(3, 3, 2)<br>x_cosine_similarity = F.cosine_similarity(x_row_dup, x_col_dup, dim=-1)<br>print(x_cosine_similarity)</pre><p>Will print.</p><pre>tensor([[1.0000, 0.9512, 0.9826],<br>       [0.9512, 1.0000, 0.9920],<br>       [0.9826, 0.9920, 1.0000]])</pre><p>This definitely looks better!</p><h4>The last trick: Broadcasting</h4><p>While we used .expand(…) here, we’d like to mention that it was strictly not necessary since most of the operations defined in PyTorch support a concept called <a href="https://pytorch.org/docs/stable/notes/broadcasting.html">dimension broadcasting</a>. From the documentation,</p><blockquote><em>If a PyTorch operation supports broadcast, then its Tensor arguments can be automatically expanded to be of equal sizes (without making copies of the data).</em></blockquote><p>Hence, the cosine similarity operation when fed with 2 tensors of shape (1, 3) and (3, 1) will broadcast both of them to (3, 3) and essentially implicitly perform the tensor expansion step that we explicitly performed above.</p><p>When we run the following code.</p><pre>x_cosine_similarity = F.cosine_similarity(x[None,:,:], x[:,None,:], dim=-1)<br># This should print the same matrix as above.<br>print(x_cosine_similarity)</pre><p>We get the following output.</p><pre>tensor([[1.0000, 0.9512, 0.9826],<br>       [0.9512, 1.0000, 0.9920],<br>       [0.9826, 0.9920, 1.0000]])</pre><p>Which is in fact the same as what we got above.</p><h4>A note on efficiency</h4><p>Compared to many of the solutions mentioned in <a href="https://discuss.pytorch.org/t/pairwise-cosine-distance/30961/11">this discussion</a>, this solution doesn’t use any explicit for loops. Every time you write an explicit for loop, you have the following problems:</p><ol><li>There’s some non-trivial CPU computation happening, potentially starving the GPU</li><li>If you’re using a for loop, you’re probably leaving some opportunity for parallel GPU execution on the table. This will impact your overall GPU utilization, and hence the time it takes to run your computation</li></ol><h3>Conclusion</h3><p>We saw how to succinctly compute the all-pairs cosine similarity between every vector in a list of vectors. Along the way we learned about indexing PyTorch tensors using None to unsqueeze the tensor along that dimension, and how PyTorch broadcast semantics work.</p><p>We hope this will help you with your deep learning adventures!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=867e722c8572" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>