<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shounak Acharya on Medium]]></title>
        <description><![CDATA[Stories by Shounak Acharya on Medium]]></description>
        <link>https://medium.com/@Shachary?source=rss-f7d903fe5201------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*bnxghgMYc3EIVmGFXNJTDw.jpeg</url>
            <title>Stories by Shounak Acharya on Medium</title>
            <link>https://medium.com/@Shachary?source=rss-f7d903fe5201------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 05:55:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@Shachary/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[TLDS: Relevance Labelling and the ESCI benchmark]]></title>
            <link>https://medium.com/@Shachary/relevance-labelling-and-the-esci-benchmark-e10483e587a6?source=rss-f7d903fe5201------2</link>
            <guid isPermaLink="false">https://medium.com/p/e10483e587a6</guid>
            <category><![CDATA[search]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[search-engines]]></category>
            <category><![CDATA[recommendation-system]]></category>
            <dc:creator><![CDATA[Shounak Acharya]]></dc:creator>
            <pubDate>Mon, 09 Nov 2026 16:12:09 GMT</pubDate>
            <atom:updated>2026-11-09T16:26:54.765Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Introduction</strong></p><p>Recommendation Systems are a powerful tool to model interaction patterns, and show the right products that will grow your platform.</p><p>Modern literature covers a diversity of designs, implementations and solutions, each bringing a unique set of strengths and weaknesses to the table. However, all of them share the same broad common objective — surfacing <strong>highly relevant</strong> results</p><p><strong><em>“ “ What does it even mean for something to be relevant? ” ”</em></strong></p><p>Funnily enough, the answer to this is a lot harder than it looks. In fact, you could implement the most state of the art AI models out there, and they will always fall short if you aren’t clear with <em>your definition of relevance.</em></p><p>A recommendation is said to be relevant, if it has a very high chance of getting the user to engage with it, while benefiting both the user (if they, for example buy it) and the host (it has a net positive impact on their business)</p><p>Now a user will do this only if your recommendation is:</p><p>a. satisfying the user’s query (if any)<br>b. is tailored to the user’s preferences <br>c. satisfies the host organization’s business objectives</p><p>Part <strong>a </strong>is easy to understand and serves as the foundation for <strong>b</strong> and <strong>c</strong>. Search is a subsystem where the user’s query is the most direct, making search queries a great lens for understanding relevance.</p><p><strong>Search Relevance and Semantic Similarity</strong></p><p>Figuring out what the user wants is much more straightforward <strong>if they’ve</strong> <strong>explicitly asked for it. </strong>It basically implies that you should surface recommendations that are <em>semantically similar</em> to the user’s query.</p><p>For example,<strong> if your user enters “<em>tshirt”</em></strong> as their query on your platform, the bare minimum expectation of your recommendation system is to <strong>surface</strong> <strong>tshirt products only.</strong></p><p>Any AI model that’s learning to recommend will definitely optimize for this. However, this surfaces a very interesting thought- <em>How do you know that a query is semantically similar to a product? And how can one measure it?</em></p><p>More often than not, it is easier to answer this intuitively. You could easily look at your query, look at a set of products, and <em>match</em> your query to the products you’d expect to be relevant, a little relevant, not that relevant and completely irrelevant.</p><p>You just <em>know</em> the level of similarity, but a lot of nuance in the decision making is lost right here. To teach a machine to understand relevance, you must have labelled data. But how would you label this data?</p><p>Using a binary label (relevant/not relevant) here is a pitfall, because reducing the definition of relevance to one of two options evaporates a lot of the <strong>fine grained details</strong> that would’ve served as the <strong>differentiating factor</strong>. A model trained on such data would barely understand relevance.</p><p>Cool then, let’s create a standardized scoring system, say from 1–10. 1 would imply the product is completely irrelevant to the user’s query, and 10 would mean its exactly what the user would’ve been looking for.</p><p>If we can agree on similarity being a spectrum, then arriving to a scoring system instead means that we are moving in the right direction. To label this data, you could-</p><ol><li>Have humans manually sift through your data and assign these scores</li><li>Look for implicit/explicit user interaction signals (like clicks, add to carts or orders) on your platform’s search results and map each observed pattern to a score</li><li>Prompt an LLM to annotate this data for you by leveraging its vast world knowledge</li></ol><p>The industry does resort to all 3, but not without addressing the flaws in them.</p><p>People’s rubrics for decision making are often subjective, noisy and unpredictable. An LLM is not so different either in such a case. The level of semantic similarity that would qualify for a certain score would vary a lot from person to person, especially without rules.</p><p>We need some rules man.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/491/1*k-SL8MRw0pJfKMjfyHR9ZA.png" /></figure><p><strong>The ESCI benchmark</strong></p><p>In 2022, Amazon open sourced a dataset consisting of e commerce query-product pairs, each assigned an ordered, relevance label based on an elegantly defined framework.</p><p>Named as the ESCI benchmark, this was a large scale, multilingual benchmark dataset created to help researchers improve e-commerce product search.</p><p>The rules used to define these elegant relevance labels is our point of interest here. Here is an excerpt of them defining these rules in their own words:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/767/1*WoqcClc9JsnByABvZvmjZA.png" /><figcaption><a href="https://arxiv.org/pdf/2206.06588">https://arxiv.org/pdf/2206.06588</a></figcaption></figure><p>These neatly defined labels can be adapted to any e commerce search related use case, with fleshed out definitions and more edge cased examples, so as to create a robust relevance labelling rubric that everyone can agree on.</p><p>Search sessions, along with these definitions can then be shared with human agents, or LLMs (in their system prompt), further unlocking the potential for large scale and consistent data labelling.</p><p><strong>Relevance Evaluation</strong></p><p>Great, now we have dependable labelled data. Our recommendation model, trained to retrieve/recommend products based on user queries can finally be rightly evaluated on its semantic understanding of search relevance.</p><p>Simply assign a ground truth score to each relevance label (for example, Exact match -&gt; 3, Substitute -&gt; 2, Complementary -&gt; 1 and Irrelevant -&gt; 0)</p><p>Using these scores, we can penalise the model for surfacing a complementary or irrelevant match over an exact match. Using the NDCG metric, we can quantify this, representing the quality of our product retrieval!</p><p>To know more about that metric, check out my previous <a href="https://medium.com/@Shachary/tlds-recommendation-ranking-metrics-cfdc4fb23c94">blog</a> breaking it down for you:P</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e10483e587a6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[TLDS: Distributed Data Parallel]]></title>
            <link>https://medium.com/@Shachary/tlds-distributed-data-parallel-de61b660a967?source=rss-f7d903fe5201------2</link>
            <guid isPermaLink="false">https://medium.com/p/de61b660a967</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[Shounak Acharya]]></dc:creator>
            <pubDate>Wed, 17 Jun 2026 22:42:43 GMT</pubDate>
            <atom:updated>2026-06-17T22:44:42.100Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PJCQQAX09hC17GNYJGbEGA.png" /><figcaption>source: google gemini</figcaption></figure><p><strong>Introduction</strong></p><p>Time is money, literally. Cloud GPU providers bill you by the hour, and it isn’t pretty if you aren’t training your money’s worth. If you find yourself with a large (and I mean large) training dataset and access to multiple GPUs, here’s a trick that cloud compute providers don’t want you to know.</p><p>The most popular pillar of parallelization for deep learning model training , is leveraging multiple GPUs to increase the throughput of data being fed into your model. Let’s talk about <strong>Distributed Data Parallel or DDP!</strong></p><p><strong>Preface</strong></p><p>In order to understand the essence of <strong>DDP, </strong>let’s have a little chat about <em>how deep learning models train</em> in the first place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/532/1*b2TguQsq71Wrg5eZYtjrag.png" /><figcaption>your model before you’re about to train it</figcaption></figure><p>The above image illustrate all the distinct ingredients required for your model training:</p><ol><li><strong>The dataset: </strong>Your training dataset contains ordered data comprised of features (information utilised by the model to learn) and labels (the ground truth, which your model will learn to predict).</li><li><strong>The model: </strong>The model you’ve designed here would be <em>comprised of learnable values</em> (also called <strong>weights</strong>), responsible for learning the complex patterns in your training data, so that it’s able to make accurate predictions.<br>This process of the model learning and predicting can be represented as a series of matrix operations (especially matrix multiplications!), which brings us to the last and most important ingredient,</li><li><strong>The GPU: </strong>As touched upon in one of my previous <a href="https://medium.com/@Shachary/tlds-automatic-mixed-precision-training-2b07c61cafdc">blogs</a>, GPUs are the machines facilitating your model training. They are specialised to be quick at computing matrix multiplication operations, the downside being that they are limited by the available memory in their RAM (the memory component where these operations occur).</li></ol><p><strong>So what happens during training?</strong></p><p>To start our training, we first move our model into the GPU<strong>,</strong> such that <strong>all of it sits in the GPU’s memory</strong> (RAM).</p><p>Next, a sample of data points from the dataset (called a <strong>batch) </strong>is sent to the same GPU’s RAM, where the model does 2 things:</p><p><strong>A</strong>. <strong>Prediction: </strong>Via a <strong>series of matrix operations</strong> involving your data points and the model <strong>weights</strong>, the GPU is able to calculate the predictions made by the model for each of the data points in the batch.</p><p><strong>B. Learning: </strong>The model then calculates the <strong>loss</strong>, representing how close it’s prediction was to the correct ground truth. The loss would be calculated for each data point, and averaged across the batch. Using this loss, the model computes a unique term for every weight, called the <strong>gradient </strong>(more matrix operations!), which tells it how much it should modify it’s weights so that it’s able to make a more accurate prediction next time.</p><p>During training, you keep creating batches via these dataset samples, sending them to the GPU and updating your model, as it keeps getting better.</p><p>In most cases, the <em>bigger the batch -&gt; the larger number of data points the model is able to learn from in a single pass-&gt; the more informative is the gradient computed from it, allowing a </em><strong><em>smoother learning curve</em></strong></p><p>Apart from the technical improvements observed by increasing the batch size, the most obvious and intuitive win is the <strong>reduction in training time! </strong>Think about it, doubling the batch size essentially means that your model will go through your training dataset twice as fast!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ntvku2-x-6k3vib54hxRBw.png" /><figcaption>the training loop</figcaption></figure><p><strong>The catch</strong></p><p>If your takeaway from the above section is to prioritise training efficiency, while reaping the operational rewards, then you’re on the right track. Unfortunately, batches (and the computation associated with them) consume GPU memory too, and while increasing the batch size endlessly, you’ll find yourself hitting the memory limit very soon.</p><p>Whats next?</p><p><strong>Enter Distributed Data Parallel</strong></p><p>Here’s a crazy idea, what if you decided to use <strong>multiple GPU</strong>s?</p><p><strong>“</strong><em>But wouldn’t they cost more?” </em>Per hour, maybe. But what if they reduced your training time proportionally? Since cloud compute is billed by time, finishing a training run 4x faster on 4 GPUs means your net expense stays roughly the same, except you get a well-trained model in a fraction of the time!</p><p>It’s a simple approach really,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zVX0Phb2zf3aImodMIGRqg.png" /><figcaption>data parallelism!</figcaption></figure><p>Say you have access to 4 GPUs, you begin by first copying your (untrained) model into each of them. Next you configure your dataset to send a unique batch of data to each of the GPUs during training.</p><p>Consequently, each of your models will calculate a separate loss (and gradient) on it’s host GPU, which recieved it’s own batch of data.</p><p>But wait, there’s a small issue here, we can’t just update the model weights using these locally calculated gradients!</p><p>Since the input batch entering each GPU was different, the calculated loss would be different and so would the gradient calculated from that loss. Suddenly, each copy of your model wouldn’t really be a copy anymore because their weights would be updated by a different gradient value!</p><p>Yikes, that’s not good. We only want 1 trained model as the end result, not 4 different models 🫩. Is there a way to ensure that our model copies stay synchronised? Yes, and its quite nifty when you take a peek under the hood.</p><p><strong>The approach</strong></p><p>Let’s take a second to remind ourselves the technical terms introduced in the <strong>preface section</strong>.</p><p>Okay so currently, we’re in a situation where we have 4 GPUs, a copy of your model in each GPU, a distinct batch of data sent to each GPU, and the loss and gradient resulting from each of those 4 batches.</p><p>DDP needs to figure out a way such that the <em>model weights in each GPU stay identical to their copies. </em>The most straightforward way to do this would be to ensure that the <em>gradients calculated for each model weight also remain identical to corresponding weight in their model copies</em>, so that <em>weights in each model copy are updated by the same values</em>. This way they’ll remain synchronised during training!</p><p>So how does DDP’s algorithm do it?</p><p>Within each GPU before training, DDP <em>organizes</em><strong> </strong><em>the weights of your model into subgroups</em> called <strong>buckets</strong>. This is done so that a specific bucket on GPU 0 always matches the exact same subgroup of weights on GPU 1, 2, and 3. <em>DDP keeps a track of these buckets and the gradients corresponding to them.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cNjexD0EBleRgXLO-VrwgQ.png" /><figcaption>a snapshot of a copy of your model, organized into buckets</figcaption></figure><p>A gradient can be said to represent what the model has “learnt” from the input data.</p><p>Now, say we have 4 different calculated gradients for the 4 copies of Bucket 0, one in each GPU. What’s the most intuitive way to ensure every GPU ends up with the exact same gradient value without discarding any information? If you thought about <strong>averaging them across all GPUs</strong>, you’re absolutely right!</p><p>This averaging operation is carried out in 2 steps,</p><p><strong>Step 1: summation</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DXdi6tTbvcucIUdQefoUMg.png" /></figure><p>Look at the illustration above. We start at GPU 0 and move sequentially from one GPU to the next (Steps 1 to 3), adding up the gradient values along the way to build a running sum. Once the final sum is calculated at the end of the chain, it is passed right back to GPU 0 (Step 4).</p><p><strong>Step 2: synchronisation</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*J05myUX1l9pq0pOUC4Xb6A.png" /></figure><p>Next, we calculate the average gradient from the sum, and set this as the gradient value for that bucket in every GPU by passing it along.</p><p>If you perform these steps for each of the buckets you’d split your model weights into, then <strong>you’ve essentially calculated the average gradient for all model weights, and synchronised them across all the GPUs!</strong></p><p>clever right? But we haven’t finished painting the picture just yet. If you’ve understood the process so far, you’ll find yourself wondering <em>why we couldn’t just average all the gradients</em> in a single 2 step operation? What was the point of viewing the weights as buckets?</p><p>Good thought. The reason for this is because gradients are calculated in a specific order of the weights, such that gradients for bucket 1 weights can only be calculated after bucket 0 because t<em>hey depend on the bucket 0 gradient values. </em>We can take this for granted within the scope of this blog, but the underlying process causing this is called <strong>backpropagation</strong>.</p><p>DDP goes a step further by <em>running this 2 step operation parallely with the gradient computation operations</em>. Let’s look at the illustration below to make it make sense:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-fnnlDYerIIfGdA52Sb92w.png" /></figure><p>Once the dataset batch hits each GPU (at the same time because its parallelised), they start computing the bucket gradients. These gradients are calculated one bucket at a time (due to backpropagation)</p><p>Once the gradient for a bucket has been computed and materialised in the GPU memory, DDP leverages this by starting the summation + synchronisation operation for that bucket instantly.</p><p>While this is happening, the GPU parallelly also starts computing the gradients for the next bucket, and continues this till gradients from all the buckets have been synchronised across the GPU’s.</p><p>It’s a very clever optimization technique that ensures that the additional DDP specific operations don’t take any extra time apart from the gradient computations. For all my nerds out there, this process is called the <strong>Ring AllReduce</strong> operation.</p><p>Once the gradients have been synchronised, each of the GPUs will update their weights using the same gradient, ensure that your <em>model copies stay identical</em> throughout training</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vDPfrdl4Y-8r0JU3DcM8NQ.png" /></figure><p>voila!</p><p><strong><em>are you winning son?</em></strong></p><p>If you’re able to reduce your training time proportionally to the number of available GPUs, while ALSO reaping the additional benefit of a smoother training, then I think it’s safe to say that you are :)</p><p><strong>Conclusion</strong></p><p>It’s a fascinating and powerful concept that’s also surprisingly intuitive once you’re able to look past all the associated bells and whistles.</p><p>I remember being very impressed (and a little intimidated) when I first came across it, and now I find myself using it all the time whenever I train large scale recommendation models on enormous datasets.</p><p>If you’re a casual reader who came here to learn something cool, I hope that this blog fulfilled that, and if you’re an enthusiast looking to learn something new in AI, then this is definitely something you should add to your toolbox of skills!</p><p>Parallelizing the delivery of data is just the beginning when it comes to efficiently training large scale models. This rabbit hole runs deep, each level more fascinating than the next. I hope to cover all of them sometime :)</p><p>Do follow me on <a href="https://www.linkedin.com/in/shounakacharya/">linkedin</a> to stay in the loop for when my next article drops :D</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de61b660a967" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[TLDS: AUC-ROC Curves]]></title>
            <link>https://medium.com/@Shachary/tlds-auc-roc-curves-9f520676a3f1?source=rss-f7d903fe5201------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f520676a3f1</guid>
            <dc:creator><![CDATA[Shounak Acharya]]></dc:creator>
            <pubDate>Mon, 16 Feb 2026 13:52:47 GMT</pubDate>
            <atom:updated>2026-02-16T13:52:47.707Z</atom:updated>
            <content:encoded><![CDATA[<p>AUC-ROC curves are a classic component of ML. Writing this article satisfies a personal vendetta; as a student enthusiast who built much of their knowledge base through self study and exploration, I struggled to grasp the intuition behind this metric.</p><p>AUC-ROC curves are more than just a graph, number or formula, and we’re here understand its essence so that we can truly appreciate it’s existence!</p><p><strong>Introduction</strong></p><p>Let’s say that you were scrolling too many reels, and stumbled upon a popular trend where viewers guessed whether an object was made of cake. Fed up of randomly guessing the answer every time, you decided to design an ML model that can process an image of an object and predict the probability of it being made of cake.</p><p>You realise that in order to make this work, your model must nail two things- <em>Accuracy</em> <em>and Reliability</em></p><p>Accuracy is a pretty easy expectation to understand, if it’s cake your model should think its cake, and if it isn’t, only then should your model not!</p><p>Let’s remember that our model predicts the <em>probability</em> that the<em> input image you fed it is made of cake. </em>This probability is a decimal that lies anywhere between 0 and 1.</p><p>Say for an image, your model predicted it to be 70% likely that it’s cake. Now if you are super confident of your model’s predictions, you could say that any probability above 50% should be <em>sufficient in trusting your model</em> that it’s cake. On the other hand if you knew that your model is prone to mistakes, you would think that any probability above 80% would be safe enough to trust the prediction. That’s reliability!</p><p>So how can you quantify these 2 properties, to evaluate them on your model?</p><p><strong>AUC-ROC Curves</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/1*y6nQ6IXFH5f92EIuPaghDQ.png" /><figcaption>model prediction range</figcaption></figure><p>The illustration above represents the range of probabilities that can be predicted by your model (ignore the breadth). Now let’s populate this with your model’s predictions assigned for a few images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/1*LElMaDRcxaVvFjLss0PlyQ.png" /><figcaption>input images and their predicted values</figcaption></figure><p>Notice the dots, each dot represents an input image. If the dot is red, it’s an image of an object, and if it’s blue then it’s cake!</p><p>Where each dot lies along the length of the illustration, represents the model’s predicted probability of the image being of a cake. The closer it is to 1, the more your model is sure of it being a cake, and vice versa for 0. Makes sense so far?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/378/1*1TIZyjHvv8P5ABAv7w9v_Q.png" /><figcaption>decision boundary at 0.5</figcaption></figure><p>Now check out this dotted line, we call this the <strong><em>decision boundary</em></strong><em>.</em></p><p>Remember when we spoke about <em>your confidence</em> in trusting the model’s prediction? That’s exactly what the decision boundary represents. If the predicted probability of an image is greater than the number demarcated by this line (or simply put, the point in the illustration is above the line), then the prediction is interpreted as a cake. Setting that decision boundary at 0.5 sounds sensible and unbiased, so lets leave it there for a bit.</p><p>Let’s talk about accuracy. The expectation is that the blue dots are above the line (implying images of cakes were correctly identified), and the red dots should be below (implying images of objects were correctly identified too).</p><p>To maximise your model’s accuracy, you would like it to predict probabilities, such that<em> as many blue dots are above the decision boundary, and as many red dots below</em>. To be specific, the % of blue dots above the boundary should be high, while simultaneously the<em> % of red dots above the boundary</em> should also be low!</p><p>We have just described two new terms that quantify this :)</p><ol><li><strong><em>True Positive Rate (TPR)</em></strong>: Of all the blue dots (images of cakes), what fraction of them were above the decision boundary? (correctly identified as cake)</li><li><strong><em>False Positive Rate (FPR)</em></strong>: Of all the red dots (images of objects), what fraction of them were above the decision boundary? (incorrectly identified as cake).</li></ol><p>The illustration above had 10 blue and 10 red dots. Let’s calculate the TPR and FPR for our model, given the decision boundary of 0.5:</p><p><em>True Positive Rate = 8 blue dots above the line/ 10 blue dots = 0.8</em></p><p><em>False Positive Rate = 2 red dots above the line / 10 red dots = 0.2</em></p><p>In an attempt to improve the TPR, you might think to yourself, “what if I lower the decision boundary just enough to ensure that all my blue dots are considered as true positives?”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/410/1*-CtMvffJPqIKg--DjR4lUg.png" /><figcaption>decision boundary at 0.3</figcaption></figure><p>Let’s recalculate our rates with this adjustment:</p><p><em>True Positive Rate = 10 blue dots above the line/ 10 blue dots = 1.0</em></p><p><em>False Positive Rate = 5 red dots above the line / 10 red dots = 0.5</em></p><p>Oh no! Although our model is assured to correctly identify all images of cakes, it is also now 50% likely to misclassify an image of an object as a cake too!</p><p>That’s okay, let’s set the decision boundary higher than 0.5, that is sure to eliminate the false positives right?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/452/1*Hn1FYmmoifuao9t1RPycMA.png" /><figcaption>decision boundary at 0.7</figcaption></figure><p><em>True Positive Rate = 5 blue dots above the line/ 10 blue dots = 0.5</em></p><p><em>False Positive Rate = 0 red dots above the line / 10 red dots = 0.0</em></p><p>Now the TPR has crashed, whoops.</p><p>In an attempt to fix our model’s accuracy, we’ve discovered that it’s not very <em>reliable</em>!</p><p>Let’s think about this for a second. Why do we even care about moving the decision boundary?</p><p>Well actually, we don’t have to, especially when you would formally deploy this model for use. But observing <em>a stark change in our prediction rates</em> with a <em>small deviation of our decision boundary</em> exposes a lack in the model’s confidence for the predictions it assigns to the images.</p><p>This lack in confidence means that if your model comes across a new, unseen image, it can be more likely to be confused and predict that image to be something it isn’t :(</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/458/1*m-gFDHjevakLR4jnIcvR6A.png" /><figcaption>ideal model’s predictions</figcaption></figure><p>An ideal model shouldn’t just correctly classify the input images, but also ensure that they’re quite <em>separable</em> (closer to the extremities of our illustration), so that the model’s prediction rates remain closer to their ideal values (1 TPR, 0 FPR), and don’t deviate much for different values of the decision boundary.</p><p>This can be detected by sweeping the decision boundary from 1 to 0, and noting down the corresponding values of your TPR and FPR.</p><p><strong>Joining the dots…</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/835/0*xNLnUTXdzB4H-uuj.png" /><figcaption>Source: <a href="https://www.listendata.com/2023/08/calculate-auc-in-r.html">https://www.listendata.com/2023/08/calculate-auc-in-r.html</a></figcaption></figure><p>Let’s imagine a two dimensional graph with the FPR on the X axis, and the TPR on the Y axis. Now, we shall sweep the decision boundary from 1 to 0, and plot every corresponding value of TPR and FPR on this graph.</p><p><strong><em>A. Decision Boundary = 1.0</em></strong></p><p>Our boundary is set so high, that every red and blue dot (from the illustrations in the previous section) would be below it! Mathematically, this would mean that our True Positive Rate is 0.0 and False Positive Rate is also 0.0 (every single image predicted as not a cake)</p><p><strong><em>B. 0.0 &lt; Decision Boundary &lt; 1.0</em></strong></p><p>As we lower the decision boundary, and note down the corresponding prediction rates, we notice that our model catches more true positives because more dots are above the decision boundary, but also at the cost of increasing the false positive rate!</p><p><em>We should want the TPR to grow much faster than the rate at which the FPR grows during this sweep</em></p><p><strong><em>C. Decision Boundary = 0.0</em></strong></p><p>Our decision boundary is so low, that every dot is above it and <em>all </em>of your model’s predictions would be treated as it being cake, even the images that actually aren’t! Hence the True Positive rate is 1.0, but so is the False Positive Rate.</p><p>The curve formed by joining these dots is called the <strong>Receiver Operating Characteristic (ROC). </strong>The shape of this curve gives you an idea of the separability of the predictions made by your models</p><p>Remember the ideal scenario discussed earlier where the model’s predictions lay at the extremities of the probability illustration? In that case, the model perfectly separates the classes. As you lower the decision boundary from 1.0, the TPR immediately jumps from 0.0 to 1.0 while the FPR stays at 0.0. And as the decision boundary reaches 0.0, the TPR and FPR immediately jump to (0,0).</p><p>Notice how the points on this perfect ROC curve travel from (0,0), shoot straight up to the top-left corner at (0,1), and then finally move to (1,1).</p><p>Wait so that means all the points of our ROC curve are found only at one of the following three locations: (0,0), (0,1) and (1,0). Intuitively, you might think about calculating the area under this curve, it’s 1. Congratulations, you have just quantified the theoretical maximum of your model’s performance!</p><p>The next part must feel like a no brainer, you then proceed to calculate the area under the curve <em>actually</em> traced by your model. Mathematically, it will always be between 0 and 1. This metric is called <strong>Area Under Curve (AUC) </strong>and is a quantifiable, bounded metric that represents the overall predictive power of your model!</p><p>This is precisely why the metric is called the <strong>ROC-AUC curve</strong>, as it presents information in 2 parts - the shape of the curve itself and the area under it!</p><p><strong>Conclusion</strong></p><p>Uncovering the essence of the AUC-ROC curve goes a long way in removing the ambiguity associated with it</p><p>Besides, I think it’s a very cool party trick to have a model that’s great at detecting objects that are made of cake :P</p><p>If you liked my blog, I would appreciate a like and a subscription to get email notifications for my posts! I would also love to entertain feedback, clarifications and corrections in my comments :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f520676a3f1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[TLDS: Recommendation Ranking Metrics]]></title>
            <link>https://medium.com/@Shachary/tlds-recommendation-ranking-metrics-cfdc4fb23c94?source=rss-f7d903fe5201------2</link>
            <guid isPermaLink="false">https://medium.com/p/cfdc4fb23c94</guid>
            <dc:creator><![CDATA[Shounak Acharya]]></dc:creator>
            <pubDate>Sun, 18 Jan 2026 18:36:43 GMT</pubDate>
            <atom:updated>2026-01-18T20:48:06.283Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*k59I3VkOUSPhPnwkJEjBZA.png" /><figcaption>Source: Nano Banana Pro</figcaption></figure><p><strong>Introduction:<br></strong>Recommendation Systems are cool. A lot goes into devising a system that attempts to show you exactly what you might have been looking for.</p><p>But this raises a more foundational question, how does one measure the “correctness” of a recommendation? Well you can always look at it and determine, but that isn’t foolproof or scalable enough. Is there a way to quantitatively measure the quality of your model’s recommendations?</p><p>Yes!, quite a few actually. In this blog we’ll cover some basic yet fundamental ones.</p><p><strong>Ranking Metrics:</strong></p><p>Let’s say you’re building a search recommendation model. This model is trained to retrieve products that are at least somewhat semantically similar to your user’s query. You really want to evaluate the recommendation quality of this model.</p><p>Good thing you already have information about a bunch of user queries, along with their corresponding products, that you’re confident are relevant to your search query.</p><p>Now, if your model processes a query, the bare minimum expectation is for it to show <em>at least 1</em> of those relevant products towards the top when it retrieves and sorts them based on relevance.</p><p>“Towards the top” sounds a little vague, so you observe different sections of the model’s recommendations and see how many of the correct ones made the cut , starting from the first most relevant product, then the top 3 most relevant products, the top 10 most relevant products and so on.</p><p>This is actually a ranking metric and is defined as <strong>Recall@k.</strong> Recall because we’re measuring the ratio of correct products that are in our observed model’s <em>top k subset</em> of recommendations. Here’s a cleaner representation of the definition:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/431/1*BrTJds9O3fGtouPE2yqQrw.png" /><figcaption>Source: Gemini 3</figcaption></figure><p>The term “recall” might sound familiar if you’re aware about how classification is measured in standard ML models. It’s the same thing!, just that its definition has been adapted for recommendation systems.</p><p>There’s a more extreme variation of this metric called <strong>hitrate</strong>, a binary label that’s 1 if at least 1 relevant item came up in your recommendations, and 0 otherwise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/332/1*96SrdalaNufyV60jjxLEUQ.png" /><figcaption>Source: Gemini 3</figcaption></figure><p><strong>Hitrate@K</strong> is quite intuitive in letting you know which queries your model failed to find anything relevant.</p><p>These metrics sound quite solid so far, but something makes this feel incomplete….</p><p>Sure, we can detect the % of correct products covered by our recommendations with these metrics, but they still don’t give us the full picture at all!</p><p><em>What if we had a way to simultaneously measure how much of the products in the recommendations are actually relevant to the input query, AND ensure that the relative ranking of those products is correct?</em></p><p>Introducing the <strong>Normalized Discounted Cumulative Gain</strong>, or <strong>NDCG</strong>. Sounds like a mouthful? That’s okay the intuition is quite simple, let’s look at the definition first. Just like recall, NDCG is defined for a subset of the top K model recommendations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/1*skyKlIbc7Hp78W0E2qIQiQ.png" /><figcaption>Source: Gemini 3</figcaption></figure><p>So this metric expects two things from us, a <em>prediction</em> and a <em>target</em>.</p><p><strong>DCG</strong> ( <strong>Discounted Cumulative Gain</strong> )<strong> </strong>is your prediction here, and represents a list of recommended products, ranked by the order of relevance decided by <em>your model, </em>based on the user’s query.</p><p>Meanwhile <strong>IDCG</strong> ( <strong>Ideal Discounted Cumulative Gain) </strong>represents the desired product recommendations and their ranks, as preferred by the user when they searched up that query. This is our target.</p><p>Let’s delve a little deeper here, <strong>what is gain</strong>?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/164/1*29L3gAvSVxXUefRqhpIzAg.png" /><figcaption>Source: Gemini 3</figcaption></figure><p>For a list of k recommended products, gain at position k is simply the relevance score between the query (that the products were recommended for), and that particular product. Your relevance score could be anything, as long as it represents how similar that query is to that product.</p><p>Naturally, <strong>cumulative gain</strong> (<strong>CG</strong>) would just be the sum of relevance scores of all the products making up your list of K recommendations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/184/1*OHSPfFm5s9N5ujz40_Sj5Q.png" /><figcaption>Source: Gemini 3</figcaption></figure><p>Up till now <strong>CG@K</strong> simply tells us the overall relevance of a list of recommended products, but it still doesn’t give us any information about the <em>relative ordering</em> of those products, you could interchange their positions in the list and the score would remain unchanged. This is why, it makes sense to add a scaling factor, that modifies the relevance score of each product based on where it ranks in the list.</p><p>Technically, you’re free to use any function as the rank varying scaling factor, depending on how much you prefer to scale the relevance scores. The vanilla implementation of this metric uses <em>log scaling</em>. We call this new term<strong> discounted cumulative gain</strong> (<strong>DCG</strong>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/249/1*RPpX3VRXIZK0eeUcYUAF4A.png" /><figcaption>Source: Gemini 3</figcaption></figure><p>Here <em>i </em>represents a rank in the list, which varies from the 1 (topmost) to k (last). The lower we go down the ranks in the list, the larger the scaling factor is, and the more the relevance score of that product is lowered for being at that rank. This DCG@K would represent your model’s <em>predicted list of recommendations</em>.</p><p>This makes sense right? If a score wants to represent the order of the products, each product at that rank should contribute proportionally to that score, with the top ranking products contributing the most!</p><p>However <strong>DCG@K</strong> is still just a number and to make it truly interpretable, <em>we get the products that your model was expected to show</em>, and we calculate the DCG@K for that ideal product list but with <em>each product at their desired rank</em>, and at their ideal relevance scores with the query.</p><p>This is your<strong> IDCG@K</strong>, and if you divide your <strong>DCG@K </strong>with this term<strong>, </strong>you have a <em>normalized term</em> ranging from 0 to 1, that represents how close your list of ordered predictions is to the actual list of desired recommendations. This is why we call it <strong>normalized DCG</strong> or <strong>NDCG</strong>!</p><p>What does this mean for us? The more relevant items appear in your predicted recommendations, and the better they are ranked relative to each other, the closer your <strong>NDCG@K</strong> score will be to <strong>1</strong>. Conversely, a completely random list of irrelevant products will result in a score much closer to <strong>0</strong>.</p><p><strong>Conclusion:</strong></p><p>No single number will tell you the truth. Recall might help show you the product coverage in your recommendations, and NDCG might show you the completeness and order of your recommended list of products, but together they are able to paint a much more complete picture of your model’s performance.</p><p>There are many more such metrics that can represent your model’s performance from a different set of lens, the trick is to pick a combination that can represent your model from every angle efficiently.</p><p>If you enjoyed this article, I’d suggest you check out this much more thorough blog by <a href="https://www.evidentlyai.com/ranking-metrics/evaluating-recommender-systems">https://www.evidentlyai.com/ranking-metrics/evaluating-recommender-systems</a> !</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cfdc4fb23c94" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[TLDS: Automatic Mixed Precision Training]]></title>
            <link>https://medium.com/@Shachary/tlds-automatic-mixed-precision-training-2b07c61cafdc?source=rss-f7d903fe5201------2</link>
            <guid isPermaLink="false">https://medium.com/p/2b07c61cafdc</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[gpu]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[optimization]]></category>
            <dc:creator><![CDATA[Shounak Acharya]]></dc:creator>
            <pubDate>Sat, 13 Dec 2025 12:41:15 GMT</pubDate>
            <atom:updated>2025-12-13T20:33:32.880Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pbD_HBhMY2lFzK2hYyaHdw.png" /><figcaption>Source: Nano Banana Pro</figcaption></figure><p>Deep Learning is all the rage nowadays. If you were a student like me, you too would’ve tried training/finetuning such models on the gpu of your laptop! And of course we’ve all been naive enough to try to train a heavy model (the bigger the better right?), only to run into painfully slow training iterations or worse, random memory crashes because the model was too big :’)</p><p>It’s okay, it’s a common occurence to feel strapped on memory, even when you’d be working with a meaty compute. GPU operations might be blazingly fast at computing matrix multiplications, but they are inherently memory bound, which is a fancy way to say that <em>they are bottlenecked much more easily due to memory unavailability</em>.</p><p>Is there a better way to utilize memory more efficiently while training heavy models? Yes! There’s a really neat solution that is popularly used to make life easier. I’m no expert, but the good thing is that it doesn’t take being one to grasp <strong>Mixed Precision Training</strong></p><p><strong>Preface</strong></p><p>Let’s say you have a model (for example, a small language model or SLM) with 500M weights that you can run locally . In all likelihood, it’s weights would be stored in a 32 bit format (FP32) when you download them from Huggingface. Quick maths says that the weights of this model alone would occupy 500M (weights) x 32 / 8 (size in bytes)= <strong>2GB</strong> of memory. <br>Let’s say we want to feed this model multiple inputs parallely in a single pass, so we set a <strong>batch size of 64</strong>.</p><p>In an attempt to train this model locally, the <strong>model weights</strong> (in FP32) would be loaded on the GPU RAM, along with their <strong>activations</strong> and <strong>gradients</strong>, all in FP32.</p><p>What are activations? They’re basically the intermediate outputs from all the weights and operations inside our LLM. They represent all the transformations made to our input text to get the output, and are required to be stored so that we can calculate the gradient for every weight during training. We need to store a gradient for every weight of the SLM.</p><p>For the sake of simplicity, let’s also say that we have as many activations as weights. This is a wildly incorrect assumption and the true number of activations depends on the internal architecture of your deep learning model.</p><p>Let’s also assume a training loop where the weights are updated by calculating the gradients using the loss calculated over a batch (mini batch gradient descent)</p><p>This means that while training, our GPU memory footprint is:<br><strong><em>Weights + Gradients + (Activations per input x Batch size)</em></strong><em> =</em></p><p><strong><em>2GB + 2GB + (2GB x 64)</em></strong></p><p>Huh? This means we require <strong>132 GB</strong> of GPU RAM just for a single forward + backpropagation pass while training our model! That’s not very ideal.</p><p>Let’s hold on to that thought for a minute..</p><p><strong>Introduction</strong></p><p>Automatic Mixed Precision is an idea that optimizes the GPU compute/memory utilized during training by restricting all the mathematical operations and memory footprint involved to a 16 bit format (FP16), rather than the traditional 32 bits (FP32)</p><p>That makes sense right? If every value is stored using 16 bits, then matrix multiplication is faster and the memory required to store them is halved! The catch is that you can’t simply cast them to 16 bits without a significant loss of information in the weights, and the gradients updating those weights.</p><p>Think about it, if the weights of an SLM and it’s calculated loss were very small, <br>the gradients would be even smaller, and trying to represent them in 16 bits (which can only represent a smaller range of numbers than 32 bits, especially decimals) could cause them to be reduced to an insignificantly small value, or even exceed that representable range, and underflow to 0!</p><p>Automatic Mixed Precision Training handles this constraint very cleverly.</p><p><strong>Algorithm</strong></p><p>Here’s the idea:</p><p><em>FP32: floating point 32 bit representation<br>FP16: floating point 16 bit representation</em></p><p>So,</p><ol><li>Let’s keep a copy of the model weights in FP32 and store it on the GPU RAM</li><li>For every pass, let’s<br>a. Convert the stored FP32 weights to FP16 and store those too on the RAM (sounds extra? stay with me here)<br>b. Now, convert the input to FP16, and do a forward pass on the entire model with FP16 weights. (The activations would be stored in FP16 too)<br>c. Get the output (in FP16), convert it to FP32. Calculate the loss (FP32)<br>d. Multiply this loss by a very large scaling factor S<br>e. Calculate the gradient for every weight in our model in FP16 via this scaled loss and activations in backpropagation<br>f. Convert these calculated gradients to FP32<br>g. Divide this gradient with the same scaling factor S we multiplied the loss with</li><li>Finally, update the model weights stored in step 1 with these gradients</li></ol><p>Note: The scaling factor is adaptive and modifies itself during training to avoid scaling any value to something outside the FP16/32 representable range.</p><p>If you’ve noticed, this algorithm tries to restrict only the matrix multiplications in FP16 (like getting the activations and the final output), but leaves the remaining operations in FP32 (the loss calculation)</p><p>The scaling operations ensures that the small value calculated for the loss, doesn’t cause the gradients to be reduced to zero due to the chain rule when they are operated in 16 bits.</p><p><strong>Results</strong></p><p>Okay but what implications would it have on our training setup discussed earlier?</p><p>Because of enabling mixed precision training, our new memory footprint would be:</p><p><strong><em>Weights (in 32 bits) + Weights (in 16 bits) + Gradients (in 16 bits) + (Activations per input in 16 bits x Batch Size)</em></strong> =</p><p><strong><em>2GB + (2GB / 2) + (2GB / 2) + ( (2GB / 2) x 64)</em></strong></p><p>Which makes it, drumroll please……… <strong>68GB ! </strong>That’s a<strong> 48.5% </strong>reduction in memory requirements!</p><p>Suddenly, training our SLM feels a lot more realistic and efficient. While the gradient updates are slightly different mathematically, the impact on the final model quality is negligible. This is super useful because it gives us the memory headroom to run larger batch sizes without risking those annoying crashes!</p><p>Integrating this to your training code is simple too, here is an example demonstrating the modifications in Pytorch:</p><pre># scales the loss and unscales gradients during mixed precision training<br>scaler = torch.cuda.amp.GradScaler() <br><br># (your training loop logic) <br><br># enables mixed precision training <br>with torch.autocast(device_type = &#39;cuda&#39;, dtype = torch.float16): <br>  loss = model.forward(inputs)<br><br># back propagating the scaled loss<br>scaler.scale(loss).backward()<br><br># updating the weights as explained in the algorithm<br>scaler.step(optimizer)<br><br># adaptively updating the scaling factor<br>scaler.update()</pre><p>You should still probably try training a smaller model on your laptop though :P</p><p><strong>Conclusion</strong></p><p>You’ll be assured that this blog covers Mixed Precision Training very well, though I did intentionally leave out some nuances to emphasize on the core intuition with minimal complexity.</p><p>If I was able to intuitively and exhaustively explain the idea behind Mixed Precision Training to you, then I recommend checking out this <a href="https://docs.nvidia.com/deeplearning/performance/mixed-precision-training/index.html">explainer blog</a> by Nvidia, it’s quite rigorous and informative :D</p><p>Happy learning!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2b07c61cafdc" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>