<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by sankho roy on Medium]]></title>
        <description><![CDATA[Stories by sankho roy on Medium]]></description>
        <link>https://medium.com/@sankhoroy?source=rss-ff631f66b7d4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*tDO95JqHjQfFrc5_BoFN_Q.jpeg</url>
            <title>Stories by sankho roy on Medium</title>
            <link>https://medium.com/@sankhoroy?source=rss-ff631f66b7d4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 07:14:03 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sankhoroy/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Wait! You Changed ReLU… Did You Change the Initialisation Too?”]]></title>
            <link>https://medium.com/@sankhoroy/wait-you-changed-relu-did-you-change-the-initialisation-too-8f1224c9cb91?source=rss-ff631f66b7d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f1224c9cb91</guid>
            <category><![CDATA[foundational-model]]></category>
            <category><![CDATA[ai-model-training]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[weight-initialization]]></category>
            <dc:creator><![CDATA[sankho roy]]></dc:creator>
            <pubDate>Thu, 01 Oct 2026 19:44:40 GMT</pubDate>
            <atom:updated>2026-10-01T19:44:40.334Z</atom:updated>
            <content:encoded><![CDATA[<h3>Wait! You Changed ReLU… Did You Change the Initialisation Too?” — The Story Behind Those Weird √(2/fan-in) looking Equations in Initialisers</h3><p><em>The hidden relationship between activation functions, variance propagation, and stable neural networks.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i5bFnGPP7Sh0EUZIn525gw.png" /><figcaption>Variance Propagation: Why changing activation needs changing initialisation?</figcaption></figure><p><em>I am changing the activation function.”</em></p><p>One scientist proudly clicks a button on the control computer.</p><p>Another scientist suddenly shouts,</p><p><strong><em>“Wait, wait!! Then we also need to change the initialization!”</em></strong></p><p>A third engineer, watching the giant machine labeled <strong>Neural Network</strong>, looks confused.</p><p><strong><em>“Why?”</em></strong></p><p>If you’ve ever built a neural network, you’ve probably been that third engineer.</p><p>After all, changing an activation function feels like changing a component inside the network. Why should the way we initialize the weights have anything to do with it?</p><p>As it turns out…</p><p><strong>Everything.</strong></p><p><em>“Wait! You changed ReLU…”</em></p><p>One engineer pauses.</p><p><em>“Did you change the initialization too?”</em></p><p>The room goes silent.</p><p>Another engineer looks puzzled.</p><p><em>“Why would changing an activation function affect the random numbers we use to initialize the weights?”</em></p><p>It’s a fair question.</p><p>Most deep learning tutorials simply hand us formulas like</p><ul><li><strong>1/fan-in</strong></li><li><strong>2/(fan-in + fan-out)</strong></li><li><strong>2/fan-in</strong></li></ul><p>and move on.</p><p>Nobody explains <strong>where these numbers come from</strong>.</p><p>They almost look like magic.</p><p>Or worse, like someone kept trying different equations until one happened to work.</p><p>But here’s the surprising part:</p><blockquote><strong><em>Every popular initialization method is derived from one simple idea — keep the signal alive as it flows through the network.</em></strong></blockquote><p>Let’s see why.</p><h3>Imagine Building a Giant Water Pipeline</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-GUg9i8OD4diHR0w-B7jNg.png" /><figcaption>pipeline analogy of neural networks</figcaption></figure><p>Think of a deep neural network as a series of water pipes.</p><p>Each layer receives water, modifies it, and passes it to the next layer.</p><p>The water is your <strong>information</strong>.</p><p>Every layer performs three operations:</p><ol><li>Multiply by weights (initialization)</li><li>Pass through an activation function</li><li>Forward the result to the next layer</li></ol><p>If every pipe preserves roughly the same amount of water, the information reaches the end of the network intact.</p><p>If every pipe leaks a little…</p><p>Eventually nothing reaches the output.</p><p>If every pipe amplifies the flow…</p><p>Soon the entire system bursts.</p><p>That “amount of water” is exactly what deep learning researchers measure using <strong>variance</strong>.</p><h3>The Goal Isn’t Large Signals</h3><p>Many beginners think the goal is to make neuron outputs as large as possible.</p><p>It’s actually the opposite.</p><p>The goal is simply:</p><blockquote><strong><em>Keep the signal approximately the same size across hundreds of layers.</em></strong></blockquote><p>Not shrinking.</p><p>Not exploding.</p><p>Just stable.</p><p>A variance near 1 throughout the network is usually a healthy target.</p><h3>The Master Equation: Variance Propagation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*boK-_PS9Bl98KzTVN9lwhg.png" /></figure><h3>Step 1. Forward Variance Propagation</h3><p>Consider a neuron with:</p><ul><li>n_in inputs (fan_in)</li><li>Inputs x_i</li><li>Weights w_i</li></ul><p>The pre-activation output is:</p><p>z = sum(w_i * x_i) for i = 1 to n_in</p><p>Assume:</p><ul><li>E[w_i] = 0, E[x_i] = 0 (Average value of input and weights are zero centered)</li><li>Weights and inputs are independent</li><li>All weights have variance sigma_w^2</li><li>All inputs have variance sigma_x^2</li></ul><p>Since the variables are independent:</p><p>Var(z) = sum(Var(w_i * x_i)) for i = 1 to n_in</p><p>Because Var(w_i * x_i) = Var(w_i) * Var(x_i), we obtain:</p><p>Var(z) = n_in * sigma_w^2 * sigma_x^2</p><p><strong>or:</strong> Var(z) = fan_in * Var(w) * Var(x)</p><p>This is the forward variance equation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8IqUL03RkLyPzzRJTrBHHg.png" /><figcaption>how these initializers are calculated?</figcaption></figure><h3>Step 2. Preserve Forward Signal</h3><p>To prevent activations from exploding or vanishing, we want:</p><p>Var(z) = Var(x)</p><blockquote>we do <strong>Var(z) = Var(x)</strong> by BatchNorm or Layernorm in LLMs</blockquote><p>Substituting:</p><p><strong>Var(x)</strong> = fan_in * Var(w) * <strong>Var(x)</strong></p><p>Canceling Var(x):</p><p>Var(w) = 1 / fan_in</p><p>This is the forward-pass requirement.</p><h3>Step 3. Backward Variance Propagation</h3><p>Now consider gradients flowing backward. Suppose:</p><ul><li>Gradient variance entering a layer is Var(delta_out)</li><li>The layer has fan_out outgoing connections</li></ul><p>Using the same independence assumptions, the gradient variance becomes:</p><p>Var(delta_in) = fan_out * Var(w) * Var(delta_out)</p><p>To preserve gradients:</p><p>Var(delta_in) = Var(delta_out)</p><p>Therefore:</p><p>Var(w) = 1 / fan_out</p><p>Now we have two different requirements:</p><ul><li>Forward pass: Var(w) = 1 / fan_in</li><li>Backward pass: Var(w) = 1 / fan_out</li></ul><p>Unless fan_in = fan_out, both cannot be satisfied exactly.</p><h3>Step 4. Xavier&#39;s Compromise</h3><p>Glorot and Bengio proposed balancing the forward and backward conditions by taking their compromise. Instead of preserving only one direction, choose a variance halfway between the two constraints:</p><blockquote>basically taking harmonic mean between two</blockquote><p>Var(w) = 2 / (fan_in + fan_out)</p><p>This is the Xavier (Glorot) initialization variance.</p><h3>Step 5. Convert Variance to Standard Deviation</h3><p>If weights are sampled from a normal distribution:</p><p>w ~ N(0, sigma^2)</p><p><strong>then:</strong> sigma = sqrt(Var(w)) = sqrt(2 / (fan_in + fan_out))</p><p><strong>So: </strong>w ~ N(0, 2 / (fan_in + fan_out))</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tXCTx5_r8W4ZpXJgoAkRTw.png" /><figcaption>how different initialisations are decided based on different activation function</figcaption></figure><h3>Step 6. Uniform Xavier Initialization</h3><p>For a uniform distribution U(-a, a), the variance is:</p><p>Var(U) = a^2 / 3</p><p>Set: a^2 / 3 = 2 / (fan_in + fan_out)</p><p>Hence: a = sqrt(6 / (fan_in + fan_out))</p><p>Therefore: w ~ U(-sqrt(6 / (fan_in + fan_out)), sqrt(6 / (fan_in + fan_out)))</p><h3>So Why All These Weird Initialisation Formulas?</h3><p>At this point, those strange-looking equations should start to feel a little less mysterious.</p><p>Imagine you have built a <strong>100-layer neural network</strong>.</p><p>You connect everything together and randomly initialize the weights.</p><p>The network starts passing information forward.</p><p>At the first layer, everything looks fine.</p><p>At the second, still fine.</p><p>But by the time the signal reaches layer 50, you discover one of two things:</p><p><strong>Either the signal has almost disappeared…</strong></p><p>or</p><p><strong>it has grown into something enormous.</strong></p><p>And the same problem happens in reverse when gradients travel backward.</p><p>This is the real problem initialization is trying to solve.</p><p>It is not trying to find the “perfect” weights.</p><p>It is simply trying to give the network a <strong>safe starting point</strong> where signals and gradients can travel through many layers without continuously shrinking or exploding.</p><p>That is why we started asking:</p><blockquote><strong><em>“How large should the weights be?”</em></strong></blockquote><p>For the forward pass, the answer depends on how many inputs are feeding a neuron — the <strong>fan-in</strong>.</p><p>For the backward pass, it depends on how many outputs are connected to it — the <strong>fan-out</strong>.</p><p>And suddenly we get equations such as: fan_in and fan_out</p><p>Not because someone invented arbitrary constants, but because they are trying to maintain the same <strong>statistical energy of the signal</strong> as it moves through the network.</p><p>Then Xavier comes along and says:</p><blockquote><strong><em>“We need both directions to behave reasonably. Let’s find a compromise.”</em></strong></blockquote><p>And we get: fan_in+fan_out</p><p>Everything is fine…</p><p>until <strong>ReLU walks in</strong>.</p><p>ReLU changes the story because it removes all negative activations.</p><p>Roughly speaking, half the signal gets switched off.</p><p>So now the initialization has to compensate for that loss.</p><h3>And That’s the Bigger Lesson</h3><p>So whenever you see a strange initialization formula in a deep-learning library, don’t think:</p><blockquote>“Why did someone come up with such a weird equation?”</blockquote><p>Think instead:</p><blockquote>“What is this formula trying to keep stable?”</blockquote><p>The activation function determines how the signal changes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BK-ka-oTPcBuB3BtUibGxg.png" /><figcaption>conclusion</figcaption></figure><p>The architecture determines how many connections contribute to that signal.</p><p>And initialization chooses the starting weight scale so the whole network can remain numerically well-behaved.</p><p>So the story is really:</p><p><strong>Change the network → change how signals behave → change the variance → sometimes change the initialization.</strong></p><p>That’s why <strong>ReLU came with He initialization</strong>, just as older activations inspired different initialization strategies.</p><blockquote><strong><em>Don’t let the signal die. Don’t let it explode. Give the neural network a stable place to start learning.</em></strong></blockquote><p>And perhaps that’s the easiest way to remember initialization:</p><p><strong>It is not about choosing random weights.</strong></p><p><strong>It is about choosing random weights that give information a fighting chance to survive the journey through the network.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f1224c9cb91" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Science Behind Million-Token Context Windows: RoPE, ALiBi, and YaRN]]></title>
            <link>https://medium.com/@sankhoroy/the-science-behind-million-token-context-windows-rope-alibi-and-yarn-f18e27d6beca?source=rss-ff631f66b7d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/f18e27d6beca</guid>
            <category><![CDATA[context-length]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[sankho roy]]></dc:creator>
            <pubDate>Sun, 12 Jul 2026 15:21:53 GMT</pubDate>
            <atom:updated>2026-07-13T14:57:03.463Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How RoPE, ALiBi, and YaRN quietly solved one of the biggest challenges in modern Transformers.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0KldZ2tBNDkHVXGvAhPZcQ.png" /><figcaption>evolution of positional embeddings</figcaption></figure><p>If you’ve ever wondered how a Large Language Model knows that <strong>“The cat chased the mouse”</strong> is different from <strong>“The mouse chased the cat,”</strong> you’re really asking a much deeper question:</p><blockquote><strong><em>How does a Transformer understand order?</em></strong></blockquote><p>Unlike recurrent neural networks, Transformers process all tokens simultaneously. They don’t naturally know which word comes first, second, or last.</p><p>Without positional information, these two sentences would look almost identical to the model.</p><pre>The cat chased the mouse.<br>The mouse chased the cat.</pre><p>The words are the same. Only their positions differ.</p><p>This seemingly simple problem has driven years of innovation in Transformer architectures. As language models evolved from processing a few hundred tokens to hundreds of thousands, positional encoding techniques evolved alongside them.</p><p>This is the story of that evolution.</p><h3>Chapter 1 — Absolute Position Embeddings</h3><p>The earliest Transformers simply assigned every position its own embedding.</p><pre>Position 0 → P0<br>Position 1 → P1<br>Position 2 → P2<br>...</pre><p>The input became</p><pre>[<br>x_i = token_i + position_i<br>]</pre><p>Simple.</p><p>Elegant.</p><p>But not scalable.</p><p>The model memorized positions rather than relationships.</p><p>A token at position 1000 looked completely unrelated to one at position 1001, even though they’re neighbors.</p><p>Even worse, if the model was trained on sequences of length 2048, it had absolutely no idea what position 2049 should look like.</p><p>Its world simply ended.</p><pre>import torch<br>import torch.nn as nn<br><br>batch_size = 2<br>seq_len = 8<br>d_model = 512<br>vocab_size = 50000<br><br># Token IDs<br>tokens = torch.randint(0, vocab_size, (batch_size, seq_len))<br><br># Embedding layers<br>token_embedding = nn.Embedding(vocab_size, d_model)<br>position_embedding = nn.Embedding(seq_len, d_model)<br><br># Position indices<br>positions = torch.arange(seq_len).unsqueeze(0)<br><br># Final transformer input<br>x = token_embedding(tokens) + position_embedding(positions)<br><br>print(x.shape)<br># (batch_size, seq_len, d_model)</pre><h3>Chapter 2 — RoPE: Turning Position into Geometry</h3><p>Researchers realized something profound.</p><p>Instead of storing positions…</p><p><strong>What if positions were rotations?</strong></p><p>Every query and key vector is divided into tiny two-dimensional planes.</p><p>Each pair rotates by an angle proportional to its position.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qR9yuDtbBW21jONaRNDrVA.png" /><figcaption>RoPE</figcaption></figure><p>Instead of adding information,</p><p>RoPE transforms the vectors themselves.</p><p>For a token at position <strong>m</strong>, RoPE rotates the <strong>Query</strong> vector according to its position:</p><pre>q_rot = R(m × θ) · q</pre><p>Similarly, for a token at position <strong>n</strong>, it rotates the <strong>Key</strong> vector:</p><pre>k_rot = R(n × θ) · k</pre><p>Here, <strong>R(θ)</strong> is the 2D rotation matrix:</p><pre>            |  cos(θ)   -sin(θ) |<br>R(θ)   = <br>            |  sin(θ)    cos(θ) |</pre><p>In simple terms, every pair of dimensions in the Query and Key vectors is rotated by an angle that depends on the token’s position. As the position increases, the rotation angle increases, allowing the Transformer to naturally encode relative distances between tokens.</p><p>Imagine every embedding as an arrow.</p><pre>Position 0 →</pre><pre>Position 1 ↗</pre><pre>Position 2 ↑</pre><pre>Position 3 ↖</pre><p>The farther apart two tokens are, the larger the angle between their vectors.</p><p>The beautiful mathematical result is</p><h3>One of the most elegant properties of RoPE is that the attention computation simplifies to:</h3><pre>(Rm · q)ᵀ (Rn · k) = qᵀ R((n − m) × θ) k</pre><p>Notice something remarkable.</p><p>The attention score depends only on the <strong>relative distance</strong> between two tokens:</p><pre>Relative Distance = (n − m)</pre><p>It does <strong>not</strong> depend on the absolute positions <strong>m</strong> or <strong>n</strong> individually.</p><p>This means two tokens separated by the same distance behave similarly, regardless of where they appear in the sequence. That’s why RoPE naturally captures <strong>relative positional information</strong>, making it far more effective than traditional absolute positional embeddings.</p><p>the relative distance.</p><p>Not absolute position.</p><p>That single observation made RoPE one of the most influential positional encoding techniques ever proposed.</p><p>Models like Llama, Qwen, Mistral, Gemma and many others rely on this elegant idea.</p><h3>Rope Code</h3><pre>import torch<br><br>def rotate_half(x):<br>    x1 = x[..., ::2]<br>    x2 = x[..., 1::2]<br><br>    return torch.stack(<br>        (-x2, x1),<br>        dim=-1<br>    ).flatten(-2)<br><br><br>def apply_rope(x, cos, sin):<br>    return x * cos + rotate_half(x) * sin<br><br><br>seq_len = 128<br>head_dim = 64<br><br>position = torch.arange(seq_len).float()<br><br>inv_freq = 1.0 / (<br>    10000 ** (<br>        torch.arange(0, head_dim, 2).float() / head_dim<br>    )<br>)<br><br>freq = torch.outer(position, inv_freq)<br><br>cos = torch.repeat_interleave(torch.cos(freq), 2, dim=-1)<br>sin = torch.repeat_interleave(torch.sin(freq), 2, dim=-1)<br><br>q = torch.randn(2, seq_len, head_dim)<br><br>q_rope = apply_rope(q, cos.unsqueeze(0), sin.unsqueeze(0))</pre><h3>But RoPE Had a Hidden Weakness</h3><p>Rotation angles continue increasing forever.</p><pre>0°<br>180°<br>360°<br>720°<br>1440°<br>2880°<br>...</pre><p>Eventually the vectors begin wrapping around the circle repeatedly.</p><p>After enough rotations, distant positions start looking confusingly similar.</p><p>If a model was trained on 8K tokens, asking it to process 64K tokens often caused a noticeable drop in quality.</p><p>The geometry remained beautiful.</p><p>Reality became messy.</p><h3>Chapter 3 — ALiBi: Maybe Rotation Isn’t Necessary</h3><p>While many researchers focused on improving rotational embeddings, another group asked a completely different question.</p><p>What if we never encoded positions inside embeddings at all?</p><p>Instead…</p><p>What if we simply encouraged attention to prefer nearby tokens?</p><p>This became ALiBi.</p><p>Attention with Linear Biases.</p><p>Instead of rotating vectors,</p><p>Instead of modifying the embeddings, <strong>ALiBi</strong> directly adjusts the attention scores.</p><p>The attention score becomes:</p><pre>Attention Score</pre><pre>           (qi · kj)<br>Score = ---------------  −  mh × (i − j)<br>             √d</pre><p>where:</p><ul><li><strong>mh</strong> = slope assigned to a particular attention head</li><li><strong>(i − j)</strong> = distance between the current token and the token being attended to</li></ul><p>The farther apart two tokens are, the larger the penalty applied to their attention score.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1Tndz9KwwYNzfAuHlfXSZg.png" /><figcaption>ALiBi</figcaption></figure><p>Notice what <strong>doesn’t</strong> change.</p><p>The Query and Key embeddings remain exactly the same.</p><p>Only the <strong>attention matrix</strong> is modified by adding a simple linear bias based on token distance.</p><p>Different attention heads use different slopes. Some heads learn to focus on nearby context, acting as <strong>local experts</strong>, while others apply a gentler penalty, allowing them to capture <strong>long-range dependencies</strong> across the sequence.</p><p>The idea is almost embarrassingly simple.</p><p>Yet this tiny modification enables models to generalise to sequence lengths far beyond those seen during training, making ALiBi one of the most elegant solutions for long-context Transformers.</p><p>Nearby tokens receive almost no penalty.</p><p>Faraway tokens receive larger penalties.</p><p>Suppose the current token is at position 10.</p><p>Nothing about the embeddings changes.</p><p>Only the attention matrix changes.</p><p>Different heads receive different slopes.</p><p>Some become local experts.</p><p>Others specialize in global context.</p><p>The idea is almost embarrassingly simple.</p><p>Yet it extrapolates surprisingly well to much longer contexts than the model ever saw during training.</p><blockquote>Sometimes, the simplest ideas are the most powerful.</blockquote><h3>Code: AliBi</h3><pre>import torch<br>import math<br><br>seq_len = 128<br>head_dim = 64<br><br>q = torch.randn(1, seq_len, head_dim)<br>k = torch.randn(1, seq_len, head_dim)<br><br># Standard attention scores<br>scores = q @ k.transpose(-2, -1)<br>scores /= math.sqrt(head_dim)<br><br># Distance matrix<br>position = torch.arange(seq_len)<br>distance = position[None, :] - position[:, None]<br><br># ALiBi slope (example)<br>slope = 0.25<br><br>alibi_bias = -slope * distance.abs()<br><br>scores += alibi_bias<br><br>attention = scores.softmax(dim=-1)</pre><h3>Chapter 4 — YaRN: Keeping RoPE Alive</h3><p>Once RoPE became the standard, researchers noticed an important limitation.</p><p>The model had only learned to rotate Query and Key vectors within its training context — say <strong>8K tokens</strong>.</p><p>Extending it to <strong>32K</strong> or <strong>128K</strong> tokens meant asking the model to perform rotations it had never seen before.</p><p>One surprisingly elegant solution was <strong>Position Interpolation (PI).</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OtlL0nCPN8N0O9m5iAE3gg.png" /><figcaption>position interpolation</figcaption></figure><p>Instead of changing the Transformer, researchers changed the <strong>positions</strong>.</p><p>Suppose an LLM was trained on <strong>8K tokens</strong>, but we now want it to process <strong>32K tokens</strong>.</p><p>Instead of using the original positions:</p><pre>0, 1, 2, 3, ..., 32767</pre><p>we compress them into the training range:</p><pre>0, 0.25, 0.5, 0.75, ..., 8191.75</pre><p>The scaling is remarkably simple:</p><pre>scaled_position =<br>original_position ×<br>(training_context / target_context)</pre><p>For an <strong>8K → 32K</strong> extension:</p><pre>scale = 8192 / 32768 = 0.25</pre><pre>scaled_position = original_position × 0.25</pre><p>Instead of applying RoPE at:</p><pre>R(m × θ)</pre><p>the model computes:</p><pre>R((scaled_position) × θ)</pre><p>Notice what happened.</p><blockquote>The Transformer is no longer forced to extrapolate beyond the rotation angles it learned during training. Instead, the entire sequence is <strong>compressed into a familiar positional space</strong>.</blockquote><p>Without Position Interpolation:</p><pre>Training (8K)</pre><pre>0° -------------------- 720°</pre><pre>Inference (32K)</pre><pre>0° ------------------------------------- 2880°</pre><p>With Position Interpolation:</p><pre>Training (8K)</pre><pre>0° -------------------- 720°</pre><pre>Inference (32K)</pre><pre>0° -------------------- 720°</pre><p>It was a beautifully simple idea.</p><p>Instead of teaching the model new positions, we simply <strong>mapped new positions onto familiar ones</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tQ9gYR_3NUSM8wfCIL_XPw.png" /><figcaption>Yarn</figcaption></figure><p>This became the foundation for many of today’s long-context techniques.</p><pre>Position<br><br>0 -------8K-------16K-------32K<br>Angles<br>0°------720°------1440°------2880°</pre><p>With YaRN</p><pre>Position<br><br>0 -------8K-------16K-------32K<br>Angles<br>0°------360°------500°-------700°</pre><p>The model no longer spins embeddings around the circle at extreme speeds.</p><p>Instead, rotations remain meaningful even for very long documents.</p><p>The elegance of RoPE remains.</p><p>The context window expands dramatically.</p><h3>Code: Yarn</h3><pre>import torch<br><br>seq_len = 32768<br>head_dim = 64<br><br># Compress positions<br>scale = 4.0<br>position = torch.arange(seq_len).float() / scale<br><br>inv_freq = 1.0 / (<br>    10000 ** (<br>        torch.arange(0, head_dim, 2).float() / head_dim<br>    )<br>)<br><br>freq = torch.outer(position, inv_freq)<br><br>cos = torch.cos(freq)<br>sin = torch.sin(freq)<br><br># Feed cos/sin into the same RoPE implementation</pre><h3>Three Different Philosophies</h3><p>What makes these methods fascinating isn’t only their mathematics.</p><p>It’s their philosophy.</p><p>RoPE says:</p><blockquote><em>Encode position by rotating vectors.</em></blockquote><p>ALiBi says:</p><blockquote><em>Don’t change vectors. Change attention instead.</em></blockquote><p>YaRN says:</p><blockquote><em>Keep rotations — but slow them down.</em></blockquote><p>Three completely different viewpoints.</p><p>Three successful solutions.</p><h3>Side-by-Side Comparison</h3><p>Imagine four engineers trying to teach a Transformer how to understand sequence order.</p><p>The first engineer hands every token a numbered badge and says, <strong>“Just remember where you are.”</strong> That’s <strong>Absolute Positional Embedding</strong>. It works well inside the training range, but when a token arrives at a position the model has never seen before, the badge means nothing.</p><p>The second engineer smiles and says, <strong>“Don’t memorize your position — rotate with it.”</strong> Every Query and Key vector spins a little more as its position increases. That’s <strong>RoPE</strong>. Instead of remembering absolute locations, the model learns relative distances through geometry, making it far better at understanding relationships between tokens.</p><p>A third engineer takes a completely different approach. <strong>“Why rotate anything?”</strong> they ask. <strong>“Just encourage the model to pay a little less attention to distant tokens.”</strong> That’s <strong>ALiBi</strong>. Rather than changing the embeddings, it quietly adjusts the attention scores with a simple linear penalty. The farther away a token is, the smaller its influence — unless its content is important enough to overcome the bias.</p><p>Finally, the fourth engineer looks at RoPE and says, <strong>“Your idea is brilliant, but you’re spinning too fast.”</strong> Instead of replacing RoPE, they slow its rotations as sequences grow longer. That’s <strong>YaRN</strong>. The geometry stays the same, but the rotations remain meaningful even across hundreds of thousands of tokens, allowing today’s long-context language models to read books, codebases, and lengthy conversations without losing their sense of position.</p><p>Four engineers. Four philosophies. One shared goal: helping Transformers understand not just <strong>what</strong> the tokens are, but <strong>where</strong> they belong.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8nq3unoEhSGGiIggp6dvWQ.png" /><figcaption>position embedding ideas in a nutshell</figcaption></figure><h3>The Bigger Lesson</h3><p>One of the most fascinating aspects of modern AI is that progress doesn’t always come from larger models.</p><p>Sometimes it comes from a tiny mathematical insight.</p><p>A rotation matrix.</p><p>A linear bias.</p><p>A scaling function.</p><p>Each occupies only a few lines of code.</p><p>Yet these ideas determine whether a language model can understand a paragraph, a chapter, an entire book — or even an entire codebase.</p><blockquote>The next breakthrough in AI may not be another trillion parameters.</blockquote><blockquote>It may simply be another very small change that changes everything.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f18e27d6beca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The ₹300 Crore Question Every Boardroom Asks — And Why Traditional Marketing Mix Models Can No…]]></title>
            <link>https://medium.com/@sankhoroy/the-300-crore-question-every-boardroom-asks-and-why-traditional-marketing-mix-models-can-no-a35c73dd8525?source=rss-ff631f66b7d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/a35c73dd8525</guid>
            <category><![CDATA[marketing-mix-modeling]]></category>
            <category><![CDATA[bayesian-machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[probabilistic-programming]]></category>
            <dc:creator><![CDATA[sankho roy]]></dc:creator>
            <pubDate>Sat, 11 Jul 2026 18:19:53 GMT</pubDate>
            <atom:updated>2026-07-15T03:41:44.720Z</atom:updated>
            <content:encoded><![CDATA[<h3>The ₹300 Crore Question Every Boardroom Asks — And Why Traditional Marketing Mix Models Can No Longer Answer It</h3><p><em>“Half the money I spend on advertising is wasted; the trouble is I don’t know which half.”</em><br> — John Wanamaker</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*55LTmAucC9Xj2VS9En-biQ.png" /><figcaption>MMM using Bayesian Neural Nets</figcaption></figure><p>More than a century has passed since Wanamaker said those words.</p><p>Companies now collect petabytes of customer data. Every click, impression, purchase, store visit, and campaign can be tracked. AI can generate images, write code, and even reason.</p><p>Yet, in boardrooms across the world, one question still brings the room to silence.</p><h3>Monday. 8:30 AM. Quarterly Business Review.</h3><ul><li>The CEO projects a dashboard on the screen.</li><li>Revenue has grown <strong>8.4%</strong>.</li><li>Marketing spend has increased by <strong>37%</strong>.</li><li>Customer acquisition cost has climbed.</li><li>Market share has barely moved.</li></ul><p>The CFO leans forward and asks a simple question.</p><blockquote><strong><em>“Which ₹1 we spent last quarter generated the highest return?”</em></strong></blockquote><p>The Marketing team opens campaign dashboards.</p><p>The Sales team believes stronger customer relationships drove conversions.</p><p>Operations points to improved product availability.</p><p>Finance questions the rising acquisition cost.</p><p>Product argues that the latest feature release increased demand.</p><p>Everyone has data.</p><p>Everyone has convincing charts.</p><p>Everyone has a different answer.</p><p>Three months later, a competitor with a <strong>smaller marketing budget</strong> reports higher profits.</p><p>They didn’t discover a magical advertising platform.</p><p>They didn’t hire more salespeople.</p><p>They simply became better at answering one question:</p><blockquote><strong><em>What truly drives business growth — and how certain are we about it?</em></strong></blockquote><p>That is the problem <strong>Marketing Mix Modeling (MMM)</strong> was built to solve.</p><p>And today, Bayesian AI is transforming how that problem is approached.</p><h3>The Hardest Business Problem Isn’t Collecting Data</h3><p>It’s Connecting Cause and Effect.</p><p>Imagine a global consumer electronics company launching a new smartphone.</p><p>Over one quarter, it invests:</p><p><em>ChannelQuarterly Spend</em></p><p><em>Television₹120 Cr</em></p><p><em>Search Advertising₹65 Cr</em></p><p><em>Social Media₹42 Cr</em></p><p><em>Influencer Campaigns₹18 Cr</em></p><p><em>Retail Promotions₹75 Cr</em></p><p><em>Trade Discounts₹55 Cr</em></p><p>At the end of the quarter:</p><ul><li>Sales increase by <strong>11%</strong></li><li>Online traffic increases by <strong>38%</strong></li><li>Store visits increase by <strong>14%</strong></li></ul><p>The obvious question becomes:</p><blockquote><strong><em>Which investments actually created those additional sales?</em></strong></blockquote><p>Unfortunately…</p><p>Customers don’t buy because of a single advertisement.</p><p>A typical customer journey looks more like this:</p><p>Week 1</p><p>📺 Watches a television advertisement.</p><p>↓</p><p>Week 2</p><p>🔍 Searches Google for product reviews.</p><p>↓</p><p>Week 3</p><p>📱 Sees Instagram advertisements.</p><p>↓</p><p>Week 4</p><p>🎥 Watches YouTube reviews.</p><p>↓</p><p>Week 5</p><p>🏪 Visits a retail store.</p><p>↓</p><p>Week 6</p><p>💳 Purchases during a festive promotion.</p><p>So who deserves the credit?</p><p>Television?</p><p>Search?</p><p>Social Media?</p><p>Retail?</p><p>Discounts?</p><p>The answer is…</p><p><strong>All of them.</strong></p><p>But not equally.</p><p>Separating those contributions is the goal of Marketing Mix Modeling.</p><h3>What Exactly Is Marketing Mix Modeling?</h3><h4>(This is for Data Science or Coders to read till “Where Traditional MMM Starts Breaking Down” others can skip)</h4><p>Marketing Mix Modeling (MMM) is a statistical framework that estimates how different business activities contribute to revenue or other business outcomes.</p><p>Think of total sales as a puzzle.</p><p>Many pieces come together:</p><ul><li>Television advertising</li><li>Digital campaigns</li><li>Search marketing</li><li>Pricing</li><li>Promotions</li><li>Distribution</li><li>Seasonality</li><li>Economic conditions</li><li>Competitor activity</li></ul><p>Mathematically, a simplified version looks like:</p><p>[<br>Sales =<br>TV +<br>Digital +<br>Search +<br>Promotions +<br>Seasonality +<br>Price +<br>Random\ Noise<br>]</p><p>The challenge isn’t writing this equation.</p><p>The challenge is estimating how much each component truly contributed.</p><p>And that’s much harder than it sounds.</p><h3>Why Traditional Regression Isn’t Enough</h3><p>Suppose your company spends:</p><ul><li>₹10 Crores on TV this week</li></ul><p>Will every customer buy this week?</p><p>Of course not.</p><p>Some customers remember the advertisement for days.</p><p>Others remember it for weeks.</p><p>Some forget immediately.</p><p>Advertising behaves much like dropping a stone into a lake.</p><p>The splash happens instantly.</p><p>The ripples continue long afterward.</p><p>Traditional MMM handles this using something called <strong>Adstock</strong>.</p><h3>Adstock: Because Advertising Has Memory</h3><p>Not every advertisement creates an immediate purchase.</p><p>Advertising leaves a memory.</p><p>A television commercial seen today may influence purchases over several weeks.</p><p>Traditional MMM models this using the Adstock equation:</p><p>Imagine this campaign:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MMmr561SdNB2no46FTvJFw.png" /></figure><p>Even though spending decreases,</p><p>the advertising effect remains.</p><p>Exactly like brand recall.</p><h3>But There’s Another Problem…</h3><p>Suppose you double your advertising budget.</p><p>Will sales double?</p><p>Almost never.</p><p>Initially,</p><p>every additional ₹1 reaches many new customers.</p><p>Eventually,</p><p>most potential customers have already seen the campaign.</p><p>The return begins to flatten.</p><p>This phenomenon is called <strong>Diminishing Returns</strong>.</p><p>Traditional MMM captures this using a <strong>Reach Function</strong> (often Hill or Logistic curves).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ax-ia8boZEOSQm0m5zuXFw.png" /></figure><p>Imagine spending on digital advertising:</p><p><em>SpendReach</em></p><p><em>₹10 Cr10 Million</em></p><p><em>₹20 Cr18 Million</em></p><p><em>₹30 Cr24 Million</em></p><p><em>₹40 Cr27 Million</em></p><p>Notice something interesting.</p><p>Spending increased <strong>4×</strong>.</p><p>Reach increased only <strong>2.7×</strong>.</p><p>The curve starts flattening.</p><p>Traditional MMM models this saturation before estimating sales contribution.</p><p>People can only watch an advertisement once or a few times before nearly everyone in the target audience has seen it. Beyond that point, increasing spend mostly repeats exposures instead of finding many new people.</p><p>A reach function captures this saturation.</p><h3>1. Adstock (Advertising Memory Effect)</h3><p>Advertising doesn’t disappear after one week. Its effect decays gradually.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hfmjxv3ccU9t3X1GTMOEPA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2f8dqFlPjm93lvo2bW6-4A.png" /><figcaption>Adstock effect</figcaption></figure><p><strong>Interpretation</strong></p><ul><li>Week 1: Impact = 100</li><li>Week 2: Consumers still remember the ad → Impact = 50</li><li>Week 3: Memory fades further → 25</li><li>Eventually the effect approaches zero.</li></ul><h4>Common Reach Function</h4><p>People can only watch an advertisement once or a few times before nearly everyone in the target audience has seen it. Beyond that point, increasing spend mostly repeats exposures instead of finding many new people.</p><p>A reach function captures this saturation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q-P5O64vdzfSMLYDItPRCA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wOXjkH9cSdZA7MxvYNMvjg.png" /><figcaption>Reach(saturation curve)</figcaption></figure><p><strong>Interpretation</strong></p><ul><li>Early spending produces large gains.</li><li>Later spending mostly shows the same ads to people who have already seen them.</li><li>The curve approaches a maximum audience.</li></ul><h3>The Traditional MMM Pipeline</h3><p>For decades, most Marketing Mix Models followed roughly this process.</p><pre>Marketing Spend<br>        │<br>        ▼<br>     Adstock<br>        │<br>        ▼<br> Reach Function<br>        │<br>        ▼<br> Linear Regression<br>        │<br>        ▼<br> Channel Contribution</pre><p>This framework transformed marketing analytics.</p><p>It was a major leap beyond intuition.</p><p>But business has changed dramatically.</p><p>Today’s customer journeys are more complex.</p><p>Marketing channels interact constantly.</p><p>Consumer behavior changes faster than ever.</p><p>And that’s where traditional MMM begins to struggle.</p><blockquote>Basically marketers what they essentially do is to forecast using multivariate regression model but also account for some extra factors in modelling. We also do in LSTM or RNN we make NN models to cater time series or CNN to cater image models. basically adding some fixed formula(inductive bias) other than learable parameters.</blockquote><h3>Where Traditional MMM Starts Breaking Down</h3><h3>1. One Decay Doesn’t Fit Everyone</h3><p>Traditional MMM often assumes a fixed advertising memory.</p><p>Reality doesn’t.</p><p>A festive campaign behaves differently from a product launch.</p><p>Luxury products have different recall than FMCG products.</p><p>Different cities respond differently.</p><p>Customer memory isn’t fixed.</p><h3>2. It Produces One Number…</h3><p>Reality Isn’t One Number.</p><p>Traditional regression might report:</p><p>TV Contribution = <strong>₹82 Crores</strong></p><p>Finished.</p><p>But how certain is that estimate?</p><p>Could the real contribution be ₹75 Crores?</p><p>₹90 Crores?</p><p>₹68 Crores?</p><p>The model doesn’t tell you.</p><p>Yet executives make multi-million-dollar decisions based on it.</p><h3>3. Marketing Channels Are Correlated</h3><p>When companies increase TV spending,</p><p>they often increase Digital spending too.</p><p>Regression struggles to separate highly correlated variables.</p><p>Credit gets distributed inconsistently.</p><p>This is one reason why MMM results sometimes change dramatically between model updates.</p><h3>4. Business Knowledge Is Ignored</h3><p>Experienced marketers know things data alone cannot capture.</p><p>For example:</p><ul><li>Television usually creates stronger long-term brand effects than Display Ads.</li><li>Search performs better during high purchase intent.</li><li>Promotions work differently during festivals.</li></ul><p>Traditional regression starts with a blank slate.</p><p>It cannot naturally incorporate expert knowledge.</p><h3>Then Bayesian Thinking Changed Everything</h3><p>Traditional statistics asks:</p><blockquote><strong><em>“What is the coefficient?”</em></strong></blockquote><p>Bayesian statistics asks:</p><blockquote><strong><em>“Given everything we know, what values are most plausible?”</em></strong></blockquote><p>At first glance,</p><p>that sounds like a subtle difference.</p><p>In reality,</p><p>it completely changes how business decisions are made.</p><h3>A Weather Forecast Explains Bayesian Thinking Better Than Mathematics</h3><p>Imagine tomorrow’s weather.</p><p>Traditional regression would say:</p><blockquote><em>Tomorrow’s temperature will be </em><strong><em>32°C.</em></strong></blockquote><p>Done.</p><p>Bayesian forecasting says:</p><ul><li>70% probability: <strong>31–33°C</strong></li><li>20% probability: <strong>34°C</strong></li><li>10% probability: Rain with lower temperatures</li></ul><p>Which forecast would you trust before planning an outdoor event?</p><p>Exactly.</p><p>Business decisions deserve the same treatment.</p><p>For more insight read my article “We Are Not Astrologers, We Are Machine Learning Scientists” in medium(link below).</p><p><a href="https://medium.com/@sankhoroy/we-are-not-astrologers-we-are-machine-learning-scientists-519bca1d8859">“</a> We Are Not Astrologers, We Are Machine Learning Scientists <a href="https://medium.com/@sankhoroy/we-are-not-astrologers-we-are-machine-learning-scientists-519bca1d8859">” https://medium.com/@sankhoroy/we-are-not-astrologers-we-are-machine-learning-scientists-519bca1d8859</a></p><h3>Traditional MMM Says</h3><pre>TV ROI = 3.2</pre><p>Finished.</p><h3>Bayesian MMM Says</h3><pre>TV ROI</pre><pre>Most Likely = 3.2</pre><pre>95% Probability Interval</pre><pre>2.8 — 3.6</pre><pre>Probability TV is the best channel</pre><pre>81%</pre><pre>Probability Search performs better</pre><pre>16%</pre><pre>Probability Social performs better</pre><pre>3%</pre><p>Now decision makers understand not only <strong>what</strong> is likely,</p><p>but <strong>how confident</strong> they should be before investing another ₹100 Crores.</p><p>That is a completely different conversation.</p><h3>Enter Pyro: Bringing Bayesian AI to Marketing Mix Modeling</h3><p>This is where <strong>Pyro</strong>, built on top of PyTorch, enters the picture.</p><p>Instead of treating model parameters as fixed numbers,</p><p>Pyro treats them as <strong>probability distributions</strong>.</p><p>Rather than saying:</p><pre>TV coefficient = 0.42</pre><p>Pyro estimates something closer to:</p><pre>TV coefficient</pre><pre>Most likely = 0.42</pre><pre>95% Credible Interval</pre><pre>0.35 – 0.49</pre><p>Suddenly,</p><p>uncertainty becomes measurable.</p><p>Instead of hiding uncertainty,</p><p>the model quantifies it.</p><p>That makes forecasts far more useful for strategic planning.</p><h3>Why This Matters Beyond Marketing</h3><p>Marketing isn’t the only team making allocation decisions.</p><p>The same philosophy applies everywhere.</p><p>Operations decides:</p><ul><li>How much inventory to build.</li></ul><p>Finance decides:</p><ul><li>Where capital should be invested.</li></ul><p>Supply Chain decides:</p><ul><li>How much stock each warehouse should receive.</li></ul><p>Sales decides:</p><ul><li>Which territories deserve additional investment.</li></ul><p>Every one of these decisions contains uncertainty.</p><p>Bayesian models don’t eliminate uncertainty.</p><p>They embrace it.</p><blockquote>Traditional MMM vs Bayesian MMM</blockquote><blockquote>Traditional Market Mix Modeling (MMM) estimates a single value for each marketing channel’s impact and relies <strong>on fixed model assumptions</strong>. It is often sensitive to correlated variables, makes it difficult to incorporate business expertise, and produces point predictions with limited insight into the uncertainty behind those estimates.</blockquote><blockquote>Bayesian Market Mix Modeling (MMM) treats <strong>model parameters as probability distributions rather than fixed values</strong>. It naturally <strong>incorporates prior business knowledge, explicitly models uncertainty from the data</strong>, is more robust to multicollinearity through probabilistic inference, and provides confidence intervals for every estimate, prediction, and business recommendation.</blockquote><h3>The Most Important Shift Isn’t Mathematical</h3><p>It’s Philosophical.</p><p>For decades,</p><p>business analytics focused on finding <strong>the answer</strong>.</p><p>Modern AI increasingly focuses on understanding <strong>the uncertainty around the answer.</strong></p><p>That distinction may sound academic.</p><p>It isn’t.</p><p>When companies allocate hundreds of crores every year,</p><p>confidence matters just as much as prediction.</p><h3>Looking Ahead</h3><p>Artificial Intelligence is changing many industries.</p><p>But one of its most profound impacts isn’t generating text or images.</p><p>It’s helping organizations make <strong>better decisions under uncertainty.</strong></p><p>Marketing Mix Modeling is evolving from a deterministic statistical exercise into a probabilistic decision engine.</p><p>Frameworks like Pyro make Bayesian inference practical.</p><p>Cloud computing makes it scalable.</p><p>Modern data pipelines make it deployable.</p><p>The companies adopting these approaches today are unlikely to spend dramatically more on marketing.</p><p>They’ll simply allocate their budgets with greater confidence.</p><p>And over time,</p><p>that difference compounds.</p><h3>Final Thoughts</h3><p>Twenty years ago, competitive advantage often came from having a larger marketing budget.</p><p>Today, nearly every company has access to similar advertising platforms, similar customer data, and similar dashboards.</p><p>The real advantage is no longer <strong>collecting data</strong>.</p><p>It’s <strong>making better decisions with the same data.</strong></p><p>Traditional Marketing Mix Modeling tells you what probably happened.</p><p>Bayesian Marketing Mix Modeling tells you <strong>how confident you should be before making the next decision.</strong></p><p>In a world where every strategic decision carries uncertainty,</p><p>that isn’t just a better statistical model.</p><p>It is a better way to run a business.</p><h3><strong>Now Coding part</strong></h3><p>Full code is here <a href="https://github.com/sankhoroy/my_medium_articles/blob/main/marketing_mix_modelling_bayesian_modelling_probabilistic_ml.py">https://github.com/sankhoroy/my_medium_articles/blob/main/marketing_mix_modelling_bayesian_modelling_probabilistic_ml.py</a></p><p>(go through code and if any issues found say in github. Thanks for time.)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a35c73dd8525" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We Are Not Astrologers, We Are Machine Learning Scientists]]></title>
            <link>https://medium.com/@sankhoroy/we-are-not-astrologers-we-are-machine-learning-scientists-519bca1d8859?source=rss-ff631f66b7d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/519bca1d8859</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[probabilistic-programming]]></category>
            <category><![CDATA[bayesian-neural-network]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[sankho roy]]></dc:creator>
            <pubDate>Thu, 25 Jun 2026 18:28:50 GMT</pubDate>
            <atom:updated>2026-06-25T18:28:50.027Z</atom:updated>
            <content:encoded><![CDATA[<h4>Why Predicting a Number Is Easy, but Predicting Uncertainty Is Science</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*46QJBdTNPhkP-1G6zDZ12A.png" /><figcaption>AI astrologer vs AI scientist</figcaption></figure><p>Imagine you visit an astrologer.</p><p>You ask:</p><p><em>“Will I get a promotion next year?”</em></p><p>The astrologer closes their eyes and says:</p><p><em>“Yes. Absolutely.”</em></p><p>No explanation.<br>No confidence interval.<br>No uncertainty.</p><p>Just one number.</p><p>Surprisingly, most machine learning models behave exactly like this.</p><h4>The Problem with Traditional Neural Networks</h4><p>Suppose we want to predict house prices.</p><p>Inputs:</p><ul><li>Area = 2000 sq ft</li><li>Bedrooms = 3</li><li>Age = 10 years</li></ul><p>Output:</p><ul><li>House Price</li></ul><p>A traditional neural network learns a function:</p><p><em>Price = f(X)</em></p><p>After training, it might predict:</p><pre>Predicted Price = $500,000</pre><p>Sounds good.</p><p>But should we trust it?</p><p>What if:</p><ul><li>training data is limited</li><li>the house is in a new neighborhood</li><li>market conditions changed</li></ul><p>The neural network( even “LLM” )still confidently says:</p><pre>$500,000</pre><p>It never says:</p><pre>I am not sure.</pre><p>This is dangerous.</p><h3>Scientists Don’t Predict Like That</h3><p>Suppose a weather scientist predicts tomorrow’s temperature.</p><p>They don’t say:</p><pre>Tomorrow = 30°C</pre><p>Instead they say:</p><pre>30°C ± 3°C</pre><p>Or:</p><pre>80% chance temperature is between 27°C and 33°C</pre><p>Scientists acknowledge uncertainty.</p><p>Machine learning models should too.</p><h3>A Story About Weight Values</h3><p>Consider the simplest regression model.</p><p>y = wx + b</p><p>Suppose after training we get:</p><pre>w = 2.5<br>b = 10</pre><p>Traditional machine learning says:</p><p>These are the correct values.</p><p>But are they really?</p><p>Maybe:</p><pre>w = 2.4</pre><p>works almost equally well.</p><p>Maybe:</p><pre>w = 2.6</pre><p>is also plausible.</p><p>There may be thousands of good solutions.</p><p>Why pretend only one is correct?</p><h3>Enter Bayesian Thinking</h3><p>Bayesian statistics says:</p><blockquote><em>Don’t learn one weight.</em></blockquote><p>Learn a probability distribution over weights.</p><p>Instead of:</p><pre>w = 2.5</pre><p>we learn:</p><pre>w ~ Normal(2.5, 0.2)</pre><p>Meaning:</p><ul><li>most likely around 2.5</li><li>but could be 2.3</li><li>could be 2.7</li></ul><p>Now the model admits uncertainty.</p><p>This is much closer to how real scientists think.</p><h3>From Neural Network to Bayesian Neural Network</h3><p>Traditional Neural Network:</p><pre>Input<br>  ↓<br>Linear Layer<br>  ↓<br>ReLU<br>  ↓<br>Linear Layer<br>  ↓<br>Output</pre><p>Weights:</p><pre>W1 = fixed number<br>W2 = fixed number</pre><p>Bayesian Neural Network:</p><pre>Input<br>  ↓<br>Distribution of W1<br>  ↓<br>ReLU<br>  ↓<br>Distribution of W2<br>  ↓<br>Output Distribution</pre><p>Weights become random variables.</p><pre>W1 ~ Normal(0,1)<br>W2 ~ Normal(0,1)</pre><p>Instead of learning one network:</p><pre>Network #1</pre><p>we effectively learn millions of possible networks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sxWPSlkgSGNVYC-LZFAwsg.png" /><figcaption>what it means to be probabilistic neural net</figcaption></figure><h3>Example 1: House Price Prediction</h3><p>full code available here: <a href="https://github.com/sankhoroy/my_medium_articles/blob/main/bayesian_nn_based_house_price_prediction.py">https://github.com/sankhoroy/my_medium_articles/blob/main/bayesian_nn_based_house_price_prediction.py</a></p><p>Suppose the input house has:</p><pre>Area = 2000 sq ft</pre><p>Traditional Neural Network:</p><pre>Price = $500,000</pre><p>Bayesian Neural Network:</p><pre>Mean Price = $500,000</pre><pre>95% Confidence Interval:<br>[$450,000 , $550,000]</pre><p>Now we know uncertainty.</p><p>This is much more useful.</p><p>A bank can make safer lending decisions.</p><p>An investor can estimate risk.</p><p>A scientist can understand confidence.</p><h3>Why Bayesian Neural Networks Are Powerful</h3><p>They answer two questions.</p><p>Question 1:</p><pre>What is the prediction?</pre><p>Question 2:</p><pre>How certain are we?</pre><blockquote>Traditional neural networks answer only Question 1.</blockquote><blockquote>Bayesian neural networks answer both.</blockquote><h3>Converting a Neural Network Regression Problem</h3><p>Let’s start with a standard PyTorch regression model.</p><pre>import torch<br>import torch.nn as nn<br><br>class Regressor(nn.Module):<br>    def __init__(self):<br>        super().__init__()<br>        self.net = nn.Sequential(<br>            nn.Linear(1,32),<br>            nn.ReLU(),<br>            nn.Linear(32,1)<br>        )<br>    def forward(self,x):<br>        return self.net(x)</pre><p>Prediction:</p><pre>y_pred = model(x)</pre><p>Single output.</p><p>No uncertainty.</p><h3>Bayesian Version Using Pyro</h3><p>Pyro is Uber’s probabilistic programming framework built on top of PyTorch.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M2RTZaQ6OT0FbvxxAVKWiw.png" /></figure><p>Instead of fixed weights:</p><pre>weight = parameter</pre><p>we define:</p><pre>weight ~ distribution</pre><h3>Bayesian Neural Network</h3><p>full code available here : <a href="https://github.com/sankhoroy/my_medium_articles/blob/main/bayesian_nn_based_house_price_prediction.py">https://github.com/sankhoroy/my_medium_articles/blob/main/bayesian_nn_based_house_price_prediction.py</a></p><pre>import torch<br>import pyro<br>import pyro.distributions as dist<br><br>def model(x, y=None):<br>    w1 = pyro.sample(<br>        &quot;w1&quot;,<br>        dist.Normal(<br>            torch.zeros(1,32),<br>            torch.ones(1,32)<br>        ).to_event(2)<br>    )<br>    b1 = pyro.sample(<br>        &quot;b1&quot;,<br>        dist.Normal(<br>            torch.zeros(32),<br>            torch.ones(32)<br>        ).to_event(1)<br>    )<br>    hidden = torch.relu(x @ w1 + b1)<br>    w2 = pyro.sample(<br>        &quot;w2&quot;,<br>        dist.Normal(<br>            torch.zeros(32,1),<br>            torch.ones(32,1)<br>        ).to_event(2)<br>    )<br>    b2 = pyro.sample(<br>        &quot;b2&quot;,<br>        dist.Normal(<br>            torch.zeros(1),<br>            torch.ones(1)<br>        ).to_event(1)<br>    )<br>    mean = hidden @ w2 + b2<br>    sigma = 5.0<br>    with pyro.plate(&quot;data&quot;, len(x)):<br>        pyro.sample(<br>            &quot;obs&quot;,<br>            dist.Normal(mean.squeeze(), sigma),<br>            obs=y<br>        )</pre><p>Notice the difference.</p><p>We never declare:</p><pre>w = 2.5</pre><p>Instead:</p><pre>w ~ Normal(...)</pre><p>The model reasons over many possible weights.</p><h3>Training with Variational Inference</h3><p>Exact Bayesian inference is expensive.</p><p>Pyro uses Variational Inference.</p><p>Think of it as:</p><pre>True posterior<br>P(weights | data)<br>≈<br>Approximate posterior<br>Q(weights)</pre><p>Training minimizes:</p><pre>Difference between P and Q</pre><p>using gradient descent.</p><h3>Training Code</h3><pre>from pyro.infer import SVI<br>from pyro.infer import Trace_ELBO<br>from pyro.optim import Adam<br><br>optimizer = Adam({&quot;lr&quot;:0.01})<br>svi = SVI(<br>    model,<br>    guide,<br>    optimizer,<br>    loss=Trace_ELBO()<br>)<br>for step in range(5000):<br>    loss = svi.step(x_train, y_train)<br>    if step % 500 == 0:<br>        print(loss)</pre><p>Looks very similar to deep learning.</p><p>But now we’re learning probability distributions.</p><h3>Prediction Time</h3><p>Traditional Neural Network:</p><pre>One forward pass</pre><p>Bayesian Neural Network:</p><pre>Many forward passes</pre><p>Each pass samples new weights.</p><pre>Prediction #1 = 495000<br>Prediction #2 = 505000<br>Prediction #3 = 490000<br>Prediction #4 = 520000</pre><p>Eventually:</p><pre>Mean = 502000</pre><pre>Std = 12000</pre><p>Now we know both:</p><ul><li>expected value</li><li>uncertainty</li></ul><h3>Why This Matters</h3><p>Imagine predicting:</p><ul><li>stock prices</li><li>medical diagnosis</li><li>autonomous driving</li><li>aircraft failures</li><li>loan defaults</li></ul><p>A wrong prediction can be expensive.</p><p>Sometimes uncertainty is more important than the prediction itself.</p><p>If a model says:</p><pre>Probability of cancer = 51%</pre><p>you would also want to know:</p><pre>How confident are you?</pre><p>Bayesian models provide that answer.</p><h3>The Real Difference</h3><p>Traditional Neural Networks say:</p><blockquote><em>“The answer is 500.”</em></blockquote><p>Bayesian Neural Networks say:</p><blockquote><em>“The answer is probably around 500, but there is a reasonable chance it lies between 450 and 550.”</em></blockquote><blockquote>One behaves like an astrologer.</blockquote><blockquote>The other behaves like a scientist.</blockquote><h3>Final Thoughts</h3><p>Machine learning is often described as prediction.</p><p>But real science is not just prediction.</p><p>Science is prediction with quantified uncertainty.</p><p>The future is inherently uncertain.</p><p>Data is noisy.</p><p>Measurements are imperfect.</p><p>Models are incomplete.</p><p>Bayesian Neural Networks embrace this reality instead of hiding it.</p><p>That is why Bayesian machine learning feels less like fortune-telling and more like science.</p><blockquote>We are not astrologers.</blockquote><blockquote>We are machine learning scientists.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=519bca1d8859" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[If Autoencoders Can Compress Data, Why Did We Invent Variational Autoencoders?]]></title>
            <link>https://medium.com/@sankhoroy/if-autoencoders-can-compress-data-why-did-we-invent-variational-autoencoders-9d2ef3f5aefd?source=rss-ff631f66b7d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d2ef3f5aefd</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[autoencoder]]></category>
            <category><![CDATA[vae]]></category>
            <category><![CDATA[unsupervised-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[sankho roy]]></dc:creator>
            <pubDate>Sat, 20 Jun 2026 23:24:51 GMT</pubDate>
            <atom:updated>2026-06-21T07:03:34.400Z</atom:updated>
            <content:encoded><![CDATA[<p><em>No worries if you don’t understand the math. Just understand the story.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g-x1BkmgtYPD3LawDSFo8Q.png" /><figcaption>what is problem with autoencoders: It does the work. but you still expect more from it</figcaption></figure><p>Imagine you are designing a city.</p><p>Every house represents a data point.</p><p>You hire an engineer and say:</p><p>“Put similar houses near each other.”</p><p>The engineer replies:</p><p>“You never asked me to.”</p><p>Instead, he scatters houses randomly across the map.</p><p>The only thing he guarantees is that every house has an address.</p><p>Congratulations.</p><p>You have just built an Autoencoder.</p><h3>The City of Random Addresses</h3><p>Suppose we train an Autoencoder on face images.</p><pre>Encoder:<br>Face → Latent Vector<br>Decoder:<br>Latent Vector → Face</pre><p>After training, the model learns how to compress and reconstruct faces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fEyiKzvHGGIZScSHy_nX2g.png" /><figcaption>latent space of Autoencoder: no smooth transition is preserved. we cannot even reconstruct back from latent space our images. No organisation. complete mess</figcaption></figure><p>Mission accomplished.</p><p>But look at the latent space:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fn0M26RzZk1h9wfFldtZTw.png" /><figcaption>Autoencoder messes up representation in latent space</figcaption></figure><p>Not really.</p><p>The Autoencoder never learned geography.</p><p>The points are scattered wherever reconstruction worked.</p><p>Two similar faces may be far apart.</p><p>Two unrelated faces may be neighbors.</p><p>Compression succeeded.</p><p>Organization failed.</p><blockquote>Problem with this is we can never reconstruct back faces if we need to. It’s complete mess.</blockquote><h3>The Dangerous Road Trip: Tuning Knob</h3><p>Suppose we pick two latent vectors.</p><pre>Male Face  ------------------- Female Face</pre><p>Now interpolate between them.</p><pre>z = (1-alpha)*z_male + alpha*z_female</pre><p>for α from 0 to 1.</p><blockquote>We basically want to have tuning knob based on that we should get more manness or womanness in face if we we roate knob one side or another side.</blockquote><p>What should happen?</p><p>Ideally:</p><pre>Male<br> ↓<br>Slightly Masculine<br> ↓<br>Neutral<br> ↓<br>Slightly Feminine<br> ↓<br>Female</pre><p>But Autoencoders often produce:</p><pre>Male<br> ↓<br>Alien<br> ↓<br>Noise<br> ↓<br>Broken Face<br> ↓<br>Female</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*opZ0cVrkK5Ls5hzY9jXReg.png" /><figcaption>male female tuning knob</figcaption></figure><p>Why?</p><p>Because the road passes through regions where no training examples exist.</p><p>The latent space contains holes.</p><h3>Enter Variational Autoencoders</h3><p>Researchers asked:</p><blockquote>“What if we force the latent space to be smooth?”</blockquote><p>Instead of encoding a face into a single point:</p><pre>Face → z</pre><p>we encode it into a distribution:</p><pre>Face → (μ, σ)</pre><p>For example:</p><pre>mu     = [1.2, -0.8]<br>sigma  = [0.3, 0.2]</pre><p>Then sample:</p><pre>z = mu + sigma * epsilon</pre><p>where</p><pre>epsilon ~ N(0,1)</pre><blockquote>No issue if maths not understood. Understand the story.</blockquote><p>Now every face occupies a small neighborhood instead of a single coordinate.</p><p>Additionally, all neighborhoods are encouraged to fit inside one giant Gaussian space.</p><h3>The Difference in One Picture</h3><p>Autoencoder latent space vs Variational Autoencoder latent space:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DNR8NvZu3no2BDgg9V5hyw.png" /><figcaption>VAE vs Autoencoder latent space</figcaption></figure><p>The VAE forces latent representations into a continuous manifold.</p><p>Now interpolation becomes meaningful.</p><h3>Real Example 1: The Masculinity-Femininity Knob</h3><p>Think about Photoshop for faces.</p><p>Wouldn’t it be amazing if we had a slider:</p><pre>Masculine  &lt;-------&gt;  Feminine</pre><p>Move the slider:</p><pre>100% Male<br>75% Male<br>50% Male<br>25% Male<br>Neutral<br>25% Female<br>50% Female<br>75% Female<br>100% Female</pre><p>This requires a smooth latent space.</p><p>VAEs naturally create such spaces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xq5l4ElOfS8BTcB4JFf0Vg.png" /><figcaption>A VAE based smooth knob of any two characterstics</figcaption></figure><p>A semantic attribute often becomes a direction in latent space.</p><pre>z_new = z_face + alpha * gender_direction</pre><p>where alpha controls the amount of transformation.</p><p>This is difficult with a standard Autoencoder because neighboring latent points are not guaranteed to represent neighboring concepts.</p><p>full code is available here : <a href="https://github.com/sankhoroy/my_medium_articles/blob/main/vae_male_female_face_transition.py">https://github.com/sankhoroy/my_medium_articles/blob/main/vae_male_female_face_transition.py</a></p><h3>Real Example 2: Small Labeled Text Classification</h3><p>Suppose we have:</p><pre>10 million unlabeled documents<br>500 labeled documents</pre><p>Typical machine learning struggles because labels are scarce.</p><p>Pipeline:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aeEljJ1VHNCoR82SWmy2Gw.png" /><figcaption>Pipeline for classificaiton with less number of labeled data</figcaption></figure><h3>Explanation:</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xl5he8T1H1dFQKiRaZSfgw.png" /><figcaption>Why VAE works here?It maps information vectors in smooth space(manifold) and we can use PCA like margin based classifiers to get predictions</figcaption></figure><p>The VAE learns the geometry of the embedding manifold using all unlabeled data.</p><p>Semantically similar documents become neighbors.</p><p>PCA then finds meaningful directions of variation.</p><p>A simple classifier can often learn from surprisingly few labels because the manifold is already organized.</p><p>Full code is available here: <a href="https://github.com/sankhoroy/my_medium_articles/blob/main/text_classification_with_few_labels_vae_pca.py">https://github.com/sankhoroy/my_medium_articles/blob/main/text_classification_with_few_labels_vae_pca.py</a></p><h3>Why PCA Benefits</h3><p>Without VAE:</p><pre>Embeddings<br> ↓<br>PCA</pre><p>PCA sees noisy, irregular geometry.</p><p>With VAE:</p><pre>Embeddings<br> ↓<br>VAE<br> ↓<br>Smooth Manifold<br> ↓<br>PCA</pre><p>PCA operates on a cleaner latent structure.</p><p>The resulting features often separate classes better.</p><h3>Minimal VAE Code</h3><p>full code here: <a href="https://github.com/sankhoroy/my_medium_articles/blob/main/vae.py">https://github.com/sankhoroy/my_medium_articles/blob/main/vae.py</a></p><p>Encoder:</p><pre>class Encoder(nn.Module):<br>    def __init__(self):<br>            super().__init__()<br>            self.fc = nn.Linear(784, 256)<br>            self.mu = nn.Linear(256, 32)<br>            self.logvar = nn.Linear(256, 32)<br>        def forward(self, x):<br>            h = F.relu(self.fc(x))<br>            mu = self.mu(h)<br>            logvar = self.logvar(h)<br>            return mu, logvar</pre><p>Reparameterization trick:</p><pre>def sample(mu, logvar):<br>    std = torch.exp(0.5 * logvar)<br>    eps = torch.randn_like(std)<br>    return mu + eps * std</pre><p>Loss:</p><pre>reconstruction_loss = mse(x_hat, x)<br><br>kl_loss = -0.5 * torch.sum(<br>    1 + logvar - mu.pow(2) - logvar.exp()<br>)<br>loss = reconstruction_loss + kl_loss</pre><p>The KL term is the secret ingredient.</p><p>It keeps the latent space smooth and connected.</p><h3>The Mental Model</h3><p>Autoencoder:</p><pre>Compression</pre><p>Variational Autoencoder:</p><pre>Compression<br>+<br>Topology Preservation<br>+<br>Interpolation<br>+<br>Generation<br>+<br>Semantic Control</pre><blockquote>Autoencoders learn a dictionary.</blockquote><blockquote>VAEs learn a map.</blockquote><blockquote>A dictionary tells you where things are stored.</blockquote><blockquote>A map tells you how everything is connected.</blockquote><blockquote>And that connectivity is the real reason Variational Autoencoders were invented.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d2ef3f5aefd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Model Scored 96% Accuracy. Think That’s Success? Think Again.]]></title>
            <link>https://medium.com/@sankhoroy/your-model-scored-96-accuracy-think-thats-success-think-again-d36e1cc493fb?source=rss-ff631f66b7d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/d36e1cc493fb</guid>
            <category><![CDATA[calibration]]></category>
            <category><![CDATA[business]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[isotonic-regression]]></category>
            <category><![CDATA[logistic-regression]]></category>
            <dc:creator><![CDATA[sankho roy]]></dc:creator>
            <pubDate>Sat, 20 Jun 2026 14:45:13 GMT</pubDate>
            <atom:updated>2026-06-20T14:45:13.488Z</atom:updated>
            <content:encoded><![CDATA[<p><em>How a highly accurate model can still lose money — and why calibration and threshold tuning are the real keys to business success</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jh8guV6a0tH5Ht6zmaqlnw.jpeg" /><figcaption>“Everyone loves my accuracy score. Why are we still losing money?”</figcaption></figure><p>Most machine learning projects end their presentations with a familiar slide:</p><ul><li>Accuracy: 96.2%</li><li>F1 Score: 0.91</li><li>ROC-AUC: 0.97</li></ul><p>Everyone in the room nods.</p><p>The model is approved.</p><p>Deployment begins.</p><p>Six months later, the business team asks a painful question:</p><p>“If the model is so good, why are we losing money?”</p><p>The uncomfortable truth is that many machine learning projects stop exactly where they should start.</p><p>The model may be statistically excellent, yet economically disastrous.</p><p>The missing piece is calibration and threshold optimization.</p><p>Let’s walk through a story.</p><h3>The Fraud Detection Team That Celebrated Too Early</h3><p>Imagine a bank building a fraud detection system.</p><p>The goal is simple:</p><p>Predict whether a transaction is fraudulent.</p><p>After months of experimentation, the team develops a logistic regression model.</p><p>Validation results look fantastic:</p><ul><li>Accuracy = 96%</li><li>Precision = 88%</li><li>Recall = 82%</li><li>F1 Score = 85%</li></ul><p>The model beats every baseline.</p><p>The team celebrates.</p><p>The model goes live.</p><p>Three months later, business stakeholders report unexpected losses.</p><p>Why?</p><p>Because metrics such as Accuracy and F1 only evaluate classification performance at a specific threshold.</p><p>Most commonly:</p><p>Prediction &gt; 0.5 → Fraud</p><p>Prediction ≤ 0.5 → Legitimate</p><p>But businesses rarely operate at a universal 0.5 threshold.</p><p>They operate on money.</p><h3>The Hidden Assumption Inside Logistic Regression</h3><p>A logistic regression model outputs probabilities.</p><p>For example:</p><p>Customer A → 0.91</p><p>Customer B → 0.78</p><p>Customer C → 0.55</p><p>Customer D → 0.18</p><p>Many practitioners immediately convert these probabilities into labels.</p><p>However, there is a crucial question:</p><p>Does a predicted probability of 0.80 actually mean an 80% chance of fraud?</p><p>Not necessarily.</p><p>A model can rank examples extremely well while producing poorly calibrated probabilities.</p><p>Consider two groups:</p><p>Predicted Probability = 0.80</p><p>Group 1:</p><ul><li>80 out of 100 actually fraudulent</li></ul><p>Perfect calibration.</p><p>Group 2:</p><ul><li>45 out of 100 actually fraudulent</li></ul><p>Poor calibration.</p><p>Both models may achieve identical Accuracy, F1, and ROC-AUC.</p><p>But their probability estimates mean completely different things.</p><p>This becomes critical when business decisions depend on risk estimates.</p><h3>The Problem With Accuracy</h3><p>Suppose a bank processes 1 million transactions.</p><p>Cost matrix:</p><ul><li>True Positive (TP): +$500 saved</li><li>True Negative (TN): $0</li><li>False Positive (FP): -$10 customer inconvenience</li><li>False Negative (FN): -$1000 fraud loss</li></ul><p>Now imagine two models.</p><p>Model A:</p><ul><li>Accuracy = 96%</li></ul><p>Model B:</p><ul><li>Accuracy = 94%</li></ul><p>Most teams would choose Model A.</p><p>Let’s calculate business value.</p><h3>Model A</h3><p>Confusion Matrix:</p><ul><li>TP = 800</li><li>TN = 95,200</li><li>FP = 3,000</li><li>FN = 1,000</li></ul><p>Business Value:</p><p>= (800 × 500)</p><ul><li>(95,200 × 0)</li><li>(3,000 × 10)</li><li>(1,000 × 1,000)</li></ul><p>= 400,000</p><ul><li>30,000</li><li>1,000,000</li></ul><p>= -$630,000</p><h3>Model B</h3><p>Confusion Matrix:</p><ul><li>TP = 1,400</li><li>TN = 92,000</li><li>FP = 5,500</li><li>FN = 400</li></ul><p>Business Value:</p><p>= (1,400 × 500)</p><ul><li>(5,500 × 10)</li><li>(400 × 1,000)</li></ul><p>= 700,000</p><ul><li>55,000</li><li>400,000</li></ul><p>= +$245,000</p><p>Model B has lower accuracy.</p><p>Yet it creates nearly $875,000 more business value.</p><p>Accuracy was optimizing the wrong objective.</p><h3>Why Calibration Comes First</h3><p>Before selecting thresholds, probabilities must be trustworthy.</p><p>If the model says:</p><p>“This customer has a 70% chance of default.”</p><p>The business expects roughly 70 out of 100 similar customers to actually default.</p><p>Calibration aligns predicted probabilities with reality.</p><p>Without calibration:</p><ul><li>Risk scores become misleading</li><li>Thresholds become unstable</li><li>Cost optimization becomes unreliable</li></ul><p>This is why many mature ML systems introduce a calibration layer after model training.</p><h3><strong>Story1: Imagine you are a weather forecaster</strong></h3><p>Every day you make predictions like:</p><ul><li>90% chance of rain</li><li>80% chance of rain</li><li>70% chance of rain</li></ul><p>Now suppose you look back after 1 year.</p><p>For all days where you predicted <strong>80% chance of rain</strong>, it actually rained only <strong>40% of the time</strong>.</p><p>That means your forecasts are <strong>not calibrated</strong>.</p><p>You are overconfident.</p><p>A calibrated weather forecaster would have:</p><ul><li>Among all days predicted at 80%, rain occurs roughly 80% of the time.</li><li>Among all days predicted at 30%, rain occurs roughly 30% of the time.</li></ul><blockquote>Calibration is simply:</blockquote><blockquote><strong>“When I say 80%, does reality also say 80%?”</strong></blockquote><blockquote><strong>In ML models like Logistic Regression, NOT ALWAYS.</strong><br>The good news is that while the probabilities may be wrong, the ordering is often still correct.<br>A customer scored 0.9 is usually riskier than one scored 0.7 — the model knows <em>who is more likely</em>, even if it doesn’t know <em>how likely. And we are going to use this property.</em></blockquote><h3>Common Calibration Methods</h3><h3>Platt Scaling</h3><p>One of the earliest approaches.</p><p>A second logistic regression model is fitted on validation predictions.</p><p>Original Prediction:</p><p>0.87</p><p>After Calibration:</p><p>0.74</p><p>The score becomes more realistic.</p><p>Platt scaling works well when calibration errors follow a sigmoid pattern.</p><h3>Isotonic Regression</h3><p>A non-parametric alternative.</p><p>Instead of assuming a sigmoid shape, isotonic regression learns a monotonic mapping.</p><p>Example:</p><p>Raw Score → Calibrated Score</p><p>0.30 → 0.12</p><p>0.55 → 0.41</p><p>0.80 → 0.72</p><p>0.95 → 0.89</p><p>Advantages:</p><ul><li>Flexible</li><li>Often produces superior calibration</li></ul><p>Disadvantages:</p><ul><li>Requires more validation data</li><li>Can overfit on small datasets</li></ul><h3>Beta Calibration</h3><p>A newer extension that generalizes Platt scaling.</p><p>Often performs well when probability distributions are skewed.</p><p>In many industrial systems, practitioners compare:</p><ul><li>Platt Scaling</li><li>Isotonic Regression</li><li>Beta Calibration</li></ul><p>and select the method with the lowest calibration error.</p><h3>Measuring Calibration</h3><p>Accuracy is not enough.</p><p>We need calibration metrics.</p><h3>Brier Score</h3><p>Measures probability quality.</p><p>Lower is better.</p><p>Example:</p><p>Model A:<br>Brier Score = 0.12</p><p>Model B:<br>Brier Score = 0.07</p><p>Model B produces more trustworthy probabilities.</p><h3>Expected Calibration Error (ECE)</h3><p>Measures mismatch between:</p><p>Predicted Probability</p><p>and</p><p>Observed Frequency</p><p>Lower ECE indicates better calibration.</p><h3>Reliability Diagram</h3><p>One of the most powerful visualizations.</p><p>If a model predicts 70% confidence:</p><p>Approximately 70% of those predictions should be correct.</p><p>The closer the curve lies to the diagonal line, the better calibrated the model.</p><h3>The Workflow Mature Teams Follow</h3><p>Many beginners do:</p><p>Train → Accuracy → Deploy</p><p>Experienced teams do:</p><p>Train</p><p>↓</p><p>Evaluate</p><ul><li>Accuracy</li><li>Precision</li><li>Recall</li><li>F1</li><li>ROC-AUC</li></ul><p>↓</p><p>Calibrate Probabilities</p><ul><li>Platt Scaling</li><li>Isotonic Regression</li><li>Beta Calibration</li></ul><p>↓</p><p>Evaluate Calibration</p><ul><li>Brier Score</li><li>ECE</li><li>Reliability Diagram</li></ul><p>↓</p><p>Optimize Threshold Using Business Costs</p><p>↓</p><p>Deploy</p><p>The order matters.</p><p>Threshold optimization before calibration is like adjusting a thermometer that already reports the wrong temperature.</p><h3>The Threshold That Maximizes Profit</h3><p>After calibration, probabilities become meaningful.</p><p>Now business teams can answer questions like:</p><p>“What probability of fraud should trigger intervention?”</p><p>Maybe:</p><p>Threshold = 0.50</p><p>produces:</p><p>Profit = $300,000</p><p>Threshold = 0.65</p><p>produces:</p><p>Profit = $520,000</p><p>Threshold = 0.78</p><p>produces:</p><p>Profit = $640,000</p><p>Threshold = 0.90</p><p>produces:</p><p>Profit = $410,000</p><p>The optimal threshold is not necessarily 0.5.</p><p>It is the threshold that maximizes business value.</p><p>Every organization has a different cost matrix.</p><p>Healthcare:</p><p>False Negatives may be catastrophic.</p><p>Advertising:</p><p>False Positives may be cheap.</p><p>Fraud Detection:</p><p>Both can be expensive.</p><p>The threshold should reflect economics, not convention.</p><h3>The Real Objective of Machine Learning</h3><p>Machine learning models are often judged by statistical metrics.</p><p>Businesses are judged by outcomes.</p><p>Customers do not care about ROC-AUC.</p><p>CFOs do not care about F1 Score.</p><p>Operations teams do not care about Accuracy.</p><p>They care about:</p><ul><li>Revenue</li><li>Cost Reduction</li><li>Risk Mitigation</li><li>Customer Satisfaction</li></ul><p>Calibration acts as the bridge between statistical performance and business performance.</p><p>A model with excellent Accuracy but poor calibration can destroy value.</p><p>A model with slightly lower Accuracy but well-calibrated probabilities can become a business success.</p><p>The next time someone shows a slide saying:</p><p>“Accuracy: 96%”</p><p>ask a simple question:</p><p>“Great. Are the probabilities calibrated?”</p><p>Because that question often determines whether the model creates value or merely creates confidence.</p><h3>Enough Talking : Let’s dive into code</h3><p>Full code here <a href="https://github.com/sankhoroy/my_medium_articles/blob/main/logistic_regression_calibration.py">https://github.com/sankhoroy/my_medium_articles/blob/main/logistic_regression_calibration.py</a></p><pre>from sklearn.datasets import make_classification<br>from sklearn.model_selection import train_test_split<br>from sklearn.linear_model import LogisticRegression<br>from sklearn.calibration import CalibratedClassifierCV<br>from sklearn.metrics import (<br>    accuracy_score,<br>    f1_score,<br>    brier_score_loss<br>)<br><br># ----------------------------------------------------<br># Create sample dataset<br># ----------------------------------------------------<br>X, y = make_classification(<br>    n_samples=10000,<br>    n_features=20,<br>    n_informative=10,<br>    random_state=42<br>)<br><br># ----------------------------------------------------<br># Train / Validation / Test Split<br># ----------------------------------------------------<br>X_train, X_temp, y_train, y_temp = train_test_split(<br>    X,<br>    y,<br>    test_size=0.4,<br>    random_state=42<br>)<br><br>X_valid, X_test, y_valid, y_test = train_test_split(<br>    X_temp,<br>    y_temp,<br>    test_size=0.5,<br>    random_state=42<br>)<br><br># ----------------------------------------------------<br># Train Logistic Regression<br># ----------------------------------------------------<br>model = LogisticRegression(<br>    max_iter=1000<br>)<br><br>model.fit(X_train, y_train)<br><br># ----------------------------------------------------<br># Standard Metrics<br># ----------------------------------------------------<br>pred_test = model.predict(X_test)<br><br>print(&quot;Accuracy:&quot;,<br>      accuracy_score(y_test, pred_test))<br><br>print(&quot;F1:&quot;,<br>      f1_score(y_test, pred_test))<br><br># Raw probabilities<br>prob_test = model.predict_proba(X_test)[:, 1]<br><br>print(<br>    &quot;Brier Score (Before Calibration):&quot;,<br>    brier_score_loss(y_test, prob_test)<br>)<br>```<br><br>At this point, most machine learning projects stop.<br><br>However, we still don&#39;t know whether a predicted probability of 0.80 truly corresponds to an 80% chance of the event occurring.<br><br>Let&#39;s calibrate.<br><br>### Isotonic Regression Calibration<br><br>```python<br>calibrated_model = CalibratedClassifierCV(<br>    estimator=model,<br>    method=&quot;isotonic&quot;,<br>    cv=&quot;prefit&quot;<br>)<br><br>calibrated_model.fit(<br>    X_valid,<br>    y_valid<br>)<br><br>calibrated_probs = calibrated_model.predict_proba(<br>    X_test<br>)[:, 1]<br><br>print(<br>    &quot;Brier Score (After Calibration):&quot;,<br>    brier_score_loss(<br>        y_test,<br>        calibrated_probs<br>    )<br>)<br>```<br><br>If calibration is successful, the Brier Score should decrease, indicating that predicted probabilities are now closer to real-world frequencies.<br><br>### Platt Scaling Calibration<br><br>```python<br>platt_model = CalibratedClassifierCV(<br>    estimator=model,<br>    method=&quot;sigmoid&quot;,<br>    cv=&quot;prefit&quot;<br>)<br><br>platt_model.fit(<br>    X_valid,<br>    y_valid<br>)<br><br>platt_probs = platt_model.predict_proba(<br>    X_test<br>)[:, 1]<br><br>print(<br>    &quot;Platt Brier Score:&quot;,<br>    brier_score_loss(<br>        y_test,<br>        platt_probs<br>    )<br>)<br>```<br><br>The best calibration method is usually selected based on calibration metrics such as:<br><br>* Brier Score<br>* Expected Calibration Error (ECE)<br>* Reliability Diagram<br><br>rather than Accuracy or F1.<br><br>### Business Threshold Optimization<br><br>Once probabilities are calibrated, threshold selection becomes meaningful.<br><br>```python<br>import numpy as np<br>from sklearn.metrics import confusion_matrix<br><br>TP_VALUE = 500<br>FP_COST = 10<br>FN_COST = 1000<br><br>best_profit = -1e18<br>best_threshold = None<br><br>for threshold in np.arange(<br>    0.01,<br>    1.00,<br>    0.01<br>):<br><br>    preds = (<br>        calibrated_probs &gt;= threshold<br>    ).astype(int)<br><br>    tn, fp, fn, tp = confusion_matrix(<br>        y_test,<br>        preds<br>    ).ravel()<br><br>    profit = (<br>        tp * TP_VALUE<br>        - fp * FP_COST<br>        - fn * FN_COST<br>    )<br><br>    if profit &gt; best_profit:<br>        best_profit = profit<br>        best_threshold = threshold<br><br>print(<br>    f&quot;Best Threshold: {best_threshold:.2f}&quot;<br>)<br><br>print(<br>    f&quot;Maximum Profit: ${best_profit:,.0f}&quot;<br>)</pre><p>This final step is where machine learning becomes business optimization.</p><p>A threshold of 0.50 might maximize Accuracy.</p><p>A threshold of 0.73 might maximize Profit.</p><p>A threshold of 0.84 might minimize Risk.</p><p>The correct threshold depends on business objectives — not on machine learning conventions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d36e1cc493fb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We never use MSE for Classification, But “WHY”?]]></title>
            <link>https://medium.com/@sankhoroy/why-mean-squared-error-fails-for-classification-a-curious-case-of-curvatures-and-4-stories-read-860bfc9a94b7?source=rss-ff631f66b7d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/860bfc9a94b7</guid>
            <category><![CDATA[loss-function]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[classification]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[linear-regression]]></category>
            <dc:creator><![CDATA[sankho roy]]></dc:creator>
            <pubDate>Thu, 18 Jun 2026 15:05:08 GMT</pubDate>
            <atom:updated>2026-06-20T13:55:42.610Z</atom:updated>
            <content:encoded><![CDATA[<p>If MSE works perfectly for regression, why does almost every classifier use Cross-Entropy instead? After all MSE values at 0 or 1 are same as 0’s or 1’s in Cross-Entropy. Is it conveneince? Is it steeper gradients? or something else? The hidden mystery answer lies deep beneath the loss function landscape. Do not worry about terms like convex, non-convex, I will explain. Just go through it, four stories will explain.</p><h3>A Question That Kept Bothering Me</h3><p>When I first learned machine learning, I had a simple question:</p><blockquote><em>“Why can’t we just use Mean Squared Error (MSE) for classification?”</em></blockquote><p>After all, MSE seems universal:</p><p><strong>Loss = ½ × (Actual − Prediction)²</strong></p><p>If the prediction is wrong, the loss increases. If the prediction is correct, the loss decreases.</p><p>Seems reasonable.</p><p>Then why do textbooks insist:</p><blockquote><em>For classification, use Cross-Entropy and avoid MSE.</em></blockquote><p>To understand why, let’s walk through a small story.</p><h3>Part 1: MSE Lives Happily in Regression</h3><p>Suppose we’re predicting house prices using a linear model:</p><p><strong>Prediction</strong></p><p>ŷ = wx + b</p><p>Assume:</p><ul><li>x = 2</li><li>y = 10</li><li>w = 3</li><li>b = 1</li></ul><p>The prediction becomes:</p><p>ŷ = (3 × 2) + 1 = 7</p><p>The MSE is:</p><p>Loss = ½ × (10 − 7)²<br>= ½ × ³²<br>= 4.5</p><p>Nothing surprising here.</p><p>Now let’s express the loss as a function of the parameter w.</p><p>Loss(w) = ½ × (y − wx − b)²</p><p>Substituting our numbers:</p><p>Loss(w) = ½ × (10 − 2w − 1)²<br>= ½ × (9 − 2w)²</p><p>Expanding:</p><p>Loss(w) = 2w² − 18w + 40.5</p><p>This is simply a quadratic equation.</p><p>The second derivative is:</p><p>d²Loss/dw² = 4</p><p>which is always positive.</p><p>The loss surface therefore looks like a smooth bowl:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wxX5R1AGnKR3s6Opg5fSVQ.png" /><figcaption>this is how convex and non-convex function looks</figcaption></figure><p>There is:</p><p>✓ One global minimum<br>✓ No deceptive valleys<br>✓ No flat plateaus</p><p>Gradient descent simply rolls downhill.</p><p>Life is easy.</p><h3>What does it means to be convex or non-convex?</h3><h3>Story1: The Ice Cream Shop Story 🍦</h3><p>Imagine you and your friend visit an ice cream shop on two different days.</p><h3>Day 1</h3><p>You buy a <strong>small cone</strong> for ₹20.</p><h3>Day 2</h3><p>You buy a <strong>large sundae</strong> for ₹100.</p><p>Now your mom says:</p><blockquote><em>“Tomorrow, let’s buy something halfway between the small cone and the large sundae.”</em></blockquote><p>You expect the middle option to cost around the average:</p><p>(20+100)/2=60</p><p>You go to the shop and discover the medium-sized ice cream costs only <strong>₹50</strong>.</p><p>You say:</p><blockquote><em>“Hey! The middle option is cheaper than the average of the small and large ones.”</em></blockquote><p>This is exactly what happens in a <strong>convex function</strong>.</p><p><strong>Rule:</strong></p><blockquote><em>The value in the middle is less than or equal to the average of the values at the ends.</em></blockquote><p>This is Jensen’s inequality:</p><p>f ⁣( (x1+x2) / 2 ) ≤ ( f(x1) + f(x2) ) / 2</p><h3>Story 2: The Amusement Park Story 🎢</h3><p>Now imagine an amusement park.</p><p>At one end of the park, you’re standing on a hill that’s <strong>10 meters high</strong>.</p><p>At the other end, you’re again on a hill that’s <strong>10 meters high</strong>.</p><p>So you think:</p><blockquote><em>“If I walk halfway between them, I should still be around 10 meters high.”</em></blockquote><p>But when you reach the middle, there’s a <strong>giant hill that’s 25 meters high!</strong></p><p>You shout:</p><blockquote><em>“Wait! The middle is higher than both sides!”</em></blockquote><p>This is a <strong>non-convex function</strong>.</p><p>Here,</p><p>Middle height:</p><p>25</p><p>Average of the ends:</p><p>10+10/ 2=10</p><p>Since</p><p>25&gt;10</p><p>the middle became higher than the average.</p><p><strong>In a short: <em>The value in the middle is less than or equal to the average of the values at the ends.</em></strong></p><blockquote><strong>This is Jensen’s inequality:</strong></blockquote><blockquote><strong>f ⁣( (x1+x2)/ 2) ≤ ( f(x1)+f(x2)) / 2</strong></blockquote><blockquote><strong>In more details this is (don’t worry if not understood , it overall means <em>For a convex function, the value at the middle is never higher than the weighted average of the values at the ends.</em>)</strong></blockquote><blockquote><strong>Jensen’s Inequality</strong></blockquote><blockquote>For any two points x1,x2​ and any</blockquote><blockquote>0≤λ≤1</blockquote><blockquote>define the weighted average:</blockquote><blockquote>x = λ*x1 + (1−λ)*x2</blockquote><blockquote>A function f(x) is <strong>convex</strong> if</blockquote><blockquote>f( λ*x1 + (1−λ)*x2 ) ≤ λ* f(x1) + (1−λ)*f(x2)</blockquote><h3>Part 2: Now Let’s Build a Classifier</h3><p>Suppose we want to classify emails:</p><p>Spam → 1<br>Not Spam → 0</p><p>A linear model can produce values like:</p><p>−8, 17, 53</p><p>These are not probabilities.</p><p>So we introduce the sigmoid function:</p><p>p = 1 / (1 + e^(−z))</p><p>where</p><p>z = wx + b</p><p>The sigmoid converts any number into a probability between 0 and 1.</p><p>For example:</p><p>z = −2</p><p>gives</p><p>p = 1 / (1 + e²) ≈ 0.119</p><p>The model says:</p><p>“There is only an 11.9% chance that this email is spam.”</p><h3>Part 3: Let’s Still Use MSE</h3><p>Assume:</p><p>Actual label: y = 1 (Spam)</p><p>Input:</p><p>x = 2</p><p>Parameters:</p><p>w = −1<br>b = 0</p><p>Then,</p><p>z = (−1 × 2) + 0 = −2</p><p>Probability:</p><p>p ≈ 0.119</p><p>The MSE becomes:</p><p>Loss = ½ × (1 − 0.119)²<br>= 0.388</p><p>Everything still looks fine.</p><p>So where does the problem begin?</p><h3>Part 4: Looking at the Gradient</h3><p>For classification:</p><p>Loss = ½ × (y − p)²</p><p>where</p><p>p = sigmoid(z)</p><p>Differentiating gives:</p><p>dLoss/dz = (p − y) × p × (1 − p)</p><p>Notice something new.</p><p>The gradient now contains an extra term:</p><p>p × (1 − p)</p><p>Let’s plug in our numbers.</p><p>p = 0.119<br>y = 1</p><p>First term:</p><p>(p − y) = −0.881</p><p>Second term:</p><p>p(1 − p)</p><p>= 0.119 × 0.881</p><p>≈ 0.105</p><p>Therefore,</p><p>dLoss/dz</p><p>= (−0.881) × 0.105</p><p>≈ −0.092</p><p>The prediction is very wrong, yet the gradient magnitude is only 0.092.</p><p>Learning is already becoming slower.</p><p>But the real problem is still hidden.</p><h3>Part 5: The Hidden Enemy — Curvature</h3><p>The second derivative becomes:</p><p>d²Loss/dz²</p><p>= [p(1 − p)]²</p><ul><li>(p − y)p(1 − p)(1 − 2p)</li></ul><p>The first term is always positive.</p><p>The second term can be:</p><p>positive,<br>zero,<br>or negative.</p><p>Let’s compute it.</p><p>p = 0.119</p><p>First term:</p><p>[p(1 − p)]²</p><p>= (0.105)²</p><p>≈ 0.011</p><p>Second term:</p><p>(0.119 − 1) × 0.105 × (1 − 0.238)</p><p>≈ −0.070</p><p>Therefore,</p><p>d²Loss/dz²</p><p>= 0.011 − 0.070</p><p>≈ −0.059</p><p>The curvature is negative.</p><p>The smooth bowl has disappeared.</p><p>The loss surface bends downward and becomes non-convex. Now storytime.</p><h3>Story3: The Lost Hiker Story 🥾</h3><p>Imagine you’re hiking and trying to reach the <strong>lowest point in a valley</strong>.</p><h3>Convex World (Smooth Bowl)</h3><p>The land is shaped like one big, smooth bowl.</p><p>No matter where you start:</p><ul><li>Every step downhill takes you closer to the bottom.</li><li>There are no surprise hills.</li><li>You never wonder, <em>“Am I going the right way?”</em></li></ul><p>Eventually, you reach the lowest point.</p><p>This is why gradient descent loves convex functions.</p><h3>Story 4: Non-Convex World (Negative Curvature Appears)</h3><p>Now imagine the landscape changes.</p><p>Instead of one smooth bowl, the land becomes full of:</p><ul><li>small hills,</li><li>small valleys,</li><li>almost flat areas.</li></ul><p>You start walking downhill.</p><p>Suddenly:</p><blockquote><em>“Great! I found a valley.”</em></blockquote><p>You stop.</p><p>But unknown to you, it’s only a <strong>small valley</strong>, not the deepest one.</p><p>Another deeper valley exists far away.</p><p>Or sometimes you enter an almost flat area and think:</p><blockquote><em>“I can’t tell which direction is downhill anymore.”</em></blockquote><p>So you move very slowly.</p><p>Sometimes you even have to go slightly uphill first to eventually reach a much deeper valley, but gradient descent only looks at the immediate slope under its feet. It doesn’t know about the bigger picture.</p><h3>This is the actual risk: Now you understand!!</h3><p>Negative curvature is nature’s way of saying:</p><blockquote><em>“The land is no longer one friendly bowl. Hidden hills and valleys now exist.”</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ol-g3j0R_Zv4VaCh5AqV5w.png" /><figcaption>The same MSE that creates a smooth bowl in regression turns into a rugged landscape when combined with sigmoid outputs in classification.</figcaption></figure><h3>Why This Matters for MSE + Sigmoid</h3><p>When MSE is combined with the sigmoid function, the loss surface stops being a single smooth bowl.</p><p>The optimizer becomes like a hiker walking through a foggy mountain range instead of rolling down one clean valley.</p><p>That’s why optimization becomes harder and why classification models usually prefer <strong>cross-entropy</strong>, which creates a much friendlier landscape for learning.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=860bfc9a94b7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[PyTorch Hooks and Integrated Gradients: Understanding What Your Neural Network Sees]]></title>
            <link>https://medium.com/@sankhoroy/pytorch-hooks-and-integrated-gradients-understanding-what-your-neural-network-sees-25364073cce5?source=rss-ff631f66b7d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/25364073cce5</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[feature-importance]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[pytorch-hooks]]></category>
            <category><![CDATA[gradient-descent]]></category>
            <dc:creator><![CDATA[sankho roy]]></dc:creator>
            <pubDate>Wed, 20 May 2026 18:50:55 GMT</pubDate>
            <atom:updated>2026-05-20T18:50:55.022Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Learn how forward hooks, backward hooks, and Integrated Gradients help interpret deep learning models in PyTorch.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0TwHp342721rgxbvQ-q7PQ.png" /><figcaption>IG(integrated gradients) using pytorch hooks. showing feature importances</figcaption></figure><h3>Introduction</h3><p>Modern neural networks can achieve remarkable accuracy, but they often behave like black boxes. A model may predict that an image contains a dog with 99% confidence, but:</p><ul><li>Why did it make that prediction?</li><li>Which parts of the image mattered most?</li><li>Was the model focusing on the object itself or some irrelevant background pattern?</li></ul><p>In this article, we will explore three powerful interpretability tools in PyTorch:</p><ol><li><strong>Forward Hooks</strong></li><li><strong>Backward Hooks</strong></li><li><strong>Integrated Gradients</strong></li></ol><p>By the end, you will understand how to inspect activations, analyze gradients, and visualize what neural networks actually pay attention to.</p><h3>What Are Hooks in PyTorch?</h3><p>Hooks are callback functions that let you attach custom logic during the execution of a neural network.</p><p>You can intercept:</p><ul><li>activations during the forward pass</li><li>gradients during the backward pass</li></ul><p>Hooks are widely used for:</p><ul><li>debugging</li><li>feature extraction</li><li>Grad-CAM</li><li>visualization</li><li>gradient analysis</li><li>interpretability research</li></ul><h3>Forward Hooks</h3><p>A forward hook executes immediately after a layer computes its output.</p><h3>Forward Hook Flow</h3><pre>Input → Layer → Hook Runs → Next Layer</pre><p>The hook receives:</p><ul><li>the module</li><li>the input tensor</li><li>the output tensor</li></ul><h3>Forward Hook Example</h3><pre>import torch<br>import torch.nn as nn</pre><pre>model = nn.Sequential(<br>    nn.Linear(4, 3),<br>    nn.ReLU(),<br>    nn.Linear(3, 1)<br>)</pre><pre># Hook function</pre><pre>def forward_hook(module, input, output):<br>    print(&quot;Layer:&quot;, module)<br>    print(&quot;Input:&quot;, input)<br>    print(&quot;Output:&quot;, output)<br>    print()</pre><pre># Attach hook<br>hook = model[0].register_forward_hook(forward_hook)</pre><pre>x = torch.randn(1, 4)</pre><pre>output = model(x)</pre><h3>What Does This Do?</h3><p>When the first Linear layer executes, the hook prints:</p><ul><li>the layer itself</li><li>incoming activations</li><li>outgoing activations</li></ul><p>This is extremely useful when debugging neural networks.</p><h3>Real Use Case: Feature Extraction</h3><p>Suppose you want embeddings from an intermediate CNN layer.</p><pre>features = {}<br></pre><pre>def save_features(module, input, output):<br>    features[&#39;conv1&#39;] = output.detach()</pre><pre>model.conv1.register_forward_hook(save_features)</pre><p>Now after running inference:</p><pre>model(image)</pre><pre>print(features[&#39;conv1&#39;].shape)</pre><p>This technique is heavily used in:</p><ul><li>transfer learning</li><li>CLIP embeddings</li><li>DINO feature extraction</li><li>diffusion models</li><li>attention visualization</li></ul><h3>Backward Hooks</h3><p>Backward hooks execute during backpropagation.</p><p>They allow you to inspect gradients flowing through the network.</p><h3>Backward Flow</h3><pre>Loss ← Gradients ← Hook Runs ← Earlier Layers</pre><h3>Backward Hook Example</h3><pre>import torch<br>import torch.nn as nn</pre><pre>model = nn.Sequential(<br>    nn.Linear(4, 3),<br>    nn.ReLU(),<br>    nn.Linear(3, 1)<br>)<br></pre><pre>def backward_hook(module, grad_input, grad_output):<br>    print(&quot;Layer:&quot;, module)<br>    print(&quot;Gradient Input:&quot;, grad_input)<br>    print(&quot;Gradient Output:&quot;, grad_output)<br>    print()</pre><pre>hook = model[0].register_full_backward_hook(backward_hook)</pre><pre>x = torch.randn(1, 4)<br>target = torch.tensor([[1.0]])</pre><pre>output = model(x)</pre><pre>loss = (output - target).pow(2).mean()</pre><pre>loss.backward()</pre><h3>Why Are Backward Hooks Useful?</h3><p>They help detect:</p><ul><li>exploding gradients</li><li>vanishing gradients</li><li>dead neurons</li><li>unstable training</li></ul><p>They are also essential for:</p><ul><li>Grad-CAM</li><li>saliency maps</li><li>adversarial attacks</li><li>gradient surgery</li></ul><h3>Tensor Hooks</h3><p>You can also attach hooks directly to tensors.</p><pre>x = torch.tensor([2.0], requires_grad=True)<br></pre><pre>def tensor_hook(grad):<br>    print(&quot;Gradient:&quot;, grad)</pre><pre>x.register_hook(tensor_hook)</pre><pre>y = x ** 2</pre><pre>y.backward()</pre><p>Output:</p><pre>Gradient: tensor([4.])</pre><p>Because:</p><p>[<br>\frac{d(x²)}{dx} = 2x<br>]</p><h3>Modifying Gradients with Hooks</h3><p>Hooks can even modify gradients.</p><pre>x = torch.tensor([2.0], requires_grad=True)<br></pre><pre>def clip_grad(grad):<br>    return grad.clamp(-1, 1)</pre><pre>x.register_hook(clip_grad)</pre><pre>y = 10 * x**2</pre><pre>y.backward()</pre><pre>print(x.grad)</pre><p>Without clipping:</p><p>[<br>20x = 40<br>]</p><p>After clipping:</p><pre>tensor([1.])</pre><h3>Integrated Gradients</h3><p>Hooks help inspect internal computations.</p><p>Integrated Gradients goes one step further:</p><blockquote><em>It explains which pixels influenced the prediction.</em></blockquote><p>Integrated Gradients is one of the most important interpretability methods in deep learning.</p><h3>The Main Idea</h3><p>Instead of computing gradients only once, Integrated Gradients:</p><ol><li>Starts from a baseline image</li><li>Slowly moves toward the real image</li><li>Accumulates gradients along the path</li></ol><p>This produces far more stable explanations.</p><h3>Integrated Gradients Formula</h3><p>For input image (x) and baseline (x’):</p><p>[<br>IG_i(x) =<br>(x_i — x_i’)<br>\int_0¹<br>\frac{\partial F(x’ + \alpha(x-x’))}{\partial x_i}<br>d\alpha<br>]</p><p>Where:</p><ul><li>(F) is the model output</li><li>(x’) is the baseline image</li><li>(\alpha) interpolates between baseline and real image</li></ul><h3>Why Is This Better Than Raw Gradients?</h3><p>Raw gradients can:</p><ul><li>saturate</li><li>become noisy</li><li>miss important regions</li></ul><p>Integrated Gradients averages information across many points, making explanations much more reliable.</p><h3>Manual Integrated Gradients in PyTorch</h3><pre>import torch<br>import torchvision.models as models</pre><pre>model = models.resnet18(pretrained=True)<br>model.eval()<br></pre><pre>def integrated_gradients(<br>    model,<br>    image,<br>    target_class,<br>    baseline=None,<br>    steps=50<br>):</pre><pre>    if baseline is None:<br>        baseline = torch.zeros_like(image)</pre><pre>    scaled_inputs = []</pre><pre>    for i in range(steps + 1):<br>        alpha = i / steps<br>        scaled = baseline + alpha * (image - baseline)<br>        scaled_inputs.append(scaled)</pre><pre>    grads = []</pre><pre>    for scaled in scaled_inputs:</pre><pre>        scaled.requires_grad_(True)</pre><pre>        output = model(scaled)</pre><pre>        score = output[0, target_class]</pre><pre>        model.zero_grad()</pre><pre>        score.backward()</pre><pre>        grads.append(scaled.grad.detach())</pre><pre>    avg_grad = torch.mean(torch.stack(grads), dim=0)</pre><pre>    integrated_grad = (image - baseline) * avg_grad</pre><pre>    return integrated_grad</pre><h3>Visualizing Attribution Maps</h3><pre>import matplotlib.pyplot as plt</pre><pre>heatmap = attr.abs().sum(dim=1)[0]</pre><pre>plt.imshow(heatmap.cpu(), cmap=&#39;hot&#39;)<br>plt.colorbar()<br>plt.show()</pre><p>Bright regions indicate pixels that strongly influenced the prediction.</p><h3>Image Credibility and Trustworthiness</h3><p>Integrated Gradients is extremely useful for evaluating whether a model behaves sensibly.</p><h3>Good Model Behavior</h3><p>If the model predicts “dog,” the attribution map should focus on:</p><ul><li>face</li><li>ears</li><li>fur</li><li>body</li></ul><h3>Suspicious Model Behavior</h3><p>If the model focuses on:</p><ul><li>watermarks</li><li>image borders</li><li>grass background</li><li>compression artifacts</li></ul><p>then the model may be learning shortcuts rather than real semantic understanding.</p><h3>Measuring Credibility</h3><p>A simple credibility metric:</p><p>[<br>Credibility =<br>\frac{<br>\sum_{pixels \in object} |IG|<br>}{<br>\sum_{all\ pixels} |IG|<br>}<br>]</p><p>Interpretation:</p><ul><li>High score → model focuses on object</li><li>Low score → model focuses elsewhere</li></ul><h3>Integrated Gradients vs Grad-CAM</h3><p>Integrated GradientsGrad-CAMPixel-level attributionFeature-map attributionMore mathematically principledMore visually intuitiveSlowerFasterFine-grained explanationsCoarse localization</p><p>Both are widely used in explainable AI.</p><h3>Best Practices</h3><h3>Use .detach() in hooks</h3><p>Avoid memory leaks:</p><pre>features.append(output.detach())</pre><h3>Use register_full_backward_hook</h3><p>Older backward hooks are deprecated.</p><p>Preferred:</p><pre>register_full_backward_hook()</pre><h3>Hooks Slow Down Training</h3><p>Because Python executes additional code every iteration.</p><p>Use them carefully in production.</p><h3>Final Thoughts</h3><p>Interpretability tools are becoming essential in deep learning.</p><p>Forward hooks and backward hooks allow you to inspect the internal mechanics of neural networks.</p><p>Integrated Gradients helps answer the most important question:</p><blockquote><em>Why did the model make this prediction?</em></blockquote><p>As AI systems become more widely deployed in healthcare, finance, autonomous driving, and security, understanding model behavior is no longer optional.</p><p>Explainability is becoming a core part of trustworthy machine learning.</p><h3>Key Takeaways</h3><ul><li>Forward hooks inspect activations</li><li>Backward hooks inspect gradients</li><li>Tensor hooks can modify gradients</li><li>Integrated Gradients explains pixel importance</li><li>Attribution maps help evaluate model credibility</li><li>Explainability helps detect dataset bias and shortcut learning</li></ul><h3>References</h3><ul><li>PyTorch Documentation</li><li>Captum Documentation</li><li>Axiomatic Attribution for Deep Networks (Integrated Gradients paper)</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=25364073cce5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NTK-Aware Interpolation in YaRN — The Missing Intuition Behind Long Context LLMs]]></title>
            <link>https://medium.com/@sankhoroy/ntk-aware-interpolation-in-yarn-the-missing-intuition-behind-long-context-llms-54fa81494b57?source=rss-ff631f66b7d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/54fa81494b57</guid>
            <category><![CDATA[neural-tangent-kernel]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[positonal-embedding]]></category>
            <category><![CDATA[yarn]]></category>
            <dc:creator><![CDATA[sankho roy]]></dc:creator>
            <pubDate>Sat, 16 May 2026 21:38:19 GMT</pubDate>
            <atom:updated>2026-05-16T21:38:19.641Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QT5RLwfD9tcE6qmHocBxWQ.png" /><figcaption>Ntk of positional embedding(RopE) vs Yarn</figcaption></figure><p>Large Language Models like GPT, Llama, Qwen, and DeepSeek can now handle context windows of 128K, 1M, or even more tokens.</p><p>But transformers were never originally designed for such huge contexts.</p><p>So how do models suddenly read books, repositories, or hours of conversations?</p><p>One of the biggest tricks is extending Rotary Positional Embeddings (RoPE) using methods like YaRN.</p><p>And inside YaRN, you often hear a mysterious phrase:</p><blockquote><em>“NTK-aware interpolation.”</em></blockquote><p>Most explanations immediately jump into advanced math and Neural Tangent Kernel theory.</p><p>But the core idea is actually intuitive.</p><p>This article explains:</p><ul><li>what a kernel really is</li><li>what Neural Tangent Kernel (NTK) means</li><li>why RoPE breaks at long contexts</li><li>why naive interpolation fails</li><li>how NTK-aware scaling works in YaRN</li><li>formulas + PyTorch code</li><li>deep intuition without assuming advanced math</li></ul><h3>What is a Kernel?</h3><p>A kernel is simply:</p><blockquote><em>a similarity function.</em></blockquote><p>Mathematically:</p><p>K(x, y)</p><p>It takes two things and tells us:</p><ul><li>large value → similar</li><li>small value → different</li></ul><p>That’s it.</p><h3>Simplest Kernel: Dot Product</h3><p>The most basic kernel is:</p><p>K(x, y) = x^T y</p><p>Example:</p><p>x = [1, 2]<br>y = [3, 4]</p><p>Then:</p><p>K(x, y) = 1×3 + 2×4 = 11</p><p>Large positive value means the vectors point in similar directions.</p><h3>PyTorch Example</h3><pre>import torch</pre><pre>x = torch.tensor([1., 2.])<br>y = torch.tensor([3., 4.])</pre><pre>kernel = torch.dot(x, y)</pre><pre>print(kernel)</pre><p>Output:</p><pre>tensor(11.)</pre><h3>Transformers Also Use Kernels</h3><p>Attention in transformers is:</p><p>Attention(Q, K, V) = softmax(QK^T / sqrt(d))V</p><p>The important part is:</p><p>QK^T</p><p>That is fundamentally a similarity computation.</p><p>Transformers constantly ask:</p><blockquote><em>“Which tokens are most similar or relevant?”</em></blockquote><h3>Example of Attention Similarity</h3><p>Suppose:</p><p>q = [1, 2]</p><p>k1 = [1, 2]<br>k2 = [5, -1]</p><p>Then:</p><p>q · k1 = 5<br>q · k2 = 3</p><p>So token 1 is considered more relevant.</p><h3>PyTorch Attention Example</h3><pre>import torch</pre><pre>q = torch.tensor([1., 2.])</pre><pre>K = torch.tensor([<br>    [1., 2.],<br>    [5., -1.]<br>])</pre><pre>scores = q @ K.T</pre><pre>print(scores)</pre><h3>What is the Neural Tangent Kernel (NTK)?</h3><p>The Neural Tangent Kernel sounds scary, but the core idea is:</p><blockquote><em>“How similarly does the network react to two inputs?”</em></blockquote><p>Suppose a neural network:</p><p>f(x, θ)</p><p>where:</p><ul><li>x = input</li><li>θ = model weights</li></ul><p>NTK measures whether two inputs affect the model’s weights in similar ways.</p><h3>NTK Formula</h3><p>K(x, x’) = ∇θ f(x)^T ∇θ f(x’)</p><p>This compares gradients.</p><p>Gradient means:</p><blockquote><em>“How much does each weight matter for this input?”</em></blockquote><p>If two inputs produce similar gradients, they influence the network similarly.</p><p>Then NTK is large.</p><h3>Intuition</h3><p>Suppose:</p><p>For input A:</p><p>∇θ f(A) = [1, 0, 5]</p><p>For input B:</p><p>∇θ f(B) = [1, 0, 4]</p><p>Dot product:</p><p>1×1 + 0×0 + 5×4 = 21</p><p>Large value → similar behavior.</p><h3>Tiny NTK PyTorch Example</h3><pre>import torch<br>import torch.nn as nn</pre><pre>model = nn.Sequential(<br>    nn.Linear(2, 3),<br>    nn.ReLU(),<br>    nn.Linear(3, 1)<br>)</pre><pre>x1 = torch.tensor([[1., 2.]])<br>x2 = torch.tensor([[2., 1.]])</pre><pre># gradients for x1<br>out1 = model(x1)</pre><pre>grads1 = torch.autograd.grad(<br>    out1,<br>    model.parameters(),<br>    retain_graph=True<br>)</pre><pre>g1 = torch.cat([g.flatten() for g in grads1])</pre><pre># gradients for x2<br>out2 = model(x2)</pre><pre>grads2 = torch.autograd.grad(<br>    out2,<br>    model.parameters()<br>)</pre><pre>g2 = torch.cat([g.flatten() for g in grads2])</pre><pre># NTK similarity<br>ntk = torch.dot(g1, g2)</pre><pre>print(ntk)</pre><h3>Where RoPE Comes In</h3><p>Transformers need positional information.</p><p>RoPE (Rotary Positional Embedding) encodes positions using rotations.</p><p>For each dimension:</p><p>θ_i(p) = p × ω_i</p><p>where:</p><ul><li>p = token position</li><li>ω_i = frequency</li></ul><p>Attention then depends on terms like:</p><p>cos((p − q)ω_i)</p><p>This means token similarity depends on positional phase.</p><h3>The Long Context Problem</h3><p>Suppose the model trained at:</p><p>4096 tokens</p><p>Now we suddenly use:</p><p>128000 tokens</p><p>Then:</p><p>θ = pω</p><p>becomes huge.</p><p>High-frequency dimensions rotate too rapidly.</p><p>Attention becomes unstable.</p><p>Tokens start looking unrelated or aliased.</p><h3>Naive Position Interpolation</h3><p>Old solution:</p><p>Compress positions.</p><p>p’ = p × (L_train / L_new)</p><p>Example:</p><p>p’ = p × (4096 / 131072)</p><p>This squeezes 128K positions into the original 4K range.</p><p>Good:</p><ul><li>avoids huge angles</li></ul><p>Bad:</p><ul><li>destroys local precision</li><li>nearby tokens become overly similar</li><li>attention geometry changes</li></ul><h3>Why This Breaks the Model</h3><p>Remember:</p><p>Transformers learned specific similarity relationships.</p><p>Originally:</p><ul><li>nearby tokens = highly related</li><li>medium distance = somewhat related</li><li>far distance = less related</li></ul><p>Bad scaling distorts these relationships.</p><p>The transformer suddenly sees a completely different positional geometry.</p><h3>NTK-Aware Scaling</h3><p>This is the key idea.</p><p>Instead of blindly scaling all frequencies equally:</p><p>ω_i’ = ω_i / s</p><p>YaRN scales dimensions differently.</p><p>Low-frequency dimensions:</p><ul><li>mostly unchanged</li></ul><p>High-frequency dimensions:</p><ul><li>scaled down more aggressively</li></ul><p>So:</p><p>ω_i’ = ω_i × s_i</p><p>where scaling factor depends on dimension.</p><h3>Why High Frequencies Matter</h3><p>High-frequency RoPE dimensions encode local detail.</p><p>They rotate very fast.</p><p>Tiny position changes create massive phase shifts.</p><p>If extended badly:</p><ul><li>local token relationships collapse</li><li>nearby words become hard to distinguish</li></ul><p>NTK-aware scaling carefully slows these dimensions.</p><h3>The Real Goal</h3><p>The actual goal is:</p><p>K_new ≈ K_old</p><p>Meaning:</p><blockquote><em>preserve the original attention similarity behavior.</em></blockquote><p>That is why it is called:</p><p>NTK-aware.</p><p>It tries to preserve the transformer’s effective kernel.</p><h3>RoPE Kernel Formula</h3><p>RoPE attention behaves roughly like:</p><p>K(p, q) = cos((p − q)ω)</p><p>If ω changes too much:</p><p>K_new ≠ K_old</p><p>Attention geometry changes.</p><p>Model quality drops.</p><h3>PyTorch Kernel Comparison</h3><pre>import torch<br></pre><pre>def rope_kernel(distance, omega):<br>    return torch.cos(distance * omega)<br></pre><pre>distances = torch.arange(0, 20)</pre><pre>omega_old = 0.1<br>omega_new = 0.08</pre><pre>old_kernel = rope_kernel(distances, omega_old)<br>new_kernel = rope_kernel(distances, omega_new)</pre><pre>kernel_difference = torch.mean(<br>    (old_kernel - new_kernel) ** 2<br>)</pre><pre>print(kernel_difference)</pre><p>Small difference means:</p><blockquote><em>new attention behaves similarly to old attention.</em></blockquote><h3>The Big Intuition</h3><p>Without NTK-aware scaling:</p><ul><li>positional relationships distort</li><li>local attention collapses</li><li>transformer behaves unpredictably</li></ul><p>With NTK-aware scaling:</p><ul><li>attention geometry stays stable</li><li>local structure preserved</li><li>long-context extension becomes smoother</li></ul><h3>5-Year-Old Analogy</h3><p>Imagine every token spins a wheel.</p><p>Some wheels spin slowly.<br>Some spin very fast.</p><p>At short context:</p><p>everything is synchronized.</p><p>At huge context:</p><p>fast wheels spin so much they become random blur.</p><p>Naive interpolation slows ALL wheels equally.</p><p>Problem:</p><p>slow wheels become too slow.</p><p>YaRN only slows the crazy-fast wheels.</p><p>So the system stays balanced.</p><h3>Final Summary</h3><p>Kernel:</p><blockquote><em>similarity function</em></blockquote><p>Neural Tangent Kernel:</p><blockquote><em>measures how similarly a neural network reacts to two inputs</em></blockquote><p>RoPE:</p><blockquote><em>changes transformer similarity using rotational phases</em></blockquote><p>NTK-aware interpolation:</p><blockquote><em>extends context while preserving the transformer’s original attention similarity behavior</em></blockquote><p>YaRN:</p><blockquote><em>selectively rescales RoPE frequencies to preserve long-context stability</em></blockquote><p>That single idea is why modern LLMs can stretch from 4K context to 128K+ without completely breaking.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=54fa81494b57" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Evolution of Deep Learning Optimizers]]></title>
            <link>https://medium.com/@sankhoroy/the-evolution-of-deep-learning-optimizers-c9fde462f41e?source=rss-ff631f66b7d4------2</link>
            <guid isPermaLink="false">https://medium.com/p/c9fde462f41e</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[optimizer]]></category>
            <dc:creator><![CDATA[sankho roy]]></dc:creator>
            <pubDate>Sat, 16 May 2026 19:51:40 GMT</pubDate>
            <atom:updated>2026-05-20T20:27:58.185Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vjU494_QFlhskUAGUevv4g.png" /><figcaption>Evolution Tree of optimisers</figcaption></figure><h3>From Gradient Descent to Muon, SOAP, and Distributed Shampoo</h3><h3>Introduction</h3><p>Training a neural network is fundamentally an optimization problem.</p><p>A model starts with random weights and gradually updates them to minimize a loss function. The algorithm responsible for deciding <em>how</em> these weights are updated is called an <strong>optimizer</strong>.</p><p>Over the past several decades, optimizers have evolved dramatically:</p><ul><li>from simple gradient descent,</li><li>to adaptive learning rate methods,</li><li>to large-scale distributed optimizers,</li><li>and finally to geometry-aware methods designed for modern large language models.</li></ul><p>This evolution mirrors the increasing complexity of machine learning systems:</p><p>EraMain ChallengeEarly MLBasic optimizationDeep LearningStability and speedTransformersMemory and scalingLLM EraDistributed geometry-aware optimization</p><p>This article walks through the entire optimizer evolution tree and explains what each generation improved.</p><h3>1. Gradient Descent (GD)</h3><p>Gradient Descent is the foundation of nearly all neural network training.</p><p>The idea is simple:</p><blockquote><em>Move parameters in the direction that reduces loss.</em></blockquote><p>Mathematically:</p><p>[<br>\theta_{t+1} = \theta_t — \eta \nabla L(\theta_t)<br>]</p><p>Where:</p><ul><li>(\theta) = model parameters</li><li>(\eta) = learning rate</li><li>(\nabla L) = gradient of the loss</li></ul><p>The gradient tells us which direction increases loss the fastest. By moving in the opposite direction, we reduce the loss.</p><h3>Limitations</h3><p>Gradient Descent computes gradients using the <em>entire dataset</em> before every update.</p><p>This causes:</p><ul><li>slow training,</li><li>large memory usage,</li><li>poor scalability.</li></ul><p>As datasets grew larger, a faster method became necessary.</p><h3>2. Stochastic Gradient Descent (SGD)</h3><p>Stochastic Gradient Descent introduced randomness into optimization.</p><p>Instead of computing gradients on the full dataset, SGD uses a single sample:</p><p>[<br>\theta_{t+1} = \theta_t — \eta \nabla L_i(\theta_t)<br>]</p><p>This made updates much faster.</p><h3>Improvements Over GD</h3><ul><li>Faster updates</li><li>Online learning possible</li><li>Better scalability</li></ul><h3>New Problem Introduced</h3><p>SGD updates became noisy.</p><p>Instead of smoothly descending toward minima, the optimizer zig-zagged heavily.</p><p>This instability motivated the next generation.</p><h3>3. Mini-Batch SGD</h3><p>Mini-batch SGD became the industry standard.</p><p>Instead of using:</p><ul><li>all samples (GD), or</li><li>one sample (SGD),</li></ul><p>it uses small batches such as:</p><ul><li>32</li><li>64</li><li>128</li><li>1024</li></ul><p>samples at a time.</p><h3>Why Mini-Batches Matter</h3><p>Mini-batching balances:</p><p>BenefitResultFaster computationGPU accelerationLess noiseStable convergenceBetter hardware utilizationParallel processing</p><p>This remains the default training setup today.</p><h3>4. Momentum</h3><p>SGD suffers from oscillation, especially in steep valleys.</p><p>Momentum adds velocity to updates.</p><p>Instead of relying only on the current gradient, it also remembers previous directions.</p><h3>Formula</h3><p>Velocity:</p><p>[<br>v_t = \beta v_{t-1} + (1-\beta)g_t<br>]</p><p>Update:</p><p>[<br>\theta_{t+1} = \theta_t — \eta v_t<br>]</p><h3>Key Innovation</h3><p>Momentum introduced:</p><blockquote><em>Optimization with memory.</em></blockquote><p>The optimizer now accumulates consistent directions and suppresses oscillations.</p><h3>Impact</h3><p>Momentum significantly accelerated deep learning optimization and became a core building block for future methods.</p><h3>5. Nesterov Momentum</h3><p>Nesterov Momentum improved classical momentum by introducing a “look-ahead” mechanism.</p><p>Instead of computing gradients at the current position, it estimates where momentum will move the parameters next.</p><h3>Formula</h3><p>[<br>g_t = \nabla L(\theta_t — \beta v_{t-1})<br>]</p><p>This anticipatory update improves stability.</p><h3>Improvement</h3><ul><li>Smarter momentum</li><li>Reduced overshooting</li><li>Faster convergence</li></ul><p>Nesterov momentum became popular in CNN training before Adam dominated transformers.</p><h3>6. AdaGrad</h3><p>AdaGrad introduced one of the most important ideas in optimizer history:</p><h3>Adaptive Learning Rates</h3><p>Instead of using one learning rate for all parameters, AdaGrad assigns different effective learning rates to different parameters.</p><h3>Formula</h3><p>[<br>G_t = G_{t-1} + g_t²<br>]</p><p>Update:</p><h3>[<br>\theta_{t+1}</h3><h3>\theta_t</h3><p>\frac{\eta}{\sqrt{G_t + \epsilon}}g_t<br>]</p><h3>Main Insight</h3><p>Parameters receiving large gradients repeatedly should slow down.</p><p>Rarely updated parameters should continue learning aggressively.</p><h3>Why AdaGrad Worked Well</h3><p>AdaGrad became extremely successful for:</p><ul><li>sparse features,</li><li>NLP,</li><li>embeddings.</li></ul><p>Rare words received larger effective learning rates.</p><h3>Problem</h3><p>The accumulated gradient term keeps growing forever:</p><p>[<br>G_t \to \infty<br>]</p><p>Eventually learning rates shrink close to zero.</p><p>Training effectively stops.</p><p>This inspired RMSProp.</p><h3>7. RMSProp</h3><p>RMSProp solved AdaGrad’s learning-rate decay problem.</p><p>Instead of summing all historical gradients, RMSProp keeps an exponentially weighted moving average.</p><h3>Formula</h3><p>[<br>v_t = \beta v_{t-1} + (1-\beta)g_t²<br>]</p><p>Update:</p><h3>[<br>\theta_{t+1}</h3><h3>\theta_t</h3><p>\frac{\eta}{\sqrt{v_t + \epsilon}}g_t<br>]</p><h3>Key Improvement</h3><p>Old gradients gradually fade away.</p><p>This prevents learning rates from collapsing.</p><h3>Impact</h3><p>RMSProp became highly effective for:</p><ul><li>RNNs,</li><li>deep networks,</li><li>non-stationary optimization.</li></ul><p>It also directly inspired Adam.</p><h3>8. Adam</h3><p>Adam (Adaptive Moment Estimation) became the dominant optimizer in modern deep learning.</p><p>It combines:</p><ul><li>Momentum</li><li>RMSProp</li></ul><p>into one framework.</p><h3>Formula</h3><p>First moment:</p><p>[<br>m_t = \beta_1 m_{t-1} + (1-\beta_1)g_t<br>]</p><p>Second moment:</p><p>[<br>v_t = \beta_2 v_{t-1} + (1-\beta_2)g_t²<br>]</p><p>Bias correction:</p><p>[<br>\hat m_t = \frac{m_t}{1-\beta_1^t}<br>]</p><p>[<br>\hat v_t = \frac{v_t}{1-\beta_2^t}<br>]</p><p>Update:</p><h3>[<br>\theta_{t+1}</h3><h3>\theta_t</h3><p>\eta<br>\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}<br>]</p><h3>Why Adam Dominated</h3><p>Adam is:</p><ul><li>fast,</li><li>stable,</li><li>adaptive,</li><li>easy to tune.</li></ul><p>It became the default optimizer for:</p><ul><li>transformers,</li><li>LLMs,</li><li>diffusion models,</li><li>reinforcement learning.</li></ul><h3>9. AdamW</h3><p>Although Adam was successful, researchers discovered a subtle problem.</p><p>Weight decay in Adam interacted incorrectly with adaptive scaling.</p><p>AdamW fixed this by decoupling weight decay from the gradient update.</p><h3>Formula</h3><h3>[<br>\theta_{t+1}</h3><h3>\theta_t</h3><h3>\eta<br>\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}</h3><p>\eta \lambda \theta_t<br>]</p><h3>Why AdamW Matters</h3><p>AdamW dramatically improved:</p><ul><li>transformer generalization,</li><li>large-scale language model training,</li><li>regularization stability.</li></ul><p>Today AdamW is the standard optimizer for transformer architectures.</p><h3>10. LAMB</h3><p>As models became larger, training required enormous batch sizes.</p><p>Adam struggled under extreme large-batch training.</p><p>LAMB (Layer-wise Adaptive Moments) solved this.</p><h3>Key Innovation</h3><p>LAMB introduces trust ratios:</p><p>[<br>r_t = \frac{||\theta_t||}{||u_t||}<br>]</p><p>where:</p><ul><li>(u_t) is the Adam update.</li></ul><p>Each layer scales updates independently.</p><h3>Impact</h3><p>LAMB enabled:</p><ul><li>batch sizes above 32K,</li><li>distributed transformer training,</li><li>efficient BERT pretraining.</li></ul><h3>11. Adafactor</h3><p>Transformer models introduced another major challenge:</p><h3>Memory Consumption</h3><p>Adam stores:</p><ul><li>first moments,</li><li>second moments.</li></ul><p>This doubles optimizer memory.</p><p>For giant language models, this becomes prohibitively expensive.</p><h3>Adafactor Innovation</h3><p>Adafactor factorizes second-moment matrices.</p><p>Instead of storing:</p><p>[<br>V \in \mathbb{R}^{m \times n}<br>]</p><p>it stores:</p><ul><li>row statistics,</li><li>column statistics.</li></ul><h3>Result</h3><p>Massive memory reduction.</p><p>Adafactor became highly important for:</p><ul><li>T5,</li><li>large transformer training,</li><li>memory-constrained optimization.</li></ul><h3>12. Lion</h3><p>Lion is a newer optimizer focused on simplicity and efficiency.</p><h3>Core Idea</h3><p>Instead of using exact gradient magnitudes, Lion updates using only the sign of momentum.</p><h3>Formula</h3><h3>[<br>\theta_{t+1}</h3><h3>\theta_t</h3><p>\eta \cdot sign(m_t)<br>]</p><h3>Why Lion Matters</h3><p>Lion reduces:</p><ul><li>memory usage,</li><li>computational overhead.</li></ul><p>It can also improve generalization in some transformer settings.</p><h3>13. BFGS</h3><p>Until now, most optimizers were first-order methods.</p><p>They relied only on gradients.</p><p>BFGS introduced:</p><h3>Second-Order Optimization</h3><p>Second-order methods attempt to understand the curvature of the loss surface.</p><h3>Hessian Matrix</h3><p>Curvature is described by the Hessian:</p><p>[<br>H = \nabla² L<br>]</p><p>Newton’s method uses:</p><h3>[<br>\theta_{t+1}</h3><p>\theta_t — H^{-1}g_t<br>]</p><p>But computing the Hessian directly is computationally infeasible for deep learning.</p><h3>BFGS Innovation</h3><p>BFGS approximates the inverse Hessian iteratively.</p><p>It learns curvature information from how gradients change over time.</p><h3>Limitation</h3><p>Memory complexity:</p><p>[<br>O(n²)<br>]</p><p>Too expensive for large neural networks.</p><h3>14. L-BFGS</h3><p>L-BFGS solved BFGS memory limitations.</p><p>Instead of storing the full Hessian approximation, it stores only recent history vectors.</p><h3>Key Idea</h3><p>Store:</p><ul><li>recent parameter changes,</li><li>recent gradient changes.</li></ul><p>This dramatically reduces memory.</p><h3>Why It Matters</h3><p>L-BFGS remains useful in:</p><ul><li>scientific optimization,</li><li>physics-informed neural networks,</li><li>smaller ML systems.</li></ul><p>However, deep learning usually relies on noisy mini-batch gradients, making SGD-style methods more practical.</p><h3>15. Shampoo</h3><p>As models continued scaling, researchers revisited second-order optimization.</p><p>Shampoo introduced matrix preconditioning.</p><h3>Core Idea</h3><p>Instead of assigning independent learning rates to each parameter, Shampoo models parameter interactions.</p><h3>Formula</h3><p>For weight matrix (W):</p><h3>[<br>W_{t+1}</h3><h3>W_t</h3><p>\eta<br>L_t^{-1/4}<br>G_t<br>R_t^{-1/4}<br>]</p><p>Where:</p><ul><li>(L_t) and (R_t) capture gradient covariance.</li></ul><h3>Key Advantage</h3><p>Shampoo understands geometry better than Adam.</p><p>It converges faster in many large-scale settings.</p><h3>Challenge</h3><p>Matrix operations are computationally expensive.</p><h3>16. Distributed Shampoo</h3><p>Distributed Shampoo scaled Shampoo across massive compute clusters.</p><h3>Main Innovations</h3><h3>1. Block Partitioning</h3><p>Large tensors are split into manageable blocks.</p><h3>2. Distributed Preconditioning</h3><p>Different accelerators compute different matrix statistics.</p><h3>3. Grafting</h3><p>Combines Shampoo’s direction with Adam-style learning-rate scaling.</p><h3>Impact</h3><p>Distributed Shampoo enabled second-order optimization for giant language models and TPU-scale training.</p><h3>17. SOAP</h3><p>SOAP combines:</p><ul><li>Shampoo geometry,</li><li>Adam stability.</li></ul><h3>Core Idea</h3><p>Gradients are projected into Shampoo eigenspaces.</p><p>Adam-style updates are then applied in the transformed basis.</p><h3>Why SOAP Matters</h3><p>SOAP improves:</p><ul><li>curvature handling,</li><li>optimization stability,</li><li>transformer scaling.</li></ul><p>It represents a practical compromise between first-order and second-order optimization.</p><h3>18. Muon</h3><p>Muon represents one of the newest directions in optimizer research.</p><p>Instead of merely scaling gradients, Muon focuses on preserving geometric structure.</p><h3>Key Innovation</h3><p>Muon orthogonalizes momentum updates.</p><p>Using methods such as Newton-Schulz iteration, it computes orthogonalized updates:</p><h3>[<br>W_{t+1}</h3><p>W_t — \eta \cdot Orth(M_t)<br>]</p><h3>Why This Is Important</h3><p>Traditional optimizers distort weight spaces.</p><p>Muon attempts to preserve:</p><ul><li>representation geometry,</li><li>orthogonality,</li><li>stable feature structure.</li></ul><p>This is especially promising for:</p><ul><li>transformers,</li><li>large language models,</li><li>deep representation learning.</li></ul><h3>The Grand Pattern of Optimizer Evolution</h3><p>Each generation solved a different bottleneck.</p><p>ProblemSolutionSlow trainingSGDOscillationMomentumUneven parameter scalingAdaGradVanishing learning ratesRMSPropStabilityAdamGeneralizationAdamWLarge batchesLAMBMemory usageAdafactorCurvature understandingShampooDistributed scalingDistributed ShampooGeometry preservationMuon</p><h3>Modern Optimizer Landscape</h3><p>Today’s deep learning ecosystem uses different optimizers depending on scale and architecture.</p><p>ApplicationCommon OptimizerCNNsSGD + MomentumTransformersAdamWHuge LLMsAdamW / AdafactorExperimental LLM researchSOAP / Muon / Shampoo</p><h3>Conclusion</h3><p>The history of optimizers reflects the evolution of machine learning itself.</p><p>Optimization began with simple gradient descent and evolved through:</p><ul><li>stochastic learning,</li><li>momentum,</li><li>adaptive learning rates,</li><li>second-order geometry,</li><li>distributed scalability,</li><li>and representation-preserving updates.</li></ul><p>Modern optimizers are no longer just about moving downhill.</p><p>They now attempt to:</p><ul><li>understand geometry,</li><li>preserve representations,</li><li>scale across thousands of accelerators,</li><li>and efficiently train trillion-parameter systems.</li></ul><p>As large language models continue growing, optimizer research is becoming one of the most important frontiers in deep learning.</p><p>The next generation of breakthroughs may come not from larger models — but from better optimization.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c9fde462f41e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>