<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Himanshu Nanda on Medium]]></title>
        <description><![CDATA[Stories by Himanshu Nanda on Medium]]></description>
        <link>https://medium.com/@himanshunanda2002?source=rss-6a2b4ec91d48------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*kQCVPW1mS4xyJb2ESJj6bw.jpeg</url>
            <title>Stories by Himanshu Nanda on Medium</title>
            <link>https://medium.com/@himanshunanda2002?source=rss-6a2b4ec91d48------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 06:43:07 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@himanshunanda2002/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Your SLM Is Small. So Why Is Inference Still Expensive?]]></title>
            <link>https://medium.com/@himanshunanda2002/your-slm-is-small-so-why-is-inference-still-expensive-593795550ed4?source=rss-6a2b4ec91d48------2</link>
            <guid isPermaLink="false">https://medium.com/p/593795550ed4</guid>
            <category><![CDATA[slm-infernce]]></category>
            <category><![CDATA[lms]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[generative-ai]]></category>
            <dc:creator><![CDATA[Himanshu Nanda]]></dc:creator>
            <pubDate>Sat, 05 Sep 2026 09:19:15 GMT</pubDate>
            <atom:updated>2026-09-05T09:19:15.411Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h4fL0WKYebUYfFvP1xM7Fw.png" /></figure><h3>A mental model for understanding Partial Loading, KV Caching, and Transformer architectures</h3><p>A 2B parameter model sounds small.</p><p>You quantize it. You fine-tune it. You put it on a GPU.</p><p>And then you expect inference to be easy.</p><p>But production has a funny way of exposing assumptions.</p><ul><li>The model fits in memory, yet latency is high.</li><li>You add more GPU capacity, yet cost rises faster than throughput.</li><li>You reduce the model size, yet long-context requests still consume a surprising amount of memory.</li></ul><p>So what exactly are we paying for during inference?</p><p>The answer isn’t simply “the model.” We’re paying for a combination of: <strong>weights, computation, memory, data movement, attention state, context length, and concurrency.</strong></p><p>The interesting part is that the balance between these changes with the architecture and the workload.</p><p>Two techniques make this particularly interesting:</p><ul><li><strong>Partial Loading: </strong>Where should the model weights live?</li><li><strong>KV Caching: </strong>What computation have I already done that I don’t need to repeat?</li></ul><p>They both optimize inference. But they solve very different problems.</p><h3>First: Think of inference as a resource problem</h3><p>At a very high level, we can think about inference as:</p><pre>T_inference ≈ T_compute + T_memory + T_movement</pre><p>This isn’t a complete latency equation, it’s a mental model.</p><p>Every inference system is constantly balancing:</p><ul><li><strong>Compute: </strong>How much work does the accelerator need to perform?</li><li><strong>Memory: </strong>How much information needs to remain available?</li><li><strong>Data movement: </strong>How much information needs to move between memory tiers?</li></ul><p>This distinction matters because an optimization that improves one dimension can make another dimension worse. That is exactly what happens with partial loading and KV caching.</p><h3>Partial Loading: “Do I need all the weights here?”</h3><p>A model’s parameters don’t necessarily have to live in the fastest memory all the time. Conceptually:</p><pre>W = W_GPU + W_CPU</pre><p>Some parameters can remain outside GPU memory and be moved closer to the accelerator when required.</p><p>The immediate benefit is obvious: <strong>GPU memory pressure goes down.</strong></p><p>But there is no free lunch. Now we have data movement:</p><pre>T_movement ≈ Data Size / Bandwidth</pre><p>So partial loading creates a trade-off: <strong>less memory pressure → more data movement.</strong></p><p>This is useful when memory capacity is your constraint. But if bandwidth or transfer latency is your constraint, moving weights around can become the bottleneck.</p><blockquote><strong><em>Inference lesson #1:</em></strong><em> Saving memory does not automatically mean saving latency.</em></blockquote><h3>KV Caching: “Why am I calculating this again?”</h3><p>Now consider autoregressive generation. Suppose the model has already processed <em>“The cat is sitting”</em> and is generating the next token.</p><p>Inside every attention layer:</p><pre>Q = X W_Q<br>K = X W_K<br>V = X W_V</pre><pre>Attention(Q, K, V) = softmax(QKᵀ / √d_k) V</pre><p>The key observation: <strong>the Keys and Values belonging to previous tokens don’t change simply because we’re generating another token.</strong> So why calculate them again?</p><p>Instead, we store them:</p><pre>K_cache = [K1, K2, ..., Kt]<br>V_cache = [V1, V2, ..., Vt]</pre><p>When the next token arrives, its new K and V are added to the cache, while the previous values are reused. That’s the basic idea behind KV caching.</p><blockquote><strong><em>The mental model:</em></strong><em> Don’t pay twice for computation you’ve already done.</em></blockquote><p>But there’s a catch — the cache itself consumes memory:</p><pre>M_KV ∝ L × H_KV × D × T</pre><p>Where L = number of transformer layers, H_KV = number of KV heads, D = head dimension, T = sequence length.</p><p>So:</p><ul><li>Context ↑ ⇒ KV memory ↑</li><li>Concurrent requests ↑ ⇒ KV memory ↑</li></ul><p>The optimization that saved computation has now created a memory problem.</p><blockquote><strong><em>Inference lesson #2:</em></strong><em> Reducing computation can increase memory pressure.</em></blockquote><h3>Partial Loading vs. KV Caching</h3><p>Partial Loading KV Caching <strong>Main question</strong> Where should weights live? What have I already computed? <strong>Primary target</strong> Memory capacity Repeated computation <strong>Main benefit</strong> Lower GPU memory footprint Faster autoregressive decoding <strong>New cost</strong> Data movement Cache memory <strong>Depends heavily on</strong> Memory hierarchy &amp; bandwidth Context length &amp; concurrency <strong>Typical bottleneck</strong> Transfer latency Memory capacity/bandwidth</p><p>Put simply: <strong>Partial loading optimizes where the parameters live. KV caching optimizes how often computation happens.</strong></p><p>This distinction becomes even more interesting once we look at architecture.</p><h3>But Transformers Don’t All Pay the Same Inference Bill</h3><p>The word “Transformer” hides an important detail — not every Transformer performs inference the same way. There are broadly three architectures worth keeping in mind:</p><ul><li>Encoder-only</li><li>Decoder-only</li><li>Encoder-decoder</li></ul><p>The original Transformer was introduced as an encoder-decoder architecture, while GPT-style systems use decoder-only architectures — and that changes what can be cached, when it can be cached, and where memory is spent.</p><h3>1. Decoder-only: the KV-cache playground</h3><p>Think GPT, Llama, Mistral, Phi, Gemma and similar autoregressive models. During generation, at step <em>t</em>:</p><pre>q_t = x_t W_Q<br>k_t = x_t W_K<br>v_t = x_t W_V</pre><p>The cache becomes K_cache = [k1, ..., kt] and V_cache = [v1, ..., vt]. At the next step, the new query attends over the accumulated K/V states.</p><p>One small but important observation: <strong>Q usually doesn’t need to persist the same way.</strong> The current query interrogates the existing context; the keys and values represent the context that needs to survive across decoding steps. Hence: <strong>KV Cache — not QKV Cache.</strong></p><h3>2. Attention architecture changes the equation</h3><p>Not every model has the same number of K/V heads:</p><ul><li>Standard Multi-Head Attention: H_KV ≈ H_Q</li><li>Multi-Query Attention: H_KV = 1</li><li>Grouped-Query Attention: somewhere between the two</li></ul><p>Since M_KV ∝ H_KV, reducing the number of KV heads directly reduces the memory required by the cache. So inference efficiency isn&#39;t only an inference-engine problem — it can also be an <em>architecture</em> problem. A model designed with fewer KV heads has a fundamentally different memory profile during inference.</p><h3>3. Encoder-only: a different game</h3><p>Think BERT-style models. The encoder processes the input sequence together rather than generating one token after another, so the classic autoregressive KV-cache story doesn’t apply in exactly the same way. There isn’t necessarily a growing autoregressive KV state for every generated token — the optimization target is different.</p><h3>4. Encoder-decoder: now there are two attention paths</h3><p>Think T5-style architectures. The encoder first processes the input (X → H_encoder), and the decoder generates tokens while attending to that representation. The decoder has two forms of attention:</p><p><strong>Decoder self-attention</strong> (Q_d, K_d, V_d from X_d) — evolves autoregressively and can be cached.</p><p><strong>Cross-attention</strong> , the decoder queries the encoder representation: Q_d from the decoder, while K_e, V_e come from H_encoder. The encoder-derived K/V can be reused while the decoder continues generating.</p><p>So the inference picture becomes: <strong>Encoder Representation → Cross-Attention + Decoder KV Cache.</strong></p><h3>The Hidden Inference Tax: Context</h3><p>Here’s where KV caching becomes especially important. A more complete estimate:</p><pre>M_KV = 2 × L × H_KV × D × T × B × S</pre><p>Where L = layers, H_KV = KV heads, D = head dimension, T = sequence length, B = batch/concurrency, S = bytes per element.</p><p>The exact implementation varies, but the relationship is what matters:</p><pre>M_KV ∝ T × B × H_KV × L</pre><p>This means a model can be small in terms of parameters and still become expensive to serve. Why? <strong>Model size and inference state are not the same thing.</strong></p><h3>And Then Comes Prefix Caching</h3><p>KV caching usually makes us think about reuse <em>within</em> one request. But what if multiple requests share the same prefix — for example, system instructions + common context + user-specific input?</p><p>If the first two components are identical across many requests, why recompute their attention state every time? That’s where prefix caching comes in: modern inference systems cache KV blocks from previously processed prefixes and reuse them when a new request shares that prefix. vLLM’s Automatic Prefix Caching documentation describes this using hashed KV-cache blocks.</p><p>This changes the question from <em>“What have I already calculated in this request?”</em> to <em>“What has the system already calculated for another request?”</em> — a much more interesting optimization at scale.</p><h3>When the KV Cache Becomes the Problem</h3><p>Here’s the paradox: KV caching exists to save compute, but the cache itself can become the memory bottleneck. This is why production inference systems care about things like:</p><ul><li>PagedAttention</li><li>Continuous batching</li><li>Cache quantization</li><li>Cache offloading</li><li>Prefix caching</li><li>GQA/MQA</li><li>Sliding-window attention</li></ul><p>PagedAttention, for example, treats KV-cache memory more like virtual memory — cache blocks can be stored non-contiguously rather than requiring one large contiguous allocation. The original vLLM work identified KV-cache memory management as a major serving bottleneck. Hugging Face also exposes cache strategies including dynamic, static, quantized, and offloaded approaches — KV caching itself is now an optimization space, not a single technique.</p><h3>What About FlashAttention?</h3><p>KV caching and FlashAttention are <strong>not</strong> the same optimization.</p><ul><li>KV caching reduces redundant computation across autoregressive decoding steps.</li><li>FlashAttention makes attention itself more efficient by being IO-aware — reducing expensive memory reads/writes between GPU memory levels through tiling.</li></ul><p>They attack different parts of the problem:</p><pre>Inference<br> ├── Where do weights live?        → Partial Loading / Offloading<br> ├── What can I reuse?             → KV / Prefix Caching<br> ├── How much KV state do I need?  → GQA / MQA / Quantization<br> └── How efficiently do I move data? → IO-aware attention / kernels</pre><h3>So What Should We Optimize?</h3><p>Before reaching for an optimization technique, ask four questions:</p><ol><li><strong>Is the model too large for the fastest memory?</strong> → weight placement, quantization, partial loading, offloading.</li><li><strong>Am I repeatedly computing the same context?</strong> → KV caching, prefix caching.</li><li><strong>Is the KV cache itself becoming the bottleneck?</strong> → GQA/MQA, quantization, cache management, context length, batching.</li><li><strong>Is the real problem computation or memory movement?</strong> → kernel optimization, FlashAttention, batching, speculative decoding, accelerator utilization.</li></ol><p>These aren’t competing tricks — they attack different terms in the inference budget.</p><h3>The Mental Model I Keep Coming Back To</h3><p>When someone says <em>“It’s only a 2B model,”</em> that tells you surprisingly little about its production inference cost. You still need to ask:</p><ul><li>How are the weights stored?</li><li>How much data is moving?</li><li>How long is the context?</li><li>How large is the KV cache?</li><li>How many requests are running concurrently?</li><li>What attention architecture does the model use?</li><li>What is being recomputed? What can be reused?</li></ul><p>A model’s parameter count tells you about the model. <strong>Inference performance tells you about the system around the model.</strong></p><p>And perhaps the simplest mental model is:</p><pre>Inference Efficiency ≈ Compute + Memory + Data Movement</pre><p>The interesting engineering isn’t simply making the model smaller — it’s figuring out which resource you’re actually paying for.</p><h3>Further Reading</h3><ul><li><a href="https://arxiv.org/abs/1706.03762">Attention Is All You Need</a> — the original Transformer paper</li><li><a href="https://huggingface.co/docs/transformers/en/kv_cache">Hugging Face — How caching works</a> — practical explanation of attention and KV caching</li><li><a href="https://tinycomputers.io/posts/partial-llm-loading-running-models-too-big-for-vram.html">Partial LLM Loading: Running Models Too Big for VRAM</a> — a hands-on look at layer streaming, CPU offloading, and the real-world latency trade-offs of partial loading</li><li><a href="https://docs.vllm.ai/en/latest/design/automatic_prefix_caching.html">vLLM — Automatic Prefix Caching</a> — how KV blocks can be reused across requests</li><li><a href="https://arxiv.org/abs/2309.06180">PagedAttention / vLLM</a> — memory management for LLM serving</li><li><a href="https://arxiv.org/abs/2205.14135">FlashAttention</a> — IO-aware attention optimization</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=593795550ed4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I Tried to Fit a Line to Some Data… and Ended Up Questioning How Reality Generates Points]]></title>
            <link>https://medium.com/@himanshunanda2002/i-tried-to-fit-a-line-to-some-data-and-ended-up-questioning-how-reality-generates-points-e003c18235a7?source=rss-6a2b4ec91d48------2</link>
            <guid isPermaLink="false">https://medium.com/p/e003c18235a7</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Himanshu Nanda]]></dc:creator>
            <pubDate>Sun, 15 Mar 2026 20:05:13 GMT</pubDate>
            <atom:updated>2026-03-15T20:05:13.714Z</atom:updated>
            <content:encoded><![CDATA[<p>(A beginner-friendly and slightly confused journey into linear models, least squares, and why Gaussian blobs secretly run the universe.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zGbKnEYPdQz0kBrv1I7dgg.png" /><figcaption>Sneaky Gaussian</figcaption></figure><h3>It Started With a Simple Goal</h3><p>I sat down with a machine learning book thinking:</p><blockquote><em>“Today I will learn how to draw a line through some data.”</em></blockquote><p>That’s it. Nothing fancy. Just a line… or so I thought</p><p>Weird, right?</p><p>But somewhere between <strong>least squares</strong>, <strong>Gaussian distributions</strong>, and <strong>decision boundaries</strong>, I realized something surprising:</p><blockquote><em>Machine learning isn’t just about fitting models, it’s about understanding </em><strong><em>how data is generated</em></strong><em>.</em></blockquote><p>And that idea completely changed how I look at ML.</p><p>Let’s walk through what happened.</p><h3>Step 1: The Classic Setup</h3><p>Imagine you have data points like this:</p><ul><li><strong>x</strong> → input features</li><li><strong>y</strong> → output values</li></ul><p>We assume a simple relationship between them:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/155/1*qwAFe2qeyyBsL86l14YnlQ.png" /></figure><p>Where:</p><ul><li><strong>X</strong> → matrix of input features</li><li><strong>β</strong> → coefficients we want to learn</li><li><strong>ε</strong> → noise (because reality is messy)</li></ul><p>In simple terms:</p><blockquote><strong><em>Observed Data = Pattern + Randomness</em></strong></blockquote><p>Our goal?</p><p>Find <strong>β</strong> such that the model predictions are as close as possible to the actual values.</p><h3>Step 2: Meet the Residual (aka The Error)</h3><p>Every prediction makes a tiny mistake.</p><p>That mistake is called a <strong>residual</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/150/1*uKgk4r4sj8A3iww-mhHi7A.png" /></figure><p>Which simply means:</p><blockquote><strong><em>Residual = Actual value − Predicted value</em></strong></blockquote><p>If we stack all residuals together, we get the <strong>residual vector</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/157/1*n7-_E8bayrRjLxdZXoN7Fw.png" /></figure><p>You can think of this as the model’s personal diary titled:</p><blockquote>“All the places I messed up.”</blockquote><h3>Step 3: Squaring Our Mistakes (Because Positivity Matters)</h3><p>If we simply summed the residuals, positive and negative errors could cancel out.</p><p>That would make our model look suspiciously good.</p><p>So instead we <strong>square the errors</strong>.</p><p>This gives us the <strong>Residual Sum of Squares (RSS)</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/318/1*ewrNoJGUecJM1mcfrozVJg.png" /></figure><p>In words:</p><blockquote><em>Add up all the squared mistakes your model made.</em></blockquote><p>Using linear algebra, we can write this much more elegantly:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/355/1*GSI-KgYXF43Z8Yke8Gvm4g.png" /></figure><p>Our optimization goal becomes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/157/1*-HUSxYbE42LIk3lxicb5Kg.png" /></figure><p>Translation:</p><blockquote><em>Find the coefficients that make the prediction errors as small as possible.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B2xy_voZMKFQvqm6Zexs7Q.png" /><figcaption>Residuals measure the vertical distance between predicted values and actual observations.</figcaption></figure><h3>Step 4: Deriving the Least Squares Solution</h3><p>Let’s expand the RSS expression:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/349/1*lTJn-jnIk-XF_87QjQC2Pw.png" /></figure><p>Multiplying it out gives:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/481/1*lblZL42Jzku7OTgIkzn42w.png" /></figure><p>Because the middle terms are scalars:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/238/1*d0Wyih7tgwx7YQjMN713kw.png" /></figure><p>So we simplify:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/425/1*_tXVo281vDHNkvfM9ZkDSA.png" /></figure><p>Now we differentiate with respect to <strong>β</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/325/1*eTBd3hVrbopXejps8fz_bg.png" /></figure><p>Follow the below rule to ease up the calculations till next Step:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/317/1*NPMpfyYwCrq3kv9EP1zJYA.png" /></figure><p>Since, for quadratic forms:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/516/1*uDGBgYg3s9yOYeJa30P80w.png" /></figure><p>Now, set the derivative to zero:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/187/1*_ONZhGy4yxtn4-CSRwA1-w.png" /></figure><p>Solving for <strong>β</strong> gives the famous <strong>Normal Equation</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/243/1*rAiNZ1Nji-p3Fd_GH0midA.png" /></figure><p>Boom.</p><p>The <strong>line of best fit has been mathematically summoned</strong>.</p><h3>Step 5: The Plot Twist — Least Squares is Actually Maximum Likelihood</h3><p>Just when I thought I understood the story, another twist appeared.</p><p>Suppose we assume the noise follows a <strong>Gaussian distribution</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/172/1*P3uXQTnvNRx4c8Vs57wjpg.png" /></figure><p>This means the noise follows a <strong>bell-shaped curve</strong> centered at zero.</p><p>Under this assumption, the output values follow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/213/1*97YUGBjdlQvp1-BChH4_bg.png" /></figure><p>Now we can ask a probabilistic question:</p><blockquote><em>What values of </em><strong><em>β</em></strong><em> make the observed data most likely?</em></blockquote><p>This is the <strong>Maximum Likelihood Estimation (MLE)</strong> approach.</p><p>The likelihood of the data becomes:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*FNn_WXyVZAt531-lIAl7wA.png" /></figure><p>Taking the log-likelihood simplifies it to:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/446/1*okcamcbInQPYNY4ebfXgvA.png" /></figure><p>And suddenly something familiar appears:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/193/1*GFhRvgZRvgg4lMJN68kXQQ.png" /></figure><p>That’s exactly the <strong>Residual Sum of Squares</strong>.</p><p>Which means:</p><blockquote><strong><em>Maximizing likelihood is equivalent to minimizing RSS.</em></strong></blockquote><p>So the least squares estimator is also the <strong>maximum likelihood estimator</strong>.</p><p>Two completely different perspectives…</p><p>Same solution.</p><blockquote>But solving for β only tells us <em>how</em> to fit a model. It says nothing about <em>why</em> the data looks the way it does.</blockquote><h3>Step 6: But Wait… Where Did the Data Come From?</h3><p>Just when I thought the story was over, the author asked a sneaky question:</p><blockquote><em>What if the real question isn’t </em>“how to fit the model”<em><br> but </em>“how the data was generated in the first place?”</blockquote><p>Mind. Slightly. Blown.</p><p>Two scenarios were proposed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*UhFTki_izjwuyhIbLYiRvA.png" /><figcaption>Throughout the following scenarios, classes are color-coded as follows:</figcaption></figure><h3>Scenario 1: The Friendly Gaussian Universe</h3><p>In this world:</p><ul><li>Each class is generated from a <strong>Gaussian distribution</strong></li><li>The distributions have <strong>different means</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lSuRyTOhT6EnLlyZhGoeyw.png" /><figcaption>Two classes generated from Gaussian distributions. Even the best classifier will make mistakes when the distributions overlap.</figcaption></figure><p>So visually we get two fuzzy clouds of points, blue points for Class 0 and orange points for Class 1.</p><p>Sometimes those clouds overlap.</p><p>When they do, even the <strong>best classifier in the universe</strong> will make mistakes.</p><p>This unavoidable error is called the <strong>Bayes Error</strong>.</p><blockquote>The Bayes Error is the theoretical minimum error any classifier can achieve on a given dataset — it exists because the distributions themselves overlap.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2UrFh6cSdcI903mdn71ZEw.png" /><figcaption>A linear model attempts to separate the two Gaussian clouds with a straight boundary. Points that fall on the wrong side are misclassified, this is unavoidable when distributions overlap.</figcaption></figure><p>Meaning:</p><blockquote><em>The data itself is noisy, errors are inevitable.</em></blockquote><p>Your model isn’t dumb.</p><p>Reality just likes chaos.</p><h3>Scenario 2: The Sneaky Mixture Universe</h3><p>Now imagine something more complicated.</p><p>Instead of <strong>one Gaussian per class</strong>, each class is actually a <strong>mixture of many Gaussians</strong>.</p><p>Think of:</p><ul><li>Multiple tiny blue and orange clusters</li><li>Scattered across the space</li></ul><p>This creates <strong>complex shapes</strong> in the data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6gCLjt59iSSYY_M7RhTerA.png" /><figcaption>When each class is a mixture of multiple Gaussians, clusters scatter across the feature space. A single straight boundary cannot separate them — a flexible model is needed.</figcaption></figure><p>Now a straight line is too simple to separate them.</p><p>The model fails not because the data is noisy…</p><p>but because the <strong>model is too rigid</strong>.</p><p>The fix isn’t better optimization, it’s a fundamentally different model that can learn curved, flexible boundaries.</p><h3>The Big Machine Learning Lesson</h3><p>Both scenarios could produce datasets that <em>look similar</em>.</p><p>But the <strong>underlying reality is completely different</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/836/1*P1y52Mw9WZZf9LfoXtXC5Q.png" /><figcaption>Gaussian Scenarios and Model Selection</figcaption></figure><p>Which leads to an important insight:</p><blockquote><em>Machine learning is not just about fitting models.<br> It’s about understanding </em><strong><em>how the world generates data</em></strong><em>.</em></blockquote><h3>What I Learned Today</h3><p>Three big takeaways:</p><p><strong>Linear regression minimizes the Residual Sum of Squares</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/323/1*yo6s1rZnvLM9iQbmKIEggQ.png" /></figure><p><strong>The least squares solution is</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/233/1*S704OE0D65V90sbI2sVUBA.png" /></figure><p><strong>Least squares is also maximum likelihood when noise is Gaussian</strong></p><blockquote>Optimization meets probability.</blockquote><h3>Final Thought</h3><p>When we build machine learning models, we often ask:</p><blockquote>“What model fits the data best?”</blockquote><p>But maybe the better question is:</p><blockquote>“What kind of world could have produced this data?”</blockquote><p>And sometimes the answer is:</p><p>A couple of <strong>mischievous Gaussian blobs</strong>.</p><h3>Connect with me</h3><p>If you enjoyed this article and want to follow my journey into <strong>statistical machine learning and AI</strong>, feel free to connect.</p><p>🔗 LinkedIn: <a href="https://www.linkedin.com/in/himanshu-nanda-8537a6225/">https://www.linkedin.com/in/himanshu-nanda-8537a6225/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e003c18235a7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>