<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Subinthapa on Medium]]></title>
        <description><![CDATA[Stories by Subinthapa on Medium]]></description>
        <link>https://medium.com/@subinthapa2092?source=rss-9f567b128564------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*SglAQIZ7Pugv5QWbL5DR1g.png</url>
            <title>Stories by Subinthapa on Medium</title>
            <link>https://medium.com/@subinthapa2092?source=rss-9f567b128564------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 01:05:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@subinthapa2092/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[GRU From First Principles: The Two Gates That Make RNNs Actually Work]]></title>
            <link>https://medium.com/@subinthapa2092/gru-from-first-principles-the-two-gates-that-make-rnns-actually-work-e25a39901682?source=rss-9f567b128564------2</link>
            <guid isPermaLink="false">https://medium.com/p/e25a39901682</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[gru]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Subinthapa]]></dc:creator>
            <pubDate>Sun, 27 Sep 2026 12:16:20 GMT</pubDate>
            <atom:updated>2026-09-27T12:16:20.706Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jIZHZbg6jULxEkGgm9W6wA.png" /></figure><h4>A short, math-first look at Gated Recurrent Units, and why they exist at all</h4><p>A vanilla RNN has one job: carry a single hidden state forward through time. That single state has to remember everything, and it gets rewritten at every step with no way to protect what actually matters. Over long sequences, this breaks down.</p><p>Take this example:</p><blockquote><em>“Subin lives in Dhading. Subin moved to Kathmandu.”</em></blockquote><p>To correctly answer “where does Subin live now?”, a network has to update its memory of Subin’s location without losing track of who “Subin” is in the first place. <strong>LSTM</strong> solved this with a separate long-term memory (a cell state) and three gates. <strong>GRU</strong>, short for <strong>Gated Recurrent Unit</strong>, asks a simpler question: can you get most of that benefit with fewer moving parts?</p><p>This is the short version of that story. The full derivation, hand-worked numerical example, NumPy implementation, and TensorFlow classifier live in the complete article on my site, linked at the end.</p><h3>The Core Problem</h3><pre>Vanilla RNN<br>    ↓<br>one hidden state, no gating, gradients vanish over long sequences<br>    ↓<br>GRU<br>    ↓<br>one hidden state, two gates, learned control over what gets kept</pre><p>A vanilla RNN’s hidden state gets pushed around by every new input, whether that input matters ten steps later or not. GRU adds two gates that let the network <strong>decide</strong>, at every timestep, how much of the old memory to keep and how much to replace.</p><h3>The Two Gates</h3><p><strong>Update gate</strong>, decides how much old memory survives versus how much gets replaced:</p><pre>z_t = σ(W_z x_t + U_z h_{t-1} + b_z)</pre><p><strong>Reset gate</strong>, decides how much old memory gets used when proposing new content:</p><pre>r_t = σ(W_r x_t + U_r h_{t-1} + b_r)</pre><p>Both are computed <strong>independently and simultaneously</strong> from the current input and the previous hidden state, not one after the other. Both squash to a value between 0 and 1, the same “how much” interpretation as any sigmoid gate:</p><pre>z_t → 0   keep old memory          z_t → 1   take the new content<br>r_t → 0   ignore old memory when proposing new content<br>r_t → 1   fully use old memory when proposing new content</pre><p>The reset gate feeds into one more equation, the <strong>candidate hidden state</strong>, the network’s proposal for what the new memory could look like:</p><pre>h̃_t = tanh(W_h x_t + U_h (r_t ⊙ h_{t-1}) + b_h)</pre><p><strong>r_t ⊙ h_{t-1}</strong> means the reset gate is applied to the old memory <em>before</em> it’s used to shape the proposal. If <strong>r_t</strong> is near 0, the candidate is proposed almost as if the old memory didn’t exist.</p><h3>The One Equation That Matters Most</h3><p>Everything above exists to feed this:</p><pre>h_t = (1 - z_t) ⊙ h_{t-1} + z_t ⊙ h̃_t</pre><p>Read it as one sentence: <strong>keep some old information, and add some new information, in a proportion the update gate learns.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*guHKon5Qwy0wkP6w-mHQeQ.png" /><figcaption><strong>The GRU update gate blends old and new memory.</strong></figcaption></figure><p>When <strong>z_t</strong> is near 0, almost nothing changes, <strong>h_t ≈ h_{t-1}</strong>. When <strong>z_t</strong> is near 1, the old memory is almost entirely replaced, <strong>h_t ≈ h̃_t</strong>. Every value in between is a learned blend.</p><p>This is also exactly why GRU needs one fewer gate than LSTM. LSTM uses <strong>two independent gates</strong>, forget and input, that don’t have to sum to anything. GRU uses <strong>one gate and its complement</strong>, <strong>z_t</strong> and <strong>1 — z_t</strong>, which always sum to exactly 1 by construction. The “how much to forget” and “how much to add” decisions are the same decision here.</p><h3>Why This Helps With Vanishing Gradients</h3><p>The direct gradient pathway through this equation is:</p><pre>∂h_t / ∂h_{t-1} ≈ (1 - z_t)</pre><p>Compare that to a vanilla RNN, where the equivalent term is a fixed weight multiplied by a squashed derivative, almost always well under 1, shrinking fast across many steps. GRU’s term is a value the network <strong>learns</strong>, and can hold close to 1 for exactly the timesteps where old memory matters, by learning to keep <strong>z_t</strong> close to 0.</p><p><strong>One honest caveat:</strong> this mitigates the vanishing gradient problem, it doesn’t eliminate it. If <strong>z_t</strong> learns to stay near 1 for a long stretch, old memory still gets overwritten, and the same decay can reappear.</p><h3>GRU vs LSTM vs Vanilla RNN</h3><pre>| Feature                       | Vanilla RNN | GRU    | LSTM    |<br>| ----------------------------- | ----------- | ------ | ------- |<br>| Cell state                    | No          | No     | Yes     |<br>| Gates                         | None        | 2      | 3       |<br>| Parameters (same hidden size) | Lowest      | Medium | Highest |<br>| Long-term dependencies        | Difficult   | Better | Better  |</pre><p>For a hidden size of 8 and input size of 16, this works out to <strong>200</strong> recurrent-layer parameters for a vanilla RNN, <strong>600</strong> for GRU, and <strong>800</strong> for LSTM. GRU sits a quarter below LSTM, for the same reason it has one fewer gate.</p><p>Neither GRU nor LSTM is strictly “better.” GRU trades some of LSTM’s independent gating flexibility for fewer parameters, and on many tasks, particularly with less training data, that trade works out in its favor.</p><h3>Two Quick Misconceptions</h3><p><strong>“GRU has a separate cell state, like LSTM.”</strong> It doesn’t. GRU merges the cell state and hidden state into one, <strong>h_t</strong>. There’s nothing else to track.</p><p><strong>“GRU completely eliminates vanishing gradients.”</strong> It doesn’t. Gating gives the network a learnable pathway that can hold a gradient near 1 across many steps, but if the network learns to keep <strong>z_t</strong> near 1 for too long, the same decay that affects a vanilla RNN can reappear.</p><h3>Read the Full Breakdown</h3><p>This covered the intuition and the core equations. The complete article goes further: a full hand-worked numerical example computed step by step, a from-scratch NumPy implementation that prints every gate at every timestep, a complete TensorFlow sentiment classifier trained end to end, and the full parameter-count derivation.</p><p><strong>Read the complete version here: </strong><a href="https://www.subinthapa.com.np/blog/gru-from-first-principles"><strong>subinthapa.com.np/blog/gru-from-first-principles</strong></a></p><p>If this helped, share it with someone working through sequence models. The full article also links back to the <a href="https://medium.com/@subinthapa2092/rnn-from-first-principles-math-numpy-and-tensorflow-d8ae87ca46cd"><strong>RNN</strong></a> and <a href="https://medium.com/@subinthapa2092/lstm-from-first-principles-math-numpy-and-tensorflow-a88384eeb8f5"><strong>LSTM</strong></a> pieces this one builds on, if you want to start from the beginning.</p><p><strong>Follow me for more on Python, Deep Learning, and Computer Vision.</strong></p><p><strong>LinkedIn:</strong> <a href="https://www.linkedin.com/in/subinthapa/">Subin Thapa</a></p><p><strong>GitHub:</strong> <a href="https://github.com/Subinthapa2092">Subinthapa2092</a></p><p><strong>YouTube:</strong> <a href="https://www.youtube.com/@codingwithsubin">Coding With Subin</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e25a39901682" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LSTM From First Principles: Math, NumPy, and TensorFlow]]></title>
            <link>https://medium.com/@subinthapa2092/lstm-from-first-principles-math-numpy-and-tensorflow-a88384eeb8f5?source=rss-9f567b128564------2</link>
            <guid isPermaLink="false">https://medium.com/p/a88384eeb8f5</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[lstm]]></category>
            <dc:creator><![CDATA[Subinthapa]]></dc:creator>
            <pubDate>Wed, 23 Sep 2026 12:39:42 GMT</pubDate>
            <atom:updated>2026-09-23T12:39:42.573Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3_rliRZCof0AKhH9Nky1QQ.png" /></figure><h4>Understanding Long Short-Term Memory Networks From a Single Block of Equations to a Working Sentiment Classifier</h4><p>A vanilla RNN carries a single hidden state h_t forward through time. That same state must capture everything the sequence has revealed so far. As sequences grow longer, repeated transformations through tanh can cause gradients to <strong>shrink during backpropagation</strong>, making long range dependencies difficult to learn.</p><p>Consider:</p><blockquote>“My name is Subin. I live in Nepal. I like the stock market. What is my name?”</blockquote><p>To answer correctly, the network needs to preserve <strong>“Subin”</strong> across several timesteps. A vanilla RNN can struggle because information and gradients must pass through the same recurrent pathway at every step.</p><p>This is the problem <strong>Long Short Term Memory (LSTM)</strong> was designed to address. Its <strong>cell state</strong> provides a dedicated memory pathway, while three gates learn what to <strong>forget, store, and output</strong>.</p><p>In this article, we will build the LSTM from its <strong>six core equations</strong>, work through a sequence by hand, unroll it through time, examine how its gradient pathway differs from a vanilla RNN, implement it from scratch in <strong>NumPy</strong>, and finish with a complete <strong>TensorFlow sentiment classifier</strong>.</p><h3>Part 1: Why Do We Need LSTM?</h3><p>The chain that leads to LSTM starts with a limitation, not a feature.</p><pre>Vanilla RNN<br>    ↓<br>repeated multiplication through time<br>    ↓<br>vanishing / exploding gradients<br>    ↓<br>long-term dependencies become difficult<br>    ↓<br>LSTM</pre><p>A vanilla RNN’s hidden state gets rewritten at every timestep by the same tanh(W_xh x_t + W_hh h_{t-1} + b_h). There is no mechanism to protect old information; every new word pushes the hidden state around, whether that information matters ten steps later or not. Information that isn&#39;t dispersed or actively protected gets overwritten, which is exactly why architectures like LSTM and GRU exist: to give the network a way to decide, explicitly, what to keep.</p><p>LSTM solves this by giving the network <strong>two</strong> memories instead of one, and by learning, at every timestep, exactly what to forget, what to store, and what to reveal.</p><h3>Part 2: The Core Idea, Cell State</h3><p>This is the single most important new concept in LSTM.</p><p>A vanilla RNN has one running memory:</p><pre>h_t</pre><p>An LSTM has two:</p><pre>c_t   ← long-term cell state (the notebook)<br>h_t   ← short-term hidden state (working memory)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tl9OtakHBlJ6pZQeMVoxVA.png" /><figcaption><strong>Figure 1. LSTM cell state and gated information flow.</strong></figcaption></figure><p>Think of c_t as a long-term notebook that mostly carries forward untouched, while h_t is the short-term working memory the network uses at each step. The cell state is what makes an LSTM different from a vanilla RNN, and nearly everything else in this article explains how c_t gets updated.</p><h3>Part 3: The Four Components</h3><p>This is where the article becomes mathematical. An LSTM has three gates plus one candidate memory, and all four are computed the same way: take the previous hidden state and the current input, concatenate them, multiply by a learned weight matrix, add a bias, and squash the result.</p><p><strong>Forget gate</strong>, decides what old information should disappear:</p><pre>f_t = σ(W_f [h_{t-1}, x_t] + b_f)</pre><p><strong>Input gate</strong>, decides what new information is allowed in:</p><pre>i_t = σ(W_i [h_{t-1}, x_t] + b_i)</pre><p><strong>Candidate cell state</strong>, proposes new content for memory:</p><pre>c̃_t = tanh(W_c [h_{t-1}, x_t] + b_c)</pre><p><strong>Output gate</strong>, decides what part of memory becomes the hidden state:</p><pre>o_t = σ(W_o [h_{t-1}, x_t] + b_o)</pre><p>These four pieces combine into the two update equations that define an LSTM cell:</p><pre>c_t = f_t ⊙ c_{t-1} + i_t ⊙ c̃_t<br><br>h_t = o_t ⊙ tanh(c_t)</pre><p>The ⊙ symbol means element-wise multiplication (also called the Hadamard product): each entry of one vector multiplies the matching entry of the other, not a matrix multiplication. [h_{t-1}, x_t] means the two vectors are concatenated into one longer vector before being multiplied by the gate&#39;s weight matrix. These six equations are the heart of this entire article. Every section from here on comes back to them.</p><h3>Shapes: Where Every Matrix Comes From</h3><p>Exactly as the RNN article’s shapes section did for W_xh and W_hh, every quantity above is a vector or a matrix with a specific shape that has to line up for the equations to be valid.</p><pre>x_t   ∈ R^(d_x)              the input dimension, e.g. an embedding size<br><br>h_t   ∈ R^(d_h)              the hidden size, a design choice you pick<br><br>c_t   ∈ R^(d_h)              the cell state, always the same size as h_t<br><br>[h_{t-1}, x_t]  ∈ R^(d_h + d_x)      the concatenated vector each gate reads<br><br>W_f, W_i, W_c, W_o  ∈ R^(d_h × (d_h + d_x))     one weight matrix per gate<br><br>b_f, b_i, b_c, b_o  ∈ R^(d_h)                    one bias vector per gate</pre><p>The flow from input to gate is the same for all four:</p><pre>[h_{t-1}, x_t]<br>       ↓<br>   W_f  →  f_t<br>   W_i  →  i_t<br>   W_c  →  c̃_t<br>   W_o  →  o_t</pre><p>Because c_t and h_t are always the same size, d_h, the cell state and hidden state can be added, multiplied, and passed through tanh together without any shape mismatch. This is exactly why LSTM(64) in Part 12 sets a single number, d_h = 64, that simultaneously fixes the size of c_t, h_t, and all four gates at once.</p><h3>Part 4: Understanding the Sigmoid Gate</h3><p>Don’t just memorize that sigmoid outputs a number between 0 and 1. Understand what that number <em>means</em> inside a gate.</p><pre>σ(z) → 0     mostly block, close the gate<br>σ(z) → 1     mostly allow, open the gate<br>σ(z) = 0.5   half open, partial information passes through</pre><p>The sigmoid’s output is not a label or a probability of a class; it is a <strong>per-dimension valve</strong>. A value near 0 in one dimension of f_t means &quot;erase this part of the cell state.&quot; A value near 1 in the matching dimension of i_t means &quot;fully accept this part of the candidate memory.&quot; Every gate answers its own question:</p><pre>Forget gate  →  what old information should I let disappear?<br>Input gate   →  what new information should I let in?<br>Output gate  →  what part of memory should I expose right now?</pre><p>This is why LSTM’s equations feel less like arbitrary math and more like a small decision-making process running at every timestep.</p><h3>Why the Candidate Uses tanh, Not Sigmoid</h3><p>The three gates use sigmoid because they answer a <strong>how much</strong> question, a value between 0 and 1. The candidate memory is different:</p><pre>c̃_t = tanh(W_c [h_{t-1}, x_t] + b_c)</pre><p>tanh produces values between −1 and 1, so the candidate can propose content that pushes the cell state in either direction:</p><pre>candidate memory<br>        ↓<br>negative  ←  0  →  positive</pre><p>This gives four clean, one-line interpretations worth memorizing together:</p><blockquote><strong><em>Candidate says what could be written; the input gate says how much of it actually gets written.</em></strong><em> </em><strong><em>Forget gate says how much old memory survives; output gate says how much current memory is exposed through </em></strong><strong><em>h_t.</em></strong></blockquote><h3>Part 5: Understanding the Cell State Update</h3><p>The cell state deserves its own close look, since it is the part of LSTM that actually fixes the vanishing gradient problem.</p><pre>c_{t-1}<br>   ↓<br>[ forget some, via f_t ]<br>   ↓<br>[ add some new information, via i_t ⊙ c̃_t ]<br>   ↓<br>c_t</pre><p>Look again at the equation:</p><pre>c_t = f_t ⊙ c_{t-1} + i_t ⊙ c̃_t</pre><p>The operation connecting the old memory to the new memory is <strong>addition</strong>, not repeated multiplication through a nonlinearity the way a vanilla RNN’s h_t is. f_t ⊙ c_{t-1} decides how much of the old notebook survives, and i_t ⊙ c̃_t writes new content alongside it, without needing to overwrite the entire notebook to do so. This additive pathway is one of the main reasons a gradient can travel through the cell-state route more effectively than through the purely multiplicative recurrence of a vanilla RNN.</p><p><strong>Be careful with this claim.</strong> LSTMs do not <em>eliminate</em> the vanishing gradient problem, they <strong>mitigate</strong> it. If the forget gate f_t stays close to 0 for many consecutive steps, the cell state still gets wiped and the same decay reappears. What LSTM adds is a <em>pathway</em> that lets the network learn to keep f_t close to 1 exactly when long-term memory actually matters, something a vanilla RNN has no equivalent mechanism for.</p><h3>Part 6: A Full Numerical Example</h3><p>Theory earns trust once real numbers move through it by hand. Take a small three-word sequence and push it through one shared LSTM cell, one timestep at a time.</p><p>This is a <strong>scalar toy example</strong>, exactly like the RNN article’s, with d_x = d_h = 1, so every gate is a single number rather than a vector. Real LSTMs work with vectors and matrices; the scalar version here exists purely so every multiplication can be checked by hand.</p><pre>t = 1   x1 = 0.10<br>t = 2   x2 = 0.30<br>t = 3   x3 = 0.50</pre><p>The same four sets of weights are reused at every timestep, exactly as <strong>parameter sharing</strong> works in a vanilla RNN:</p><pre>W_f: (0.50, 0.60), b_f = 0.10<br>W_i: (0.40, 0.50), b_i = 0.00<br>W_c: (0.30, 0.80), b_c = 0.00<br>W_o: (0.60, 0.40), b_o = 0.10<br><br>h_0 = 0<br>c_0 = 0</pre><p>Each weight pair is (weight on h_{t-1}, weight on x_t).</p><h3>Step 1, t = 1</h3><pre>f_1 = σ(0.50·0 + 0.60·0.10 + 0.10) = σ(0.160) = 0.5399<br>i_1 = σ(0.40·0 + 0.50·0.10 + 0.00) = σ(0.050) = 0.5125<br>c̃_1 = tanh(0.30·0 + 0.80·0.10 + 0.00) = tanh(0.080) = 0.0798<br>c_1 = f_1·c_0 + i_1·c̃_1 = (0.5399)(0) + (0.5125)(0.0798) = 0.0409<br>o_1 = σ(0.60·0 + 0.40·0.10 + 0.10) = σ(0.140) = 0.5349<br>h_1 = o_1·tanh(c_1) = (0.5349)(0.0409) = 0.0219</pre><h3>Step 2, t = 2</h3><pre>f_2 = σ(0.50·0.0219 + 0.60·0.30 + 0.10) = σ(0.2909) = 0.5722<br>i_2 = σ(0.40·0.0219 + 0.50·0.30 + 0.00) = σ(0.1588) = 0.5396<br>c̃_2 = tanh(0.30·0.0219 + 0.80·0.30 + 0.00) = tanh(0.2466) = 0.2417<br>c_2 = f_2·c_1 + i_2·c̃_2 = (0.5722)(0.0409) + (0.5396)(0.2417) = 0.1538<br>o_2 = σ(0.60·0.0219 + 0.40·0.30 + 0.10) = σ(0.2331) = 0.5580<br>h_2 = o_2·tanh(c_2) = (0.5580)(0.1527) = 0.0852</pre><h3>Step 3, t = 3</h3><pre>f_3 = σ(0.50·0.0852 + 0.60·0.50 + 0.10) = σ(0.4426) = 0.6089<br>i_3 = σ(0.40·0.0852 + 0.50·0.50 + 0.00) = σ(0.2841) = 0.5705<br>c̃_3 = tanh(0.30·0.0852 + 0.80·0.50 + 0.00) = tanh(0.4256) = 0.4016<br>c_3 = f_3·c_2 + i_3·c̃_3 = (0.6089)(0.1538) + (0.5705)(0.4016) = 0.3228<br>o_3 = σ(0.60·0.0852 + 0.40·0.50 + 0.10) = σ(0.3511) = 0.5869<br>h_3 = o_3·tanh(c_3) = (0.5869)(0.3120) = 0.1831</pre><pre>t   f_t       i_t       c̃_t      c_t       o_t       h_t<br>1   0.5399    0.5125    0.0798    0.0409    0.5349    0.0219<br>2   0.5722    0.5396    0.2417    0.1538    0.5580    0.0852<br>3   0.6089    0.5705    0.4016    0.3228    0.5869    0.1831</pre><p>Notice c_t climbs steadily: 0.0409 → 0.1538 → 0.3228. The cell state is accumulating, additively, rather than being squashed and reset the way a vanilla RNN&#39;s h_t is. That gentle, additive climb is the entire point of Part 5.</p><h3>Part 7: One Timestep to Multiple Timesteps</h3><p>Unrolling an LSTM works exactly like unrolling a vanilla RNN, except now two lines run through the diagram instead of one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eEneAASp3B8DfgKXOSpybg.png" /><figcaption><strong>Figure 2. LSTM unrolled across time.</strong></figcaption></figure><p>Just like a vanilla RNN, an LSTM <strong>reuses the same parameters at every timestep</strong>. The same W_f, W_i, W_c, and W_o matrices and their biases are applied throughout the sequence. A ten word sentence and a hundred word sentence use the <strong>same weights</strong>, simply applied more times. This is <strong>parameter sharing</strong> across time.</p><h3>Part 8: Why the Gradient Survives Better</h3><p>You don’t need to derive every gate’s enormous derivative to understand why LSTM helps. The important pathway is the cell state itself.</p><pre>c_t<br> ↑<br>c_{t-1}<br> ↑<br>c_{t-2}<br> ↑<br>c_{t-3}</pre><p>Recall the cell-state update:</p><pre>c_t = f_t ⊙ c_{t-1} + i_t ⊙ c̃_t</pre><p>Differentiating this with respect to c_{t-1}, in the simplified element-wise view where f_t is treated as locally constant, gives:</p><pre>∂c_t / ∂c_{t-1} ≈ f_t</pre><p>In a vanilla RNN, the gradient repeatedly passes through<br>∂h_t/∂h_{t-1} ≈ W_hh · (1 − tanh²(z)), causing it to shrink across many timesteps. In an LSTM, the direct cell-state pathway has<br>∂c_t/∂c_{t-1} ≈ f_t, where the forget gate f_t is learned and can stay near 1 when long-term memory matters.</p><p><strong>One caveat:</strong> This describes the direct cell-state pathway. The full gradient also includes indirect paths through the gates and hidden state. The direct pathway is the key intuition, while full backpropagation accounts for all of these paths.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NJ10rzniUhkqx6z8JqPlwA.png" /><figcaption><strong>Figure 3. Vanilla RNN vs LSTM gradient flow.</strong></figcaption></figure><p>LSTM does not eliminate vanishing gradients. It provides a learnable pathway that can help keep gradients flowing across long sequences.</p><h3>Part 9: LSTM vs Vanilla RNN</h3><pre>| Feature                       | Vanilla RNN | LSTM             |<br>| ----------------------------- | ----------- | ---------------- |<br>| Hidden state                  | Yes         | Yes              |<br>| Cell state                    | No          | Yes              |<br>| Forget gate                   | No          | Yes              |<br>| Input gate                    | No          | Yes              |<br>| Output gate                   | No          | Yes              |<br>| Long-term dependencies        | Difficult   | Better supported |<br>| Parameters (same hidden size) | Fewer       | Roughly 4× more  |<br>| Sequential computation        | Yes         | Yes              |</pre><p><strong>LSTM is not automatically “better” in every situation.</strong> It is a trade-off: more parameters, more computation per step, and a more complex architecture, in exchange for a gradient pathway that survives longer sequences more reliably. For short sequences, a vanilla RNN can be perfectly sufficient and considerably cheaper to train.</p><h3>Part 10: Parameter Count</h3><p>The same accounting used for a vanilla RNN extends naturally to LSTM. An LSTM computes four affine transformations from the same concatenated vector [h_{t-1}, x_t], followed by different nonlinearities, sigmoid for the three gates and tanh for the candidate, and the four results interact through the cell-state update rather than staying fully independent. It is not, technically, four separate recurrent layers running side by side; it is one cell whose four internal transformations happen to have the same shape as a vanilla RNN&#39;s single recurrence.</p><pre>Vanilla RNN recurrent layer:  d_h·d_x + d_h·d_h + d_h<br>LSTM recurrent layer:         4 · (d_h·d_x + d_h·d_h + d_h)</pre><p>Each of the four gates, forget, input, candidate, output, needs its own full set of weights: one matrix over x_t, one matrix over h_{t-1}, and one bias. That&#39;s exactly why the factor of <strong>4</strong> appears.</p><p>For a concrete example, using the same shapes as the RNN article’s TensorFlow model:</p><pre>d_x = 16   (embedding size)<br>d_h = 8    (hidden size)<br><br>Vanilla RNN: (8)(16) + (8)(8) + 8 = 128 + 64 + 8 = 200<br>LSTM:        4 × 200                             = 800</pre><p><strong>200</strong> parameters for the RNN, <strong>800</strong> for the LSTM at the same hidden size. This is exactly why an LSTM layer is often described as costing roughly four times as many parameters as a vanilla RNN layer of the same size, a fact that connects directly back to the RNN article’s own parameter-count section.</p><p><strong>One implementation note.</strong> The four weight matrices W_f, W_i, W_c, W_o are written separately here because that is what makes each gate&#39;s role easy to understand. Framework implementations, including TensorFlow&#39;s, commonly stack these into one larger matrix, W ∈ R^(4d_h × (d_h + d_x)), and compute all four transformations in a single matrix multiplication for efficiency. The math is identical either way; only the bookkeeping changes.</p><h3>Part 11: LSTM From Scratch With NumPy</h3><p>This reproduces the exact hand-worked example from Part 6, one gate at a time.</p><pre>import numpy as np<br><br>def sigmoid(z):<br>    return 1 / (1 + np.exp(-z))<br><br>def lstm_step(x_t, h_prev, c_prev, W):<br>    &quot;&quot;&quot;One LSTM timestep. W holds (w_h, w_x, b) for each of the four gates.&quot;&quot;&quot;<br>    Wf_h, Wf_x, b_f = W[&quot;f&quot;]<br>    Wi_h, Wi_x, b_i = W[&quot;i&quot;]<br>    Wc_h, Wc_x, b_c = W[&quot;c&quot;]<br>    Wo_h, Wo_x, b_o = W[&quot;o&quot;]<br>    f_t = sigmoid(Wf_h * h_prev + Wf_x * x_t + b_f)       # forget gate<br>    i_t = sigmoid(Wi_h * h_prev + Wi_x * x_t + b_i)       # input gate<br>    c_tilde = np.tanh(Wc_h * h_prev + Wc_x * x_t + b_c)   # candidate memory<br>    c_t = f_t * c_prev + i_t * c_tilde                    # new cell state<br>    o_t = sigmoid(Wo_h * h_prev + Wo_x * x_t + b_o)       # output gate<br>    h_t = o_t * np.tanh(c_t)                              # new hidden state<br>    return h_t, c_t, (f_t, i_t, c_tilde, o_t)<br><br>def run_sequence(inputs, W, h0=0.0, c0=0.0, verbose=True):<br>    h, c = h0, c0<br>    hidden_states, cell_states = [], []<br>    for t, x_t in enumerate(inputs, start=1):<br>        h, c, (f_t, i_t, c_tilde, o_t) = lstm_step(x_t, h, c, W)<br>        hidden_states.append(h)<br>        cell_states.append(c)<br>        if verbose:<br>            print(f&quot;t={t}&quot;)<br>            print(f&quot;  forget gate  f_t = {f_t:.4f}&quot;)<br>            print(f&quot;  input gate   i_t = {i_t:.4f}&quot;)<br>            print(f&quot;  candidate    c~t = {c_tilde:.4f}&quot;)<br>            print(f&quot;  cell state   c_t = {c:.4f}&quot;)<br>            print(f&quot;  output gate  o_t = {o_t:.4f}&quot;)<br>            print(f&quot;  hidden state h_t = {h:.4f}&quot;)<br>    return hidden_states, cell_states<br><br>W = {<br>    &quot;f&quot;: (0.50, 0.60, 0.10),<br>    &quot;i&quot;: (0.40, 0.50, 0.00),<br>    &quot;c&quot;: (0.30, 0.80, 0.00),<br>    &quot;o&quot;: (0.60, 0.40, 0.10),<br>}<br>inputs = [0.10, 0.30, 0.50]<br>hidden_states, cell_states = run_sequence(inputs, W)</pre><pre>t=1<br>  forget gate  f_t = 0.5399<br>  input gate   i_t = 0.5125<br>  candidate    c~t = 0.0798<br>  cell state   c_t = 0.0409<br>  output gate  o_t = 0.5349<br>  hidden state h_t = 0.0219<br>t=2<br>  forget gate  f_t = 0.5722<br>  input gate   i_t = 0.5396<br>  candidate    c~t = 0.2417<br>  cell state   c_t = 0.1538<br>  output gate  o_t = 0.5580<br>  hidden state h_t = 0.0852<br>t=3<br>  forget gate  f_t = 0.6089<br>  input gate   i_t = 0.5705<br>  candidate    c~t = 0.4016<br>  cell state   c_t = 0.3228<br>  output gate  o_t = 0.5869<br>  hidden state h_t = 0.1831</pre><p>Printing every gate, not just the final h_t and c_t, is what makes this implementation correspond directly to the six equations in Part 3: each printed line is one equation, evaluated with real numbers.</p><p>This matches Part 6’s hand calculation exactly, to four decimal places. <strong>Forward propagation first</strong> is the right way to build intuition here; a full manual backpropagation-through-time derivation for LSTM involves eight separate gradient paths per timestep, and Part 8’s simplified ∂c_t/∂c_{t-1} ≈ f_t view already captures the important insight without that overhead.</p><h3>Part 12: The Same Model in TensorFlow</h3><p>Once the manual version makes sense, TensorFlow is a matter of not writing the loop yourself.</p><pre>from tensorflow import keras<br><br>model = keras.Sequential([<br>    keras.Input(shape=(max_length,)),<br>    keras.layers.Embedding(input_dim=vocab_size, output_dim=100),<br>    keras.layers.LSTM(64),<br>    keras.layers.Dense(1, activation=&quot;sigmoid&quot;),<br>])<br>model.compile(optimizer=&quot;adam&quot;, loss=&quot;binary_crossentropy&quot;, metrics=[&quot;accuracy&quot;])</pre><p>The data flow reads the same way the RNN article’s did, only the recurrent layer changed:</p><pre>Embedding<br>   ↓<br>word vectors<br>   ↓<br>LSTM<br>   ↓<br>final hidden representation<br>   ↓<br>Dense<br>   ↓<br>sentiment probability</pre><p>LSTM(64) sets d_h = 64, exactly like SimpleRNN(64) did, except internally it now maintains both h_t and c_t, and carries four gate weight matrices instead of one recurrence weight matrix. Everything from Part 10&#39;s parameter count applies directly: swap in d_x = 100 (the embedding size) and d_h = 64, and model.summary() will report a layer with roughly four times the parameters a SimpleRNN(64) layer would have had.</p><p><strong>On exact numbers:</strong> as with the RNN article’s TensorFlow run, actual accuracy and loss values from training depend on the library version, the random seed, and even CPU versus GPU execution. Treat any specific number from a real training run as illustrative of the pattern, not as a value to reproduce bit-for-bit.</p><h3>A Complete, Runnable Training Example</h3><p>The architecture above is real, but a full workflow needs data flowing through it end to end, exactly like the RNN article’s Part 14. This reuses the same small 16-sentence toy dataset from that article, swapping only SimpleRNN for LSTM, so the two articles stay directly comparable.</p><pre>import numpy as np<br>import tensorflow as tf<br>from tensorflow import keras<br>from tensorflow.keras.preprocessing.text import Tokenizer<br>from tensorflow.keras.preprocessing.sequence import pad_sequences<br><br>tf.random.set_seed(42)<br>np.random.seed(42)<br># 1. The same small labeled dataset used in the RNN article, 1 = positive, 0 = negative<br>texts = [<br>    &quot;I love this movie&quot;, &quot;This film was amazing&quot;, &quot;What a wonderful story&quot;,<br>    &quot;I really enjoyed this&quot;, &quot;Best movie I have seen&quot;, &quot;I hate this movie&quot;,<br>    &quot;This film was terrible&quot;, &quot;What a boring story&quot;, &quot;I really disliked this&quot;,<br>    &quot;Worst movie I have seen&quot;, &quot;The acting was great&quot;, &quot;The acting was awful&quot;,<br>    &quot;A truly beautiful film&quot;, &quot;A truly disappointing film&quot;,<br>    &quot;I would watch this again&quot;, &quot;I would never watch this again&quot;,<br>]<br>labels = [1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0]<br># 2. Tokenization and padding, identical to the RNN pipeline<br>tokenizer = Tokenizer(num_words=1000, oov_token=&quot;&lt;OOV&gt;&quot;)<br>tokenizer.fit_on_texts(texts)<br>sequences = tokenizer.texts_to_sequences(texts)<br>max_length = max(len(seq) for seq in sequences)<br>padded = pad_sequences(sequences, maxlen=max_length, padding=&quot;post&quot;)<br># 3. Train/test split<br>X, y = np.array(padded), np.array(labels)<br>X_train, X_test = X[:12], X[12:]<br>y_train, y_test = y[:12], y[12:]<br># 4. Build the model: Embedding -&gt; LSTM -&gt; Dense<br>vocab_size = len(tokenizer.word_index) + 1<br>model = keras.Sequential([<br>    keras.Input(shape=(max_length,)),<br>    keras.layers.Embedding(input_dim=vocab_size, output_dim=16),<br>    keras.layers.LSTM(8),<br>    keras.layers.Dense(1, activation=&quot;sigmoid&quot;),<br>])<br>model.compile(optimizer=&quot;adam&quot;, loss=&quot;binary_crossentropy&quot;, metrics=[&quot;accuracy&quot;])<br>model.summary()<br># 5. Fit<br>history = model.fit(X_train, y_train, epochs=30, verbose=0)<br>print(&quot;final training accuracy:&quot;, round(history.history[&quot;accuracy&quot;][-1], 4))<br># 6. Evaluate<br>test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)<br>print(f&quot;test loss: {test_loss:.4f}, test accuracy: {test_acc:.4f}&quot;)<br># 7. Predict on a brand new sentence<br>new_sentence = [&quot;I really loved this film&quot;]<br>new_seq = tokenizer.texts_to_sequences(new_sentence)<br>new_padded = pad_sequences(new_seq, maxlen=max_length, padding=&quot;post&quot;)<br>prediction = model.predict(new_padded, verbose=0)[0][0]<br>print(f&quot;prediction for &#39;{new_sentence[0]}&#39;: {prediction:.4f}&quot;)</pre><pre>Layer (type)              Output Shape         Param #<br>embedding                 (None, 6, 16)         544<br>lstm                       (None, 8)             800<br>dense                      (None, 1)             9<br>Total params: 1353</pre><p>Notice the recurrent layer’s parameter count: <strong>800</strong>, exactly the number Part 10 computed by hand for d_x = 16, d_h = 8, and exactly four times the RNN article&#39;s SimpleRNN(8) layer, which reported 200 for the same shapes.</p><p>This is the same 16-sentence, educational-only dataset used in the RNN article, so the same honesty applies here too: with only 12 training examples, a model reaching near-perfect training accuracy is demonstrating that the pipeline runs correctly end to end, tokenization, padding, an LSTM layer training without error, and a genuinely new sentence receiving a prediction, not that it has learned real sentiment understanding. Scaling this exact pipeline to a real dataset of thousands of labeled reviews is what turns it into something worth deploying.</p><h3>return_sequences vs return_state</h3><p>Part 14 already covers return_sequences=True for stacking layers. A closely related, separately useful flag is return_state:</p><pre>lstm_layer = keras.layers.LSTM(<br>    64,<br>    return_sequences=True,<br>    return_state=True,<br>)</pre><pre>return_sequences=True   →  return every hidden state, h_1 through h_T<br>return_state=True       →  also return the final hidden state and final cell state separately</pre><p>return_sequences controls what comes out at <em>every timestep</em>. return_state controls whether the layer <em>additionally</em> hands back its final h_T and c_T as their own outputs, useful when a later part of the model, an encoder-decoder setup, for instance, needs to initialize a second LSTM with the first one&#39;s exact final memory rather than just its sequence of outputs.</p><h3>Part 13: Bidirectional LSTM</h3><p>Everything the RNN article said about Bidirectional RNNs carries over directly, since LSTM is still fundamentally a recurrence, just a gated one.</p><pre>keras.layers.Bidirectional(<br>    keras.layers.LSTM(64)<br>)</pre><pre>Forward LSTM   →<br>Backward LSTM  ←<br>        ↓<br>   concatenated</pre><p>A forward LSTM reads the sequence left to right, a backward LSTM reads it right to left, and their final hidden states are concatenated into one combined vector, exactly the [forward_h_t ; backward_h_t] pattern from the RNN article. The same limitation applies too: a Bidirectional LSTM needs the entire sequence available before it can process any single word, so it works well for classifying a complete sentence but is unsuitable for strictly causal, real-time prediction, where the future word genuinely doesn&#39;t exist yet.</p><h3>Part 14: Stacked LSTM</h3><p>Stacking works the same way it does for a vanilla RNN: one layer’s sequence of hidden states becomes the next layer’s input sequence.</p><pre>model = keras.Sequential([<br>    keras.layers.Embedding(vocab_size, 100),<br>    keras.layers.LSTM(64, return_sequences=True),<br>    keras.layers.LSTM(32),<br>    keras.layers.Dense(1, activation=&quot;sigmoid&quot;),<br>])</pre><p><strong>return_sequences=True matters here.</strong> By default, an LSTM layer returns only its <em>final</em> hidden state, a single vector summarizing the whole sequence. But the second LSTM layer needs a full sequence of hidden states to process, one per timestep, not a single collapsed vector. Setting return_sequences=True on the first layer tells it to output h_1, h_2, ..., h_T instead of just h_T, which is exactly what a stacked recurrent layer requires as its input.</p><h3>Part 15: LSTM for Different Tasks</h3><p>The same many-to-one, many-to-many, and one-to-many categories from the RNN article apply directly to LSTM, since these categories describe the task, not the recurrence itself.</p><pre>Many → One    sentiment classification, only the final h_T is used<br>Many → Many   sequence labeling, every h_t is used, same-length input and output<br>Many → Many   machine translation, every h_t is used, different-length input and output<br>One → Many    text generation, one seed token in, a full sequence out</pre><p>Which of these a task falls into determines whether the surrounding model discards intermediate hidden states or keeps every one of them; the LSTM cell computing h_t and c_t at each step never changes.</p><h3>Part 16: LSTM for Next-Word Prediction</h3><p>Just like the vanilla RNN case, next-word prediction estimates a full probability distribution over the vocabulary at every step, only now that estimate is built from an LSTM’s hidden state instead of a vanilla RNN’s.</p><pre>&quot;I want to go&quot;<br>       ↓<br>     LSTM<br>       ↓<br>    Dense<br>       ↓<br>   Softmax<br>       ↓<br>   &quot;home&quot;</pre><pre>P(x_{t+1} | x_1, ..., x_t)</pre><p>The softmax layer turns the Dense layer’s raw scores into a valid probability distribution across the vocabulary, and the word with the highest probability becomes the prediction, exactly as in the RNN article’s Part 16. The only thing that changed is which recurrence produced the hidden state feeding into that Dense layer.</p><h3>Part 17: Limitations of LSTM</h3><p>LSTM fixes a real problem, but it isn’t free, and it isn’t the end of the story.</p><pre>More parameters           roughly 4x a vanilla RNN at the same hidden size<br>More computation           four gates to compute at every single timestep<br>Still sequential            h_t and c_t at step t still require step t-1, no parallelism<br>Slower on long sequences    the extra gate computations compound the RNN&#39;s own slowness<br>More complex architecture   six equations to reason about instead of two<br>GRU exists                  a simpler gated alternative with fewer parameters<br>Transformers exist          generally preferred for many large-scale modern NLP tasks</pre><p>That last point is worth sitting with, since it gives the whole family of architectures historical context:</p><pre>RNN<br> ↓<br>LSTM / GRU<br> ↓<br>Transformer</pre><p>LSTM solved the vanilla RNN’s memory problem, but it didn’t solve the sequential computation problem: h_t and c_t still mathematically require h_{t-1} and c_{t-1}, so training remains inherently step-by-step. This is one of the main reasons Transformer-based architectures, which process every position in a sequence in parallel, became the dominant choice for many large-scale NLP tasks.</p><h3>Part 18: LSTM vs GRU</h3><p>A brief introduction is enough here, since GRU deserves its own full treatment.</p><pre>| Feature      | LSTM                                | GRU                         |<br>| ------------ | ----------------------------------- | --------------------------- |<br>| Cell state   | Yes, separate from hidden state     | No separate cell state      |<br>| Gates        | 3 main gates: forget, input, output | 2 main gates: reset, update |<br>| Parameters   | More                                | Fewer                       |<br>| Architecture | More complex                        | Simpler                     |</pre><p>GRU merges the cell state and hidden state into one, and replaces LSTM’s three gates with two, trading a small amount of expressive power for a lighter, faster architecture. The next article picks up exactly here, deriving GRU from the same vanishing-gradient problem that motivated LSTM in the first place.</p><h3>Common Misconceptions</h3><p><strong>“LSTM completely solves the vanishing gradient problem.”</strong> It does not. LSTM gives the network a <em>learnable, largely additive</em> pathway through the cell state, which makes it much easier to preserve long-range information than a vanilla RNN can. But if the forget gate learns to stay near 0 for a stretch of timesteps, the cell state is still wiped, and the same decay can reappear. LSTM mitigates the problem; it doesn’t eliminate it.</p><p><strong>“The cell state and the hidden state are basically the same thing.”</strong> They are not. c_t is a long-term notebook that updates additively and changes slowly. h_t is recomputed from c_t at every step through the output gate and tanh, and is what the rest of the network (a Dense layer, for instance) actually reads. Confusing the two is one of the most common LSTM mistakes.</p><p><strong>“More gates always means a better model.”</strong> Not automatically. Every additional gate is more parameters to learn and more computation per timestep. On short sequences or small datasets, a vanilla RNN or GRU can match or outperform an LSTM while training faster and overfitting less.</p><p><strong>“LSTM removes the sequential computation bottleneck of RNNs.”</strong> It does not. h_t and c_t still depend on h_{t-1} and c_{t-1}, so an LSTM is exactly as sequential as a vanilla RNN, just with more work happening at each step. This is precisely why Transformer architectures, which allow full parallelism across positions, replaced LSTM in many large-scale settings.</p><h3>Key Formulas</h3><p><strong>The Core LSTM Equations</strong></p><pre>f_t = σ(W_f [h_{t-1}, x_t] + b_f)<br>i_t = σ(W_i [h_{t-1}, x_t] + b_i)<br>c̃_t = tanh(W_c [h_{t-1}, x_t] + b_c)<br>c_t = f_t ⊙ c_{t-1} + i_t ⊙ c̃_t<br>o_t = σ(W_o [h_{t-1}, x_t] + b_o)<br>h_t = o_t ⊙ tanh(c_t)</pre><p><strong>Simplified Gradient Pathway</strong></p><pre>∂c_t / ∂c_{t-1} ≈ f_t</pre><p><strong>Parameter Count</strong></p><pre>Vanilla RNN:  d_h·d_x + d_h·d_h + d_h<br>LSTM:         4 · (d_h·d_x + d_h·d_h + d_h)</pre><p><strong>Next Word Prediction</strong></p><pre>P(x_{t+1} | x_1, ..., x_t)</pre><h3>Complete Summary Table</h3><pre>| Concept                  | What It Is                     | What It Does                                                                                 |<br>| ------------------------ | ------------------------------ | -------------------------------------------------------------------------------------------- |<br>| Cell state (c_t)         | Long-term memory pathway       | Carries important information across timesteps                                               |<br>| Hidden state (h_t)       | Current output state           | Carries information to the next timestep and produces the output                             |<br>| Forget gate (f_t)        | Sigmoid gate                   | Decides what information to retain from c_{t-1}                                              |<br>| Input gate (i_t)         | Sigmoid gate                   | Controls how much new information is written into the cell state                             |<br>| Candidate memory (c̃_t)  | Tanh-generated candidate       | Provides new information that can be added to the cell state                                 |<br>| Output gate (o_t)        | Sigmoid gate                   | Controls what part of c_t becomes h_t                                                        |<br>| Element-wise product (⊙) | Dimension-wise multiplication  | Applies gate values independently to each dimension                                          |<br>| Parameter sharing        | Same weights at every timestep | Reuses the same weights across timesteps and allows sequences of different lengths           |<br>| Gradient pathway         | Direct cell-state pathway      | Helps gradients flow across many timesteps                                                   |<br>| return_sequences=True    | Returns h_t for every timestep | Produces an output for every timestep instead of only the final output                       |<br>| Bidirectional LSTM       | Forward + backward LSTM        | Processes the sequence in both directions                                                    |<br>| Parameter count          | Four main transformations      | Gives an LSTM roughly 4× the recurrent transformation parameters of a comparable vanilla RNN |<br>| GRU                      | Gated recurrent architecture   | Provides a simpler alternative to LSTM with fewer gates and typically fewer parameters       |</pre><h3>Final Thoughts</h3><p>LSTM comes down to <strong>six equations</strong>, applied at every timestep with the same <strong>four sets of parameters</strong> reused across the sequence.</p><p>The key difference is the <strong>cell state</strong>: a long-term memory pathway updated mainly through <strong>addition</strong>, while the <strong>gates</strong> learn what to forget, keep, and reveal.</p><p>LSTM does not eliminate the <strong>vanishing gradient problem</strong>. Instead, it provides a <strong>learnable pathway that helps gradients flow across many timesteps</strong>, controlled largely by the <strong>forget gate</strong>.</p><p>LSTM is not human-like memory. It is a mathematical system that learns how to update its internal states from data.</p><p>Once these equations are clear, the path to <strong>GRUs, Bidirectional LSTMs, Attention, and Transformers</strong> becomes much easier to understand.</p><p>If this article helped make <strong>LSTMs</strong> click, share it with someone learning sequence models.</p><p><strong>Follow me for more on Python, Deep Learning, and Computer Vision.</strong></p><p><strong>LinkedIn:</strong> <a href="https://www.linkedin.com/in/subinthapa/">Subin Thapa<br></a><strong>GitHub:</strong> <a href="https://github.com/Subinthapa2092">Subinthapa2092</a><br><strong>YouTube:</strong> <a href="https://www.youtube.com/@codingwithsubin">Coding With Subin</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a88384eeb8f5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RNN From First Principles: Math, NumPy, and TensorFlow]]></title>
            <link>https://medium.com/@subinthapa2092/rnn-from-first-principles-math-numpy-and-tensorflow-d8ae87ca46cd?source=rss-9f567b128564------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8ae87ca46cd</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[recurrent-neural-network]]></category>
            <dc:creator><![CDATA[Subinthapa]]></dc:creator>
            <pubDate>Sat, 19 Sep 2026 16:18:53 GMT</pubDate>
            <atom:updated>2026-09-19T16:18:53.336Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JRNbBQEXzm27vx8rMgr9tQ.png" /></figure><h4>Understanding Recurrent Neural Networks From a Single Equation to a Working Sentiment Classifier</h4><p>A feedforward network looks at an input and produces an output. Nothing about that input remembers what came before it. That works perfectly for a photo, since every pixel matters at once, but it breaks down the moment order itself carries meaning.</p><p>“The phone is very fast” and “Is the phone very fast?” contain the exact same words. Only their order tells you one is a statement and the other is a question. A network with no memory of what it just read cannot tell them apart. This is the problem a <strong>recurrent neural network</strong>, or <strong>RNN</strong>, exists to solve.</p><p>This article builds the RNN from a single equation, works through one sentence by hand, unrolls it through time, derives why its gradients vanish or explode, implements it from scratch in NumPy, and ends with a complete, runnable TensorFlow sentiment classifier.</p><h3>Part 1: What Is Sequence Data</h3><p><strong>Sequence data</strong> is data where order changes meaning. Shuffle the words in a sentence and the meaning breaks. Shuffle the days in a stock price history and the trend disappears. Shuffle the frames of audio and it stops sounding like speech.</p><pre>Text          &quot;I love this movie&quot;: word order carries meaning<br>Time series   stock prices, temperature readings, day by day<br>Speech        an audio waveform, sampled moment by moment<br>Sensor data   a heart rate monitor, one reading after another</pre><p>A standard feedforward network processes each input independently, with no built in concept of “what came before this.” For sequence data, that is exactly the information you cannot afford to throw away.</p><h3>Part 2: Basic RNN Intuition</h3><pre>x1 → RNN → h1<br>       ↓<br>x2 → RNN → h2<br>       ↓<br>x3 → RNN → h3</pre><p>At every timestep, the RNN receives two things: the <strong>current input</strong> x_t, and the <strong>previous hidden state</strong> h_{t-1}, a running summary of everything the network has seen so far. It combines them into a <strong>current hidden state</strong> h_t, which becomes both this step&#39;s memory and next step&#39;s input.</p><pre>current input           x_t<br>previous hidden state   h_{t-1}<br>current hidden state    h_t<br>output                  y_t</pre><p>Think of h_t as the network&#39;s short term memory. At each word, it updates that memory using the new word plus whatever it already remembered, then passes the updated memory forward.</p><h3>Part 3: RNN Mathematics</h3><p>Every RNN, no matter how it is eventually used, is built from two equations.</p><pre>h_t = tanh(W_xh * x_t + W_hh * h_{t-1} + b_h)<br>y_t = W_hy * h_t + b_y</pre><p>Every symbol here is doing specific, necessary work.</p><p><strong>x_t</strong> is the input at the current timestep: a word, a price, a sensor reading, whatever the sequence is made of.</p><p><strong>h_{t-1}</strong> is the previous hidden state, the network&#39;s memory carried in from the last timestep. At the very first timestep there is nothing to carry in yet, so h_0 is initialized to zero.</p><p><strong>W_xh</strong> is the weight matrix applied to the current input, learned during training, controlling how much the new input should influence the new memory.</p><p><strong>W_hh</strong> is the weight matrix applied to the previous hidden state, controlling how much of the old memory should carry forward.</p><p><strong>b_h</strong> is the bias term for the hidden state, a learned constant added before the activation function.</p><p><strong>tanh</strong> is the activation function, squashing its input into the range negative one to positive one. This keeps each individual activation bounded, but it does not by itself guarantee that the overall recurrent dynamics stay well behaved. It does not prevent the <strong>vanishing</strong> or <strong>exploding gradient</strong> problems covered in Parts 9 and 10, since those come from repeated multiplication across timesteps, not from the range of a single activation.</p><p><strong>h_t</strong> is the new hidden state, the network&#39;s updated memory, built from the current input and everything remembered so far.</p><p><strong>W_hy</strong> and <strong>b_y</strong> turn the hidden state into an actual output: a predicted next word, a sentiment score, whatever the task requires.</p><p><strong>y_t</strong> is that output.</p><p>The single most important detail hiding in this equation is that W_xh, W_hh, and b_h do not get new values at every timestep. The exact same three learned parameters are reused at every single position in the sequence. This is called <strong>parameter sharing</strong>, and it is the reason an RNN can handle sequences of any length with a fixed number of weights.</p><h3>Shapes: From Scalars to Real Vectors</h3><p>Everything above is written as if x_t and h_t were single numbers, which is a useful simplification for arithmetic you can follow by hand. A real RNN almost never works with scalars. Every quantity in the equation is a vector or a matrix, with a specific shape that has to line up correctly for the math to even be valid.</p><pre>x_t   is a vector in R^(d_x)         the input dimension, e.g. an embedding size<br><br>h_t   is a vector in R^(d_h)         the hidden size, a design choice you pick<br><br>y_t   is a vector in R^(d_y)         the output dimension, e.g. 1 for binary sentiment<br><br>W_xh  is a matrix in R^(d_h x d_x)    maps an input vector into hidden space<br><br>W_hh  is a matrix in R^(d_h x d_h)    maps the previous hidden state into hidden space<br><br>W_hy  is a matrix in R^(d_y x d_h)    maps the hidden state into output space<br><br>b_h   is a vector in R^(d_h)<br><br>b_y   is a vector in R^(d_y)</pre><p>W_xh * x_t takes a d_x-dimensional input and produces a d_h-dimensional result. W_hh * h_{t-1} takes a d_h-dimensional hidden state and produces another d_h-dimensional result: the same shape, so the two can be added together directly before tanh is applied. This is exactly why W_hh has to be <strong>square</strong>, d_h by d_h. Its job is to map the hidden space back into that same hidden space, timestep after timestep.</p><p>This is the detail that makes the jump from a scalar toy example to a real TensorFlow model make sense. SimpleRNN(64) is choosing d_h = 64. An Embedding layer with output_dim = 100 is choosing d_x = 100. Nothing about the underlying equation changes, only how many numbers x_t and h_t actually hold.</p><h3>Part 4: A Full Numerical Example</h3><p>Theory is easier to trust once you have pushed real numbers through it by hand. Take the sentence <strong>“I love this movie”</strong> and represent each word with a simple numeric index for this toy calculation.</p><p>This is deliberately a <strong>scalar toy example</strong>: d_x = d_h = d_y = 1 in the shapes just introduced, using a raw word index directly as x_t. Real NLP models never do this. Feeding a bare index like 1 or 104 into the math would imply a numeric relationship between words that does not exist, since &quot;movie&quot; is not four times &quot;I.&quot; Part 15 replaces this with proper embedding vectors. The scalar version here exists purely so every multiplication can be checked by hand, one number at a time.</p><pre>t=1   x1 = 1   (&quot;I&quot;)<br>t=2   x2 = 2   (&quot;love&quot;)<br>t=3   x3 = 3   (&quot;this&quot;)<br>t=4   x4 = 4   (&quot;movie&quot;)</pre><p>To keep the arithmetic small enough to follow by hand, both weights are single numbers instead of matrices, and the bias is zero.</p><pre>W_xh = 0.5<br>W_hh = 0.8<br>h_0  = 0</pre><h3>Step 1, t = 1</h3><pre>h_1 = tanh(W_xh * x_1 + W_hh * h_0)<br>    = tanh((0.5)(1) + (0.8)(0))<br>    = tanh(0.5)<br>    = 0.4621</pre><h3>Step 2, t = 2</h3><pre>h_2 = tanh(W_xh * x_2 + W_hh * h_1)<br>    = tanh((0.5)(2) + (0.8)(0.4621))<br>    = tanh(1.0 + 0.3697)<br>    = tanh(1.3697)<br>    = 0.8786</pre><h3>Step 3, t = 3</h3><pre>h_3 = tanh(W_xh * x_3 + W_hh * h_2)<br>    = tanh((0.5)(3) + (0.8)(0.8786))<br>    = tanh(1.5 + 0.7029)<br>    = tanh(2.2029)<br>    = 0.9759</pre><h3>Step 4, t = 4</h3><pre>h_4 = tanh(W_xh * x_4 + W_hh * h_3)<br>    = tanh((0.5)(4) + (0.8)(0.9759))<br>    = tanh(2.0 + 0.7807)<br>    = tanh(2.7807)<br>    = 0.9923</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-OKyyuW2gQUVvfyZmERWCA.png" /><figcaption><strong>Figure 1. RNN hidden-state flow across four timesteps.</strong></figcaption></figure><p>Notice how <strong>h_t</strong> moves closer to 1 as the sentence progresses. Each new word contributes to the <strong>current hidden state</strong> while carrying information from the previous timestep. This is the basic idea of the <strong>accumulating memory</strong> in an RNN.</p><p>If the final hidden state feeds into a <strong>sentiment classifier</strong>, the true label might be <strong>y = 1</strong> for positive sentiment, while the model predicts <strong>ŷ = 0.90</strong>. A prediction close to the true label produces a smaller loss, while a prediction farther away produces a larger loss. Part 8 explores this loss in detail.</p><h3>Part 5: Unrolling Through Time and the Computational Graph</h3><p>“Unrolling” simply means drawing the same RNN cell once per timestep instead of drawing one box with a loop arrow back into itself.</p><pre>h_0 → h_1 → h_2 → h_3</pre><p>Once unrolled, it becomes visually obvious that W_xh, W_hh, and W_hy are reused at every single timestep, not relearned fresh each time.</p><pre>Timestep 1 uses:  W_xh, W_hh, b_h<br>Timestep 2 uses:  W_xh, W_hh, b_h    (the exact same numbers)<br>Timestep 3 uses:  W_xh, W_hh, b_h    (still the exact same numbers)</pre><p>This is exactly what makes an RNN able to handle sequences of any length with a fixed number of parameters. A sentence with 4 words and a sentence with 40 words use the same three weight matrices, just applied more times.</p><h3>The Full Computational Graph</h3><p>The unrolled chain above only draws the hidden state. A complete picture needs to show the output at every step too, and be precise about which quantities are the same thing and which only look similar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QoovMd2UMUH6i_YAerLsPQ.png" /><figcaption><strong>Figure 2. RNN unrolled through time.</strong></figcaption></figure><p>Four distinctions are worth keeping clear:</p><p><strong>Hidden state.</strong> h_1, h_2, and h_3 are the network’s <strong>internal memory</strong>, updated at every timestep as information moves through the sequence.</p><p><strong>Output.</strong> y_1, y_2, and y_3 are the <strong>outputs produced at each timestep</strong> from their corresponding hidden states through W_hy. Not every task uses every output.</p><p><strong>Final hidden state.</strong> h_3 is simply the <strong>last hidden state</strong>, computed by the same recurrence as every other h_t. Its importance comes from having accumulated information from the sequence up to that point.</p><p><strong>Many-to-one vs. many-to-many.</strong> The RNN itself does not change; the difference is <strong>which outputs the task keeps</strong>. Sentiment classification is many-to-one, using the final hidden state for a single prediction. Sequence labeling is many-to-many, keeping an output for every timestep.</p><h3>Part 6: RNN Architectures</h3><p>RNNs get classified by how many inputs they take versus how many outputs they produce.</p><p><strong>One to one.</strong> One input, one output. Not really recurrent in any meaningful sense, closer to a standard feedforward network, included mostly for completeness.</p><p><strong>One to many.</strong> One input, a full sequence of outputs. Image captioning: one image in, a whole sentence describing it out, one word at a time.</p><p><strong>Many to one.</strong> A full sequence in, a single output. Sentiment classification: an entire review goes in, one label, positive or negative, comes out.</p><p><strong>Many to many.</strong> A sequence in, a sequence out. Machine translation: an English sentence in, a French sentence out. Video frame labeling: one label produced per frame across a whole clip.</p><h3>Part 7: Input and Output Sequence Configurations</h3><p>The same four shapes from Part 6 map directly onto real task categories.</p><pre>Many → One    sentiment classification: a review in, one label out<br>Many → Many   sequence labeling: a sentence in, one tag per word out<br>One → Many    language generation: one seed word in, a full sequence out</pre><p><strong>Sentiment classification</strong> reads an entire sentence and produces a single positive or negative score at the end. <strong>Language generation</strong> starts from a single seed and produces one new word at a time, each new word feeding back in as the next input. <strong>Video captioning</strong> is many-to-many at a different scale entirely: many video frames in, a full sentence out. <strong>Sequence labeling</strong>, tagging each word in a sentence with a part of speech, is many-to-many at the same length: one tag per input word.</p><h3>Part 8: Backpropagation Through Time</h3><p>Ordinary backpropagation was built for networks with no loops, where every layer is used exactly once per forward pass. An RNN reuses the same weights at every timestep, so a single ordinary backward pass is not enough. <strong>Backpropagation Through Time</strong>, or <strong>BPTT</strong>, is what handles this correctly.</p><p>The idea is to unroll the RNN across every timestep exactly as in Part 5, then run backpropagation across that entire unrolled chain, all the way from the final loss back to the first timestep.</p><pre>&quot;I&quot; → &quot;love&quot; → &quot;this&quot; → &quot;movie&quot; → y_hat → Loss</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QGhTSxYgDjH8cAl2zsMhdA.png" /><figcaption><strong>Figure 3. BPTT: gradients flow backward through the unrolled RNN.</strong></figcaption></figure><p>The question BPTT answers at every step is always the same one: <strong>if I slightly change this weight, how much does the loss change?</strong> For the hidden state at the final timestep, that question reads:</p><pre>dL/dh_4 = how much the loss changes if h_4 changes slightly</pre><p>Since h_4 was computed from h_3, and h_3 from h_2, and so on, the <strong>chain rule</strong> has to trace backward through every one of those dependencies to reach an earlier hidden state.</p><pre>dL/dh_3 = dL/dh_4 * dh_4/dh_3</pre><p>And to reach h_2, that same chain extends further:</p><pre>dL/dh_2 = dL/dh_4 * dh_4/dh_3 * dh_3/dh_2</pre><p>This is the same idea as a chain of introductions. If you know something only because a friend told you, and that friend only knows it because someone else told them, tracing the information back to its original source means following every link in the chain, not jumping straight to the start. <strong>That is exactly what BPTT does, one link and one timestep at a time.</strong></p><p>Because W_hh is reused at every timestep, its total gradient is <strong>the sum of its contributions across the entire sequence</strong>, rather than just one of these terms.</p><p>Here is the key idea behind how an RNN learns over long sequences: each term like dh_4/dh_3 involves multiplying by W_hh and the derivative of tanh again. Repeating these multiplications across many timesteps can cause gradients to <strong>shrink toward zero or grow very large</strong>, depending on the repeated factors. This is the foundation of the <strong>vanishing and exploding gradient problems</strong> discussed in the next two sections.</p><h3>Part 9: The Vanishing Gradient Problem</h3><p><strong>tanh</strong> has a maximum derivative of exactly 1, and in practice its derivative is usually smaller than 1. Multiply several numbers smaller than 1 together repeatedly, and the result shrinks quickly.</p><pre>1.0  * 0.5 = 0.5<br>0.5  * 0.5 = 0.25<br>0.25 * 0.5 = 0.125</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gz0ckh5WAowXPylJ19HUBQ.png" /><figcaption><strong>Figure 4. Vanishing gradients through time.</strong></figcaption></figure><p>This is the <strong>vanishing gradient problem</strong>. dL/dh_t becomes extremely small when it has to travel backward through enough timesteps. A gradient that has nearly vanished gives the network very little information about how to adjust weights at earlier timesteps.</p><p>Here is why this is a practical problem, not just a mathematical curiosity.</p><blockquote>“The movie I watched yesterday with my brother after dinner at the new cinema was amazing.”</blockquote><p>The word that “was amazing” actually depends on, <strong>“movie,”</strong> appears many words earlier in the sentence. By the time the gradient signal from the end of the sentence travels backward to “movie,” it may have been multiplied by shrinking factors so many times that very little useful signal remains. The network therefore struggles to learn this <strong>long-distance dependency</strong>, not because the relationship is absent, but because the gradient carrying that information has weakened along the way.</p><p>This limitation is one of the key motivations behind architectures such as <strong>LSTM</strong> and <strong>GRU</strong>.</p><h3>Part 10: The Exploding Gradient Problem</h3><p>The opposite failure happens when those repeated factors are larger than 1 instead of smaller.</p><pre>1.0 * 2.0 = 2.0<br>2.0 * 2.0 = 4.0<br>4.0 * 2.0 = 8.0</pre><p>Instead of decaying toward zero, the gradient grows without bound. In practice this shows up as a training run that suddenly produces NaN losses or wildly unstable weight updates, the numeric equivalent of a value spiraling out of control.</p><p>The standard fix is <strong>gradient clipping</strong>: if the gradient’s magnitude exceeds some chosen threshold, rescale it back down to that threshold before applying the update.</p><pre>if gradient_norm &gt; threshold:<br>    gradient = gradient * (threshold / gradient_norm)</pre><p>This does not fix vanishing gradients. It specifically targets the exploding case, capping the damage a single unstable timestep can do to the whole network’s weights.</p><p><strong>One precision note.</strong> In practice, clipping is almost always applied to the <strong>global norm</strong> of the gradient, computed across every parameter in the network at once, not to each weight’s gradient individually. Clipping weight by weight would distort the relative direction of the update; clipping the global norm rescales the whole update vector uniformly and preserves its direction. This is also what tf.clip_by_global_norm and PyTorch&#39;s clip_grad_norm_ do by default.</p><h3>Part 11: Bidirectional RNN</h3><p>Every RNN covered so far only looks backward, at everything that has already happened. A <strong>Bidirectional RNN</strong>, or <strong>BiRNN</strong>, runs two RNNs over the same sequence at once: one reads forward, while the other reads backward. Their hidden states are then combined at every timestep.</p><pre>Forward:   x1 → x2 → x3 → x4<br>Backward:  x1 ← x2 ← x3 ← x4<br><br>h_t = [forward_h_t ; backward_h_t]</pre><p>The square brackets denote <strong>concatenation</strong>: simply placing the forward and backward hidden states side by side into one combined vector.</p><p>Here is why this matters, using a genuinely ambiguous sentence:</p><blockquote>“The bank near the river flooded.”</blockquote><p>Reading only left to right, when the model reaches “bank,” it has not yet seen “river,” which provides important context for interpreting the word. The backward RNN can carry that future context back from later words. Combining both directions gives the network access to <strong>context from both sides</strong> of every word, rather than only what came before it.</p><p><strong>This comes with an important limitation.</strong> A Bidirectional RNN needs the entire sequence available before processing it, since the backward pass starts from the end. Consider trying to predict the next word live, as someone types:</p><blockquote>“I am going to ___”</blockquote><p>There is no future word yet to read backward from because the next word is exactly what is being predicted. Bidirectional RNNs are therefore suited to tasks where the <strong>full sequence is available upfront</strong>, such as classifying a complete sentence, but not to real-time causal prediction where the sequence unfolds one step at a time.</p><h3>Part 12: RNN Variants at a Glance</h3><pre>Vanilla RNN            the basic single-direction recurrence covered in Parts 3 through 5<br>Bidirectional RNN       reads forward and backward, covered in Part 11<br>Stacked, or deep RNN    multiple RNN layers, each layer&#39;s hidden states feed the next layer as its input<br>Stateful RNN            hidden state carries over between separate training batches<br>Stateless RNN           hidden state resets to zero at the start of every batch, the default assumption elsewhere in this article</pre><p>A <strong>stacked RNN</strong> simply feeds the sequence of hidden states from one RNN layer in as the input sequence to a second RNN layer stacked on top, letting the network build more abstract, higher level representations the same way stacking convolutional layers does for images.</p><pre>Layer 2:   h1(2) → h2(2) → h3(2)<br>                │       │       │<br>Layer 1:   h1(1) → h2(1) → h3(1)<br>                │       │       │<br>Input:        x1      x2      x3</pre><h3>Part 13: RNN From Scratch With NumPy</h3><p>Nothing builds confidence in an equation faster than watching code reproduce numbers you already worked out by hand. This implements the exact example from Part 4.</p><pre>import numpy as np<br><br>class RNNCell:<br>    def __init__(self, wx, wh, b=0.0):<br>        self.wx = wx<br>        self.wh = wh<br>        self.b = b<br>    def step(self, x_t, h_prev):<br>        z = self.wx * x_t + self.wh * h_prev + self.b<br>        h_t = np.tanh(z)<br>        return h_t<br><br>def run_sequence(cell, inputs, h0=0.0):<br>    &quot;&quot;&quot;Runs the RNN forward across a full sequence, returning every hidden state.&quot;&quot;&quot;<br>    h = h0<br>    hidden_states = []<br>    for x_t in inputs:<br>        h = cell.step(x_t, h)<br>        hidden_states.append(h)<br>    return hidden_states<br><br># &quot;I love this movie&quot;, encoded as simple word positions<br>inputs = [1, 2, 3, 4]<br>cell = RNNCell(wx=0.5, wh=0.8)<br>hidden_states = run_sequence(cell, inputs, h0=0.0)<br>for t, h in enumerate(hidden_states, start=1):<br>    print(f&quot;h{t} = {h:.4f}&quot;)</pre><pre>h1 = 0.4621<br>h2 = 0.8786<br>h3 = 0.9759<br>h4 = 0.9923</pre><p>This matches Part 4’s hand calculation exactly, to four decimal places.</p><h3>Adding an Output and a Loss</h3><pre>def predict(hidden_states, why, by):<br>    &quot;&quot;&quot;Many-to-one prediction: only the final hidden state is used.&quot;&quot;&quot;<br>    h_final = hidden_states[-1]<br>    y_hat = why * h_final + by<br>    return y_hat<br><br>def mse_loss(y_true, y_pred):<br>    return (y_true - y_pred) ** 2<br><br>y_hat = predict(hidden_states, why=0.9, by=0.0)<br>loss = mse_loss(y_true=1.0, y_pred=y_hat)<br>print(f&quot;y_hat = {y_hat:.4f}&quot;)<br>print(f&quot;loss  = {loss:.4f}&quot;)</pre><pre>y_hat = 0.8931<br>loss  = 0.0114</pre><p>A y_hat close to 1 with a true label of 1 produces a small loss, matching the <strong>“good prediction, small loss”</strong> relationship from Part 4.</p><p><strong>A quick note on the loss function used here.</strong> This walkthrough uses <strong>mean squared error (MSE)</strong> purely because its derivative is simple enough to work through by hand, keeping the manual BPTT code in the next section readable. A real binary sentiment classifier typically uses <strong>binary cross-entropy</strong>, which is what the TensorFlow model in Part 14 uses. The two losses behave differently in practice: MSE penalizes confident wrong predictions less strongly than cross-entropy. The mechanics of BPTT remain the same either way. Only the first gradient, dL/dy_hat, changes.</p><h3>Implementing BPTT Manually</h3><p>Extending the scalar version from Part 8 into actual code, walking the gradient backward one timestep at a time.</p><pre>def bptt(inputs, hidden_states, y_true, y_hat, wx, wh, why):<br>    &quot;&quot;&quot;<br>    Manual backpropagation through time for the scalar RNN above.<br>    Returns the accumulated gradients for wx, wh, and why.<br>    &quot;&quot;&quot;<br>    n = len(inputs)<br>    dL_dyhat = -2 * (y_true - y_hat)          # derivative of MSE loss<br>    dL_dwhy = dL_dyhat * hidden_states[-1]<br>    dL_dh_next = dL_dyhat * why               # gradient flowing into the last hidden state<br><br>dL_dwx = 0.0<br>    dL_dwh = 0.0<br>    for t in reversed(range(n)):<br>        h_t = hidden_states[t]<br>        h_prev = hidden_states[t - 1] if t &gt; 0 else 0.0<br>        dtanh = 1 - h_t ** 2                  # derivative of tanh(z) is 1 - tanh(z)^2<br>        dL_dz = dL_dh_next * dtanh<br>        dL_dwx += dL_dz * inputs[t]<br>        dL_dwh += dL_dz * h_prev<br>        dL_dh_next = dL_dz * wh               # pass the gradient one step further back<br>    return dL_dwx, dL_dwh, dL_dwhy<br><br>dwx, dwh, dwhy = bptt(inputs, hidden_states, y_true=1.0, y_hat=y_hat,<br>                       wx=0.5, wh=0.8, why=0.9)<br>print(f&quot;dL/dWx  = {dwx:.4f}&quot;)<br>print(f&quot;dL/dWh  = {dwh:.4f}&quot;)<br>print(f&quot;dL/dWhy = {dwhy:.4f}&quot;)</pre><pre>dL/dWx  = -0.0121<br>dL/dWh  = -0.0030<br>dL/dWhy = -0.2121</pre><p>The for t in reversed(range(n)) loop is <strong>BPTT </strong>in its most literal form: start at the last timestep, and walk backward, exactly the direction Part 8 described in words, now written as code that actually runs.</p><h3>Part 14: The Same Model in TensorFlow</h3><p>Once the manual version above makes sense, the library version is mostly a matter of not having to write the loop yourself.</p><pre>import tensorflow as tf<br>from tensorflow import keras<br><br>model = keras.Sequential([<br>    keras.layers.SimpleRNN(64, input_shape=(None, 1)),<br>    keras.layers.Dense(1, activation=&quot;sigmoid&quot;),<br>])<br>model.compile(optimizer=&quot;adam&quot;, loss=&quot;binary_crossentropy&quot;, metrics=[&quot;accuracy&quot;])</pre><p>SimpleRNN(64) creates 64 hidden units, meaning h_t is now a 64-dimensional vector instead of the single scalar used for hand calculation, with W_xh, W_hh, and b_h sized to match, exactly the shapes introduced in Part 3. Internally, SimpleRNN runs the exact loop from Part 13&#39;s run_sequence function, once per timestep, across whatever sequence length it is given, and model.fit runs the exact BPTT process from Part 13&#39;s bptt function automatically, across every weight, for every batch, for as many epochs as you train. Nothing about the underlying math changes, only who is writing the loop.</p><h3>A Complete, Runnable Training Example</h3><p>The architecture above is real, but it is not yet a full workflow. This version takes a small labeled dataset all the way from raw text to a prediction on a brand new sentence, and every number below is copied directly from actually running this code, not estimated.</p><pre>import numpy as np<br>import tensorflow as tf<br>from tensorflow import keras<br>from tensorflow.keras.preprocessing.text import Tokenizer<br>from tensorflow.keras.preprocessing.sequence import pad_sequences<br><br>tf.random.set_seed(42)<br>np.random.seed(42)<br># 1. A small labeled dataset, 1 = positive, 0 = negative<br>texts = [<br>    &quot;I love this movie&quot;, &quot;This film was amazing&quot;, &quot;What a wonderful story&quot;,<br>    &quot;I really enjoyed this&quot;, &quot;Best movie I have seen&quot;, &quot;I hate this movie&quot;,<br>    &quot;This film was terrible&quot;, &quot;What a boring story&quot;, &quot;I really disliked this&quot;,<br>    &quot;Worst movie I have seen&quot;, &quot;The acting was great&quot;, &quot;The acting was awful&quot;,<br>    &quot;A truly beautiful film&quot;, &quot;A truly disappointing film&quot;,<br>    &quot;I would watch this again&quot;, &quot;I would never watch this again&quot;,<br>]<br>labels = [1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 0]<br># 2. Tokenization: words to integer IDs<br>tokenizer = Tokenizer(num_words=1000, oov_token=&quot;&lt;OOV&gt;&quot;)<br>tokenizer.fit_on_texts(texts)<br>sequences = tokenizer.texts_to_sequences(texts)<br># 3. Padding: every sequence forced to the same length<br>max_length = max(len(seq) for seq in sequences)<br>padded = pad_sequences(sequences, maxlen=max_length, padding=&quot;post&quot;)<br># 4. Train/test split<br>X, y = np.array(padded), np.array(labels)<br>X_train, X_test = X[:12], X[12:]<br>y_train, y_test = y[:12], y[12:]<br># 5. Build the model: Embedding -&gt; SimpleRNN -&gt; Dense<br>vocab_size = len(tokenizer.word_index) + 1<br>model = keras.Sequential([<br>    keras.Input(shape=(max_length,)),<br>    keras.layers.Embedding(input_dim=vocab_size, output_dim=16),<br>    keras.layers.SimpleRNN(8),<br>    keras.layers.Dense(1, activation=&quot;sigmoid&quot;),<br>])<br># 6. Compile<br>model.compile(optimizer=&quot;adam&quot;, loss=&quot;binary_crossentropy&quot;, metrics=[&quot;accuracy&quot;])<br>model.summary()<br># 7. Fit<br>history = model.fit(X_train, y_train, epochs=30, verbose=0)<br>print(&quot;final training accuracy:&quot;, round(history.history[&quot;accuracy&quot;][-1], 4))<br># 8. Evaluate<br>test_loss, test_acc = model.evaluate(X_test, y_test, verbose=0)<br>print(f&quot;test loss: {test_loss:.4f}, test accuracy: {test_acc:.4f}&quot;)<br># 9. Predict on a brand new sentence<br>new_sentence = [&quot;I really loved this film&quot;]<br>new_seq = tokenizer.texts_to_sequences(new_sentence)<br>new_padded = pad_sequences(new_seq, maxlen=max_length, padding=&quot;post&quot;)<br>prediction = model.predict(new_padded, verbose=0)[0][0]<br>print(f&quot;prediction for &#39;{new_sentence[0]}&#39;: {prediction:.4f}&quot;)</pre><pre>Layer (type)              Output Shape         Param #<br>embedding                 (None, 6, 16)         544<br>simple_rnn                (None, 8)             200<br>dense                     (None, 1)             9<br>Total params: 753<br><br>final training accuracy: 1.0<br>test loss: 0.6791, test accuracy: 0.7500<br>prediction for &#39;I really loved this film&#39;: 0.4775</pre><p><em>These exact numbers came from one run on one machine with the random seed fixed. TensorFlow, Keras, and NumPy version differences, and even CPU versus GPU execution, can shift the last few decimal places. Treat the values above as illustrative of the pattern rather than figures to expect bit-for-bit.</em></p><p>Sixteen sentences is nowhere near enough data for a model anyone would deploy. A training accuracy of 1.0 here mostly reflects memorization of 12 examples, not genuine mastery. That is the honest reading of the result.</p><p>What this example actually demonstrates is the complete pipeline working end to end: tokenization producing consistent integer sequences, padding making them uniform in length, the model training successfully, and a brand-new sentence, “I really loved this film,” receiving a prediction.</p><p>Scaling this same pipeline to thousands of labeled reviews is what separates this toy example from a genuinely useful classifier.</p><h3>Counting the Parameters</h3><p>The same idea used to count a fully connected layer’s parameters applies to an RNN’s own layers, and the model just trained above gives real numbers to check it against.</p><pre>Recurrent layer parameters = d_h * d_x + d_h * d_h + d_h<br>Output layer parameters    = d_y * d_h + d_y</pre><p>The first term maps the input into hidden space, the second maps the previous hidden state into hidden space, and the lone d_h is the hidden bias, matching the three learned pieces named back in Part 3.</p><p>For the model above, the embedding produces d_x = 16, SimpleRNN(8) sets d_h = 8, and the final Dense(1) sets d_y = 1.</p><pre>Recurrent layer: (8)(16) + (8)(8) + 8 = 128 + 64 + 8 = 200<br>Output layer:    (1)(8) + 1           = 9</pre><p><strong>200</strong> and <strong>9</strong> are exactly what model.summary() printed above, not a coincidence. The formula and the framework are describing the same weights. This is also worth keeping in mind going forward: an <strong>LSTM</strong> layer replaces this single set of weights with four separate sets, one for each of its gates, which is exactly why an LSTM with the same hidden size carries roughly four times as many parameters as the vanilla RNN counted here.</p><h3>Part 15: A Real NLP Pipeline</h3><p>A real sentence has to pass through several steps before an RNN ever sees a number.</p><pre>Text<br>  → Cleaning<br>  → Tokenization<br>  → Token IDs<br>  → Padding<br>  → Embedding<br>  → RNN<br>  → Dense<br>  → Output</pre><p>Walking “I love this movie” through each stage:</p><pre>Text:          &quot;I love this movie&quot;<br>Cleaning:      lowercase, punctuation stripped → &quot;i love this movie&quot;<br>Tokenization:  [&quot;i&quot;, &quot;love&quot;, &quot;this&quot;, &quot;movie&quot;]<br>Token IDs:     [3, 27, 5, 104]        (each word&#39;s position in a vocabulary)<br>Padding:       [3, 27, 5, 104, 0, 0]  (padded to a fixed sequence length)<br>Embedding:     each ID becomes a dense vector, not just the integer itself<br>RNN:           processes the sequence of embedding vectors, timestep by timestep<br>Dense:         maps the final hidden state to a prediction<br>Output:        0.91  →  positive</pre><p><strong>Embedding</strong> replaces a bare integer ID with a learned, dense vector, since feeding raw word indices like 3 and 104 straight into the math would imply a numeric relationship between words that does not actually exist. A common embedding size is 100 dimensions, meaning each word becomes a vector in a 100 dimensional space, x_t is an element of R^100, while a 64-unit RNN&#39;s hidden state h_t is an element of R^64, a completely different, learned dimensionality that has nothing to do with the embedding size other than both being design choices.</p><pre>model = keras.Sequential([<br>    keras.layers.Embedding(input_dim=vocab_size, output_dim=100),<br>    keras.layers.SimpleRNN(64),<br>    keras.layers.Dense(1, activation=&quot;sigmoid&quot;),<br>])</pre><h3>Part 16: RNN for Next Word Prediction</h3><p>Instead of producing one label for a whole sequence, next word prediction produces a probability distribution over the entire vocabulary at every step.</p><pre>&quot;I want to go to&quot;<br>        ↓<br>       RNN<br>        ↓<br>probability distribution over every word in the vocabulary<br>        ↓<br>     &quot;school&quot;</pre><p>Formally, the model is estimating:</p><pre>P(x_{t+1} | x_1, x_2, ..., x_t)</pre><p>Read this as: the probability of the next word, given every word that came before it. The RNN’s final hidden state, having compressed the entire history into one vector, feeds into a <strong>softmax</strong> layer, which converts raw scores into a valid probability distribution across the whole vocabulary, one probability per possible next word, all summing to 1. The word with the highest probability, “school” in this example, becomes the model’s prediction.</p><h3>Part 17: Training an RNN</h3><p>Several pieces come together during actual training, beyond the forward and backward pass already covered.</p><pre>Loss function      measures how wrong a prediction is, MSE for regression, cross entropy for classification<br>Cross entropy      the standard loss for next word prediction and sentiment classification<br>Optimizer          Adam or SGD, the algorithm that turns gradients into weight updates<br>Learning rate      how large each update step is<br>Epochs             how many full passes through the training data<br>Batch              how many sequences are processed before one weight update<br>Sequence length    how many timesteps each training example is unrolled across</pre><p><strong>Teacher forcing</strong> is specific to sequence generation tasks like next word prediction. During training, instead of feeding the model’s own, possibly wrong, previous prediction back in as the next input, the actual correct previous word from the training data is fed in instead. This keeps early training stable, since the model is not forced to build every prediction on top of its own mistakes from the very first epoch, though it does create a mismatch with inference time, when the model’s own predictions are all it has to work with.</p><h3>Part 18: Limitations of Vanilla RNN</h3><pre>Vanishing gradients             long-distance dependencies are hard to learn, Part 9<br>Exploding gradients             training can become numerically unstable, Part 10<br>Long-term dependency struggles  a direct consequence of the vanishing gradient problem<br>Sequential computation          each timestep depends on the last, so timesteps cannot be parallelized<br>Training limitations            slow on long sequences, and prone to the two gradient problems above</pre><p>The sequential computation limitation is worth sitting with, since it is easy to miss. Unlike a convolution, which can process every position of an image at once, an RNN’s hidden state at timestep 5 mathematically requires the hidden state at timestep 4, which requires timestep 3, and so on. This chain cannot be broken apart and computed in parallel, which makes long sequences meaningfully slower to train than their length alone would suggest.</p><p><strong>These limitations motivated architectures such as LSTM and GRU</strong>, which replace the single recurrence covered here with gated mechanisms specifically designed to control what gets remembered and what gets forgotten.</p><h3>Part 19: RNN vs Traditional Feedforward Neural Network</h3><pre>| Feature                             | Feedforward NN | RNN                        |<br>| ----------------------------------- | -------------- | -------------------------- |<br>| Sequence memory                     | No             | Yes                        |<br>| Uses previous timestep              | No             | Yes                        |<br>| Handles variable-length sequences   | Limited        | Yes                        |<br>| Captures temporal information       | No             | Yes                        |<br>| Parameters reused across positions  | No             | Yes, via parameter sharing |<br>| Processes all positions in parallel | Yes            | No, inherently sequential  |</pre><h3>Part 20: Final Summary</h3><pre>Sequence<br>   ↓<br>   x_t<br>   ↓<br>Previous hidden state h_{t-1}<br>   ↓<br>RNN equation<br>   ↓<br>Current hidden state h_t<br>   ↓<br>Output<br>   ↓<br>Loss<br>   ↓<br>BPTT<br>   ↓<br>Problem: Vanishing / Exploding Gradients<br>   ↓<br>LSTM / GRU</pre><p>Every part of this article sits somewhere on this one chain. An input arrives, it combines with memory, that memory updates, an output gets produced, the output gets scored against the truth, and that score’s gradient has to travel all the way back through every timestep it came from. Where that gradient survives the trip, the network learns. Where it does not, vanilla RNNs hit a wall that gated architectures like <strong>LSTM and GRU </strong>were built specifically to climb over.</p><h3>Key Formulas</h3><p><strong>The Core RNN Equations</strong></p><pre>h_t = tanh(W_xh * x_t + W_hh * h_{t-1} + b_h)<br>y_t = W_hy * h_t + b_y</pre><p><strong>Shapes</strong></p><pre>x_t in R^(d_x)              W_xh in R^(d_h x d_x)<br>h_t in R^(d_h)              W_hh in R^(d_h x d_h)<br>y_t in R^(d_y)              W_hy in R^(d_y x d_h)</pre><p><strong>Chain Rule Through Time</strong></p><pre>dL/dh_{t-1} = dL/dh_t * dh_t/dh_{t-1}</pre><p><strong>Vanishing and Exploding Gradients</strong></p><pre>Repeated factors &lt; 1   →  gradient shrinks toward zero   (vanishing)<br>Repeated factors &gt; 1   →  gradient grows without bound    (exploding)</pre><p><strong>Gradient Clipping</strong></p><pre>if gradient_norm &gt; threshold:<br>    gradient = gradient * (threshold / gradient_norm)</pre><p><strong>Bidirectional Hidden State</strong></p><pre>h_t = [forward_h_t ; backward_h_t]</pre><p><strong>Next Word Prediction</strong></p><pre>P(x_{t+1} | x_1, x_2, ..., x_t)</pre><p><strong>Parameter Count</strong></p><pre>Recurrent layer = d_h * d_x + d_h * d_h + d_h<br>Output layer    = d_y * d_h + d_y</pre><h3>Complete Summary Table</h3><pre>| Concept                  | What It Is                                           | What It Does                                                                |<br>| ------------------------ | ---------------------------------------------------- | --------------------------------------------------------------------------- |<br>| Sequence data            | Data where order changes meaning                     | The reason RNNs exist                                                       |<br>| Hidden state             | The network’s running memory                         | Carries information from earlier timesteps forward                          |<br>| Parameter sharing        | The same weights reused at every timestep            | Lets one fixed set of weights handle any sequence length                    |<br>| Vector and matrix shapes | `x_t`, `h_t`, `y_t` as vectors; `W`’s as matrices    | Makes the equations work with real embeddings, not just scalars             |<br>| Unrolling                | The recurrence drawn as a chain over time            | Makes BPTT possible to reason about and compute                             |<br>| Computational graph      | Hidden state and output shown at every timestep      | Clarifies many-to-one vs. many-to-many                                      |<br>| BPTT                     | Backpropagation extended across every timestep       | Computes how each weight contributed to the loss                            |<br>| Vanishing gradient       | Repeated multiplication by factors under 1           | Makes long-distance dependencies hard to learn                              |<br>| Exploding gradient       | Repeated multiplication by factors over 1            | Can cause unstable or NaN training                                          |<br>| Gradient clipping        | Rescaling an oversized gradient                      | Directly addresses exploding gradients                                      |<br>| Bidirectional RNN        | Two RNNs, one forward and one backward               | Uses both past and future context when available                            |<br>| Embedding                | A learned dense vector for each token                | Replaces raw token IDs with meaningful numeric representations              |<br>| Parameter count          | `d_h × d_x + d_h × d_h + d_h`, plus the output layer | Shows why RNNs remain compact and why LSTMs need roughly 4× more parameters |<br>| Teacher forcing          | Feeding the true previous token during training      | Stabilizes early training on generation tasks                               |<br>| Softmax                  | Converts raw scores into probabilities               | Used for next-word prediction over a vocabulary                             |</pre><h3>Common Misconceptions</h3><p>A few RNN misunderstandings are worth clearing up:</p><p><strong>“</strong><strong>tanh prevents vanishing and exploding gradients.”</strong> It does not. The problems come from repeatedly multiplying W_hh and the derivative of tanh across timesteps.</p><p><strong>“A bigger hidden state fixes long-range dependencies.”</strong> More hidden units add capacity, but they do not change the gradient flow. LSTM and GRU change the recurrence itself.</p><p><strong>“The final hidden state </strong><strong>h_T is special.”</strong> It is not. It is computed by the same recurrence as every other h_t; many-to-one tasks simply use the final state for their prediction.</p><p><strong>“RNNs process the whole sequence at once.”</strong> They do not. Each h_t depends on h_{t-1}, making the computation sequential.</p><p><strong>“Raw word indices are fine as inputs.”</strong> They are not. Token IDs do not carry meaningful numeric relationships; real pipelines use embeddings.</p><h3>Final Thoughts</h3><p>An RNN is built from one simple recurrence, applied repeatedly with the same weights. The hidden state carries information forward, while BPTT traces the resulting gradients backward through time.</p><p>That simple structure explains both the <strong>strengths and limitations of vanilla RNNs</strong>, and why architectures such as <strong>LSTM and GRU</strong> were developed.</p><p>If this article helped make RNNs click, share it with someone learning sequence models.</p><p><strong>Follow me for more on Python, deep learning, and computer vision.</strong></p><p><strong>LinkedIn:</strong> <a href="https://www.linkedin.com/in/subinthapa">Subin Thapa</a></p><p><strong>GitHub:</strong> <a href="https://github.com/Subinthapa2092">Subinthapa2092</a></p><p><strong>YouTube:</strong> <a href="https://www.youtube.com/@codingwithsubin">Coding With Subin</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8ae87ca46cd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From 82% to 98%: Building a Tumor Detector With a CNN]]></title>
            <link>https://medium.com/@subinthapa2092/from-82-to-98-building-a-tumor-detector-with-a-cnn-6802a6db707f?source=rss-9f567b128564------2</link>
            <guid isPermaLink="false">https://medium.com/p/6802a6db707f</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[medical-ai]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Subinthapa]]></dc:creator>
            <pubDate>Mon, 14 Sep 2026 04:12:10 GMT</pubDate>
            <atom:updated>2026-09-14T04:12:10.144Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-xLMx5w0Fd5KEI8pCmAIlg.png" /></figure><h4>A real histopathology dataset, an unstable first run, and what finally worked</h4><p><strong>By epoch 5</strong>, validation AUC had dropped from <strong>0.9331 to 0.6645.</strong> The loss had gone from <strong>0.36 to 1.54</strong>. Whatever the model had learned in the first three epochs, it was actively forgetting it by the fifth.</p><p>That run wasn’t the failure. It was the warning. The real run, twenty epochs with early stopping and a restored best checkpoint, landed at <strong>0.9830 test ROC-AUC</strong>. This is the story of the gap between those two numbers, and the longer story of getting that trained model to actually load inside a Docker container.</p><p>Full project on GitHub: <a href="https://github.com/Subinthapa2092/histopathologic-cancer-detection">Subinthapa2092/histopathologic-cancer-detection</a></p><h3>Part 1: The Dataset</h3><p>The <a href="https://www.kaggle.com/competitions/histopathologic-cancer-detection/overview">Kaggle Histopathologic Cancer Detection</a> competition, a de-duplicated version of the <strong>PatchCamelyon (PCam)</strong> benchmark, itself built from CAMELYON16 lymph-node scans.</p><pre>Total patches:     220,025<br>Patch size:        96x96, RGB<br>Label rule:        1 if the CENTER 32x32px region contains tumor tissue</pre><p>That labeling rule matters more than it looks. The tumor signal isn’t “<strong>does this image look abnormal.</strong>” It’s specifically whether the center of the patch crosses a threshold. Two models trained on this data can legitimately disagree on identical-looking edges of a patch and still both be right, because the label was never about the whole image. Everything downstream, the baseline’s center-region features, the CNN’s architecture, reads differently once that rule is the starting point instead of an afterthought.</p><p>Split, stratified:</p><pre>Train:       154,017<br>Validation:   33,004<br>Test:         33,004</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1bOsStNFVDW3hcvkr_EqBQ.png" /><figcaption><strong>Figure 1. Sample tissue patches: normal vs. tumor.</strong></figcaption></figure><h3>Part 2: Why a Baseline Came First</h3><p>Before any CNN code, a <strong>color-histogram + Logistic Regression</strong> model went in.</p><pre>def extract_color_features(filepath, bins=16):<br>    img = load_image_as_array(filepath).astype(np.float32) / 255.0<br>    features = []<br>    for channel in range(3):<br>        hist, _ = np.histogram(img[:, :, channel].ravel(), bins=bins, range=(0.0, 1.0))<br>        hist = hist / hist.sum()<br>        features.extend(hist.tolist())<br>    # center 32x32 gets its own mean/std features, matching the label rule<br>    ...</pre><p>Not a strawman. The center-region features were built in on purpose, matching exactly what the competition’s own labeling rule uses. If a linear model over color statistics could solve this, a CNN would have been the wrong tool for it.</p><p>It couldn’t. But it wasn’t nothing either.</p><pre>Baseline Test ROC-AUC:  0.8937<br>Baseline Test Accuracy: 82.10%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*Bh4SfVILHKExzkFutxVPkQ.png" /><figcaption><strong>Figure 2. Class distribution across the real dataset.</strong></figcaption></figure><p>A histogram gets you to 82%. That’s the number every later result has to actually beat, not just edge past.</p><h3>Part 3: The CNN</h3><pre>Input (96x96x3)<br>  -&gt; Rescaling(1/255)<br>  -&gt; [Conv3x3 -&gt; BatchNorm -&gt; Conv3x3 -&gt; BatchNorm -&gt; MaxPool] x4   (32, 64, 128, 256 filters)<br>  -&gt; GlobalAveragePooling2D<br>  -&gt; Dropout(0.5)<br>  -&gt; Dense(256) -&gt; ReLU<br>  -&gt; Dropout(0.3)<br>  -&gt; Dense(1, sigmoid)</pre><p>Two choices worth explaining.</p><p><strong>GlobalAveragePooling2D instead of Flatten.</strong> Flatten into a big Dense layer adds enormous parameter count for a dataset this size relative to a typical ImageNet-scale training set. Global average pooling collapses each feature map to a single number instead, which means far fewer parameters heading into the dense head and less to overfit with.</p><p><strong>BatchNorm after every Conv2D, not just some.</strong> Consistent normalization at every stage kept training stable across all four resolution levels. That choice mattered more than expected once the wobble described in Part 5 showed up.</p><p>Compiled with <strong>ROC-AUC</strong> tracked as a metric during training, not just computed at the end, since AUC is the actual competition metric.</p><pre>model.compile(<br>    optimizer=keras.optimizers.Adam(learning_rate=1e-3),<br>    loss=&quot;binary_crossentropy&quot;,<br>    metrics=[keras.metrics.AUC(name=&quot;auc&quot;), &quot;accuracy&quot;],<br>)</pre><p>Watching AUC live rather than just loss is what made the early-stopping decision in Part 5 meaningful in the first place.</p><h3>Part 4: The Data Pipeline</h3><p>Two details here that aren’t obvious until you hit them.</p><p><strong>TIFF doesn’t decode through </strong><strong>tf.io.decode_image.</strong> The Kaggle files ship as .tif. TensorFlow&#39;s native image decoder supports PNG, JPEG, GIF, and BMP, not TIFF. The fix routes every file through <strong>PIL</strong> inside a tf.py_function:</p><pre>def _load_image_numpy(filepath, image_size):<br>    from PIL import Image<br>    with Image.open(filepath.numpy().decode(&quot;utf-8&quot;)) as img:<br>        img = img.convert(&quot;RGB&quot;)<br>        return np.array(img, dtype=np.uint8)</pre><p>Anyone reusing a generic tf.data image pipeline on this dataset will hit a silent wrong decode or an outright crash here. It&#39;s a TIFF problem, not a TensorFlow problem, and it doesn&#39;t announce itself as one.</p><p><strong>Everything loads lazily.</strong> 220,025 patches at 96x96x3 is multiple gigabytes as raw arrays. tf.data.Dataset.from_tensor_slices runs on filepaths, not pixels, with .map() doing the decode per batch. Memory stayed flat regardless of dataset size.</p><p>Augmentation (horizontal flip, vertical flip, brightness jitter) applied only to the training split. Tissue patches have no canonical “up,” so flips cost nothing and add real regularization.</p><p>Class weights, computed from the real training split:</p><pre>{0: 0.84, 1: 1.23}</pre><p>Close to balanced already, this dataset doesn’t have the severe imbalance some medical datasets do. Still computed properly rather than assumed, because “probably fine” isn’t the same as checked.</p><h3>Part 5: The Unstable First Run</h3><p>The first training attempt used 5 epochs, no learning rate schedule beyond the default.</p><pre>Epoch 1: val_auc 0.9331, val_loss 0.3640<br>Epoch 3: val_auc 0.9675, val_loss 0.2307   (best)<br>Epoch 4: val_auc 0.9102, val_loss 0.5782<br>Epoch 5: val_auc 0.6645, val_loss 1.5407</pre><p>Peak at epoch 3, collapse by epoch 5. restore_best_weights=True meant the saved model was still the epoch 3 version, nothing was actually lost, but the collapse needed explaining before trusting a longer run.</p><p>The fix was <strong>ReduceLROnPlateau</strong>, watching val_loss:</p><pre>callbacks = [<br>    keras.callbacks.EarlyStopping(monitor=&quot;val_auc&quot;, mode=&quot;max&quot;, patience=5, restore_best_weights=True),<br>    keras.callbacks.ReduceLROnPlateau(monitor=&quot;val_loss&quot;, factor=0.5, patience=2, min_lr=1e-6),<br>]</pre><p>The second run, 20 epochs:</p><pre>Epoch  1: val_auc 0.9567   lr 5.0e-4<br>Epoch  9: val_auc 0.9828   lr 2.5e-4   (best)<br>Epoch 14: val_auc 0.9726   lr 6.25e-5  (stopped here, no improvement since epoch 9)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*B2G4OJ6q4hLx1CszMpbmFw.png" /><figcaption><strong>Figure 3. Training and validation curves over 20 epochs.</strong></figcaption></figure><p>No collapse this time. The learning rate stepped down three times as val_loss plateaued, and training stayed stable through all fourteen epochs instead of falling apart at five. <strong>Early stopping</strong> ended it automatically once five straight epochs passed without beating epoch 9, not a number chosen in advance, a number the training run decided on its own.</p><h3>Part 6: What “20,000 Images” Actually Means</h3><p>Worth stating plainly, because it changes how the final number should be read.</p><p>Training ran on <strong>CPU only</strong>. TensorFlow dropped native Windows GPU support as of 2.11. A plain pip install tensorflow on Windows gets CPU, full stop, unless WSL2 is set up with a CUDA-enabled Linux side install specifically. At roughly 430 to 575 seconds per epoch on the full 154,017-image training set, twenty epochs would have cost the better part of a day.</p><p>The training set was cut to a 20,000-image random subsample, 3,000 for validation, to make that practical.</p><p><strong>The test set was not subsampled.</strong> All 33,004 real test images went into the final evaluation, untouched by any of the training-time sampling decisions. The 0.9830 test AUC below is a real number on real, full-size held-out data. It’s the training data that was reduced, not the number being reported.</p><p>This is the honest ceiling question: a from-scratch CNN trained on 20,000 images already beats an 82%-accuracy baseline by more than twelve points. Training on the full 154,017, or on a GPU where the full set becomes practical, is the most likely lever left for pushing past 0.9830, not a change to the architecture.</p><h3>Part 7: Final Results</h3><pre>Test ROC-AUC:  0.9830<br>Test Accuracy: 94.39%</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*zpAasndJmauE9GXxRacYGw.png" /><figcaption><strong>Figure 4. ROC curve on the test set.</strong></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*SwDY8ANfmboNAkv8cr7clw.png" /><figcaption><strong>Figure 5. Confusion matrix on the test set.</strong></figcaption></figure><pre>| Metric        | Baseline |    CNN |<br>| ------------- | -------: | -----: |<br>| Val ROC-AUC   |   0.8962 | 0.9828 |<br>| Test ROC-AUC  |   0.8937 | 0.9830 |<br>| Test Accuracy |   82.10% | 94.39% |</pre><p>Per-class, test set (n = 33,004):</p><pre>| Class  | Precision | Recall |   F1 | Support |<br>| ------ | --------: | -----: | ---: | ------: |<br>| Normal |      0.95 |   0.96 | 0.95 |  19,637 |<br>| Tumor  |      0.94 |   0.92 | 0.93 |  13,367 |</pre><pre>[[18794   843]<br> [ 1008 12359]]</pre><p><strong>Tumor recall</strong> sits at 0.92, meaning roughly 8% of real tumor patches in this test set were missed. In a screening context that number, not accuracy, is the one that would matter most: a missed tumor patch is a worse failure mode than a false alarm on a normal one. Precision and recall stay close between the two classes, 0.95 and 0.94, which means the model isn’t quietly trading one kind of error for the other to inflate overall accuracy. That’s a real risk with imbalanced medical data, one this dataset’s near-even split happens to sidestep, but it’s worth checking explicitly rather than assuming.</p><h3>Part 8: The Docker Problem That Took Four Rebuilds to Fix</h3><p>This section is longer than the modeling section, on purpose. It cost more time.</p><p><strong>Attempt 1</strong> built cleanly, then crashed on load:</p><pre>TypeError: GlorotUniform.__init__() got an unexpected keyword argument &#39;input_axes&#39;</pre><p>The model file was saved by a newer <strong>Keras</strong> than the one Docker had installed. requirements.txt had tensorflow with no version pin, which meant the venv that trained the model and the container trying to load it could silently resolve two different releases.</p><p><strong>Fix attempt:</strong> pin the exact venv version.</p><pre>tensorflow==2.21.0</pre><p>Rebuilt. Same crash, byte for byte identical traceback. Pinning TensorFlow alone wasn’t enough. keras is a separate package that TensorFlow depends on but doesn&#39;t pin, so Docker could still land on a different Keras release than the venv had.</p><p><strong>Second fix:</strong> pin Keras too.</p><pre>tensorflow==2.21.0<br>keras==3.15.1</pre><p>Rebuild failed outright this time, with a cleaner error than before:</p><pre>ERROR: Could not find a version that satisfies the requirement keras==3.15.1<br>Requires-Python &gt;=3.11</pre><p>The Dockerfile was still on python:3.10-slim. Keras 3.15.1 requires 3.11+. The venv had already been upgraded to Python 3.11 earlier in the project specifically to get TensorFlow working locally. The Dockerfile had never been updated to match.</p><pre>FROM python:3.11-slim</pre><p>Rebuild. Install succeeded. Run succeeded.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D5bpsHyNqNAWyz6Israifw.png" /><figcaption><strong>Figure 6. Successful Docker evaluation run.</strong></figcaption></figure><p>Same numbers as the notebook, to four decimal places. Same confusion matrix. That match is the actual proof the fix worked, not just “no crash,” but bit for bit identical output to the environment where the model was trained. Three separate bugs, one shared root cause: nothing had ever pinned the container to match the machine that trained the model.</p><h3>Part 9: Project Structure</h3><pre>histopathologic-cancer-cnn/<br>├── data/<br>│   ├── README.md<br>│   ├── sample_submission.csv<br>│   └── train_labels.csv<br>├── models/<br>│   ├── color_hist_logreg_baseline.joblib<br>│   ├── histopath_cnn.keras<br>│   └── README.md<br>├── notebooks/<br>│   └── Histopathologic_Cancer_Detection_CNN.ipynb<br>├── outputs/<br>│   ├── class_distribution_real.png<br>│   ├── confusion_matrix_baseline_real.png<br>│   ├── confusion_matrix.png<br>│   ├── roc_curve.png<br>│   ├── sample_patches_real.png<br>│   └── training_curves.png<br>├── src/<br>│   ├── __init__.py<br>│   ├── baseline.py         # color-histogram + LogReg<br>│   ├── data_loader.py       # merges train_labels.csv with image filepaths<br>│   ├── dataset.py            # lazy tf.data pipeline, TIFF via PIL<br>│   ├── evaluate.py            # ROC-AUC, ROC curve, confusion matrix<br>│   ├── model.py                 # CNN and transfer-learning backbones<br>│   ├── predict.py                # single-image CLI prediction<br>│   ├── preprocessing.py           # splits, class weights, image I/O<br>│   └── train.py                    # trains baseline always, CNN if TF is importable<br>├── .dockerignore<br>├── .gitignore<br>├── config.py                # every path and hyperparameter, one place<br>├── docker-compose.yml<br>├── Dockerfile<br>├── LICENSE<br>├── README.md<br>└── requirements.txt</pre><p>src/train.py checks whether TensorFlow is importable before attempting to train the CNN. It trains the baseline regardless, and exits cleanly with a message rather than crashing if TensorFlow isn&#39;t installed yet. That single check made it safe to run the same command at every stage of this project, including before the environment was fully set up.</p><pre>python -m src.train<br>python -m src.evaluate<br>python -m src.predict --image_path data/train/&lt;some_id&gt;.tif</pre><pre>docker compose up --build<br>docker compose run histopathologic-cancer-cnn python -m src.evaluate</pre><h3>Key Formulas</h3><p>Binary Crossentropy</p><pre>Loss = -[y * log(p) + (1 - y) * log(1 - p)]</pre><p>Precision and Recall</p><pre>Precision = TP / (TP + FP)<br>Recall    = TP / (TP + FN)</pre><p><strong>ROC-AUC</strong>, in one line: the probability that the model ranks a randomly chosen tumor patch above a randomly chosen normal one. It’s threshold independent, which is exactly why it, not accuracy, is the competition’s actual scoring metric.</p><h3>What I Would Improve Next</h3><p><strong>Train on the full 154,017-image set.</strong> The single biggest lever left untouched. Everything reported here came from 13% of the available training data.</p><p><strong>Get a GPU into the loop.</strong> Colab, a cloud instance, or WSL2 with CUDA specifically. Any of these turns “full dataset” from an overnight problem into an afternoon one.</p><p><strong>Actually run the transfer-learning path.</strong> build_transfer_model (MobileNetV2, EfficientNetB0, ResNet50) is implemented and untested. Published results on this exact competition tend to favor transfer learning over a from-scratch CNN at this dataset size, worth checking directly rather than assuming.</p><p><strong>Add a fixed random seed.</strong> Every run in this project, the baseline and both CNN attempts, used whatever seed TensorFlow picked. Two runs of identical code could plausibly land at slightly different numbers, and right now there’s no way to tell how much of the 0.9830 is signal versus luck.</p><h3>Summary Table</h3><pre>| Concept                                | What It Did Here                                                                                   |<br>| -------------------------------------- | -------------------------------------------------------------------------------------------------- |<br>| Color-histogram baseline               | Real floor (82.10%), not a strawman. The CNN had to clear it.                                      |<br>| BatchNorm on every conv block          | Kept the second training run stable where the first one wasn’t.                                    |<br>| ReduceLROnPlateau                      | Helped recover from the epoch 3-to-5 collapsein the first run.                                     |<br>| EarlyStopping + `restore_best_weights` | Stopped at epoch 14 and automatically kept epoch 9’s weights.                                      |<br>| 20k-image training subsample           | Kept training feasible under the CPU-only constraint. The test set stayed full-size and untouched. |<br>| `tf.py_function` + PIL                 | Enabled TIFF decoding because `tf.io.decode_image` couldn’t decode them.                           |<br>| Pinned TensorFlow and Keras versions   | Fixed the dependency mismatch behind four Docker rebuilds.                                         |</pre><h3>What I Actually Learned</h3><p><strong>A baseline is a claim, not a formality.</strong> 82% from a color histogram means the CNN’s 94% is doing real work. That comparison is the whole point of building the baseline first, not an afterthought tacked on after the fact.</p><p><strong>“It crashed the same way twice” is informative, not just frustrating.</strong> The identical traceback after the first version pin was the signal that TensorFlow alone wasn’t the fix. It pointed straight at Keras as the actual unpinned dependency, and saved a third blind guess.</p><p><strong>Docker version pins your dependencies. It doesn’t version pin your Dockerfile.</strong> Pinning requirements.txt fixed half the problem. The base image&#39;s Python version was a completely separate thing that had drifted out of sync with the venv, and nothing about pinning packages would have caught that on its own.</p><h3>Final Thoughts</h3><p>None of these problems are exotic. Missing version pins and a Dockerfile that drifted out of sync with the venv are common, not rare, and they rarely get written down because “<strong>then I fixed a dependency version</strong>” doesn’t feel like the interesting part of a project.</p><p>It was, this time. The modeling decisions, GlobalAveragePooling over Flatten, graduated learning-rate reduction, a real baseline, got the<strong> CNN to 0.9830.</strong> Getting that same <strong>0.9830</strong> to reproduce inside a container took longer, and taught more about how these projects actually fail once they leave a single laptop.</p><p><strong>Full project:</strong> <a href="https://github.com/Subinthapa2092/histopathologic-cancer-detection">github.com/Subinthapa2092/histopathologic-cancer-detection</a></p><p>Follow me for more on Python, deep learning, and computer vision.</p><p><strong>LinkedIn:</strong> <a href="https://www.linkedin.com/in/subinthapa">Subin Thapa</a></p><p><strong>GitHub:</strong> <a href="https://github.com/Subinthapa2092">Subinthapa2092</a></p><p><strong>YouTube:</strong> <a href="https://www.youtube.com/@codingwithsubin">Coding With Subin</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6802a6db707f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SQL Server From the Ground Up: A Practical Guide to T-SQL, Joins, CTEs, and Views]]></title>
            <link>https://medium.com/@subinthapa2092/sql-server-from-the-ground-up-a-practical-guide-to-t-sql-joins-ctes-and-views-e5070eddc31d?source=rss-9f567b128564------2</link>
            <guid isPermaLink="false">https://medium.com/p/e5070eddc31d</guid>
            <category><![CDATA[sql-server]]></category>
            <category><![CDATA[data-analytics]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[t-sql]]></category>
            <category><![CDATA[sql]]></category>
            <dc:creator><![CDATA[Subinthapa]]></dc:creator>
            <pubDate>Sat, 12 Sep 2026 10:34:21 GMT</pubDate>
            <atom:updated>2026-09-12T10:34:21.555Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8G5Za3DFMMRpj-4KUVLjDw.png" /></figure><h4>From CREATE TABLE to Practical Analytical Queries, Built on One Consistent Database</h4><p>Every <strong>SQL tutorial</strong> teaches SELECT and WHERE, then jumps into a new example database for every new topic. By the time you reach <strong>JOINs</strong>, you have forgotten the schema from the aggregation section, and by the time you reach <strong>Views</strong>, you are learning another database from scratch.</p><p>This guide takes a different approach. <strong>One database. Three tables: Customers, Products, and Orders.</strong> The schema is built once at the beginning and reused throughout the entire guide, from basic <strong>SELECT statements</strong> to <strong>JOINs, aggregations, subqueries, CTEs, and Views</strong>. By the end, you will not just have seen dozens of isolated code snippets. You will understand how a <strong>realistic relational schema</strong> supports the analytical workflow a data professional actually uses.</p><p>The guide covers the essential <strong>SQL Server and T-SQL foundations</strong> needed to work confidently with data: <strong>relational database fundamentals, filtering, sorting, aggregation, grouping, JOINs, conditional logic, subqueries, CTEs, and Views</strong>. Each concept builds on the same database, making it easier to understand not only how individual SQL statements work, but also how they connect together to solve real-world data problems.</p><h3>Part 1: SQL Server and Relational Database Fundamentals</h3><h3>What Is SQL, and What Is T-SQL</h3><p><strong>SQL</strong>, Structured Query Language, is the language used to communicate with <strong>relational databases</strong>. It lets you create databases, create tables, insert and query data, and manage who can access what.</p><p><strong>SQL Server</strong> is Microsoft’s relational database management system, and it speaks <strong>T-SQL</strong>, Transact-SQL, Microsoft’s extension of standard SQL. T-SQL adds procedural features like variables, control-of-flow statements, and error handling on top of the SQL standard, but the vast majority of what you write day to day, SELECT, JOIN, GROUP BY, is standard SQL that works the same way across most relational databases.</p><h3>RDBMS, Database, and Schema</h3><p>A <strong>RDBMS</strong>, Relational Database Management System, is the software that stores data in tables and enforces the relationships between them. SQL Server, PostgreSQL, and MySQL are all RDBMS products.</p><p>A <strong>database</strong> is a named container for related tables, views, and other objects. A <strong>schema</strong> is a logical grouping inside a database, a way of organizing tables into namespaces, commonly used to separate concerns like Sales, HR, or Inventory inside a single database.</p><pre>CREATE DATABASE CommerceDB;<br>GO<br><br><br>USE CommerceDB;<br>GO<br><br>CREATE SCHEMA Sales;<br>GO</pre><p>GO is not part of SQL itself. It is a batch separator recognized by SQL Server client tools, telling them to send everything above it to the server as one batch before moving on.</p><h3>Tables, Rows, and Columns</h3><p>A <strong>table</strong> stores data in rows and columns. Each <strong>column</strong> holds one type of value, a name, a date, a price. Each <strong>row</strong> is one record, one customer, one order, one product.</p><pre>Customers<br>CustomerID   FirstName   LastName    City         Country<br>1            Ram         Shrestha    Kathmandu    Nepal<br>2            Sita        Gurung      Pokhara      Nepal</pre><p>Here, CustomerID, FirstName, LastName, City, and Country are columns. The row 1, Ram, Shrestha, Kathmandu, Nepal is a single record.</p><h3>Primary Keys and Foreign Keys</h3><p>A <strong>primary key</strong> uniquely identifies every row in a table. No two rows can share one, and it cannot be NULL. A <strong>foreign key</strong> is a column in one table that references the primary key of another table, and this is exactly what creates a relationship between tables.</p><pre>Customers.CustomerID   -&gt;  primary key of Customers<br>Orders.CustomerID      -&gt;  foreign key, references Customers.CustomerID</pre><p>An order cannot exist without pointing to a real, existing customer. That is the relationship a foreign key enforces.</p><h3>Common Data Types</h3><pre>INT              whole numbers                 1, 42, -7<br>VARCHAR(n)       variable length text, up to n characters<br>DECIMAL(p, s)    exact decimal numbers, p total digits, s after the decimal<br>DATE             calendar date, no time component<br>BIT              0 or 1, used for true/false flags<br>NULL             not a data type, a marker for missing or unknown values</pre><p>DECIMAL(10,2) is the standard choice for money: 10 total digits, 2 of them after the decimal point, giving you exact values like 199.99 with no floating point rounding error.</p><h3>Building the Database Used Throughout This Guide</h3><p>Three tables, used consistently in every part of this article. A customer places orders. Each order is for one product. This is as close to a real, small e-commerce schema as a teaching example gets.</p><pre>CREATE TABLE Customers (<br>    CustomerID   INT PRIMARY KEY,<br>    FirstName    VARCHAR(50)   NOT NULL,<br>    LastName     VARCHAR(50)   NOT NULL,<br>    City         VARCHAR(50),<br>    Country      VARCHAR(50),<br>    Email        VARCHAR(100),<br>    Phone        VARCHAR(20)   NULL,<br>    SignupDate   DATE<br>);<br><br><br>CREATE TABLE Products (<br>    ProductID      INT PRIMARY KEY,<br>    ProductName    VARCHAR(100)   NOT NULL,<br>    Category       VARCHAR(50),<br>    Price          DECIMAL(10,2)  NOT NULL,<br>    Discount       DECIMAL(4,2)   NULL,<br>    StockQuantity  INT<br>);<br>CREATE TABLE Orders (<br>    OrderID       INT PRIMARY KEY,<br>    CustomerID    INT REFERENCES Customers(CustomerID),<br>    ProductID     INT REFERENCES Products(ProductID),<br>    OrderDate     DATE,<br>    Quantity      INT,<br>    TotalAmount   DECIMAL(10,2)<br>);</pre><p>Notice Orders.CustomerID and Orders.ProductID are both foreign keys. Products.Discount and Customers.Phone are deliberately left nullable, they will matter directly in Part 8.</p><h3>Inserting the Data</h3><pre>INSERT INTO Customers (CustomerID, FirstName, LastName, City, Country, Email, Phone, SignupDate)<br>VALUES<br>    (1, &#39;Ram&#39;,   &#39;Shrestha&#39;, &#39;Kathmandu&#39;, &#39;Nepal&#39;, &#39;ram@example.com&#39;,   &#39;9800000001&#39;, &#39;2023-01-15&#39;),<br>    (2, &#39;Sita&#39;,  &#39;Gurung&#39;,   &#39;Pokhara&#39;,   &#39;Nepal&#39;, &#39;sita@example.com&#39;,  &#39;9800000002&#39;, &#39;2023-02-20&#39;),<br>    (3, &#39;John&#39;,  &#39;Smith&#39;,    &#39;London&#39;,    &#39;UK&#39;,    &#39;john@example.com&#39;,  NULL,         &#39;2023-03-05&#39;),<br>    (4, &#39;Maria&#39;, &#39;Garcia&#39;,   &#39;Madrid&#39;,    &#39;Spain&#39;, &#39;maria@example.com&#39;, &#39;341234567&#39;,  &#39;2023-04-10&#39;),<br>    (5, &#39;Amit&#39;,  &#39;Verma&#39;,    &#39;Delhi&#39;,     &#39;India&#39;, &#39;amit@example.com&#39;,  &#39;9910000005&#39;, &#39;2023-05-22&#39;),<br>    (6, &#39;Laxmi&#39;, &#39;Thapa&#39;,    &#39;Kathmandu&#39;, &#39;Nepal&#39;, &#39;laxmi@example.com&#39;, NULL,         &#39;2023-06-30&#39;);<br><br>INSERT INTO Products (ProductID, ProductName, Category, Price, Discount, StockQuantity)<br>VALUES<br>    (101, &#39;Wireless Mouse&#39;,      &#39;Electronics&#39;, 15.99,  NULL,  200),<br>    (102, &#39;Mechanical Keyboard&#39;, &#39;Electronics&#39;, 45.50,  10.00, 120),<br>    (103, &#39;Office Chair&#39;,        &#39;Furniture&#39;,   89.99,  NULL,  40),<br>    (104, &#39;Standing Desk&#39;,       &#39;Furniture&#39;,   199.99, 15.00, 15),<br>    (105, &#39;Notebook Pack&#39;,       &#39;Stationery&#39;,  5.49,   0.00,  500),<br>    (106, &#39;Desk Lamp&#39;,           &#39;Furniture&#39;,   24.99,  NULL,  60),<br>    (107, &#39;USB-C Hub&#39;,           &#39;Electronics&#39;, 29.99,  5.00,  0);<br><br>INSERT INTO Orders (OrderID, CustomerID, ProductID, OrderDate, Quantity, TotalAmount)<br>VALUES<br>    (1001, 1, 101, &#39;2024-01-10&#39;, 2, 31.98),<br>    (1002, 1, 103, &#39;2024-01-15&#39;, 1, 89.99),<br>    (1003, 2, 102, &#39;2024-01-18&#39;, 1, 45.50),<br>    (1004, 3, 104, &#39;2024-02-01&#39;, 1, 199.99),<br>    (1005, 3, 105, &#39;2024-02-01&#39;, 3, 16.47),<br>    (1006, 4, 106, &#39;2024-02-10&#39;, 2, 49.98),<br>    (1007, 5, 101, &#39;2024-02-14&#39;, 1, 15.99),<br>    (1008, 5, 102, &#39;2024-03-01&#39;, 1, 45.50),<br>    (1009, 2, 105, &#39;2024-03-05&#39;, 5, 27.45),<br>    (1010, 1, 106, &#39;2024-03-20&#39;, 1, 24.99);</pre><p>Look closely and two details are already planted on purpose. Customer 6, Laxmi, has no orders at all. Product 107, the USB-C Hub, has zero stock and has never been ordered. Both come back deliberately in Part 7.</p><h3>Basic SELECT</h3><pre>SELECT * FROM Customers;<br>SELECT FirstName, LastName, City FROM Customers;</pre><p>SELECT * returns every column. Naming columns explicitly is almost always the better habit, it is clearer to read, and it will not silently change output if someone adds a column to the table later.</p><h3>Part 2: SELECT and Filtering</h3><h3>WHERE and Comparison Operators</h3><pre>SELECT ProductName, Price<br>FROM Products<br>WHERE Price &gt; 30;</pre><pre>ProductName            Price<br>Mechanical Keyboard    45.50<br>Office Chair           89.99<br>Standing Desk          199.99</pre><p>Standard comparison operators: =, &lt;&gt; or != for not equal, &lt;, &gt;, &lt;=, &gt;=.</p><h3>AND, OR, NOT</h3><pre>SELECT ProductName, Category, Price<br>FROM Products<br>WHERE Category = &#39;Electronics&#39; AND Price &lt; 30;</pre><pre>ProductName       Category       Price<br>Wireless Mouse    Electronics    15.99</pre><p>AND requires both conditions to be true. OR requires at least one. NOT negates a condition entirely.</p><pre>SELECT ProductName<br>FROM Products<br>WHERE NOT Category = &#39;Furniture&#39;;</pre><p>A common mistake is chaining AND conditions on the same column expecting an OR-like result:</p><pre>/* Wrong: no row can have two different categories at once, returns nothing */<br>SELECT * FROM Products<br>WHERE Category = &#39;Electronics&#39; AND Category = &#39;Furniture&#39;;<br><br>/* Corrected */<br>SELECT * FROM Products<br>WHERE Category = &#39;Electronics&#39; OR Category = &#39;Furniture&#39;;</pre><h3>IN</h3><pre>SELECT ProductName, Category<br>FROM Products<br>WHERE Category IN (&#39;Electronics&#39;, &#39;Stationery&#39;);</pre><p>IN is shorthand for a chain of OR conditions on the same column, and it reads far more cleanly once you are checking against three or more values.</p><h3>BETWEEN</h3><pre>SELECT ProductName, Price<br>FROM Products<br>WHERE Price BETWEEN 20 AND 100;</pre><p>BETWEEN is inclusive on both ends, so a price of exactly 20 or exactly 100 would both be included.</p><h3>LIKE</h3><pre>SELECT FirstName, LastName<br>FROM Customers<br>WHERE LastName LIKE &#39;G%&#39;;</pre><pre>FirstName   LastName<br>Sita        Gurung<br>Maria       Garcia</pre><p>% matches any number of characters, _ matches exactly one. LIKE &#39;%a&#39; finds anything ending in &quot;a&quot;. LIKE &#39;_it_&#39; finds four-letter strings with &quot;i&quot; and &quot;t&quot; in the middle positions.</p><h3>IS NULL and IS NOT NULL</h3><pre>SELECT FirstName, LastName, Phone<br>FROM Customers<br>WHERE Phone IS NULL;</pre><pre>FirstName   LastName   Phone<br>John        Smith      NULL<br>Laxmi       Thapa      NULL</pre><p><strong>This is one of the most common beginner mistakes in SQL.</strong> NULL represents an unknown value, and unknown compared to anything, including another NULL, is itself unknown, not true or false.</p><pre>/* Wrong: this will never match a NULL phone, even though it looks like it should */<br>SELECT * FROM Customers WHERE Phone = NULL;<br>/* Correct */<br>SELECT * FROM Customers WHERE Phone IS NULL;</pre><h3>Part 3: Sorting and Result Control</h3><h3>ORDER BY, ASC and DESC</h3><pre>SELECT ProductName, Price<br>FROM Products<br>ORDER BY Price DESC;</pre><pre>ProductName            Price<br>Standing Desk          199.99<br>Office Chair           89.99<br>Mechanical Keyboard    45.50<br>USB-C Hub              29.99<br>Desk Lamp              24.99<br>Wireless Mouse         15.99<br>Notebook Pack          5.49</pre><p>ASC, ascending, is the default if you omit it. DESC reverses the order.</p><h3>DISTINCT</h3><pre>SELECT DISTINCT Category<br>FROM Products;</pre><pre>Category<br>Electronics<br>Furniture<br>Stationery</pre><p>DISTINCT removes duplicate rows from the result. Without it, this query would return Electronics three times, once per matching product.</p><h3>TOP</h3><pre>SELECT TOP 3 ProductName, Price<br>FROM Products<br>ORDER BY Price DESC;</pre><p>TOP limits how many rows come back. It is almost always paired with ORDER BY, otherwise &quot;top 3&quot; has no defined meaning since table row order is not guaranteed.</p><h3>Column Aliases</h3><pre>SELECT<br>    ProductName AS Product,<br>    Price * 1.13 AS PriceWithTax<br>FROM Products;</pre><p>AS renames a column in the output only, it does not touch the underlying table. Aliases make computed columns and joined result sets far more readable, especially once two tables both have a column called Name or ID.</p><h3>Part 4: Aggregate Functions</h3><p>Aggregate functions collapse many rows into a single summary value.</p><pre>COUNT   how many rows<br>SUM     total of a numeric column<br>AVG     average of a numeric column<br>MIN     smallest value<br>MAX     largest value</pre><h3>COUNT</h3><pre>SELECT COUNT(*) AS TotalOrders<br>FROM Orders;</pre><pre>TotalOrders<br>10</pre><p>COUNT(*) counts rows regardless of NULLs. COUNT(ColumnName) counts only rows where that specific column is not NULL, which matters the moment a column is nullable.</p><pre>SELECT COUNT(Phone) AS CustomersWithPhone<br>FROM Customers;</pre><pre>CustomersWithPhone<br>4</pre><p>Six customers total, but only four have a phone number on file, so COUNT(Phone) correctly excludes the two NULLs.</p><h3>SUM and AVG</h3><pre>SELECT<br>    SUM(TotalAmount) AS TotalRevenue,<br>    AVG(TotalAmount) AS AverageOrderValue<br>FROM Orders;</pre><pre>TotalRevenue   AverageOrderValue<br>547.84         54.784</pre><h3>MIN and MAX</h3><pre>SELECT<br>    MIN(Price) AS CheapestProduct,<br>    MAX(Price) AS MostExpensiveProduct<br>FROM Products;</pre><pre>CheapestProduct   MostExpensiveProduct<br>5.49              199.99</pre><p>A practical business framing ties all five together in one query: <strong>“Give me a one-line summary of our order history.”</strong></p><pre>SELECT<br>    COUNT(*)            AS TotalOrders,<br>    SUM(TotalAmount)     AS TotalRevenue,<br>    AVG(TotalAmount)     AS AvgOrderValue,<br>    MIN(TotalAmount)     AS SmallestOrder,<br>    MAX(TotalAmount)     AS LargestOrder<br>FROM Orders;</pre><h3>Part 5: GROUP BY and HAVING</h3><p>Aggregate functions get far more useful once they are computed per group instead of across the entire table.</p><h3>GROUP BY</h3><pre>SELECT<br>    CustomerID,<br>    COUNT(*)         AS OrderCount,<br>    SUM(TotalAmount) AS TotalSpent<br>FROM Orders<br>GROUP BY CustomerID<br>ORDER BY TotalSpent DESC;</pre><pre>CustomerID   OrderCount   TotalSpent<br>3            2            216.46<br>1            3            146.96<br>2            2             72.95<br>5            2             61.49<br>4            1             49.98</pre><p>Every column in the SELECT list that is not wrapped in an aggregate function must appear in the GROUP BY clause. SQL Server will raise an error otherwise, since it would not know which single value to display for an ungrouped column when multiple rows are being collapsed into one.</p><h3>HAVING</h3><p><strong>WHERE filters rows before grouping happens. HAVING filters groups after aggregation has already happened.</strong> This is the single most important distinction in this entire section.</p><pre>SELECT<br>    CustomerID,<br>    SUM(TotalAmount) AS TotalSpent<br>FROM Orders<br>GROUP BY CustomerID<br>HAVING SUM(TotalAmount) &gt; 100;</pre><pre>CustomerID   TotalSpent<br>1            146.96<br>3            216.46</pre><p>A common mistake is trying to filter on an aggregate using WHERE:</p><pre>/* Wrong: WHERE cannot see the result of SUM, since aggregation has not happened yet */<br>SELECT CustomerID, SUM(TotalAmount) AS TotalSpent<br>FROM Orders<br>WHERE SUM(TotalAmount) &gt; 100<br>GROUP BY CustomerID;<br><br>/* Corrected */<br>SELECT CustomerID, SUM(TotalAmount) AS TotalSpent<br>FROM Orders<br>GROUP BY CustomerID<br>HAVING SUM(TotalAmount) &gt; 100;</pre><h3>A Real Analytical Query: Revenue by Product Category</h3><p>This requires a join, jumping slightly ahead to Part 6, but it is worth seeing here as motivation for why GROUP BY matters in practice.</p><pre>SELECT<br>    p.Category,<br>    COUNT(o.OrderID)   AS OrdersInCategory,<br>    SUM(o.TotalAmount) AS RevenueInCategory<br>FROM Orders o<br>JOIN Products p ON o.ProductID = p.ProductID<br>GROUP BY p.Category<br>ORDER BY RevenueInCategory DESC;</pre><pre>Category       OrdersInCategory   RevenueInCategory<br>Furniture      4                  364.95<br>Electronics    4                  138.97<br>Stationery     2                  43.92</pre><h3>Part 6: SQL JOINs</h3><p>A <strong>join</strong> combines rows from two or more tables based on a related column, typically a foreign key pointing back to a primary key.</p><h3>INNER JOIN</h3><p>Returns only rows that have a match in both tables.</p><pre>SELECT<br>    c.FirstName,<br>    c.LastName,<br>    o.OrderID,<br>    o.TotalAmount<br>FROM Customers c<br>INNER JOIN Orders o ON c.CustomerID = o.CustomerID;</pre><pre>FirstName   LastName   OrderID   TotalAmount<br>Ram         Shrestha   1001      31.98<br>Ram         Shrestha   1002      89.99<br>Sita        Gurung     1003      45.50<br>John        Smith      1004      199.99<br>...</pre><p>Laxmi does not appear here at all. She has no orders, and INNER JOIN only returns rows present on both sides.</p><h3>LEFT JOIN</h3><p>Returns every row from the left table, and matching rows from the right table where they exist. Where there is no match, the right side’s columns come back as NULL.</p><pre>SELECT<br>    c.FirstName,<br>    c.LastName,<br>    o.OrderID<br>FROM Customers c<br>LEFT JOIN Orders o ON c.CustomerID = o.CustomerID;</pre><pre>FirstName   LastName   OrderID<br>Ram         Shrestha   1001<br>Ram         Shrestha   1002<br>...<br>Laxmi       Thapa      NULL</pre><p>Now Laxmi shows up, with NULL in place of an order, because <strong>LEFT JOIN</strong> keeps every row from Customers no matter what.</p><h3>RIGHT JOIN</h3><p>The mirror image of LEFT JOIN, every row from the right table is kept, with NULLs on the left where there is no match. In practice, most people write every join as a LEFT JOIN and simply swap which table comes first, so <strong>RIGHT JOIN </strong>appears far less often in real codebases, but it is worth recognizing.</p><pre>SELECT<br>    p.ProductName,<br>    o.OrderID<br>FROM Orders o<br>RIGHT JOIN Products p ON o.ProductID = p.ProductID;</pre><p>This keeps every product, including the USB-C Hub that has never been ordered, with NULL in the OrderID column for it.</p><h3>FULL OUTER JOIN</h3><p>Keeps every row from both tables, matched where possible, NULL on whichever side has no match.</p><pre>SELECT<br>    c.FirstName,<br>    o.OrderID<br>FROM Customers c<br>FULL OUTER JOIN Orders o ON c.CustomerID = o.CustomerID;</pre><p>This is the join to reach for when you need a complete picture of both sides at once, every customer and every order, matched wherever a relationship exists.</p><pre>INNER JOIN        only matching rows on both sides<br>LEFT JOIN         everything on the left, matched where possible<br>RIGHT JOIN        everything on the right, matched where possible<br>FULL OUTER JOIN   everything on both sides, matched where possible</pre><h3>Join Conditions and Multi-Table Relationships</h3><p>A join condition tells SQL Server which columns actually relate the two tables. Get it wrong, and you get a <strong>Cartesian product</strong>, every row in the first table paired with every row in the second, almost never what you actually want.</p><pre>/* Wrong: missing join condition, produces 6 x 7 = 42 nonsensical rows */<br>SELECT c.FirstName, p.ProductName<br>FROM Customers c, Products p;<br><br>/* Corrected, using the Orders table as the bridge between them */<br>SELECT c.FirstName, p.ProductName<br>FROM Customers c<br>JOIN Orders o ON c.CustomerID = o.CustomerID<br>JOIN Products p ON o.ProductID = p.ProductID;</pre><p>This last query is the real shape of a three-table relationship: Customers and Products do not connect directly, they connect through Orders.</p><h3>Part 7: Advanced JOIN Practice</h3><h3>Customer Analysis: Full Purchase History</h3><pre>SELECT<br>    c.FirstName,<br>    c.LastName,<br>    p.ProductName,<br>    o.OrderDate,<br>    o.TotalAmount<br>FROM Customers c<br>JOIN Orders o ON c.CustomerID = o.CustomerID<br>JOIN Products p ON o.ProductID = p.ProductID<br>ORDER BY c.CustomerID, o.OrderDate;</pre><h3>Product Analysis: Which Products Actually Sell</h3><pre>SELECT<br>    p.ProductName,<br>    COUNT(o.OrderID)   AS TimesOrdered,<br>    SUM(o.Quantity)     AS UnitsSold,<br>    SUM(o.TotalAmount)  AS Revenue<br>FROM Products p<br>JOIN Orders o ON p.ProductID = o.ProductID<br>GROUP BY p.ProductName<br>ORDER BY Revenue DESC;</pre><h3>Sales Analysis: Revenue by Country</h3><pre>SELECT<br>    c.Country,<br>    COUNT(o.OrderID)   AS OrderCount,<br>    SUM(o.TotalAmount)  AS TotalRevenue<br>FROM Customers c<br>JOIN Orders o ON c.CustomerID = o.CustomerID<br>GROUP BY c.Country<br>ORDER BY TotalRevenue DESC;</pre><h3>Finding Customers With No Orders</h3><p>This is the classic <strong>LEFT JOIN plus IS NULL</strong> pattern, and it is one of the most practically useful query shapes in all of SQL.</p><pre>SELECT c.FirstName, c.LastName<br>FROM Customers c<br>LEFT JOIN Orders o ON c.CustomerID = o.CustomerID<br>WHERE o.OrderID IS NULL;</pre><pre>FirstName   LastName<br>Laxmi       Thapa</pre><p>Read this pattern as: keep every customer, attach their orders if any exist, then keep only the ones where nothing attached.</p><h3>Finding Products That Have Never Been Ordered</h3><p>The exact same pattern, applied to Products instead of Customers.</p><pre>SELECT p.ProductName, p.StockQuantity<br>FROM Products p<br>LEFT JOIN Orders o ON p.ProductID = o.ProductID<br>WHERE o.OrderID IS NULL;</pre><pre>ProductName   StockQuantity<br>USB-C Hub     0</pre><p>Notice this single query would flag the USB-C Hub for two completely different business reasons at once, zero stock and zero sales, which is exactly the kind of signal a real inventory report needs to surface.</p><h3>Part 8: CASE and NULL Handling</h3><h3>CASE WHEN</h3><p>CASE adds conditional logic directly inside a query, evaluated row by row.</p><pre>SELECT<br>    OrderID,<br>    TotalAmount,<br>    CASE<br>        WHEN TotalAmount &gt;= 100 THEN &#39;Large Order&#39;<br>        WHEN TotalAmount &gt;= 40  THEN &#39;Medium Order&#39;<br>        ELSE &#39;Small Order&#39;<br>    END AS OrderSize<br>FROM Orders;</pre><pre>OrderID   TotalAmount   OrderSize<br>1001      31.98         Small Order<br>1002      89.99         Medium Order<br>1004      199.99        Large Order<br>...</pre><p>Conditions are checked top to bottom, and the first one that matches wins, the rest are never evaluated. ELSE is the catch-all for anything that matched none of the conditions above it, and omitting it simply returns NULL for those rows instead of an error.</p><h3>ISNULL and COALESCE</h3><pre>SELECT<br>    ProductName,<br>    Discount,<br>    ISNULL(Discount, 0) AS DiscountOrZero<br>FROM Products;</pre><pre>ProductName            Discount   DiscountOrZero<br>Wireless Mouse         NULL       0.00<br>Mechanical Keyboard    10.00      10.00<br>Office Chair           NULL       0.00</pre><p>ISNULL(a, b) returns a if it is not NULL, otherwise b. It is SQL Server specific. COALESCE does the same job but is standard SQL, works across database platforms, and accepts more than two arguments, returning the first one that is not NULL.</p><pre>SELECT<br>    FirstName,<br>    COALESCE(Phone, Email, &#39;No contact on file&#39;) AS BestContact<br>FROM Customers;</pre><pre>FirstName   BestContact<br>Ram         9800000001<br>John        john@example.com</pre><p>For John, Phone is NULL, so COALESCE falls through to Email. If both were NULL, it would fall through to the literal string.</p><h3>NULLIF</h3><p>NULLIF(a, b) returns NULL if a equals b, otherwise it returns a. It is the reverse instinct of the other two functions, deliberately turning a specific value into NULL rather than replacing NULL with something else.</p><pre>SELECT<br>    ProductName,<br>    Discount,<br>    NULLIF(Discount, 0) AS TreatZeroAsNoDiscount<br>FROM Products;</pre><pre>ProductName     Discount   TreatZeroAsNoDiscount<br>Notebook Pack   0.00       NULL</pre><p>This is genuinely useful for data cleaning, if an explicit 0 and a missing value should be treated identically downstream, NULLIF collapses them into the same NULL representation.</p><h3>Part 9: Subqueries</h3><p>A <strong>subquery</strong> is a query nested inside another query, used wherever a single value, a list of values, or a filtered result set is needed as an input.</p><h3>Scalar Subqueries</h3><p>Returns exactly one value, usable anywhere a single value is expected.</p><pre>SELECT ProductName, Price<br>FROM Products<br>WHERE Price &gt; (SELECT AVG(Price) FROM Products);</pre><pre>ProductName            Price<br>Mechanical Keyboard    45.50<br>Office Chair           89.99<br>Standing Desk          199.99</pre><h3>Subqueries With IN</h3><pre>SELECT FirstName, LastName<br>FROM Customers<br>WHERE CustomerID IN (<br>    SELECT CustomerID FROM Orders WHERE TotalAmount &gt; 100<br>);</pre><p>Returns every customer who placed at least one order over 100, without needing a JOIN or GROUP BY at all.</p><h3>EXISTS</h3><p>EXISTS checks whether a subquery returns at least one matching row. It is especially useful when you only need to know whether a related record exists. Depending on the query, indexes, and execution plan, SQL Server may optimize EXISTS and IN differently, so you should not assume one is always faster.</p><pre>SELECT c.FirstName, c.LastName<br>FROM Customers c<br>WHERE EXISTS (<br>    SELECT 1 FROM Orders o WHERE o.CustomerID = c.CustomerID<br>);</pre><p>This returns every customer who has placed at least one order, the direct opposite of the LEFT JOIN plus IS NULL pattern from Part 7.</p><h3>Correlated Subqueries</h3><p>A <strong>correlated subquery</strong> references a column from the outer query, meaning it cannot run on its own, it is evaluated once per row of the outer query.</p><pre>SELECT<br>    o.OrderID,<br>    o.CustomerID,<br>    o.TotalAmount<br>FROM Orders o<br>WHERE o.TotalAmount &gt; (<br>    SELECT AVG(o2.TotalAmount)<br>    FROM Orders o2<br>    WHERE o2.CustomerID = o.CustomerID<br>);</pre><p>This finds every order that is larger than that specific customer’s own average order, a genuinely different question than comparing against the global average across all customers.</p><h3>Subqueries With Aggregate Functions</h3><pre>SELECT ProductName<br>FROM Products<br>WHERE ProductID = (<br>    SELECT TOP 1 ProductID<br>    FROM Orders<br>    GROUP BY ProductID<br>    ORDER BY SUM(Quantity) DESC<br>);</pre><p>This answers a genuinely useful business question in one query: which single product has sold the most total units.</p><h3>Part 10: CTEs</h3><p>A <strong>CTE</strong>, Common Table Expression, is a named, temporary result set defined with WITH, that exists only for the duration of the query that follows it.</p><h3>A Single CTE</h3><pre>WITH CustomerTotals AS (<br>    SELECT<br>        CustomerID,<br>        SUM(TotalAmount) AS TotalSpent<br>    FROM Orders<br>    GROUP BY CustomerID<br>)<br>SELECT c.FirstName, c.LastName, ct.TotalSpent<br>FROM Customers c<br>JOIN CustomerTotals ct ON c.CustomerID = ct.CustomerID<br>WHERE ct.TotalSpent &gt; 50;</pre><p>Read this top to bottom like a short story: first define CustomerTotals, then use it exactly like a regular table in the query underneath.</p><h3>Multiple CTEs</h3><pre>WITH CustomerTotals AS (<br>    SELECT CustomerID, SUM(TotalAmount) AS TotalSpent<br>    FROM Orders<br>    GROUP BY CustomerID<br>),<br>HighValueCustomers AS (<br>    SELECT CustomerID<br>    FROM CustomerTotals<br>    WHERE TotalSpent &gt; 50<br>)<br>SELECT c.FirstName, c.LastName<br>FROM Customers c<br>JOIN HighValueCustomers hvc ON c.CustomerID = hvc.CustomerID;</pre><p>Multiple CTEs are separated by commas, and each one can reference any CTE defined above it, letting you build a multi-step calculation as a readable sequence instead of one deeply nested query.</p><h3>CTE vs Subquery</h3><p>The two can often solve the exact same problem. The difference is almost entirely about readability and reuse.</p><pre>/* As a subquery, the logic is nested and read inside-out */<br>SELECT c.FirstName, c.LastName<br>FROM Customers c<br>JOIN (<br>    SELECT CustomerID, SUM(TotalAmount) AS TotalSpent<br>    FROM Orders<br>    GROUP BY CustomerID<br>) ct ON c.CustomerID = ct.CustomerID<br>WHERE ct.TotalSpent &gt; 50;<br><br><br>/* As a CTE, the same logic reads top to bottom */<br>WITH CustomerTotals AS (<br>    SELECT CustomerID, SUM(TotalAmount) AS TotalSpent<br>    FROM Orders<br>    GROUP BY CustomerID<br>)<br>SELECT c.FirstName, c.LastName<br>FROM Customers c<br>JOIN CustomerTotals ct ON c.CustomerID = ct.CustomerID<br>WHERE ct.TotalSpent &gt; 50;</pre><p>Both return an identical result. A CTE can also be referenced more than once in the same query, something a subquery cannot do without repeating the entire nested block. This is exactly why CTEs improve readability once queries grow past two or three steps of logic, each step gets its own name instead of another layer of parentheses.</p><h3>Part 11: SQL Server Views</h3><h3>What Is a View</h3><p>A <strong>view</strong> is a saved, named SELECT query that behaves like a virtual table. It does not store data itself, every time you query a view, SQL Server runs the underlying query fresh.</p><h3>Why Views Are Useful</h3><p>Views hide complexity behind a simple name, standardize a calculation so every report uses the same definition of “revenue” or “active customer,” and can restrict which columns or rows a given user is allowed to see.</p><h3>CREATE VIEW</h3><pre>CREATE VIEW CustomerOrderSummary AS<br>SELECT<br>    c.CustomerID,<br>    c.FirstName,<br>    c.LastName,<br>    COUNT(o.OrderID)   AS OrderCount,<br>    SUM(o.TotalAmount) AS TotalSpent<br>FROM Customers c<br>LEFT JOIN Orders o ON c.CustomerID = o.CustomerID<br>GROUP BY c.CustomerID, c.FirstName, c.LastName;</pre><h3>Querying a View</h3><pre>SELECT * FROM CustomerOrderSummary<br>WHERE TotalSpent &gt; 50<br>ORDER BY TotalSpent DESC;</pre><p>From the outside, this looks exactly like querying an ordinary table. The LEFT JOIN, the GROUP BY, all of it stays hidden inside the view&#39;s own definition.</p><h3>CREATE OR ALTER VIEW</h3><pre>CREATE OR ALTER VIEW CustomerOrderSummary AS<br>SELECT<br>    c.CustomerID,<br>    c.FirstName,<br>    c.LastName,<br>    COUNT(o.OrderID)    AS OrderCount,<br>    SUM(o.TotalAmount)  AS TotalSpent,<br>    AVG(o.TotalAmount)  AS AvgOrderValue<br>FROM Customers c<br>LEFT JOIN Orders o ON c.CustomerID = o.CustomerID<br>GROUP BY c.CustomerID, c.FirstName, c.LastName;</pre><p>CREATE OR ALTER updates the view&#39;s definition if it already exists, or creates it if it does not, sparing you from writing a separate DROP statement first every time you need to tweak one line.</p><h3>DROP VIEW</h3><pre>DROP VIEW IF EXISTS CustomerOrderSummary;</pre><h3>Views With WHERE, JOIN, and GROUP BY</h3><p>A view can wrap any query, filtered, joined, or aggregated.</p><pre>CREATE VIEW ElectronicsOrders AS<br>SELECT<br>    o.OrderID,<br>    c.FirstName,<br>    c.LastName,<br>    p.ProductName,<br>    o.TotalAmount<br>FROM Orders o<br>JOIN Customers c ON o.CustomerID = c.CustomerID<br>JOIN Products p ON o.ProductID = p.ProductID<br>WHERE p.Category = &#39;Electronics&#39;;</pre><h3>WHERE vs HAVING When Querying a View</h3><p>The same WHERE vs HAVING distinction from Part 5 still applies once you query on top of a view. If the view has already aggregated the data, filtering on one of its aggregate columns from the outside is a plain WHERE, not a HAVING, because as far as the outer query is concerned, TotalSpent is just an ordinary column now, the aggregation already happened inside the view.</p><pre>SELECT * FROM CustomerOrderSummary WHERE TotalSpent &gt; 100;</pre><h3>ORDER BY Considerations in Views</h3><p>SQL Server does not guarantee row order in a view. The reliable approach is to apply ORDER BY when querying the view, not inside the view&#39;s own definition.</p><pre>SELECT FirstName, LastName, TotalSpent<br>FROM CustomerOrderSummary<br>ORDER BY TotalSpent DESC;</pre><p>The right habit is simple: sort when you query the view, not inside the view itself.</p><h3>View vs Table vs CTE vs Subquery</h3><pre>Table     stores actual data on disk<br>View      stores a saved query definition, no data of its own, reusable across many queries<br>CTE       temporary, named result set, exists only for one query, cannot be reused elsewhere<br>Subquery  nested inline, no name, exists only inside the one query it is written in</pre><p>Reach for a table when you need to persist data. Reach for a view when the same complex query gets reused across many reports and should have one single definition everyone shares. Reach for a CTE when you need readable, named steps inside a single query. Reach for a subquery for a small, one-off filter that does not need its own name.</p><h3>Basic Security and Use-Case Considerations</h3><p>A view can expose only selected columns or rows from underlying tables, which can help simplify data access and reduce unnecessary exposure of sensitive information. For example, a reporting user could work with customer order totals without needing direct access to raw contact information. However, a view is not automatically a complete security boundary, appropriate SQL Server permissions and security design are still required.</p><h3>Key SQL Concepts Covered</h3><pre>SQL and T-SQL             the language, and Microsoft&#39;s extension of it<br>RDBMS, Database, Schema   the containers data lives inside<br>Primary and foreign keys  what makes rows unique and tables related<br>SELECT and WHERE          reading and filtering rows<br>ORDER BY, TOP, DISTINCT   controlling and shaping result sets<br>Aggregate functions        COUNT, SUM, AVG, MIN, MAX<br>GROUP BY and HAVING        summarizing by category, then filtering those summaries<br>JOIN types                 INNER, LEFT, RIGHT, FULL OUTER<br>CASE, ISNULL, COALESCE, NULLIF   conditional logic and NULL handling<br>Subqueries                 scalar, IN, EXISTS, correlated<br>CTEs                       named, readable, multi-step queries<br>Views                      saved, reusable, access-controlled queries</pre><h3>Important Lessons Learned</h3><p><strong>NULL is not a value, it is the absence of one.</strong> Every comparison against NULL using = silently returns nothing. IS NULL and IS NOT NULL exist specifically because of this.</p><p><strong>WHERE and HAVING run at different stages of a query, not different syntaxes for the same job.</strong> WHERE filters raw rows before grouping. HAVING filters groups after aggregation.</p><p><strong>A join without a correct join condition is not a small query, it is a different, wrong query.</strong> A missing or incorrect condition produces a Cartesian product, not an error, which makes it dangerous precisely because it still returns rows.</p><p><strong>Views do not store data.</strong> Every query against a view re-runs its underlying definition. A view is a convenience and an access boundary, not a performance shortcut on its own.</p><p><strong>CTEs and subqueries are often interchangeable, readability is the real deciding factor.</strong> Reach for a CTE the moment a query needs to be read top to bottom by a human being other than the one who wrote it.</p><h3>Common Mistakes to Avoid</h3><p>Comparing against NULL with = instead of IS NULL.</p><p>Filtering an aggregate with WHERE instead of HAVING.</p><p>Joining two tables without a join condition, or with the wrong columns, producing a Cartesian product.</p><p>Selecting a column that is not aggregated and not in the GROUP BY list.</p><p>Assuming a view is faster than its underlying query. It is not, it is the same query, saved under a name.</p><p>Relying on row order from a table or view without an explicit ORDER BY on the final query.</p><h3>SQL Server Cheat Sheet</h3><pre>CREATE DATABASE dbname;<br><br>CREATE SCHEMA schemaname;<br><br>CREATE TABLE t (col TYPE constraints, ...);<br><br>INSERT INTO t (cols) VALUES (...);<br><br>SELECT cols FROM t WHERE condition ORDER BY col ASC|DESC;<br><br>SELECT DISTINCT col FROM t;<br><br>SELECT TOP n cols FROM t ORDER BY col;<br><br>COUNT(*), COUNT(col), SUM(col), AVG(col), MIN(col), MAX(col)<br>SELECT col, AGG(col2) FROM t GROUP BY col HAVING AGG(col2) condition;<br><br>SELECT a.*, b.* FROM a<br>INNER JOIN b ON a.key = b.key<br>LEFT JOIN b ON a.key = b.key<br>RIGHT JOIN b ON a.key = b.key<br>FULL OUTER JOIN b ON a.key = b.key<br>CASE WHEN condition THEN value ELSE other END<br>ISNULL(col, default)<br>COALESCE(col1, col2, default)<br>NULLIF(col, value)<br>SELECT * FROM t WHERE col &gt; (SELECT AGG(col) FROM t2);<br><br>SELECT * FROM t WHERE col IN (SELECT col FROM t2);<br><br>SELECT * FROM t WHERE EXISTS (SELECT 1 FROM t2 WHERE t2.key = t.key);<br><br>WITH cte_name AS (SELECT ...)<br>SELECT * FROM cte_name;<br><br>CREATE VIEW viewname AS SELECT ...;<br><br>CREATE OR ALTER VIEW viewname AS SELECT ...;<br><br>DROP VIEW IF EXISTS viewname;</pre><h3>How This Connects to Analytics, Power BI, Azure SQL, and Data Engineering</h3><p>Everything in this guide is the foundation underneath every layer built on top of a database.</p><p><strong>Power BI</strong> connects directly to SQL Server, and every join, aggregation, and filter covered here can either happen inside your SQL query before the data ever reaches Power BI, or be recreated less efficiently inside Power Query and DAX afterward. Writing the logic in SQL first is almost always faster and easier to maintain.</p><p><strong>Azure SQL Database</strong> and <strong>Azure SQL Managed Instance</strong> run the same T-SQL covered in this guide, in the cloud, which is exactly why this is Phase 1 of DP-800 preparation rather than a standalone SQL course. The syntax you write locally is the syntax you write against a cloud-hosted database.</p><p><strong>Views</strong> are frequently the exact layer a BI tool or a downstream data pipeline is pointed at, rather than raw tables, precisely because they let you change what “customer revenue” means in one place and have every report update automatically.</p><p><strong>CTEs</strong> show up constantly in <strong>data engineering</strong> pipelines, breaking a complex transformation into named, testable steps instead of one unreadable nested query, which is the same readability argument from Part 10, just at production scale.</p><h3>Conclusion</h3><p>None of the individual pieces in this guide are advanced on their own. <strong>SELECT, WHERE, JOIN, and GROUP BY</strong> are each small, learnable ideas. What makes SQL powerful is how these ideas <strong>compose together</strong>. A <strong>JOIN</strong> feeds a <strong>GROUP BY</strong>. A <strong>GROUP BY</strong> feeds a <strong>HAVING</strong>. A <strong>CTE</strong> names a step so that a later part of the same query can build on it. A <strong>View</strong> packages the resulting logic under one name, allowing reports and applications to work with the result without repeatedly exposing the underlying query complexity.</p><p>The database in this guide, <strong>Customers, Products, and Orders</strong>, is deliberately small. The questions built on top of it, such as <strong>revenue by category, customers with no orders, and each customer’s spending compared with their own average</strong>, are not toy examples. They represent the same kinds of questions commonly encountered in real <strong>analytics and data engineering</strong> work, simplified here into a schema small enough to understand from the first CREATE TABLE statement to the final CREATE VIEW.</p><p>Together, these concepts provide a strong <strong>T-SQL foundation for working with SQL Server data</strong>. The goal is not simply to memorize individual SQL commands, but to understand how they work together to transform raw relational data into meaningful, reusable, and analytical results.</p><h3>What Comes Next</h3><p>This guide built a strong foundation in <strong>SQL Server and T-SQL</strong>, covering <strong>SELECT, filtering, aggregation, JOINs, subqueries, CTEs, and Views</strong>.</p><p>The next step is to explore how SQL Server is <strong>administered, secured, optimized, and monitored</strong>, including <strong>database architecture, indexes, security, backup and restore, Azure SQL, and performance</strong>.</p><p>If this guide helped you understand SQL more clearly, consider sharing it with someone learning <strong>SQL, data analytics, or data engineering</strong>.</p><p>Follow me for more content on <strong>Python, SQL, data science, machine learning, data engineering, and AI</strong>.</p><p><strong>LinkedIn:</strong> <a href="https://www.linkedin.com/in/subinthapa/">Subin Thapa</a></p><p><strong>GitHub:</strong> <a href="https://github.com/Subinthapa2092">Subinthapa2092</a></p><p><strong>YouTube:</strong> <a href="https://www.youtube.com/@codingwithsubin">Coding With Subin</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e5070eddc31d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Cat vs Dog Classifier That Works the Same Way Twice]]></title>
            <link>https://medium.com/@subinthapa2092/building-a-cat-vs-dog-classifier-that-works-the-same-way-twice-fe4de3c2843c?source=rss-9f567b128564------2</link>
            <guid isPermaLink="false">https://medium.com/p/fe4de3c2843c</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[cnn]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Subinthapa]]></dc:creator>
            <pubDate>Mon, 07 Sep 2026 13:23:36 GMT</pubDate>
            <atom:updated>2026-09-07T13:23:36.024Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Fbx6g_tsGrKEMshkWIgccA.png" /></figure><h4>L2 regularization, graduated dropout, augmentation, and Docker: what each one actually did, and what it didn’t</h4><p>By epoch twenty, training accuracy reached 79% in the Docker run.</p><p>Impressive, until you look at the <strong>validation curve</strong>. Validation accuracy had already peaked three epochs earlier and was quietly sliding backward. The model wasn’t learning cats and dogs. It was <strong>memorizing</strong> photographs.</p><p>Four techniques later, I found what actually fixed it. And one thing I <em>thought</em> would help didn’t, at least not provably. This is the honest version of that story, including the part where I can’t fully credit one of my own design choices.</p><p><strong>Full project on GitHub:</strong> <a href="https://github.com/Subinthapa2092/cat-dog-cnn-classifier.git">Subinthapa2092/cat-dog-cnn-classifier</a></p><p><em>The base architecture and workflow in this project follow a class assignment taught by my mentor, Angat Sitoula. The regularization fixes, Docker setup, reproducibility investigation, and debugging process described below are my own extension of that foundation.</em></p><h3>Part 1: The Dataset</h3><p>The dataset is Kaggle’s <strong>Cat and Dog</strong> set (tongpython/cat-and-dog), pre-split into training and test folders.</p><pre>Training set:  8,005 images<br>Test set:      2,023 images<br>Classes:       cats, dogs</pre><p>Every image is resized to <strong>150x150 pixels</strong> on load. The test set uses shuffle=False, so predictions always line up with the correct labels, a small detail that matters a lot once you&#39;re building a <strong>confusion matrix</strong> later.</p><pre>train_dataset = tf.keras.utils.image_dataset_from_directory(<br>    train_dir, image_size=(150, 150), batch_size=32, shuffle=True,<br>)<br>test_dataset = tf.keras.utils.image_dataset_from_directory(<br>    test_dir, image_size=(150, 150), batch_size=32, shuffle=False,<br>)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*L9ygjtina7j_zjQDysHT9A.png" /><figcaption><strong>Figure 1. Sample cat and dog images.</strong></figcaption></figure><h3>Part 2: The Architecture</h3><p>Most tutorials build the simplest network first, watch it <strong>overfit</strong>, then patch it. This project did the opposite. Anti-overfitting measures were built in from the start.</p><pre>Input (150x150x3)<br>  -&gt; Data Augmentation<br>  -&gt; Rescaling (0 to 1)<br>  -&gt; Conv2D(32)  -&gt; ReLU -&gt; MaxPooling -&gt; Dropout(0.2)<br>  -&gt; Conv2D(64)  -&gt; ReLU -&gt; MaxPooling -&gt; Dropout(0.2)<br>  -&gt; Conv2D(128) -&gt; ReLU -&gt; MaxPooling -&gt; Dropout(0.3)<br>  -&gt; Flatten<br>  -&gt; Dense(128)  -&gt; ReLU -&gt; Dropout(0.5)<br>  -&gt; Dense(1)    -&gt; Sigmoid</pre><p>Two choices separate this from a bare minimum CNN: <strong>L2 regularization</strong> on every layer, and <strong>graduated dropout</strong> that gets heavier with depth.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*priHFuJFZnRn9aWrOUildw.png" /><figcaption><strong>Figure 2. L2 regularization and dropout prevent overfitting.</strong></figcaption></figure><h3>Why L2 Regularization</h3><p>Every Conv2D and Dense(128) layer carries an <strong>L2 kernel regularizer</strong>.</p><pre>layers.Conv2D(32, kernel_size=(3, 3), activation=&quot;relu&quot;,<br>              kernel_regularizer=regularizers.l2(1e-4))</pre><p>This adds a penalty to the loss proportional to the squared weight values.</p><pre>Loss = Data Loss + λ × Σw²</pre><p><strong>Why L2 and not L1.</strong> L1 can push weights to exactly zero, killing convolutional filters entirely. L2 shrinks weights without switching any off, keeping the full <strong>receptive field</strong> intact. λ was set to <strong>0.0001</strong>, deliberately mild.</p><h3>Why Graduated Dropout</h3><p><strong>Dropout</strong> randomly disables units during training, forcing the network to not rely on any single path.</p><pre>Dropout(0.2)  after Conv Block 1   (light, early features must flow)<br>Dropout(0.2)  after Conv Block 2<br>Dropout(0.3)  after Conv Block 3<br>Dropout(0.5)  after Dense(128)     (heaviest, most prone to memorization)</pre><p>Early layers detect <strong>edges</strong> and <strong>textures</strong>. Dropping too many too early starves the deeper layers. So the rate starts light and climbs to <strong>0.5</strong> only at the dense layer, where memorization is most likely to take hold.</p><h3>Part 3: The Augmentation Pipeline</h3><p><strong>Data augmentation</strong> is built as a Keras layer inside the model itself.</p><pre>def build_augmentation():<br>    return keras.Sequential([<br>        layers.RandomFlip(&quot;horizontal&quot;),<br>        layers.RandomRotation(0.1),<br>        layers.RandomZoom(0.2),<br>        layers.RandomTranslation(height_factor=0.1, width_factor=0.1),<br>        layers.RandomContrast(0.1),<br>    ], name=&quot;data_augmentation&quot;)</pre><p>Placing it inside the model means Keras automatically turns it off during <strong>evaluation</strong> and <strong>inference</strong>. No separate flag needed. No risk of augmented images leaking into the test pass.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/793/1*CPvNjb4DrXJkhiy3Vz55yA.png" /><figcaption><strong>Figure 3. Data-augmented variations of a training image.</strong></figcaption></figure><h3>Part 4: Training Setup</h3><pre>model.compile(<br>    optimizer=&quot;adam&quot;,<br>    loss=&quot;binary_crossentropy&quot;,<br>    metrics=[&quot;accuracy&quot;],<br>)<br><br>early_stopping = keras.callbacks.EarlyStopping(<br>    monitor=&quot;val_loss&quot;, patience=3, restore_best_weights=True,<br>)<br>history = model.fit(<br>    train_dataset,<br>    validation_data=test_dataset,<br>    epochs=20,<br>    callbacks=[early_stopping],<br>)</pre><p><strong>Important limitation: test set reuse.</strong> This pipeline uses the test set as validation during training. The same data guiding <strong>early stopping</strong> is later used to report final accuracy. A cleaner version would carve out a separate validation split and keep the test set completely untouched. Read the Part 7 number with this in mind.</p><h3>Part 5: The Training Curves</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Aq7Icny0WYXJM4SUCHg5mw.png" /><figcaption><strong>Figure 4. Accuracy rises while loss falls steadily.</strong></figcaption></figure><p>Both accuracy lines climb from around <strong>50%</strong> to the <strong>mid seventies</strong>. Validation loss tracks training loss throughout rather than pulling away.</p><p>That diverging pattern, training climbs while validation stalls, is the signature of a model that stopped learning and started memorizing. It does not appear here.</p><p>This is the visible payoff of building regularization in before training, not as a patch after watching it fail.</p><h3>Part 6: How Much Work Is L2 Actually Doing?</h3><p>Worth asking directly, because Figure 4 alone does not answer it.</p><p><strong>The honest answer:</strong> the ablation was never run. Removing only the L2 term while keeping everything else identical would actually answer this. Without it, crediting L2 for the clean curve in Figure 4 overstates what the evidence shows.</p><p>The reasonable case for keeping it anyway:</p><p><strong>The capacity argument.</strong> Roughly <strong>4.8 million parameters</strong> trained on <strong>8,000 images</strong>. That ratio gives a network more than enough freedom to memorize if nothing holds it back.</p><p><strong>The magnitude argument.</strong> λ = 0.0001 is mild. It likely acts as cheap insurance on top of two mechanisms doing the real work: <strong>four dropout layers</strong> and <strong>five augmentation types</strong> reshaping every training image every epoch.</p><p><strong>The experiment that would settle this.</strong> Same architecture, same dropout, same augmentation, L2 removed. Close curves means L2 was insurance. A visible gap means it was pulling real weight.</p><p><strong>The defensible position:</strong> keep it, but don’t overclaim it.</p><h3>Part 7: Final Results</h3><p>One number first.</p><pre>Test Accuracy:  76.22%<br>Test Loss:      0.5382<br>Epochs trained: 20</pre><p>Then the more interesting numbers underneath it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*THyt4nZls02WP0cOShsZbQ.png" /><figcaption><strong>Figure 5. Confusion matrix of cat and dog classification.</strong></figcaption></figure><p><strong>Precision, Recall, F1</strong></p><pre>| Class | Precision | Recall |   F1 |<br>| ----- | --------: | -----: | ---: |<br>| Cats  |      0.73 |   0.84 | 0.78 |<br>| Dogs  |      0.81 |   0.68 | 0.74 |</pre><p>A real asymmetry sits in this table.</p><p><strong>Cats are identified correctly 84% of the time. Dogs only 68%.</strong></p><p>The confusion matrix shows why: <strong>320 dogs</strong> are misclassified as cats, nearly twice the <strong>161 cats</strong> misclassified as dogs.</p><p>The model carries a consistent <strong>bias toward guessing cat</strong> when uncertain. This never shows up in overall accuracy, which is exactly why the <strong>confusion matrix</strong> is reported here.</p><h3>Part 8: Docker and the Reproducibility Gap</h3><p>The notebook version landed at <strong>74.05% accuracy</strong>. The Docker version landed at <strong>79.09% accuracy</strong>.</p><p>Same architecture. Same data. Same hyperparameters. Different number.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OmFpT1mPcwof4gvw8DVlTQ.png" /><figcaption><strong>Figure 6. Containerized training run completed successfully.</strong></figcaption></figure><p>Neither is wrong. The difference came from <strong>randomness that was never seeded</strong>. Weight initialization, augmentation ordering, and shuffling were not fixed. Two runs of identical code will land in different places.</p><p><strong>Docker</strong> does not fix that. It was never designed to.</p><p>What Docker actually guarantees: the <strong>Python version</strong>, the <strong>TensorFlow build</strong>, and every installed package are identical on every machine that runs the container. The whole class of “it worked on my machine” problems from silent version drift disappears. That is a completely different problem from the 74.05 vs 79.09 gap.</p><pre>docker compose up --build<br>docker compose run cat-dog-classifier python -m src.evaluate</pre><p><strong>The honest verdict.</strong> A virtual environment would have sufficed for one person on one machine. Docker’s real value is letting anyone else run the project with a single command. That matters for a portfolio project.</p><h3>Part 9: Project Structure</h3><p>Rather than one long notebook, the logic lives in small separate modules.</p><pre>| File              | Main Responsibility                 |<br>| ----------------- | ----------------------------------- |<br>| `config.py`       | Configuration and training settings |<br>| `data_loader.py`  | Loads and prepares image datasets   |<br>| `augmentation.py` | Applies image augmentation          |<br>| `model.py`        | Builds and compiles the CNN         |<br>| `train.py`        | Trains the model and saves results  |<br>| `evaluate.py`     | Evaluates model performance         |<br>| `predict.py`      | Predicts a single image             |</pre><pre>python -m src.train<br>python -m src.evaluate<br>python -m src.predict --image path/to/photo.jpg --show</pre><p>Each step runs independently. No need to re-execute an entire notebook to reach the part you care about.</p><h3>Key Formulas</h3><p><strong>L2 Regularization</strong></p><pre>Loss = Data Loss + λ × Σw²</pre><p><strong>Binary Crossentropy</strong></p><pre>Loss = −[y × log(p) + (1 − y) × log(1 − p)]</pre><p><strong>Precision and Recall</strong></p><pre>Precision = TP / (TP + FP)<br>Recall    = TP / (TP + FN)</pre><h3>What I Would Improve Next</h3><p><strong>Run the L2 ablation.</strong> Same dropout, same augmentation, no L2. Compare curves directly.</p><p><strong>Fix the train/test split.</strong> Reserve a genuine validation set so the test set stays untouched until the very end.</p><p><strong>Add a fixed random seed.</strong> This is the actual fix for the 74.05 vs 79.09 gap, not a container change.</p><p><strong>Investigate the dog recall gap.</strong> Find which specific dog photos are misclassified as cats and why.</p><h3>Summary Table</h3><pre>| Concept               | What It Did Here                                        |<br>| --------------------- | ------------------------------------------------------- |<br>| L2 regularization     | Mild safety margin (λ = 0.0001); ablation not run       |<br>| Graduated dropout     | 0.2 → 0.5 with depth; likely the primary regularizer    |<br>| In-model augmentation | Auto-off at evaluation; no leakage risk                 |<br>| Early stopping        | Patience 3; restores best weights                       |<br>| Docker                | Removed setup friction; did not fix run-to-run variance |<br>| Confusion matrix      | Revealed cat-leaning bias invisible in overall accuracy |</pre><h3>What I Actually Learned</h3><p><strong>Validation accuracy is the only accuracy that matters.</strong> Training at 79.09% looked good. It meant nothing by itself.</p><p><strong>Build against overfitting before training, not after.</strong> Patching a model that already overfits teaches you less than designing correctly from the start.</p><p><strong>Be honest about what you cannot measure.</strong> The L2 ablation is unrun. Crediting it for the clean curve without the experiment is not honesty, it’s narrative.</p><p><strong>Reproducibility is not one problem.</strong> Docker fixes version drift. Seeding fixes run to run variance. These are different problems. Fixing one does not fix the other.</p><h3>Final Thoughts</h3><p>None of these choices are unusual. <strong>L2 regularization</strong>, <strong>graduated dropout</strong>, <strong>augmentation</strong>, and <strong>Docker</strong> are all standard tools.</p><p>What matters is being clear about what each one actually did.</p><p><strong>Dropout</strong> and <strong>augmentation</strong> are the more likely reason this model does not overfit. <strong>Docker</strong> removed setup friction. The accuracy gap came from <strong>unseeded randomness</strong>, not from the container.</p><p>The result: mid seventies accuracy, a clean training curve, and an honest account of which choices earned their place.</p><p>Feel free to open a GitHub issue if you run the L2 ablation before I do.</p><p>Special thanks to my mentor, Angat Sitoula, for teaching the foundational workflow this project builds on.</p><h3>What Comes Next</h3><p>This article focused on understanding how CNNs actually see and learn from images, from individual pixels and small kernels to feature maps, receptive fields, and final predictions.</p><p>But CNNs are only the beginning of modern computer vision.</p><p>The next step is to go deeper into how these networks can be improved, optimized, and applied to real world problems: <strong>transfer learning</strong>, <strong>batch normalization</strong>, deeper architectures, and modern computer vision models. The mathematics stays the same. The architecture gets more powerful.</p><p>If this guide helped you understand <strong>CNNs</strong>, <strong>convolution</strong>, <strong>kernels</strong>, <strong>feature maps</strong>, <strong>pooling</strong>, <strong>backpropagation</strong>, and <strong>gradient descent</strong> more clearly, consider sharing it with someone learning machine learning, deep learning, computer vision, or AI.</p><p>I’d also love to hear your thoughts. Which concept finally made CNNs click for you? Understanding pixels as numbers, seeing how a kernel detects an edge, watching convolution create a feature map, understanding receptive fields, or seeing how gradient descent learns the filters automatically?</p><p>Follow me for more content on Python, mathematics for machine learning, data science, machine learning, deep learning, computer vision, and AI.</p><p><strong>LinkedIn:</strong> <a href="https://www.linkedin.com/in/subinthapa/">Subin Thapa</a></p><p><strong>GitHub:</strong> <a href="https://github.com/Subinthapa2092">Subinthapa2092</a></p><p><strong>YouTube:</strong> <a href="https://www.youtube.com/@codingwithsubin">Coding With Subin</a></p><p><strong>Full project:</strong> <a href="https://github.com/Subinthapa2092/cat-dog-cnn-classifier.git">github.com/Subinthapa2092/cat-dog-cnn-classi</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fe4de3c2843c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Convolutional Neural Networks Actually See]]></title>
            <link>https://medium.com/@subinthapa2092/how-convolutional-neural-networks-actually-see-db3a0875d20a?source=rss-9f567b128564------2</link>
            <guid isPermaLink="false">https://medium.com/p/db3a0875d20a</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Subinthapa]]></dc:creator>
            <pubDate>Sun, 06 Sep 2026 05:15:22 GMT</pubDate>
            <atom:updated>2026-09-06T05:57:24.707Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WHdgMSeAdZ5f3b6YNDePPA.png" /></figure><h4>The Complete Guide to Pixels, Kernels, Convolution, Learning, and Building a CNN From First Principles</h4><p>A photo, to a computer, is not a picture. It is a <strong>grid of numbers</strong>.</p><p>Before a <strong>convolutional neural network</strong>, commonly shortened to <strong>CNN</strong>, can tell a cat from a dog, it has to solve a more basic problem: how do you take a grid of numbers and turn it into an understanding of <strong>shapes, edges, and eventually objects</strong>? And how does the network figure any of that out on its own?</p><p>This article builds that answer from the ground up. It starts with why a <strong>plain neural network</strong> struggles with images at all. It walks through what a <strong>kernel</strong> and a <strong>convolution</strong> actually are, with real numbers. It covers <strong>padding, stride, and pooling</strong>. Then it goes further, into how a CNN learns its own <strong>filters through backpropagation</strong>, how to count its <strong>parameters</strong>, how <strong>classification</strong> actually works, and ends with a <strong>complete, working Python implementation</strong>.</p><h3>Part 1: Why a Regular Neural Network Struggles With Images</h3><h3>An Image Is Just Numbers</h3><p>A <strong>greyscale image</strong> is a grid of numbers, one number per pixel. Each value represents how dark or bright that pixel is. A 100 by 100 pixel greyscale image is really just:</p><pre>100 x 100 = 10,000 numbers</pre><p>To feed this into a standard neural network, every one of those 10,000 numbers has to be flattened into a single long list.</p><pre>image = [x1, x2, x3, ..., x10000]</pre><h3>The Dimensionality Problem</h3><p>Here is where the trouble starts. A <strong>fully connected layer</strong> connects every input value to every neuron in the next layer.</p><p>Even a tiny example makes the cost obvious. Two input points connected to two neurons already needs four connections, since each input must reach both neurons.</p><pre>2 points, 2 neurons -&gt; 2 x 2 = 4 connections</pre><p>Scale that up to a real image. Connecting all 10,000 pixel values to a hidden layer of just 10,000 neurons requires:</p><pre>10,000 x 10,000 = 100,000,000 weights</pre><p>One hundred million weights, for a small, low resolution, black and white image, before the network has learned anything at all.</p><p>This is usually called the <strong>dimensionality problem</strong>. But it helps to be precise about what a CNN actually fixes.</p><p>It is not only that images have a lot of numbers. A <strong>fully connected layer</strong> also throws away something valuable: it treats every pixel as equally related to every other pixel. In reality, nearby pixels are far more likely to belong to the same edge or shape than pixels on opposite corners of the image.</p><p>The real ideas behind a CNN are these three.</p><pre>Local connectivity      -&gt; look at small neighborhoods, not the whole image at once<br>Parameter sharing       -&gt; reuse the same small kernel across the entire image<br>Spatial structure kept  -&gt; keep the 2D layout intact instead of flattening it away</pre><p>A smaller parameter count is the visible result of these three ideas. It is not the whole story.</p><blockquote>A<strong> CNN</strong> isn’t cheaper because it’s simpler. It’s cheaper because it refuses to treat two pixels on opposite corners of the image as <strong>equally related</strong>.</blockquote><p><strong><em>Those three ideas</em></strong><em> didn’t appear out of nowhere. They came from a much older question about </em><strong><em>how vision itself works</em></strong><em>.</em></p><h3>Part 2: A Short History of Why CNNs Look the Way They Do</h3><p>The design of a CNN did not come out of nowhere.</p><p>In the <strong>1950s and 60s</strong>, research into the brain’s <strong>visual cortex</strong> showed that neurons respond to small, localized parts of what the eyes see, not the entire visual field at once. That was an early hint that vision might work by examining small regions rather than everything simultaneously.</p><p>In the <strong>1980s</strong>, <strong>backpropagation</strong> made it possible to actually train networks with this kind of structure. The analogy of the eyes being connected to the brain in a structured, localized way carried directly into network design.</p><p><strong>Yann LeCun</strong> built on this with early work on handwritten digit recognition. That work led to <strong>LeNet-5</strong>, and later to architectures like <strong>AlexNet</strong>. Progress was genuinely slowed for years by one very practical constraint: there was no powerful GPU yet to train these networks at any meaningful scale. Once GPUs caught up, CNNs became practical for real, large images, not just small digit scans.</p><p><em>With the history out of the way, it’s worth zooming all the way in on what a single pixel actually is.</em></p><h3>Part 3: What a Digital Image Actually Looks Like Up Close</h3><p>In a greyscale image, every pixel holds a single number.</p><pre>0   = pure black<br>255 = pure white</pre><p>Values in between are shades of gray, darker as they approach 0 and lighter as they approach 255. A small patch of image that is uniformly dark might look like this:</p><pre>20  20  20<br>20  20  20<br>20  20  20</pre><p>And a patch that is uniformly bright might look like this:</p><pre>240  240  240<br>240  240  240<br>240  240  240</pre><p>Neither patch has anything visually interesting happening inside it. The values are constant.</p><p>But place a dark patch directly beside a bright patch, and something changes.</p><pre>20   20   240<br>20   20   240<br>20   20   240</pre><p>That sudden jump, from 20 to 240, is exactly what an <strong>edge</strong> is.</p><pre>Edge = the place where pixel values change significantly</pre><p>An edge is just a big, sudden numeric change. That single idea is the seed the rest of convolution grows out of.</p><p><em>Detecting that jump is exactly what a kernel is built to do.</em></p><h3>Part 4: Kernels, Small Matrices That Look for Patterns</h3><p>A <strong>kernel</strong>, also called a <strong>filter</strong>, is a small matrix of numbers the network uses to detect a pattern in an image. A common <strong>3 by 3</strong> kernel for detecting a vertical edge looks like this:</p><pre>-1   0   +1<br>-1   0   +1<br>-1   0   +1</pre><p>Read column by column, this kernel is really asking one question: are the pixels on the right very different from the pixels on the left?</p><pre>(left x -1) + (middle x 0) + (right x +1)  =  right - left</pre><p>This particular kernel is hand designed. It is a useful teaching example because the numbers are easy to reason about.</p><p>But it is worth being upfront: this is not how most kernels in a real, trained CNN come to exist. A kernel is more precisely a small matrix of <strong>learnable weights</strong>. The network starts with random numbers and adjusts them through training, until they happen to detect whatever patterns actually help it make correct predictions. <strong>Part 15</strong> covers exactly how that learning happens.</p><p>For now, treating the kernel as a fixed, hand built pattern detector is the clearest way to understand what convolution computes.</p><h3>Testing It on a Flat Region</h3><p>Apply this kernel to a patch where nothing changes, left to right.</p><pre>20   20   20<br>-1    0   +1<br><br>-20 + 0 + 20 = 0<br></pre><p>The output is zero. No difference between left and right means no edge. The kernel correctly reports nothing interesting here.</p><h3>Testing It on an Actual Edge</h3><p>Now apply the same kernel to the patch above, where the left side is dark and the right side is bright.</p><pre>20   20   240<br>-1    0   +1<br><br>(20)(-1) + (20)(0) + (240)(+1) = -20 + 0 + 240 = 220</pre><p><strong>220</strong> is a large number. A large output means a large difference between left and right, which means a strong edge sits right there. Repeat this across all three rows of the full patch and sum the result:</p><pre>-20 + 0 + 240 = 220   (row 1)<br>-20 + 0 + 240 = 220   (row 2)<br>-20 + 0 + 240 = 220   (row 3)<br><br>Total = 660<br><br>Small output -&gt; probably no edge<br>Large output -&gt; probably a strong edge</pre><p>A horizontal edge works the same way, just rotated. Instead of comparing left to right, this kernel compares top to bottom.</p><pre>-1   -1   -1<br> 0    0    0<br>+1   +1   +1</pre><p><em>Sliding either kernel across a full image, not just one patch, is the operation that gives convolution its name.</em></p><h3>Part 5: Convolution, Formally</h3><p><strong>Convolution</strong> is the process of sliding a kernel across an image, one position at a time. At every position, it computes a weighted sum called a <strong>dot product</strong>. Each position produces exactly one output number.</p><pre>Convolution output = dot product of the kernel and the image patch under it</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*emwLvEeWcs_g_tm__sjjqA.png" /><figcaption><strong>Figure 1. Convolution: kernel to feature map.</strong></figcaption></figure><h3>A Full Worked Example</h3><p>Take a 5 by 5 greyscale image and a 3 by 3 kernel.</p><pre>X = | 1  2  0  3  1 |        K = | 1   0  -1 |<br>    | 4  1  2  1  0 |            | 1   0  -1 |<br>    | 1  3  1  2  2 |            | 1   0  -1 |<br>    | 0  1  4  1  3 |<br>    | 2  2  1  0  1 |</pre><p>The top left 3 by 3 patch of the image lines up with the kernel like this:</p><pre>| 1  2  0 |     | 1   0  -1 |<br>| 4  1  2 |  x  | 1   0  -1 |<br>| 1  3  1 |     | 1   0  -1 |</pre><p>Multiply each pair of overlapping numbers and add everything together:</p><pre>(1)(1) + (2)(0) + (0)(-1)<br>+ (4)(1) + (1)(0) + (2)(-1)<br>+ (1)(1) + (3)(0) + (1)(-1)<br>= 3</pre><p>That single number, 3, becomes the top left value of the output. The kernel then slides one column to the right and computes the next value. It keeps going until it has covered the entire image, producing a complete output grid called a <strong>feature map</strong>.</p><h3>The Output Size Formula</h3><p>Notice the output is smaller than the input. A 5 by 5 image convolved with a 3 by 3 kernel produces a 3 by 3 output, not another 5 by 5 grid.</p><pre>Output size = (n - k + 1) x (n - k + 1)<br><br>n = input size, k = kernel size<br>(5 - 3 + 1) = 3, so output = 3 x 3</pre><p><em>A shrinking output isn’t always welcome. Padding exists to control exactly that.</em></p><h3>Part 6: Padding, Because Convolution Shrinks the Image</h3><p>Shrinking the image on every convolution is not always what you want. This becomes especially clear once several convolutional layers are stacked one after another.</p><p><strong>Padding</strong> solves this by adding extra pixels, usually zeros, around the border of the image before convolution runs.</p><pre>Zero padding = a ring of zeros added around the image&#39;s edge</pre><p>Adding <strong>1 padding</strong> means adding one extra row on top and bottom, and one extra column on left and right. A 5 by 5 image with 1 padding effectively becomes a 7 by 7 image before the kernel touches it. That is exactly enough to let a 3 by 3 kernel produce a 5 by 5 output again, the same size as the original input.</p><h3>Part 7: Stride, Skipping Positions on Purpose</h3><p><strong>Stride</strong> controls how many pixels the kernel jumps between each position. A stride of 1 checks nearly every position, moving one pixel at a time. A stride of 2 skips every other position, covering the image more coarsely and producing a smaller output.</p><p>Combining padding and stride gives the general formula for output size.</p><pre>Output size = floor( (n - k + 2p) / s ) + 1<br><br>n = input size<br>k = kernel size<br>p = padding<br>s = stride</pre><p>Set <strong>p to 0</strong> and<strong> s to 1</strong>, and this collapses back down to the simpler formula from Part 5.</p><h3>Part 8: Color Images Have Three Channels</h3><p>Everything so far assumed a greyscale image, a single grid of numbers. A color image is not one grid. It is three, stacked on top of each other, one each for <strong>Red, Green, and Blue</strong>, commonly called <strong>RGB</strong>.</p><pre>Greyscale image: height x width          (a single 2D grid)<br>Color image:     height x width x 3      (three 2D grids stacked together)</pre><p>A 5 by 5 color image is really shaped 5 by 5 by 3. To convolve over it, the kernel needs a matching depth too, becoming 3 by 3 by 3 instead of a flat 3 by 3. That lets it slide across all three color channels at once and combine them into a single output value at each position.</p><h3>Part 9: Multiple Filters and Output Channels</h3><p>Everything up to this point used exactly one kernel. And one kernel can only ever detect one kind of pattern.</p><p>A real convolutional layer applies many kernels to the same input at once. Each one is initialized differently and free to specialize in detecting something different.</p><p>Every individual filter, no matter how many channels the input has, still produces exactly one 2D feature map. Use 32 separate filters, and the layer produces 32 separate feature maps, stacked together into a single output volume.</p><pre>Input:    32 x 32 x 3     (a small color image)<br>Filters:  32, each 3x3x3<br><br>Output:   30 x 30 x 32</pre><p>The <strong>width</strong> and <strong>height</strong> shrink for the same reason a single filter shrinks them, governed by the formula from Part 7. The <strong>depth</strong> of the output has nothing to do with the input’s depth. It is set entirely by how many filters were used. Three input channels became 32 output channels here simply because 32 filters were chosen. Each of those filters looked across all 3 input channels to produce its own single feature map.</p><p>This is also why a second convolutional layer stacked on top needs its filters to be 3 by 3 by 32, not 3 by 3 by 3. Its input depth is whatever the previous layer’s output depth was, 32 here, not the original image’s 3 color channels.</p><p><em>All these filters sound expensive. The actual parameter count says otherwise.</em></p><h3>Part 10: Counting Parameters, Why CNNs Are So Efficient</h3><p>Part 1 showed a fully connected layer needing <strong>100,000,000 </strong>weights just to connect two layers of <strong>10,000</strong> neurons each. It is worth actually counting how few parameters a convolutional layer needs to do comparable work. This is where local connectivity and parameter sharing pay off directly.</p><pre>Parameters = (kernel height x kernel width x input channels x number of filters) + biases</pre><p>Take the layer from Part 9: a 3 by 3 kernel, 3 input channels, and 32 filters.</p><pre>3 x 3 x 3 x 32 = 864<br>+ 32 biases (one per filter)<br>= 896 parameters</pre><p><strong>896 parameters</strong>, for a convolutional layer capable of scanning an entire image for 32 different patterns. Compare that to the fully connected example from Part 1.</p><pre>Fully connected layer: 100,000,000 weights<br>Convolutional layer:           896 weights</pre><p>That is not a <strong>rounding error</strong>. It is roughly five orders of magnitude.</p><p>Every one of those 896 weights gets reused at every position the kernel slides to across the entire image. That is exactly what <strong>parameter sharing</strong> means in practice.</p><p>The network does not need a separate weight for the top left corner and a separate weight for the bottom right corner. The same 3 by 3 pattern detector checks both locations using identical weights.</p><blockquote>The same <strong>3×3 detector</strong> checks the top-left corner and the bottom-right corner using the exact same <strong>9 numbers</strong>. That reuse, not cleverness, is where the <strong>five orders of magnitude</strong> go.</blockquote><p><em>Raw convolution output isn’t the end of the line for a </em><strong><em>feature map</em></strong><em>. It still has to pass through an </em><strong><em>activation function</em></strong><em> first.</em></p><h3>Part 11: ReLU, Keeping Only What Matters</h3><p>After a convolution produces a feature map, the values pass through an activation function called <strong>ReLU</strong>, short for Rectified Linear Unit.</p><pre>ReLU(x) = max(0, x)</pre><p>Every positive value passes through unchanged. Every zero or negative value gets clipped to exactly zero.</p><pre>Positive -&gt; keep it<br>Zero or negative -&gt; change to 0</pre><p>Applied to a feature map, ReLU zeroes out the negative entries and leaves everything else alone. The result is often called an <strong>activated feature map</strong>.</p><pre>Image -&gt; Kernel -&gt; Feature Map -&gt; ReLU -&gt; Activated Feature Map</pre><h3>Part 12: Feature Maps and Going Deeper</h3><p>A single kernel only detects one kind of pattern, a vertical edge, say, or a horizontal one. Real CNNs apply many different kernels to the same image at once, exactly as Part 9 described. Each one produces its own feature map, tuned to notice something different.</p><pre>Largest difference in a patch -&gt; strong edge</pre><p>Stacking convolutional layers lets the network build understanding in stages. The first layer’s kernels tend to pick up simple things: edges and corners.</p><p>Feed those feature maps into a second convolutional layer, and its kernels can combine edges into more complex shapes, the outline of an ear, the curve of a nose. Keep stacking, and the network works its way up from raw pixels to increasingly abstract, higher level features, eventually reaching something specific enough to base a classification decision on.</p><h3>Receptive Field: How Much of the Image Each Neuron Actually Sees</h3><p>That progression from <strong>edges to shapes to objects</strong> has a precise geometric explanation. It is called the <strong>receptive field</strong>: how large a region of the original input image a single value in a given feature map is actually influenced by.</p><p>A neuron in the very first convolutional layer, built from a 3 by 3 kernel, only ever looks at a 3 by 3 patch of the original image. Its receptive field is small and local by construction.</p><pre>Layer 1 neuron -&gt; sees a 3x3 patch of the original image</pre><p>A neuron in a second convolutional layer, also built from a 3 by 3 kernel, looks at a 3 by 3 patch of the first layer’s output. But each of those first layer values already summarized its own 3 by 3 patch of the original image. So the second layer neuron’s effective view of the original image is larger than 3 by 3, closer to 5 by 5 once the overlap is accounted for.</p><pre>Early layer  -&gt; small local region   (edges)<br>Middle layer -&gt; larger region        (shapes, textures)<br>Deep layer   -&gt; very large region    (whole objects)</pre><p>This is exactly why deeper layers recognize whole objects rather than just edges. It is not that deeper layers use fundamentally different math.</p><p>Their effective view of the original image has simply grown large enough to take in an entire ear, an entire face, or an entire object, just by stacking enough small, local operations on top of each other.</p><h3>Part 13: Pooling, Shrinking Without Losing the Point</h3><p><strong>Pooling</strong> reduces the size of a feature map while keeping the information that matters most. The most common form is <strong>max pooling</strong>.</p><pre>Max pooling = look at a small area, keep only the single biggest number</pre><p>The reasoning is direct. The biggest number in a small region represents the most significant, most strongly activated feature detected there. Keeping it and discarding the rest loses very little that actually matters.</p><p>Take a 4 by 4 feature map and pool it down using small 2 by 2 windows. The result is a smaller grid holding only the strongest value from each window, roughly a quarter of the original size.</p><p><strong>Average pooling</strong> works the same way structurally, but instead of keeping the maximum, it takes the average of each small region. Max pooling is better suited to pulling out one strong, dominant feature. Average pooling is better suited to preserving overall, general information across a region rather than spotlighting a single peak.</p><pre>Want the strongest feature   -&gt; max pooling<br>Want overall information     -&gt; average pooling</pre><p>Pooling is not the only way modern architectures shrink a feature map. Many newer designs use a convolution with a stride greater than 1 to downsample instead, folding the shrinking step directly into the convolution itself. Pooling is a foundational, easy to reason about technique. It is not the only correct way to do this.</p><h3>Part 14: The Full CNN Pipeline</h3><p>Put every piece together, and a typical CNN follows the same repeating structure: convolve, activate, pool, and repeat, before finally making a decision.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4MWgGiqR5VDibnEX7gMsFQ.png" /><figcaption><strong>Figure 2. CNN Pipeline: From Image to Prediction.</strong></figcaption></figure><h3>Watching the Shape Change at Every Stage</h3><p>Concrete numbers make this easier to follow than the diagram alone. Starting from a 32 by 32 color image:</p><pre>Input:                     32 x 32 x 3<br>Conv (32 filters, 3x3):    30 x 30 x 32<br>MaxPool (2x2):             15 x 15 x 32<br>Conv (64 filters, 3x3):    13 x 13 x 64<br>MaxPool (2x2):              6 x  6 x 64<br>Flatten:                   2,304 values   (6 x 6 x 64)<br>Dense (128 units):           128 values<br>Output (1 unit, sigmoid):      1 value</pre><p>Every convolution shrinks width and height a little, per the formula from Part 7. Depth grows to match however many filters were used, per Part 9.</p><p>Every pooling step cuts width and height further, without touching depth at all. By the time the data reaches the flatten step, it is still a small, deep grid. <strong>Flatten</strong> simply lays it out as one long list.</p><h3>Flatten, Turning a Grid Into a List</h3><p>The <strong>fully connected</strong>, or <strong>dense</strong>, layer near the end of the network needs a plain 1D list as input. A <strong>flatten</strong> step reshapes the grid into one.</p><pre>Flatten:  | 1  2 |   -&gt;   [1, 2, 3, 4]<br>          | 3  4 |<br><br>Multi dimensional -&gt; single dimension</pre><p>From there, the flattened vector feeds into one or more dense layers. These combine everything the convolutional layers detected into a final prediction.</p><p><em>Everything up to here assumed the kernels already knew what to look for. The next part is about where those numbers actually come from.</em></p><h3>Part 15: How a CNN Actually Learns</h3><p>Everything so far described what happens when an image passes through a CNN, assuming the kernels already contain the right numbers. None of it explains where those numbers come from. This is the part most introductions to CNNs skip, and it may be the most important one.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_5vWC5FseiwfVdE54tAbXA.png" /><figcaption><strong>Figure 3. The CNN training loop: from random filters to better features.</strong></figcaption></figure><h3>The Forward Pass</h3><p>A <strong>forward pass</strong> is simply running an image through the network exactly as described in Parts 1 through 14: convolution, ReLU, pooling, repeated, flattened, and passed through a dense layer to produce a prediction.</p><pre>Image -&gt; Convolution -&gt; ReLU -&gt; Pooling -&gt; ... -&gt; Flatten -&gt; Dense -&gt; Prediction</pre><p>At the very start of training, every filter is filled with small random numbers. The first forward pass through the network is, essentially, a guess.</p><h3>The Loss Function</h3><p>Once the network produces a prediction, the <strong>loss function</strong> measures how wrong that prediction was compared to the true label.</p><pre>Prediction -&gt; compare to true label -&gt; Loss</pre><p>A high loss means the prediction was far from correct. A low loss means it was close.</p><p>Training a CNN, at its core, means searching for filter values that make this loss as small as possible across the entire training set. That is the same goal covered in the earlier gradient descent article on this blog, just applied here to convolutional filters instead of a simple linear model’s weights.</p><h3>Backpropagation</h3><p><strong>Backpropagation</strong> works backward from the loss, through every layer the image passed through. It figures out exactly how much each individual weight, including every number inside every kernel, contributed to that error.</p><pre>Loss -&gt; Backpropagation -&gt; how much each filter contributed to the error</pre><h3>Gradient Descent, Updating the Filters</h3><p>Once backpropagation has calculated each filter’s contribution to the error, <strong>gradient descent</strong> nudges every one of those numbers slightly in the direction that would have reduced the loss. It is the same update rule from the gradient descent article, just applied at the scale of an entire kernel’s worth of weights at once.</p><pre>Randomly Initialized Filters<br>        |<br>        v<br>   Forward Pass<br>        |<br>        v<br>    Prediction<br>        |<br>        v<br>       Loss<br>        |<br>        v<br> Backpropagation<br>        |<br>        v<br> Gradient Descent<br>        |<br>        v<br>  Updated Filters<br>        |<br>        v<br>  Better Features</pre><p>This loop repeats, batch after batch, epoch after epoch. No one hand designs the edge detecting kernel from Part 4 inside a real, trained CNN. It emerges on its own.</p><p>A kernel that happens to respond strongly to edges turns out to reduce the loss, so <strong>gradient descent</strong> keeps nudging every filter toward whatever shape reduces the loss the most.</p><blockquote><strong><em>Nobody writes the edge detector by hand. It shows up because a random filter that happened to respond to edges reduced the loss, and gradient descent kept pulling every filter in whatever direction paid that debt down.</em></strong></blockquote><p><em>That loop produces a prediction. What that prediction actually looks like depends on how many classes the network is choosing between.</em></p><h3>Part 16: Binary vs Multiclass Classification</h3><p>Part 15 mentioned a prediction without saying exactly what that prediction looks like. The answer depends on how many classes the network needs to choose between.</p><h3>Binary Classification: Sigmoid</h3><p>For a two class problem, cat versus dog, spam versus not spam, the final layer is a single neuron with a <strong>sigmoid</strong> activation. It produces one probability between 0 and 1.</p><pre>Sigmoid output: 0.83  -&gt;  interpreted as 83% dog, 17% cat</pre><h3>Multiclass Classification: Softmax</h3><p>For a problem with more than two classes, cat, dog, or horse, a single probability is not enough. The final layer instead has one neuron per class. A <strong>softmax</strong> activation converts their raw outputs into a full set of probabilities that all add up to 1.</p><pre>Cat      0.72<br>Dog      0.18<br>Horse    0.10<br>        ------<br>Total    1.00</pre><p>The class with the highest probability, cat in this example, becomes the model’s prediction.</p><h3>The Matching Loss Function</h3><p>Each of these pairs with its own loss function. Both are variations on the same underlying idea, <strong>cross entropy</strong>, which measures the gap between the predicted probabilities and the true label.</p><pre>Binary classification     -&gt; Sigmoid  -&gt; Binary Crossentropy<br>Multiclass classification -&gt; Softmax  -&gt; Categorical Crossentropy</pre><h3>Part 17: Data Augmentation, Manufacturing More Variation</h3><p><strong>Data augmentation</strong> means artificially modifying training images to create more variation, without collecting a single new photo.</p><pre>Data augmentation = artificial modification of existing training data</pre><p>Common techniques include <strong>flipping</strong> an image horizontally so it is mirrored, <strong>rotating</strong> it slightly so it appears at a small angle, <strong>zooming</strong> in or out, <strong>cropping</strong>, and <strong>adjusting brightness</strong>. Applied during training, these transformations expose the network to far more <strong>variation</strong> than the original dataset contains on its own.</p><p>It is important to understand how this usually works in practice. <strong>Data augmentation is commonly applied randomly, on the fly, each time an image is used during training.</strong> The augmented images are not necessarily created as a fixed collection of extra copies in advance.</p><p>A given image might be <strong>flipped</strong> during one training pass and <strong>rotated</strong> during another. The dataset does not necessarily grow into a permanently larger collection of stored images. Instead, the model sees a <strong>different random variation</strong> of the same image each time it encounters it.</p><p>Augmentation should be applied only to the <strong>training data</strong>, not to the <strong>validation or test data</strong>. The goal is to make the model more <strong>robust to natural variation</strong> during learning while keeping validation and test data representative of the real-world data the model will encounter.</p><p><em>With the theory covered, the next step is to turn everything into an actual </em><strong><em>training run</em></strong><em>.</em></p><h3>Part 18: Setting Up a Real Training Run</h3><p>With the theory in place, an actual project comes down to a handful of practical settings, and one structural decision that is easy to get wrong: how the data gets split.</p><pre>Training set   -&gt; what the model actually learns from<br>Validation set -&gt; used during training to tune and choose the model<br>Test set       -&gt; touched exactly once, at the very end, for a final honest score</pre><p>Reusing the <strong>test set</strong> as if it were a <strong>validation set</strong>, checking it repeatedly while still making decisions about the model, quietly lets that data influence those decisions. That makes the final reported number less trustworthy than it looks.</p><pre>train_dir = &quot;path/to/training/images&quot;<br>val_dir = &quot;path/to/validation/images&quot;<br>test_dir = &quot;path/to/test/images&quot;Batch size controls how many images pass through the network before its weights update once, commonly a number like 32. Image size fixes every incoming photo to the same width and height, regardless of its original resolution, since the network needs a consistent input shape. Epochs counts how many complete passes through the entire training dataset the model makes.<br><br>batch_size = 32<br>image_size = (150, 150)<br>epochs = 20</pre><h3>Part 19: A Complete Python Implementation</h3><p>Putting every piece from this article together, here is a small but complete CNN, from loading data to making a prediction.</p><pre>import tensorflow as tf<br>from tensorflow import keras<br>from tensorflow.keras import layers<br><br># 1. Load the dataset<br>train_ds = tf.keras.utils.image_dataset_from_directory(<br>    &quot;data/train&quot;, image_size=(150, 150), batch_size=32<br>)<br>val_ds = tf.keras.utils.image_dataset_from_directory(<br>    &quot;data/val&quot;, image_size=(150, 150), batch_size=32<br>)<br>test_ds = tf.keras.utils.image_dataset_from_directory(<br>    &quot;data/test&quot;, image_size=(150, 150), batch_size=32, shuffle=False<br>)<br># 2. Preprocess: augmentation and rescaling, built into the model itself<br>data_augmentation = keras.Sequential([<br>    layers.RandomFlip(&quot;horizontal&quot;),<br>    layers.RandomRotation(0.1),<br>    layers.RandomZoom(0.1),<br>])<br># 3. Build the CNN<br>model = keras.Sequential([<br>    layers.Input(shape=(150, 150, 3)),<br>    data_augmentation,<br>    layers.Rescaling(1.0 / 255),<br>    layers.Conv2D(32, (3, 3), activation=&quot;relu&quot;),<br>    layers.MaxPooling2D(2, 2),<br>    layers.Conv2D(64, (3, 3), activation=&quot;relu&quot;),<br>    layers.MaxPooling2D(2, 2),<br>    layers.Flatten(),<br>    layers.Dense(128, activation=&quot;relu&quot;),<br>    layers.Dense(1, activation=&quot;sigmoid&quot;),   # binary classification<br>])<br># 4. Compile<br>model.compile(<br>    optimizer=&quot;adam&quot;,<br>    loss=&quot;binary_crossentropy&quot;,<br>    metrics=[&quot;accuracy&quot;],<br>)<br># 5. Train<br>history = model.fit(<br>    train_ds,<br>    validation_data=val_ds,<br>    epochs=20,<br>)<br># 6. Evaluate, once, on the untouched test set<br>test_loss, test_accuracy = model.evaluate(test_ds)<br>print(f&quot;Test accuracy: {test_accuracy:.4f}&quot;)<br># 7. Predict on a single new image<br>img = tf.keras.utils.load_img(&quot;new_photo.jpg&quot;, target_size=(150, 150))<br>img_array = tf.keras.utils.img_to_array(img)<br>img_array = tf.expand_dims(img_array, 0)<br>prediction = model.predict(img_array)[0][0]<br>label = &quot;dog&quot; if prediction &gt; 0.5 else &quot;cat&quot;<br>print(f&quot;Prediction: {label} ({prediction:.2f})&quot;)</pre><p>Every line here maps back to a part of this article. <strong>Augmentation</strong> comes from <strong>Part 17</strong>. <strong>Convolution and pooling</strong> come from <strong>Parts 5 through 13</strong>. <strong>Flatten and dense</strong> come from <strong>Part 14</strong>. <strong>Sigmoid</strong> comes from <strong>Part 16</strong>. The <strong>compile and fit step</strong> triggers exactly the <strong>forward pass, loss, backpropagation, and gradient descent loop</strong> from <strong>Part 15</strong>, repeated automatically for every <strong>batch and epoch</strong>.</p><h3>Part 20: Evaluating a Trained Model</h3><p>Accuracy alone rarely tells the full story. This is especially true if the two classes are not perfectly balanced, or if certain kinds of mistakes matter more than others.</p><pre>Accuracy  = correct predictions / total predictions<br>Precision = of everything predicted positive, how much was actually positive<br>Recall    = of everything actually positive, how much did the model catch</pre><p>A <strong>confusion matrix</strong> lays out every combination of <strong>predicted versus actual label</strong> in a single table. It is usually the fastest way to spot whether a model has a bias toward one class, exactly the kind of asymmetry a plain accuracy score can hide.</p><p><em>That covers every idea in this guide. Everything below is reference — the formulas and terms worth bookmarking.</em></p><h3>Key Formulas</h3><p><strong>Output Size Without Padding</strong></p><pre>Output size = (n - k + 1) x (n - k + 1)</pre><p><strong>Output Size With Padding and Stride</strong></p><pre>Output size = floor( (n - k + 2p) / s ) + 1</pre><p><strong>Vertical Edge Kernel</strong></p><pre>-1   0   +1<br>-1   0   +1<br>-1   0   +1</pre><p><strong>Horizontal Edge Kernel</strong></p><pre>-1   -1   -1<br> 0    0    0<br>+1   +1   +1</pre><p><strong>Convolutional Layer Parameter Count</strong></p><pre>Parameters = (kernel height x kernel width x input channels x number of filters) + biases</pre><p><strong>ReLU Activation</strong></p><pre>ReLU(x) = max(0, x)</pre><p><strong>Max Pooling</strong></p><pre>Output = the largest value inside each pooling window</pre><p><strong>Sigmoid, for Binary Classification</strong></p><pre>Sigmoid(x) = 1 / (1 + e⁻ˣ)</pre><p><strong>Softmax, for Multiclass Classification</strong></p><pre>Softmax(xᵢ) = eˣⁱ / ∑ⱼ eˣʲ</pre><h3>Complete Summary Table</h3><pre>| Concept             | What It Is                                                 | What It Does                                                            |<br>| ------------------- | ---------------------------------------------------------- | ----------------------------------------------------------------------- |<br>| Local connectivity  | Looking at small neighborhoods instead of the whole image  | Reflects that nearby pixels are more related than distant ones          |<br>| Parameter sharing   | Reusing the same kernel at every position                  | The core reason CNNs need far fewer weights than fully connected layers |<br>| Kernel              | A small matrix of learnable weights                        | Detects a pattern learned from data rather than hand-designed           |<br>| Convolution         | Sliding a kernel across an image                           | Produces a feature map by computing a dot product at every position     |<br>| Padding             | Extra pixels, usually zeros, added around the image border | Prevents the image from shrinking on every convolution                  |<br>| Stride              | How many pixels the kernel moves per step                  | Controls how coarsely the image gets scanned                            |<br>| Multiple filters    | Applying many kernels to the same input                    | Produces multiple feature maps stacked into the output depth            |<br>| Parameter count     | Kernel size × input channels × filters, plus biases        | Shows why CNNs use far fewer weights than fully connected layers        |<br>| ReLU                | Activation function, max(0, x)                             | Keeps positive values and zeroes out negative ones                      |<br>| Receptive field     | How much of the original image a neuron is influenced by   | Grows with depth, explaining edges → shapes → objects                   |<br>| Pooling             | Shrinking a feature map by summarizing small regions       | Reduces size while keeping the strongest or average signal              |<br>| Flatten             | Reshaping a grid into a single list                        | Prepares convolutional output for a dense layer                         |<br>| Forward pass        | Running an image through the network                       | Produces a prediction                                                   |<br>| Loss function       | Measures how wrong a prediction is                         | The quantity gradient descent tries to minimize                         |<br>| Backpropagation     | Works backward from the loss                               | Calculates each weight’s contribution to the error                      |<br>| Gradient descent    | Updates every weight, including kernels                    | Turns random filters into useful ones                                   |<br>| Sigmoid vs. softmax | Binary vs. multiclass output activation                    | Produces one probability or a full distribution across classes          |<br>| Data augmentation   | Randomly modifying training images                         | Exposes the model to more variation without new data                    |<br>| Validation set      | Held-out data used during training                         | Used to tune and choose the model, not for the final score              |<br>| Confusion matrix    | Full breakdown of predictions by class                     | Reveals biases and errors that an accuracy score alone can hide         |</pre><h3>Final Thoughts</h3><p>Every prediction a <strong>CNN</strong> makes starts with <strong>weights</strong>. Every improvement comes from the same fundamental process: <strong>measure the error, calculate how each parameter contributed to it, and update the parameters to reduce the loss</strong>.</p><p>That is the foundation of <strong>backpropagation and gradient descent</strong>. The same learning process that updates a simple neural network also updates the thousands or millions of parameters inside a CNN. The difference is that CNNs organize those parameters around the <strong>spatial structure of images</strong>, allowing small filters to learn useful visual patterns.</p><p>You have now followed that process from <strong>pixels and kernels to convolution, feature maps, pooling, backpropagation, and working Python code</strong>. Understanding these foundations makes more advanced computer vision architectures much easier to understand.</p><p>A CNN may look complicated when you first see the architecture, but underneath all those layers, the core idea remains surprisingly simple:</p><p><strong>Look at the image → detect patterns → make a prediction → measure the error → update the weights → repeat.</strong></p><p>That simple loop is what turns random parameters into a model capable of recognizing increasingly complex visual patterns.</p><h3>What Comes Next</h3><p>This article focused on understanding <strong>how CNNs actually see and learn from images</strong>, from individual pixels and small kernels to feature maps, receptive fields, and final predictions.</p><p>But CNNs are only the beginning of modern computer vision.</p><p>The next step is to go deeper into how these networks can be <strong>improved, optimized, and applied to real world problems</strong>. We can move beyond a basic CNN and explore techniques such as <strong>transfer learning, regularization, batch normalization, dropout, deeper architectures, and modern computer vision models</strong>.</p><p>The <strong>mathematics stays the same</strong>. The <strong>architecture gets more powerful</strong>.</p><p>If this guide helped you understand <strong>CNNs, convolution, kernels, feature maps, pooling, backpropagation, and gradient descent</strong> more clearly, consider sharing it with someone learning <strong>Machine Learning, Deep Learning, Computer Vision, or AI</strong>.</p><p>I’d also love to hear your thoughts. <strong>Which concept finally made CNNs click for you?</strong> Was it <strong>understanding pixels as numbers</strong>, <strong>seeing how a kernel detects an edge</strong>, <strong>watching convolution create a feature map</strong>, <strong>understanding receptive fields</strong>, or seeing how <strong>gradient descent learns the filters automatically</strong>?</p><p>Follow me for more content on <strong>Python, Mathematics for Machine Learning, Data Science, Machine Learning, Deep Learning, Computer Vision, and AI</strong>.</p><p>📌 <strong>LinkedIn:</strong> <a href="https://www.linkedin.com/in/subinthapa/">Subin Thapa<br></a>📌 <strong>GitHub:</strong> <a href="https://github.com/Subinthapa2092">Subinthapa2092</a><br>📌 <strong>Youtube:</strong> <a href="https://www.youtube.com/@codingwithsubin">Coding With Subin</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db3a0875d20a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Neural Networks Actually Learn: Gradient Descent, Backpropagation, the Chain Rule, and SGD]]></title>
            <link>https://medium.com/@subinthapa2092/how-neural-networks-actually-learn-gradient-descent-backpropagation-the-chain-rule-and-sgd-9a3661708902?source=rss-9f567b128564------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a3661708902</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Subinthapa]]></dc:creator>
            <pubDate>Mon, 31 Aug 2026 10:46:56 GMT</pubDate>
            <atom:updated>2026-08-31T10:46:56.628Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jPXR8Cf7-6EmbXYsI4HEsQ.png" /></figure><h4>A Complete Guide From First Principles to Python Implementation</h4><p>Every time you use <strong>Google Translate</strong>, every time <strong>Spotify recommends a song</strong>, every time your phone <strong>unlocks with your face</strong>, a <strong>neural network</strong> made a prediction. But before any of those networks could predict anything useful, they had to <strong>learn</strong>. And the way they learned is through one repeated loop: <strong>make a prediction, measure how wrong it was, figure out which parameters caused the error, and adjust those parameters in the direction that reduces the error.</strong></p><p>That loop has a name. It is called <strong>training</strong>. And the mathematics that makes it work are what this article is about.</p><p>This explanation builds from <strong>a single number</strong> all the way to <strong>a complete Python implementation</strong>. Every formula is applied to the same example throughout, so you can check every calculation yourself.</p><h3>1. The Basic Model</h3><p>We start with the simplest possible model: <strong>a single linear equation</strong>.</p><pre>ŷ = wx + b</pre><p>Reading each symbol:</p><pre>| Symbol | Meaning                                                           |<br>| ------ | ----------------------------------------------------------------- |<br>| x      | Input, the data point we give the model                           |<br>| w      | Weight, controls how strongly x influences the prediction         |<br>| b      | Bias, shifts the prediction up or down regardless of x            |<br>| ŷ      | Predicted output (y-hat)                                          |<br>| y      | Actual target value we want the model to predict                  |</pre><p>For our example throughout this entire article:</p><pre>x = 2    (input)<br>y = 5    (actual target)<br>w = 1    (starting weight, chosen randomly)<br>b = 0    (starting bias, chosen randomly)<br>η = 0.1  (learning rate)</pre><p>First prediction:</p><pre>ŷ = wx + b<br>  = 1(2) + 0<br>  = 2</pre><p>The model predicted 2. The true answer is 5. The model is <strong>wrong by 3</strong>. But how wrong, precisely, and how do we make it less wrong? That is what the <strong>loss function</strong> measures.</p><h3>2. The Loss Function</h3><p>The loss tells the model exactly <strong>how wrong</strong> its prediction was.</p><pre>L = (ŷ - y)²</pre><p>Reading each symbol:</p><pre>| Symbol   | Meaning                                                      |<br>| -------- | ------------------------------------------------------------ |<br>| L        | Loss, a single number measuring prediction error             |<br>| ŷ        | Predicted value                                              |<br>| y        | Actual target value                                          |<br>| (ŷ − y)² | Squared error, always positive, larger errors penalized more |</pre><p>Applying to our example:</p><pre>L = (2 - 5)²<br>  = (-3)²<br>  = 9</pre><p>The loss is 9. A perfect prediction would give loss <strong>0</strong>. The higher the loss, the worse the model.</p><p>The goal of the entire training process is to <strong>minimize this loss</strong>, to find values of w and b that make L as small as possible.</p><h3>3. Derivatives: Measuring How Change Propagates</h3><p>To minimize the loss, the model needs to know: if I change w slightly, does the loss go up or down? And by how much?</p><p>That question is answered by the <strong>derivative</strong>.</p><p>A derivative measures the <strong>rate of change</strong>: how much does the output change when the input changes by a tiny amount?</p><pre>dy/dx = rate of change of y with respect to x</pre><p>For our loss function, we want to know how much the loss changes when we change each parameter.</p><h3>Partial Derivatives</h3><p>A neural network has many parameters: w, b, and in deeper networks w1, w2, w3, b1, b2, and so on. A <strong>partial derivative</strong> measures the effect of changing one parameter while holding all others fixed.</p><pre>∂L/∂w = how much does the loss change when w changes?<br>∂L/∂b = how much does the loss change when b changes?</pre><p>The symbol ∂ (del) instead of d signals that we are holding other variables constant. This is the <strong>core mathematical tool of backpropagation</strong>.</p><h3>4. The Gradient</h3><p>The <strong>gradient</strong> is the collection of all partial derivatives of the loss with respect to every parameter.</p><pre>∇L = [∂L/∂w, ∂L/∂b]</pre><p>For a model with many parameters:</p><pre>∇L = [∂L/∂w1, ∂L/∂w2, ..., ∂L/∂b]</pre><p>The gradient has two important properties:</p><p><strong>Direction:</strong> the gradient points in the direction of steepest increase of the loss. The negative gradient points in the direction of steepest decrease.</p><p><strong>Magnitude:</strong> a large gradient means the loss is changing rapidly with respect to that parameter. A small gradient means the parameter has little effect on the loss right now.</p><pre>Gradient &gt; 0  →  increasing w increases loss<br>Gradient &lt; 0  →  increasing w decreases loss<br>Gradient = 0  →  changing w does not change loss (at a minimum or flat region)</pre><h3>5. Why Gradient Descent?</h3><p>The loss function is like a <strong>hilly landscape</strong>. Every combination of w and b gives a different loss value, a different point on the landscape. The <strong>global minimum</strong> of the landscape is where loss is smallest, which is where we want to end up.</p><p><strong>Gradient descent</strong> is the algorithm that walks this landscape. At every step it asks: which direction is downhill from where I am standing right now? It takes a small step in that direction, then asks again.</p><p>Because the gradient points <strong>uphill</strong>, moving in the <strong>negative gradient direction</strong> moves downhill.</p><pre>New position = current position - step size × gradient</pre><p>Applied to model parameters:</p><pre>w_new = w_old - η × ∂L/∂w<br>b_new = b_old - η × ∂L/∂b</pre><p>The <strong>minus sign</strong> is the entire logic of gradient descent. We subtract the gradient because the gradient points toward increasing loss, and we want decreasing loss.</p><h3>Learning Rate</h3><p>The <strong>learning rate η</strong> controls the size of each step.</p><pre>η too small  →  model learns very slowly, many steps needed<br>η too large  →  model overshoots the minimum, loss may increase<br>η just right →  model converges steadily toward minimum loss</pre><p>Common starting values: 0.1, 0.01, 0.001. For our example we use η = 0.1.</p><h3>6. The Chain Rule: Connecting Parameters to Loss</h3><p>Here is the problem: w does not directly affect L. w affects ŷ, and ŷ affects L. The <strong>chain of influence</strong> is:</p><pre>w → ŷ → L</pre><p>To find ∂L/∂w, we need to trace how a change in w propagates through ŷ and eventually changes L. This is exactly what the <strong>chain rule</strong> does.</p><pre>∂L/∂w = ∂L/∂ŷ × ∂ŷ/∂w</pre><p>Reading it in plain English: how much does w affect L equals how much does ŷ affect L multiplied by how much does w affect ŷ.</p><p>Similarly for bias:</p><pre>∂L/∂b = ∂L/∂ŷ × ∂ŷ/∂b</pre><p>There is <strong>no minus sign</strong> in the chain rule itself. The minus sign only appears in the gradient descent update formula, where we subtract the gradient.</p><h3>Calculating Each Part</h3><p><strong>Part 1: How does the prediction affect the loss?</strong></p><pre>L = (ŷ - y)²<br>∂L/∂ŷ = 2(ŷ - y)</pre><p>This is the standard power rule derivative. The 2 comes from bringing the exponent down.</p><p>Applying to our example where ŷ = 2 and y = 5:</p><pre>∂L/∂ŷ = 2(2 - 5) = 2(-3) = -6</pre><p><strong>Part 2: How does the weight affect the prediction?</strong></p><pre>ŷ = wx + b<br><br>∂ŷ/∂w = x</pre><p>When we differentiate wx+b with respect to w, treating b as a constant, we get x. The bias term disappears because it does not contain w.</p><p>Applying to our example where x=2:</p><pre>∂ŷ/∂w = x = 2</pre><p><strong>Part 3: How does the bias affect the prediction?</strong></p><pre>ŷ = wx + b<br><br>∂ŷ/∂b = 1</pre><p>When we differentiate wx+b with respect to b, treating w and x as constants, we get 1.</p><pre>∂ŷ/∂b = 1</pre><h3>7. What Is Backpropagation?</h3><p><strong>Backpropagation</strong> is the algorithm that applies the chain rule systematically to calculate the gradient of the loss with respect to every parameter in the network.</p><p>In the <strong>forward pass</strong>, information flows from input to output:</p><pre>x  →  ŷ = wx + b  →  L = (ŷ - y)²</pre><p>In the <strong>backward pass</strong> (backpropagation), the error flows backward from loss to parameters:</p><pre>L  →  ∂L/∂ŷ  →  ∂L/∂w  and  ∂L/∂b</pre><p>Backpropagation and gradient descent are <strong>two different things</strong> that work together. <strong>Backpropagation calculates the gradients.</strong> <strong>Gradient descent uses those gradients to update the parameters.</strong></p><pre>Backpropagation  →  calculates gradients<br>Gradient Descent →  uses gradients to update w and b</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tK4s5Di54UcYngcjgzFrpQ.png" /><figcaption><strong>Figure 1. Forward and backward passes in backpropagation</strong></figcaption></figure><h3>8. The Complete Mathematical Example</h3><p>Using x = 2, y = 5, w = 1, b = 0, η = 0.1. Every step calculated explicitly.</p><p><strong>Step 1: Forward Pass: Prediction</strong></p><pre>ŷ = wx + b<br>  = 1(2) + 0<br>  = 2</pre><p><strong>Step 2: Forward Pass: Loss</strong></p><pre>L = (ŷ - y)²<br>  = (2 - 5)²<br>  = 9</pre><p><strong>Step 3: Backward Pass: Gradient of Loss With Respect to Prediction</strong></p><pre>∂L/∂ŷ = 2(ŷ - y)<br>       = 2(2 - 5)<br>       = -6</pre><p><strong>Step 4: Backward Pass: Gradient of Weight</strong></p><pre>∂L/∂w = ∂L/∂ŷ × ∂ŷ/∂w<br>       = (-6) × x<br>       = (-6) × 2<br>       = -12</pre><p><strong>Step 5: Backward Pass: Gradient of Bias</strong></p><pre>∂L/∂b = ∂L/∂ŷ × ∂ŷ/∂b<br>       = (-6) × 1<br>       = -6</pre><p><strong>Step 6: Update Weight</strong></p><pre>w_new = w_old - η × ∂L/∂w<br>      = 1 - 0.1 × (-12)<br>      = 1 + 1.2<br>      = 2.2</pre><p>The gradient was <strong>negative (-12)</strong>, so subtracting a negative number <strong>increases w</strong>. The model needed a larger w to predict closer to 5.</p><p><strong>Step 7: Update Bias</strong></p><pre>b_new = b_old - η × ∂L/∂b<br>      = 0 - 0.1 × (-6)<br>      = 0 + 0.6<br>      = 0.6</pre><p><strong>Step 8: New Prediction</strong></p><pre>ŷ_new = w_new × x + b_new<br>      = 2.2 × 2 + 0.6<br>      = 4.4 + 0.6<br>      = 5.0</pre><p><strong>Step 9: New Loss</strong></p><pre>L_new = (5.0 - 5)²<br>      = 0</pre><p>In this simple single-example case, <strong>one update was enough</strong> to reach the target exactly. With a real dataset of many examples, <strong>many updates across many epochs</strong> are needed.</p><h3>9. From Gradient Descent to SGD</h3><p>In the example above, we used one training example and performed one update. This is the foundation of <strong>Stochastic Gradient Descent</strong>.</p><p>But first, understand the <strong>three variants</strong>.</p><p><strong>Batch Gradient Descent</strong></p><p>Uses the <strong>entire dataset</strong> to calculate the gradient before making a single update.</p><pre>All training examples<br>         ↓<br>   Calculate gradient<br>   (average over all)<br>         ↓<br>   One parameter update</pre><p><strong>Stable but extremely slow</strong> for large datasets. With 1 million examples, you compute 1 million predictions and losses before updating w even once.</p><p><strong>Stochastic Gradient Descent (SGD)</strong></p><p>Uses <strong>one randomly selected training example</strong> per update.</p><pre>Example 1 → prediction → loss → gradient → update w,b<br>Example 2 → prediction → loss → gradient → update w,b<br>Example 3 → prediction → loss → gradient → update w,b</pre><p>The word stochastic means <strong>random</strong>. Examples are shuffled randomly and the model updates after each single example. <strong>Fast and frequent updates but noisy.</strong></p><p><strong>Mini-Batch Gradient Descent</strong></p><p>Uses a <strong>small batch</strong> (typically 32 or 64 examples) per update. The <strong>practical middle ground</strong> used in almost all real deep learning.</p><pre>Batch 1 (32 examples) → gradient → update<br>Batch 2 (32 examples) → gradient → update</pre><pre>| Method        | Data per Update     | Speed    | Stability   |<br>| ------------- | ------------------- | -------- | ----------- |<br>| Batch GD      | Entire dataset      | Slowest  | Most stable |<br>| SGD           | One example         | Fastest  | Noisiest    |<br>| Mini-Batch GD | Small batch (32/64) | Balanced | Balanced    |</pre><h3>Advantages of SGD</h3><p><strong>Computationally efficient for large datasets</strong></p><p>With 10 million training examples, Batch GD computes gradients using all 10 million examples before making a single update. SGD updates after <strong>every individual example</strong>, allowing the model to improve continuously instead of waiting for the entire dataset.</p><p><strong>Frequent parameter updates</strong></p><p>Because the model updates after every example, it can <strong>adapt quickly</strong>. In one epoch, SGD makes n updates instead of one. This frequent updating can be particularly useful <strong>early in training</strong>, when the model is far from a good solution.</p><p><strong>Noise can help explore the loss landscape</strong></p><p>Because each update is based on a single example rather than the average gradient across the entire dataset, SGD introduces <strong>noise into the optimization process</strong>. This noise can sometimes help the optimizer move away from <strong>poor or flat regions</strong> of the loss landscape.</p><p><strong>Memory efficient</strong></p><p>SGD processes <strong>one example at a time</strong>, requiring very little memory. This makes it practical for datasets that are too large to fit entirely into RAM.</p><h3>Disadvantages of SGD</h3><p><strong>Noisy updates</strong></p><p>Each update is based on a single example, which may not represent the entire dataset well. The resulting gradient is therefore a <strong>noisy estimate of the true gradient</strong>, and individual updates may point in less useful directions.</p><p><strong>Loss fluctuates instead of decreasing smoothly</strong></p><p>Unlike Batch GD, where the loss typically follows a smoother trajectory, SGD loss can <strong>bounce up and down</strong> between individual updates. The overall trend may be downward even though individual steps temporarily increase the loss.</p><p><strong>Can overshoot the minimum</strong></p><p>If a training example produces an unusually large gradient, the resulting parameter update can <strong>overshoot a good solution</strong>. This is one reason why choosing an appropriate learning rate is important when using SGD.</p><p><strong>Sensitive to the learning rate</strong></p><p>SGD can be <strong>highly sensitive to the learning rate</strong>. A learning rate that is too large can cause unstable updates or divergence, while one that is too small can make training unnecessarily slow.</p><p>If a training example produces an unusually large gradient, the resulting parameter update can <strong>overshoot a good solution</strong>. This is one reason why choosing an appropriate learning rate is important when using SGD.</p><p><strong>Sensitive to the learning rate</strong></p><p>SGD can be <strong>highly sensitive to the learning rate</strong>. A learning rate that is too large can cause unstable updates or divergence, while one that is too small can make training unnecessarily slow.</p><h3>10. Complete SGD Example: Multiple Steps</h3><p>Dataset:</p><pre>x    y<br>2    5<br>3    7<br>4    9<br>1    3<br>5   11</pre><p>Starting with w = 1, b = 0, η = 0.1. Each step uses the updated w and b from the previous step.</p><p><strong>SGD Step 1: Example (x = 2, y = 5)</strong></p><pre>ŷ = 1(2) + 0 = 2<br>L = (2-5)² = 9<br><br>∂L/∂ŷ = 2(2-5) = -6<br>∂L/∂w = -6 × 2 = -12<br>∂L/∂b = -6 × 1 = -6<br>w = 1 - 0.1(-12) = 2.2<br>b = 0 - 0.1(-6)  = 0.6</pre><p><strong>SGD Step 2: Example (x = 3, y = 7)</strong></p><pre>ŷ = 2.2(3) + 0.6 = 7.2<br>L = (7.2-7)² = 0.04<br><br>∂L/∂ŷ = 2(7.2-7) = 0.4<br>∂L/∂w = 0.4 × 3 = 1.2<br>∂L/∂b = 0.4 × 1 = 0.4<br>w = 2.2 - 0.1(1.2) = 2.08<br>b = 0.6 - 0.1(0.4) = 0.56</pre><p><strong>SGD Step 3: Example (x = 4, y = 9) Starting from w = 2.08, b = 0.56</strong></p><pre>ŷ = 2.08(4) + 0.56 = 8.88<br>L = (8.88-9)² = 0.0144<br><br>∂L/∂ŷ = 2(8.88-9) = -0.24<br>∂L/∂w = -0.24 × 4 = -0.96<br>∂L/∂b = -0.24 × 1 = -0.24<br>w = 2.08 - 0.1(-0.96) = 2.176<br>b = 0.56 - 0.1(-0.24) = 0.584</pre><p>Notice what is happening: loss is dropping <strong>rapidly</strong> from 9 to 0.04 to 0.014. The model is learning with <strong>every single example</strong>.</p><h3>11. Epochs and Iterations</h3><p><strong>One pass through the entire dataset = one epoch</strong></p><p>For our 5-example dataset with pure SGD:</p><pre>Epoch 1:<br>  Step 1: (x=2, y=5)  → update w,b<br>  Step 2: (x=3, y=7)  → update w,b<br>  Step 3: (x=4, y=9)  → update w,b<br>  Step 4: (x=1, y=3)  → update w,b<br>  Step 5: (x=5, y=11) → update w,b<br>  → 5 updates = 1 epoch<br><br>Epoch 2:<br>  Step 6:  (x=3, y=7)  → update w,b    ← shuffled order<br>  Step 7:  (x=5, y=11) → update w,b<br>  ...<br>  → 5 more updates = 1 epoch</pre><p><strong>Why shuffle?</strong></p><p>If the model always sees examples in the same order, it can <strong>learn patterns from the ordering</strong> rather than from the data itself. Shuffling at the start of each epoch ensures the learning process is genuinely stochastic and does not develop <strong>ordering bias</strong>.</p><p><strong>Epoch vs Iteration:</strong></p><pre>One iteration = one parameter update = one example (in pure SGD)<br>One epoch     = one full pass through the dataset<br>              = n iterations for n training examples</pre><p>With 1,000 training examples and 10 epochs:</p><pre>Total iterations = 1,000 × 10 = 10,000 parameter updates</pre><h3>12. SGD and Backpropagation in Neural Networks</h3><p>The same principle that worked for one weight and one bias extends to networks with <strong>millions of parameters</strong>. A deeper network has a longer chain:</p><pre>x → layer1 → layer2 → ... → output → loss</pre><p>Backpropagation applies the chain rule backward through every layer:</p><pre>∂L/∂w_output = ∂L/∂output × ∂output/∂w_output<br>∂L/∂w_layer2 = ∂L/∂output × ∂output/∂layer2 × ∂layer2/∂w_layer2<br>∂L/∂w_layer1 = ... × ∂layer1/∂w_layer1</pre><p>Each layer’s gradient is computed from the layer above it, which is why it goes backward. SGD then uses all of these gradients to update every weight in the network simultaneously.</p><pre>Backpropagation  →  gives us ∂L/∂w for every w in the network<br>SGD update rule  →  w ← w - η × ∂L/∂w for every w</pre><p>This is <strong>not two different algorithms</strong>. Backpropagation is the method for calculating the gradients. SGD is the method for using those gradients to update parameters. They <strong>always work together</strong>.</p><h3>13. Python Implementation From Scratch</h3><p>No PyTorch. No sklearn. <strong>Pure Python and NumPy</strong> to show that the math you just calculated by hand is exactly what the code does.</p><pre>import numpy as np<br>import matplotlib.pyplot as plt<br><br># Dataset<br>X = np.array([2, 3, 4, 1, 5], dtype=float)<br>Y = np.array([5, 7, 9, 3, 11], dtype=float)<br># Initial parameters<br>w = 1.0<br>b = 0.0<br># NOTE ON LEARNING RATE:<br># For single-feature squared-error SGD, an update stays stable roughly<br># when learning_rate * x^2 &lt; 2. The largest x in this dataset is 5, so<br># learning_rate must be well under 2/5^2 = 0.08. A value like 0.1 causes<br># the x=5 example alone to produce an oversized weight swing that<br># compounds across steps instead of damping out -- loss oscillates<br># between roughly 9 and several thousand instead of shrinking.<br># 0.05 stays comfortably under the stability threshold and converges cleanly.<br>learning_rate = 0.05<br>n_epochs = 20<br>loss_history = []<br>w_history = [w]<br>b_history = [b]<br>for epoch in range(n_epochs):<br>    # Shuffle the data at the start of each epoch<br>    indices = np.random.permutation(len(X))<br>    X_shuffled = X[indices]<br>    Y_shuffled = Y[indices]<br>    epoch_loss = 0<br>    for x, y in zip(X_shuffled, Y_shuffled):<br>        # Forward pass<br>        y_pred = w * x + b<br>        loss = (y_pred - y) ** 2<br>        # Backpropagation (chain rule)<br>        dL_dy_pred = 2 * (y_pred - y)   # ∂L/∂ŷ<br>        dL_dw = dL_dy_pred * x           # ∂L/∂w = ∂L/∂ŷ × ∂ŷ/∂w<br>        dL_db = dL_dy_pred * 1           # ∂L/∂b = ∂L/∂ŷ × ∂ŷ/∂b<br>        # SGD update<br>        w = w - learning_rate * dL_dw<br>        b = b - learning_rate * dL_db<br>        epoch_loss += loss<br>        w_history.append(w)<br>        b_history.append(b)<br>    avg_loss = epoch_loss / len(X)<br>    loss_history.append(avg_loss)<br>    print(f&quot;Epoch {epoch+1:2d} | Loss: {avg_loss:.6f} | w: {w:.4f} | b: {b:.4f}&quot;)<br>print(f&quot;\nFinal w: {w:.4f}  (true value: 2.0)&quot;)<br>print(f&quot;Final b: {b:.4f}  (true value: 1.0)&quot;)<br># Test prediction<br>x_test = 6<br>y_pred = w * x_test + b<br>print(f&quot;\nPrediction for x={x_test}: {y_pred:.4f}  (expected: 13.0)&quot;)<br># Plot loss<br>plt.figure(figsize=(12, 4))<br>plt.subplot(1, 3, 1)<br>plt.plot(loss_history, marker=&#39;o&#39;)<br>plt.title(&#39;Average Loss per Epoch&#39;)<br>plt.xlabel(&#39;Epoch&#39;)<br>plt.ylabel(&#39;Loss&#39;)<br>plt.grid(True)<br>plt.subplot(1, 3, 2)<br>plt.plot(w_history, alpha=0.7)<br>plt.axhline(y=2.0, color=&#39;r&#39;, linestyle=&#39;--&#39;, label=&#39;True w=2&#39;)<br>plt.title(&#39;Weight w over SGD Steps&#39;)<br>plt.xlabel(&#39;SGD Step&#39;)<br>plt.ylabel(&#39;w&#39;)<br>plt.legend()<br>plt.grid(True)<br>plt.subplot(1, 3, 3)<br>plt.plot(b_history, alpha=0.7)<br>plt.axhline(y=1.0, color=&#39;r&#39;, linestyle=&#39;--&#39;, label=&#39;True b=1&#39;)<br>plt.title(&#39;Bias b over SGD Steps&#39;)<br>plt.xlabel(&#39;SGD Step&#39;)<br>plt.ylabel(&#39;b&#39;)<br>plt.legend()<br>plt.grid(True)<br>plt.tight_layout()<br>plt.savefig(&#39;sgd_convergence.png&#39;, dpi=100)</pre><p><strong>Actual output:</strong></p><pre>Epoch  1 | Loss: 2.472745 | w: 2.0962 | b: 0.5427<br>Epoch  2 | Loss: 0.043160 | w: 2.0000 | b: 0.5519<br>Epoch  3 | Loss: 0.098573 | w: 1.9530 | b: 0.5685<br>Epoch  4 | Loss: 0.074227 | w: 2.0695 | b: 0.6407<br>Epoch  5 | Loss: 0.052259 | w: 2.1621 | b: 0.6939<br>...<br>Epoch 16 | Loss: 0.013085 | w: 2.0573 | b: 0.8615<br>Epoch 17 | Loss: 0.005935 | w: 2.0180 | b: 0.8736<br>Epoch 18 | Loss: 0.006555 | w: 2.0445 | b: 0.8951<br>Epoch 19 | Loss: 0.003825 | w: 2.0394 | b: 0.8980<br>Epoch 20 | Loss: 0.006591 | w: 2.0296 | b: 0.9112<br><br>Final w: 2.0296  (true value: 2.0)<br>Final b: 0.9112  (true value: 1.0)<br>Prediction for x=6: 13.0888  (expected: 13.0)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WBx99_ZL94sjWLyz0m_nBQ.png" /><figcaption><strong>Figure 2: SGD convergence as loss decreases and parameters approach their true values.</strong></figcaption></figure><p>The model recovered <strong>w ≈ 2.03</strong> and <strong>b ≈ 0.91</strong>, close to the true values <strong>w = 2</strong> and <strong>b = 1</strong>. Using SGD and backpropagation from scratch, it successfully learned the relationship <strong>y = 2x + 1</strong>.</p><h3>14. Common Mistakes</h3><p><strong>1. Forgetting the minus sign</strong><br> Use w = w − η × gradient. Using + makes the model move toward higher loss.</p><p><strong>2. Resetting w and b during SGD</strong><br> Updated parameters must carry forward from one example to the next. Resetting them destroys learning.</p><p><strong>3. Confusing backpropagation with gradient descent</strong><br> Backpropagation <strong>calculates gradients</strong>; gradient descent <strong>uses them to update parameters</strong>.</p><p><strong>4. Expecting loss to decrease every step</strong><br> SGD is noisy. Individual losses can increase temporarily; focus on the <strong>average loss per epoch</strong>.</p><p><strong>5. Confusing epochs and iterations</strong><br> One <strong>epoch</strong> is a complete pass through the dataset. One <strong>iteration</strong> is a single parameter update.</p><p><strong>6. Using a learning rate that is too large</strong><br> A large learning rate can cause unstable updates or divergence. If the loss keeps increasing, try a smaller value.</p><p><strong>7. Ignoring input scale</strong><br> Learning-rate stability depends on the magnitude of x. For squared-error SGD, keeping learning_rate × x² sufficiently small helps prevent unstable updates. <strong>Feature normalization</strong> is the standard solution in real-world models.</p><h3>Key Formulas</h3><p><strong>Model</strong></p><pre>ŷ = wx + b</pre><p><strong>Loss</strong></p><pre>L = (ŷ - y)²</pre><p><strong>Chain Rule</strong></p><pre>∂L/∂w = ∂L/∂ŷ × ∂ŷ/∂w<br>∂L/∂b = ∂L/∂ŷ × ∂ŷ/∂b</pre><p><strong>Computed Gradients</strong></p><pre>∂L/∂ŷ = 2(ŷ - y)<br>∂L/∂w = 2(ŷ - y) × x<br>∂L/∂b = 2(ŷ - y)</pre><p><strong>SGD Update</strong></p><pre>w ← w - η × ∂L/∂w<br>b ← b - η × ∂L/∂b</pre><h3>Complete Summary Table</h3><pre>| Concept            | What It Is                    | What It Does                                   |<br>| ------------------ | ----------------------------- | ---------------------------------------------- |<br>| Model ŷ = wx + b   | Linear prediction function    | Maps x to predicted output ŷ                   |<br>| Loss L = (ŷ − y)²  | Error measurement             | Quantifies prediction error                    |<br>| Derivative ∂L/∂w   | Rate of change                | Measures how w affects loss                    |<br>| Partial derivative | Derivative for one variable   | Isolates one parameter’s effect                |<br>| Gradient ∇L        | Vector of partial derivatives | Gives direction and magnitude of loss increase |<br>| Chain rule         | Connects derivatives          | Computes gradients through intermediate steps  |<br>| Forward pass       | x → ŷ → L                     | Computes prediction and loss                   |<br>| Backpropagation    | L → ∂L/∂ŷ → ∂L/∂w             | Computes gradients backward                    |<br>| Gradient descent   | w ← w − η × ∂L/∂w             | Updates parameters toward lower loss           |<br>| Learning rate η    | Step size                     | Controls update magnitude                      |<br>| SGD                | One example per update        | Provides fast, noisy updates                   |<br>| Epoch              | One full pass through data    | Contains n iterations for n examples           |<br>| Iteration          | One parameter update          | Represents one SGD step                        |<br>| Data shuffling     | Random reordering             | Reduces ordering bias during training          |</pre><h3>The Complete Mental Model</h3><pre>Training Example (x, y)<br>          ↓<br>    ŷ = wx + b          ← Forward Pass<br>          ↓<br>    L = (ŷ - y)²        ← Loss<br>          ↓<br>    ∂L/∂ŷ = 2(ŷ-y)     ← Backpropagation<br>          ↓              ← Chain Rule<br>    ∂L/∂w  and  ∂L/∂b<br>          ↓<br>    w ← w - η × ∂L/∂w  ← Gradient Descent (SGD)<br>    b ← b - η × ∂L/∂b<br>          ↓<br>    Better prediction<br>          ↓<br>        REPEAT</pre><p>And the key distinction to remember:</p><pre>Backpropagation  →  calculates the gradients<br>SGD              →  uses the gradients to update parameters</pre><p>These two always work together. Neither is complete without the other.</p><h3>Final Thoughts</h3><p>Every prediction a neural network makes starts with <strong>weights</strong>. Every improvement comes from the same fundamental process: <strong>measure the error, calculate how each parameter contributed to it, and update the parameters to reduce the loss</strong>.</p><p>That is the foundation of <strong>backpropagation and gradient descent</strong>. The <strong>chain rule</strong> makes the gradient calculation possible, while <strong>SGD</strong> makes learning practical on large datasets.</p><p>You have now followed this process from <strong>first principles to working Python code</strong>. Understanding this foundation makes the more complex architectures of deep learning much easier to understand.</p><h3>What Comes Next</h3><p>The next article moves from this foundation to <strong>CNNs</strong>, where <strong>kernels</strong> learn patterns from <strong>images</strong>.</p><p>The <strong>mathematics stays the same</strong>. The <strong>architecture gets more powerful</strong>.</p><p>If this guide helped you better understand <strong>Backpropagation</strong>, <strong>Gradient Descent</strong>, <strong>SGD</strong>, <strong>Chain Rule</strong>, and how neural networks learn from their errors, consider sharing it with someone learning <strong>Machine Learning, Deep Learning, or AI</strong>.</p><p>I’d also love to hear your thoughts. <strong>Which concept finally made the learning process click for you?</strong> Was it <strong>the chain rule</strong>, <strong>backpropagation</strong>, <strong>gradient descent</strong>, <strong>SGD</strong>, or understanding how <strong>weights and biases are updated</strong>?</p><p>Follow me for more content on <strong>Python, Mathematics for Machine Learning, Data Science, Machine Learning, Deep Learning, and AI</strong>.</p><p>📌 <strong>LinkedIn:</strong> <a href="https://www.linkedin.com/in/subinthapa/">Subin Thapa<br></a>📌 <strong>GitHub:</strong> <a href="https://github.com/Subinthapa2092">Subinthapa2092</a><br>📌 <strong>Facebook:</strong> <a href="https://www.facebook.com/profile.php?id=61590930345491">Coding With Subin</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a3661708902" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Regularization Explained: The Complete Guide to L1, L2, and Elastic Net]]></title>
            <link>https://medium.com/@subinthapa2092/regularization-explained-the-complete-guide-to-l1-l2-and-elastic-net-231b7a8a88df?source=rss-9f567b128564------2</link>
            <guid isPermaLink="false">https://medium.com/p/231b7a8a88df</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[mathematics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Subinthapa]]></dc:creator>
            <pubDate>Thu, 20 Aug 2026 12:08:39 GMT</pubDate>
            <atom:updated>2026-08-20T12:08:39.704Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IfWPCa6BQE8I63azFlQjPw.png" /></figure><h4>Why Models Overfit, What Regularization Actually Does, and How to Choose Between Lasso, Ridge, and Elastic Net</h4><p>Imagine a student preparing for an exam by <strong>memorizing past papers instead of understanding the concepts</strong>. They may perform perfectly on familiar questions but struggle when faced with new ones because they learned specific answers rather than the underlying patterns.</p><p>A machine learning model can fall into the same trap. With too much flexibility, it may <strong>memorize the training data instead of learning patterns that generalize to unseen data</strong>. <strong>Regularization helps prevent this by discouraging unnecessary model complexity.</strong></p><p>This article builds regularization from the ground up: <strong>why overfitting happens, how regularization modifies the loss function, how L1 and L2 differ mathematically and geometrically, when to use Lasso, Ridge, or Elastic Net, and how to implement and compare them in Python.</strong></p><h3>Part 1: Why Overfitting Happens</h3><h3>Three Ways a Model Can Fit Data</h3><p>When a model learns from data, there are three broad outcomes.</p><pre>Underfitting  = model is too simple, misses the pattern entirely<br>Good fit      = model captures the real pattern<br>Overfitting   = model memorizes noise along with the pattern</pre><p>An underfit model performs poorly everywhere, on the training data and on new data alike, because it never captured enough structure to make good predictions in the first place. An overfit model does the opposite: it performs almost perfectly on the data it was trained on, but poorly on anything new, because part of what it learned was noise specific to that training set rather than a pattern that generalizes.</p><h3>Training Error vs Validation Error</h3><p>The clearest signal of overfitting comes from comparing two numbers: how wrong the model is on the data it was trained on, and how wrong it is on data it has never seen.</p><pre>Training error   = error measured on the data the model learned from<br>Validation error = error measured on held out data the model never saw</pre><p>A healthy model has training error and validation error that are both low and reasonably close to each other. An overfit model has training error that keeps dropping while validation error starts climbing back up. That gap between the two is exactly what regularization is designed to close.</p><h3>Model Complexity Is the Real Culprit</h3><p>The deeper cause behind overfitting is often model complexity. A model with many parameters, or with weights that are free to grow too large, has enough flexibility to fit individual training points, including noisy or unusual ones that should have been ignored.</p><pre>Model complexity = how much freedom the model has to fit the data</pre><p>A straight line has limited flexibility. A high degree polynomial, however, can bend enough to follow nearly every training point, including random noise. This is the essence of overfitting: the model learns the noise instead of the underlying pattern.</p><p><strong>Figure 1</strong> illustrates this tradeoff. As model complexity increases, training error continues to decrease, while validation error eventually reaches a minimum and then rises. The best model lies near this minimum, complex enough to capture the real pattern but not so complex that it memorizes noise.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lcBBCAvGMx_ShCvP9A32Zw.png" /><figcaption><strong>Figure 1. Model complexity, underfitting, and overfitting.</strong></figcaption></figure><h3>Part 2: What Regularization Actually Does</h3><h3>The Core Idea</h3><p>Regularization works by discouraging a model from using unnecessarily large weights. Instead of only minimizing how wrong the predictions are, regularization adds a second goal: keep the weights small and simple, unless the data gives a strong reason not to.</p><pre>Total loss = Data loss + Regularization penalty</pre><p>The data loss term is the familiar part: it measures how far the model’s predictions are from the actual values, exactly the loss function used in ordinary linear regression. The regularization penalty is new: it grows whenever the model’s weights grow, which pushes the optimizer to prefer smaller, simpler weights whenever a smaller weight explains the data almost as well as a larger one.</p><h3>Why Penalizing Weight Size Reduces Overfitting</h3><p>Large weights are usually a sign that the model is leaning heavily on a specific feature, sometimes too heavily, to explain quirks in the training data that will not appear again in new data. By making large weights costly, regularization forces the model to only use a large weight when the evidence for it is strong across the whole dataset, not just a few unusual points.</p><pre>Large weight  → model reacts strongly to small changes in input → sensitive to noise<br>Small weight  → model reacts gently to changes in input          → more stable, generalizes better</pre><h3>The Role of Lambda</h3><p>The strength of the regularization penalty is controlled by a single number, usually written as the Greek letter lambda, λ.</p><pre>Total loss = Data loss + λ × Regularization penalty</pre><p>Lambda decides how much the model should care about keeping weights small, relative to how much it cares about fitting the training data exactly.</p><pre>λ = 0        → no regularization at all, identical to ordinary regression<br>λ small      → mild regularization, weights shrink only slightly<br>λ large      → strong regularization, weights are pushed hard toward zero<br>λ too large  → weights collapse toward zero, model underfits</pre><p>If lambda is zero, the penalty term disappears entirely and the model behaves exactly like ordinary linear regression, free to overfit if it wants to. As lambda increases, the model is increasingly pressured to keep its weights small, even if that means fitting the training data slightly worse. Push lambda too far, and the weights get squeezed so hard that the model loses the ability to fit even the real pattern, sliding from a good fit back into underfitting.</p><p>Choosing the right lambda is, in practice, the single most important decision when using L1 or L2 regularization, and Part 11 of this article covers exactly how to do that with cross validation.</p><h3>Part 3: L1 Regularization, Lasso</h3><h3>The Formula</h3><p>L1 regularization, commonly known as Lasso (Least Absolute Shrinkage and Selection Operator), adds a penalty equal to the sum of the absolute values of the weights.</p><pre>L1 Loss = Data Loss + λ Σ|w|</pre><p>Here, w represents each individual weight in the model, and the summation adds up the absolute value of every single weight. Absolute value is used specifically so that both a very negative weight and a very positive weight are penalized equally; the penalty only cares about size, not direction.</p><h3>The Mathematical Intuition</h3><p>Because the penalty term uses the absolute value of each weight, its gradient with respect to any weight is a constant: either positive one or negative one, regardless of how large or small that weight currently is.</p><pre>d|w|/dw = 1   if w &gt; 0<br>d|w|/dw = -1  if w &lt; 0</pre><p>This constant gradient is the key mathematical detail that separates L1 from L2. No matter whether a weight is large or already tiny, the L1 penalty pulls it toward zero by the same fixed amount at every step. For a weight that is already close to zero, this constant pull is often enough to push it all the way to exactly zero, and once a weight reaches zero, it can stay there.</p><h3>Why Weights Can Become Exactly Zero</h3><p>This is the property that makes Lasso genuinely different from Ridge. The L1 penalty applies a constant-magnitude force to nonzero weights, which can push small weights all the way to zero instead of merely shrinking them. Geometrically, this comes from the diamond-shaped L1 constraint, whose sharp corners lie directly on the axes. The optimal solution can therefore land on a corner, making one or more weights exactly zero.</p><p>Look at the left panel of <strong>Figure 2</strong>. The diamond represents the combinations of weights allowed by the L1 constraint, while the curved contours represent different levels of data loss. As the contours move inward, they eventually reach the constraint boundary. With the sharp corners of the diamond, this point of contact often occurs on an axis, meaning one of the weights becomes exactly zero.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x3-X9Domtkphd4nPGugMSg.png" /><figcaption><strong>Figure 2. L1 vs. L2 constraint geometry.</strong></figcaption></figure><h3>Feature Selection</h3><p>Because <strong>Lasso</strong> can push weights all the way to zero, it effectively performs feature selection as a side effect of training. Any feature whose weight becomes exactly zero is, for all practical purposes, removed from the model; its input no longer influences the prediction at all.</p><pre>weight = 0  →  feature has no effect on the prediction  →  feature is effectively removed</pre><p>This makes Lasso especially useful when a dataset has many features and you suspect that only a handful of them actually matter. Instead of manually deciding which features to drop, Lasso does it automatically as part of the optimization process.</p><h3>Advantages and Disadvantages</h3><p><strong>Lasso’s</strong> biggest strength is its ability to produce a <strong>sparse model</strong>, where many weights become exactly zero. This makes the model easier to interpret and can reduce the number of features needed for prediction. It is especially useful when working with hundreds or thousands of features, such as in genomics or text data, where identifying the most important features is valuable.</p><p>Its main limitation appears when features are highly correlated. Lasso may select one feature from a correlated group while setting the others to zero, even when those features contain useful information. As a result, its feature selection can be unstable, with small changes in the data sometimes leading to a different feature being selected.</p><h3>When to Use L1</h3><p><strong>Lasso</strong> is a strong choice when you have many features and suspect that some are irrelevant or redundant. It can automatically identify important features by shrinking some weights to <strong>exactly zero</strong>. This makes L1 especially useful when you value <strong>interpretability, feature selection, and a compact model</strong> over marginal gains in predictive accuracy.</p><h3>Part 4: L2 Regularization, Ridge</h3><h3>The Formula</h3><p>L2 regularization, commonly known as Ridge regression, adds a penalty equal to the sum of the squared values of the weights.</p><pre>L2 Loss = Data Loss + λ Σw²</pre><p>Just like <strong>L1</strong>, the summation runs over every weight in the model. The key difference is that each weight is squared before being added up, rather than having its absolute value taken.</p><h3>The Mathematical Intuition</h3><p>Squaring changes the shape of the penalty in an important way. The gradient of w² with respect to w is:</p><pre>d(w²)/dw = 2w</pre><p>Unlike <strong>L1’s</strong> constant gradient, the <strong>L2</strong> gradient is proportional to the weight itself. A large weight produces a large gradient, and therefore a strong pull toward zero. But as the weight shrinks, the gradient shrinks along with it, so the pull toward zero gets weaker and weaker the closer the weight gets to zero.</p><h3>Why Weights Shrink Toward Zero, Not To Zero</h3><p>This proportional relationship explains why <strong>L2 regularization shrinks weights smoothly rather than eliminating them</strong>. When a weight becomes smaller, the penalty on it also becomes smaller, so there is usually not enough force to push it all the way to zero. As a result, weights become <strong>smaller but typically remain nonzero</strong>.</p><p>Geometrically, the L2 constraint forms a <strong>smooth circle with no sharp corners</strong>. The loss contours usually touch this smooth boundary at a point where the weights are reduced but still nonzero. Thus, L2 encourages <strong>gradual weight shrinkage</strong> rather than automatic feature elimination.</p><pre>L1 constraint region  →  diamond, has corners   →  solution often lands on a corner   →  some weights become exactly 0<br>L2 constraint region  →  circle, no corners      →  solution lands on a smooth curve   →  weights shrink but rarely reach 0</pre><h3>Advantages and Disadvantages</h3><p><strong>Ridge’s main strength is stability</strong>, especially when features are highly correlated. Instead of arbitrarily selecting one feature from a correlated group, Ridge tends to <strong>share the weight across them</strong>, shrinking each coefficient moderately. This makes its coefficients less sensitive to small changes in the training data than Lasso’s.</p><p>Its main weakness is that <strong>Ridge does not perform feature selection</strong>. Coefficients are usually reduced rather than set exactly to zero, so the model remains <strong>dense</strong> and continues to use all features. If your goal is to identify the most important features and eliminate the rest, <strong>Ridge alone is not the right tool</strong>.</p><h3>When to Use L2</h3><p>Ridge is the right choice when you expect most or all of your features to carry at least some genuine signal, when your features are correlated with each other, or when you simply want to reduce overfitting without changing which features the model uses at all.</p><h3>Part 5: L1 vs L2, The Key Difference</h3><p>The table below summarizes the core distinction between the two approaches, all of which traces back to the single geometric difference shown in <strong>Figure 2:</strong> a diamond has corners, a circle does not.</p><pre>| Aspect              | L1 (Lasso)                 | L2 (Ridge)                    |<br>| ------------------- | -------------------------- | ----------------------------- |<br>| Penalty             | λΣ|w|                      | λΣw²                          |<br>| Gradient            | Constant                   | Proportional to w             |<br>| Weights             | Can become exactly zero    | Shrinks toward zero           |<br>| Feature Selection   | Yes                        | No                            |<br>| Model               | Sparse                     | Dense                         |<br>| Correlated Features | Less stable                | More stable                   |<br>| Constraint Region   | Diamond with sharp corners | Circle with a smooth boundary |</pre><h3>Part 6: Elastic Net</h3><h3>Why Combine L1 and L2</h3><p><strong>Lasso and Ridge</strong> each solve a different half of the same problem. <strong>Lasso</strong> is excellent at cutting irrelevant features but becomes unstable when features are correlated. <strong>Ridge</strong> is stable with correlated features but never actually removes any of them. Elastic Net exists to get the useful part of both behaviors in a single model.</p><h3>The Formula</h3><p>Elastic Net adds both penalties to the loss function at the same time, each with its own weighting.</p><pre>Elastic Net Loss = Data Loss + λ1 Σ|w| + λ2 Σw²</pre><p>A common way to express this is with a single overall regularization strength λ and a mixing parameter, often written as α (alpha) or r, that controls the balance between the L1 and L2 portions of the penalty.</p><pre>Elastic Net Loss = Data Loss + λ [ r Σ|w| + (1 - r) Σw² ]</pre><p>When r is 1, Elastic Net behaves exactly like pure Lasso. When r is 0, it behaves exactly like pure Ridge. Anywhere in between, it blends both effects.</p><h3>How It Works</h3><p>The L1 portion still gives Elastic Net the ability to push some weights to exactly zero, preserving feature selection. The L2 portion adds back the stability that pure Lasso lacks: when a group of features is correlated, the L2 term encourages the model to spread weight across the whole group instead of arbitrarily keeping one and dropping the rest.</p><h3>When Elastic Net Is Useful</h3><p><strong>Elastic Net</strong> is especially useful for datasets with <strong>many features and strong correlations between them</strong>. It combines L1’s feature selection with L2’s stability, making it a strong alternative when plain Lasso may behave unpredictably.</p><p>It works well for <strong>genomic data, text data with overlapping vocabulary</strong>, and other datasets where groups of features tend to move together.</p><h3>Lasso vs Ridge vs Elastic Net</h3><pre>Lasso        → best when you want automatic feature selection and features are mostly independent<br>Ridge        → best when most features matter and some are correlated with each other<br>Elastic Net  → best when you want feature selection AND you have correlated features</pre><h3>Part 7: The Mathematical Intuition, Revisited</h3><p>It helps to see all three loss functions side by side, since the only thing that changes between them is the penalty term added to the same underlying data loss.</p><pre>No regularization   →  Loss = Data Loss<br>L1 (Lasso)          →  Loss = Data Loss + λ Σ|w|<br>L2 (Ridge)          →  Loss = Data Loss + λ Σw²<br>Elastic Net         →  Loss = Data Loss + λ [ r Σ|w| + (1 - r) Σw² ]</pre><p>Without regularization, gradient descent focuses only on <strong>reducing prediction error</strong>. It does not directly penalize large model weights as long as they improve the fit to the training data.</p><p>Adding a regularization penalty changes the optimization objective. Gradient descent now balances two goals: <strong>minimizing prediction error</strong> and <strong>keeping weights small</strong>. The type of penalty determines how this balance affects the weights, which is why L1 and L2 behave differently.</p><p>The key difference lies in their gradients. <strong>L1 applies a constant pull toward zero</strong>, allowing weights to reach exactly zero. <strong>L2 applies a pull proportional to the weight</strong>, so its effect weakens as the weight approaches zero. As a result, L2 typically <strong>shrinks weights without eliminating them</strong>.</p><h3>Part 8: Python Implementation</h3><p>The example below fits four models, ordinary Linear Regression, Ridge, Lasso, and Elastic Net, on the same small synthetic dataset, so their behavior can be compared directly.</p><pre>import numpy as np<br>from sklearn.linear_model import LinearRegression, Ridge, Lasso, ElasticNet<br>from sklearn.model_selection import train_test_split<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.metrics import mean_squared_error<br><br># Create a synthetic dataset with some genuinely useful features<br># and some irrelevant, noisy features mixed in<br>rng = np.random.RandomState(42)<br>n_samples, n_features = 200, 10<br><br>X = rng.randn(n_samples, n_features)<br>true_weights = np.array([5, -3, 0, 0, 2, 0, 0, 0, 4, 0])<br>y = X @ true_weights + rng.randn(n_samples) * 2<br><br>X_train, X_test, y_train, y_test = train_test_split(<br>    X, y, test_size=0.25, random_state=42<br>)<br><br># Scale features before applying L1 or L2 regularization<br>scaler = StandardScaler()<br>X_train_scaled = scaler.fit_transform(X_train)<br>X_test_scaled = scaler.transform(X_test)<br><br>models = {<br>    &quot;Linear Regression&quot;: LinearRegression(),<br>    &quot;Ridge (L2)&quot;: Ridge(alpha=1.0),<br>    &quot;Lasso (L1)&quot;: Lasso(alpha=0.1),<br>    &quot;Elastic Net&quot;: ElasticNet(alpha=0.1, l1_ratio=0.5),<br>}<br><br>results = {}<br>for name, model in models.items():<br>    model.fit(X_train_scaled, y_train)<br>    train_pred = model.predict(X_train_scaled)<br>    test_pred = model.predict(X_test_scaled)<br>    results[name] = {<br>        &quot;train_error&quot;: mean_squared_error(y_train, train_pred),<br>        &quot;test_error&quot;: mean_squared_error(y_test, test_pred),<br>        &quot;coefficients&quot;: model.coef_,<br>    }</pre><p>Note that alpha here is scikit-learn’s name for the regularization strength, the same quantity referred to as lambda throughout this article. For Elastic Net, l1_ratio plays the role of r from Part 6, controlling the balance between the L1 and L2 penalties.</p><h3>Part 9: Practical Comparison</h3><p>Running the code on the synthetic dataset produces results similar to the table below. The true weights are <strong>5, −3, 0, 0, 2, 0, 0, 0, 4, 0</strong>, so only <strong>4 of the 10 features</strong> actually matter.</p><pre>| Model             | Training Error   | Test Error             | Nonzero Coefficients |<br>| ----------------- | ---------------- | ---------------------- | -------------------: |<br>| Linear Regression | Lowest           | Highest                |              10 / 10 |<br>| Ridge (L2)        | Slightly higher  | Lower than Linear      |              10 / 10 |<br>| Lasso (L1)        | Higher           | Lowest or near-lowest  |              ~4 / 10 |<br>| Elastic Net       | Similar to Lasso | Competitive with Lasso |            ~5–6 / 10 |</pre><p>Plain Linear Regression achieves the lowest training error because it has <strong>no regularization penalty</strong>. However, its low training error does not translate to the test set. Its extra flexibility can cause it to fit noise rather than the underlying signal.</p><p>Ridge reduces test error by <strong>shrinking every coefficient</strong>, but it keeps all ten features active, including the six irrelevant ones. Lasso goes further by driving many irrelevant coefficients <strong>exactly to zero</strong>, producing a smaller and more interpretable model based largely on the four features that actually matter.</p><p>Elastic Net falls between the two. It retains much of <strong>Lasso’s feature-selection ability</strong> while providing additional stability when features are correlated.</p><p>A side-by-side coefficient plot makes this pattern easy to see: Linear Regression and Ridge retain nonzero coefficients across all features, while <strong>Lasso and Elastic Net drive several irrelevant-feature coefficients to zero</strong>.</p><h3>Part 10: How to Choose Lambda</h3><h3>The Basic Tradeoff</h3><pre>Small λ  →  weak regularization    →  closer to the original overfitting problem<br>Large λ  →  strong regularization  →  risk of underfitting if pushed too far</pre><p>There is no universally correct value for lambda; the right value depends entirely on the dataset. The only reliable way to find a good value is to test a range of candidates and measure which one generalizes best.</p><h3>Cross Validation</h3><p><strong>Cross-validation</strong> is the standard way to choose the regularization strength. Instead of relying on a single train-validation split, the training data is divided into several folds. Each candidate value of λ is evaluated across these folds, with every fold taking a turn as the validation set.</p><p>The value that achieves the <strong>best average validation performance</strong> is then selected. This gives a more reliable estimate than depending on one particular split.</p><p>Scikit-learn provides built-in versions of Ridge, Lasso, and Elastic Net that can perform this search automatically.</p><pre>from sklearn.linear_model import RidgeCV, LassoCV, ElasticNetCV<br><br>alphas = [0.001, 0.01, 0.1, 1.0, 10.0, 100.0]<br><br>ridge_cv = RidgeCV(alphas=alphas, cv=5)<br>lasso_cv = LassoCV(alphas=alphas, cv=5)<br>elastic_cv = ElasticNetCV(<br>    alphas=alphas,<br>    l1_ratio=[0.1, 0.5, 0.7, 0.9],<br>    cv=5<br>)<br><br>ridge_cv.fit(X_train_scaled, y_train)<br>lasso_cv.fit(X_train_scaled, y_train)<br>elastic_cv.fit(X_train_scaled, y_train)<br><br>print(&quot;Best Ridge alpha:&quot;, ridge_cv.alpha_)<br>print(&quot;Best Lasso alpha:&quot;, lasso_cv.alpha_)<br>print(<br>    &quot;Best Elastic Net alpha:&quot;,<br>    elastic_cv.alpha_,<br>    &quot;l1_ratio:&quot;,<br>    elastic_cv.l1_ratio_<br>)</pre><p>Each CV model tests the candidate values automatically and stores the selected regularization strength in the <strong>alpha_</strong> attribute after fitting. For Elastic Net, the selected <strong>l1_ratio_</strong> is also available.</p><h3>Part 11: Feature Scaling, A Practical Consideration</h3><p>Regularization is <strong>sensitive to feature scale</strong>, which makes scaling an important step when using Ridge, Lasso, or Elastic Net.</p><p>Consider two features with very different ranges:</p><pre>Age:    18 to 60<br>Income: 20,000 to 500,000</pre><p>Because L1 and L2 penalties are applied directly to the model weights, features with different scales can receive very different coefficient magnitudes even when they have similar predictive importance. Without scaling, the regularization penalty may therefore <strong>shrink some features more than others simply because of their numerical scale</strong>.</p><p>The standard solution is to <strong>standardize the features</strong> so that each one has a mean of zero and a standard deviation of one.</p><pre>from sklearn.preprocessing import StandardScaler<br><br>scaler = StandardScaler()<br><br>X_train_scaled = scaler.fit_transform(X_train)<br>X_test_scaled = scaler.transform(X_test)</pre><p>Always fit the scaler on the <strong>training data only</strong>, then use that fitted scaler to transform the test data.</p><p>Feature scaling is important for <strong>Ridge, Lasso, and Elastic Net</strong>. Skipping it is one of the most common mistakes when applying regularized linear models.</p><h3>Part 12: Common Mistakes</h3><p>Several mistakes can lead to poor results when applying regularization.</p><h4>1. Assuming Regularization Always Improves Performance</h4><p>Regularization helps when a model is <strong>overfitting</strong>. If the model is already underfitting, adding more regularization can make the problem worse.</p><h4>2. Using an Unnecessarily Large λ</h4><p>A λ that is too large can <strong>shrink the weights too aggressively</strong>, causing underfitting instead of improving generalization.</p><h4>3. Forgetting Feature Scaling</h4><p>As discussed in Part 11, unscaled features can cause the regularization penalty to <strong>affect coefficients unevenly because of their different scales</strong>.</p><h4>4. Comparing Models Only on Training Error</h4><p>Training error alone cannot reveal overfitting. A less-regularized model may achieve lower training error while performing <strong>worse on unseen data</strong>.</p><h4>5. Choosing λ Using the Test Set</h4><p>The test set should provide a <strong>final, unbiased estimate of performance</strong>. Using it to select λ means the test data has influenced the modeling process.</p><p>Choose λ using <strong>cross-validation on the training data</strong>, and evaluate the final model on the test set only after all modeling decisions are complete.</p><h3>Key Formulas</h3><p><strong>Total Loss With Regularization</strong></p><pre>Total loss = Data loss + λ × Regularization penalty</pre><p><strong>L1 Loss, Lasso</strong></p><pre>L1 Loss = Data Loss + λ Σ|w|</pre><p><strong>L2 Loss, Ridge</strong></p><pre>L2 Loss = Data Loss + λ Σw²</pre><p><strong>Elastic Net Loss</strong></p><pre>Elastic Net Loss = Data Loss + λ [ r Σ|w| + (1 - r) Σw² ]</pre><p><strong>L1 Penalty Gradient</strong></p><pre>d|w|/dw = 1   if w &gt; 0<br>d|w|/dw = -1  if w &lt; 0</pre><p><strong>L2 Penalty Gradient</strong></p><pre>d(w²)/dw = 2w</pre><h3>Complete Summary Table</h3><pre>| Concept          | What It Is                                 | What It Does                                           |<br>| ---------------- | ------------------------------------------ | ------------------------------------------------------ |<br>| Overfitting      | Model learns noise with the pattern        | Low training error, high validation error              |<br>| Underfitting     | Model is too simple to capture the pattern | High training and validation error                     |<br>| Regularization   | Extra penalty added to the loss            | Discourages unnecessarily large weights                |<br>| Lambda (λ)       | Regularization strength                    | Controls how strongly weights are penalized            |<br>| L1 (Lasso)       | Penalty on absolute weights                | Can set weights to zero; performs feature selection    |<br>| L2 (Ridge)       | Penalty on squared weights                 | Shrinks weights smoothly; usually keeps them nonzero   |<br>| Sparse Model     | Model with many zero weights               | Easier to interpret; often produced by Lasso           |<br>| Dense Model      | Model with mostly nonzero weights          | More stable with correlated features; typical of Ridge |<br>| Elastic Net      | Combination of L1 and L2                   | Feature selection plus stability                       |<br>| Cross-Validation | Repeated train-validation splits           | Helps select the best λ                                |<br>| Feature Scaling  | Rescales features to comparable ranges     | Makes regularization fair across features              |</pre><h3>Final Thoughts</h3><p>Regularization is built on a simple idea: a model that focuses only on fitting the training data can eventually <strong>learn the noise instead of the underlying pattern</strong>. Regularization adds a second goal: <strong>keep the model’s weights under control</strong> so that it generalizes better to unseen data.</p><p><strong>L1 (Lasso)</strong> encourages sparsity by pulling weights toward zero and can set some of them <strong>exactly to zero</strong>, making it useful for feature selection.</p><p><strong>L2 (Ridge)</strong> penalizes large weights more strongly and <strong>shrinks them smoothly</strong>, usually keeping all features in the model.</p><p><strong>Elastic Net</strong> combines both approaches, providing <strong>feature selection from L1 and stability from L2</strong>, especially when features are correlated.</p><p>The goal is not to prevent the model from learning useful patterns. It is to encourage the model to <strong>use only the complexity it actually needs</strong>.</p><p>If this guide helped you better understand <strong>Regularization</strong>, <strong>Overfitting</strong>, <strong>L1 (Lasso)</strong>, <strong>L2 (Ridge)</strong>, <strong>Elastic Net</strong>, and how regularization helps models generalize better, consider sharing it with someone learning <strong>Machine Learning, Data Science, or AI</strong>.</p><p>I’d also love to hear your thoughts. <strong>Which concept made regularization finally click for you?</strong> Was it <strong>L1’s feature selection</strong>, <strong>L2’s weight shrinkage</strong>, <strong>Elastic Net</strong>, <strong>cross-validation</strong>, or understanding <strong>how regularization prevents overfitting</strong>?</p><p>Follow me for more content on <strong>Python, Mathematics for Machine Learning, Data Science, Machine Learning, Deep Learning, and AI</strong>.</p><p>📌 <strong>LinkedIn:</strong> <a href="https://www.linkedin.com/in/subinthapa/">Subin Thapa</a><br> 📌 <strong>GitHub:</strong> <a href="https://github.com/Subinthapa2092">Subinthapa2092</a><br> 📌 <strong>Facebook:</strong> <a href="https://www.facebook.com/profile.php?id=61590930345491">Coding With Subin</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=231b7a8a88df" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Drawing the Line: The Math Behind Support Vector Machines]]></title>
            <link>https://medium.com/@subinthapa2092/drawing-the-line-the-math-behind-support-vector-machines-4b598ebc1874?source=rss-9f567b128564------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b598ebc1874</guid>
            <category><![CDATA[svm-tutorial]]></category>
            <category><![CDATA[support-vector-machine]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Subinthapa]]></dc:creator>
            <pubDate>Fri, 24 Jul 2026 06:55:26 GMT</pubDate>
            <atom:updated>2026-07-24T07:54:44.020Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*byZPeqFqjmU_Hvh4_HGjvQ.png" /></figure><h4>How a Simple Line, a Margin, and a Trick With Dimensions Let Machines Separate Data</h4><p>Imagine two groups of people standing on a field, loosely divided into <strong>two clusters</strong>. Someone hands you a rope and asks you to place it on the ground so that it separates the two groups as cleanly as possible, leaving the largest possible empty space on both sides. Among all the possible ways to position the rope, one creates the widest gap between the groups.</p><p>This is the core idea behind a <strong>Support Vector Machine (SVM)</strong>. Instead of finding just any line that separates two classes, an <strong>SVM</strong> searches for the one that maximizes the margin, which is the distance between the decision boundary and the nearest data points from each class. A larger margin generally makes the model more robust and improves its ability to classify unseen data correctly.</p><p>In this article, we will build the intuition behind <strong>SVM </strong>step by step before exploring the mathematics that powers it. You will learn how decision boundaries, margins, and support vectors work, why soft margins are needed for noisy data, how kernel functions enable <strong>SVM</strong> to solve non linear problems, how to tune its key hyperparameters, and how <strong>SVM</strong> compares with Logistic Regression and Decision Trees. We will also explore its regression counterpart, <strong>Support Vector Regression (SVR)</strong>, and discuss where <strong>SVM</strong> is used in real world applications.</p><p>The foundations of SVM were developed by <strong>Vladimir Vapnik</strong> and his collaborators, with the modern soft margin formulation introduced in 1995. For many years, <strong>SVM</strong> was one of the most successful machine learning algorithms for tasks such as text classification, image recognition, and bioinformatics. Although deep learning now dominates many large scale applications, the principles introduced by <strong>SVM</strong>, including margins, support vectors, and kernel methods, remain essential concepts in modern machine learning.</p><h3>Part 1: The Decision Boundary</h3><h3>Two Classes, One Line</h3><p>Suppose we are classifying data into two categories. In SVM, these two categories are conventionally labeled <strong>+1 </strong>and<strong> -1</strong>, not <strong>0</strong> and <strong>1 </strong>as in many other classifiers.</p><pre>Class +1  →  one group of points<br>Class -1  →  the other group of points</pre><p>The line that separates the two classes is called the <strong>decision boundary</strong>. Every point on one side of this line is predicted as +1, and every point on the other side is predicted as -1.</p><p>On either side of the decision boundary, SVM draws two more lines, parallel to it, that just barely touch the closest points of each class. These are called the <strong>positive margin</strong> and the <strong>negative margin</strong>. The gap between them is the region SVM is trying to make as wide as possible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T25XL2EA7iJK2NV_j1M14A.png" /><figcaption><strong>Figure 1: Decision Boundary with Positive and Negative Margins</strong></figcaption></figure><p>A two-dimensional binary classification plot showing <strong>blue stars </strong>(<strong>Class +1</strong>) and <strong>orange circles</strong> (<strong>Class</strong> <strong>−1</strong>). The solid line represents the decision boundary, while the parallel dashed lines indicate the positive and negative margins. The shaded region between them illustrates the maximum margin used to separate the two classes.</p><h3>A Small Dataset</h3><p>To make this concrete, consider four labeled points, each with two features x₁ and x₂:</p><pre>Point    x1    x2    Class<br> P1       2     2     +1<br> P2       3     3     +1<br> P3       0     1     -1<br> P4       1     0     -1</pre><p>Written more generally, a data point is a vector <strong>x = (x₁, x₂)</strong>, and its label y belongs to the set <strong>{+1, -1}</strong>.</p><pre>x → (x1, x2)<br>y → {+1, -1}</pre><p>The objective of<strong> SVM</strong> is to find the line that separates the two classes, and among all the lines that could do this job, to pick the one with the widest margin.</p><h3>Part 2: The Equation of the Decision Boundary</h3><h3>Working in Two Dimensions</h3><p>In two-dimensional space, a line can be written as:</p><pre>w1·x1 + w2·x2 + b = 0</pre><p>Here, <strong>w</strong> is the weight vector, and <strong>b</strong> is the bias.</p><pre>w = [w1, w2]<br>b = bias</pre><p>Written using vector notation, the same equation becomes:</p><pre>wᵀx + b = 0</pre><p>Each part of this equation plays a distinct geometric role. The weight vector w controls the slope, meaning the direction the line points in. The bias b controls where the line sits, shifting it left, right, up, or down without changing its angle. The vector <strong>x</strong> simply represents whichever data point is being tested.</p><pre>w  →  controls the slope (direction of the line)<br>b  →  controls where the line sits (shifts it)<br>x  →  the data point being tested</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xM6zkMa4mUedu9Z5QuaX6g.png" /><figcaption><strong>Figure 2: Effect of w and b on the Decision Boundary</strong></figcaption></figure><p>Three side-by-side plots illustrate how <strong>w</strong> and <strong>b</strong> affect a linear decision boundary. A smaller <strong>‖w‖</strong> results in a gentler slope, while a larger <strong>‖w‖</strong> produces a steeper line. Changing <strong>b</strong> shifts the line’s position without altering its angle.</p><p>The weight vector <strong>w</strong> is always perpendicular to the line, controlling its orientation. The bias <strong>b</strong> acts as a translation term, moving the boundary across the plane without rotation.</p><h3>Solving for w and b Using the Dataset</h3><p>Using the four points from <strong>Part 1</strong>, and treating <strong>P3 = (0, 1)</strong> and <strong>P4 = (1, 0)</strong> as the closest negative points, and <strong>P1 = (2, 2) </strong>as the closest positive point, we can solve for the maximum-margin line by symmetry.</p><p>Because <strong>P3</strong> and <strong>P4</strong> sit on the line <strong>x₁ + x₂ = 1</strong>, and the closest positive point <strong>P1</strong> sits on <strong>x₁ + x₂ = 4</strong>, the boundary is placed exactly halfway between them:</p><pre>c = (1 + 4) / 2 = 2.5<br><br>Unscaled boundary:  x1 + x2 - 2.5 = 0</pre><p>To make this a valid SVM boundary, w and b must be scaled so that the closest points land exactly on <strong>w·x + b = +1</strong> and <strong>w·x + b = -1</strong>. Checking the distance from the unscaled line to a support vector gives a scaling factor of <strong>1.5</strong>, so:</p><pre>w = [2/3, 2/3]<br>b = -5/3</pre><p><strong>Verifying with the data:</strong></p><pre>P1 = (2,2):  (2/3)(2) + (2/3)(2) - 5/3 = 4/3 + 4/3 - 5/3 = 1     → on positive margin ✓<br>P3 = (0,1):  (2/3)(0) + (2/3)(1) - 5/3 = 2/3 - 5/3 = -1          → on negative margin ✓<br>P4 = (1,0):  (2/3)(1) + (2/3)(0) - 5/3 = 2/3 - 5/3 = -1          → on negative margin ✓<br>P2 = (3,3):  (2/3)(3) + (2/3)(3) - 5/3 = 4 - 5/3 = 7/3  (&gt;1)     → correctly classified, not a support vector</pre><p><strong>P1</strong>, <strong>P3</strong>, and <strong>P4 </strong>sit exactly on the margin lines. These are the <strong>support vectors</strong>, the only points that actually determine where the boundary and margin sit. <strong>P2</strong>, sitting further away, could be moved and the boundary would not change at all. This is the single most important property of <strong>SVM</strong>: only the closest points matter.</p><p><strong>Simplifying the boundary equation:</strong></p><pre>(2/3)x1 + (2/3)x2 - 5/3 = 0<br><br>Multiply by 3:<br>2x1 + 2x2 - 5 = 0<br><br>Divide by 2:<br>x1 + x2 - 2.5 = 0<br>→ x2 = 2.5 - x1</pre><h3>The Prediction Rule</h3><p>Once w and b are known, classifying a new point is simple. Plug the point into the linear function:</p><pre>f(x) = wᵀx + b<br>     = w1·x1 + w2·x2 + b</pre><p>For a <strong>linear</strong> <strong>SVM</strong>, this form is convenient and cheap to evaluate directly. For <strong>kernel SVMs</strong> <strong>(Part 5)</strong>, the equivalent decision function is instead expressed as a weighted sum over only the support vectors, a form covered in full in <strong>Part 6</strong>.</p><p>The sign of <strong>f(x)</strong> tells you which side of the decision boundary the point falls on:</p><pre>ŷ = sign(wᵀx + b)<br><br>wᵀx + b &gt; 0   →  ŷ = +1<br>wᵀx + b &lt; 0   →  ŷ = -1<br>wᵀx + b = 0   →  exactly on the boundary</pre><p><strong>Worked example.</strong> Using f(x) = x₁ + x₂ — 3:</p><pre>Test point A = (3, 2):<br>f(A) = 3 + 2 - 3 = 2   → 2 &gt; 0   → ŷ = +1<br><br>Test point B = (1, 0):<br>f(B) = 1 + 0 - 3 = -2  → -2 &lt; 0  → ŷ = -1</pre><p>Point A lands on the positive side, point B lands on the negative side, exactly as their positions relative to the line would suggest.</p><h3>Part 3: The Margin, and Why SVM Maximizes It</h3><h3>Back to the Rope</h3><p>Remember the rope from the opening. Laying it down so it merely <em>separates</em> the two groups is the easy part, almost any reasonable placement does that. The harder, more valuable task is finding the placement that leaves the most slack on both sides, so that if either group shifts slightly, the rope still holds. That “slack on both sides” is exactly what the math in this section is measuring.</p><h3>Two Boundary Lines</h3><p>SVM does not just find one decision boundary at wᵀx + b = 0. It also defines two support lines, one on each side:</p><pre>wᵀx + b =  0    (decision boundary)<br>wᵀx + b = +1    (positive margin line)<br>wᵀx + b = -1    (negative margin line)</pre><p>Every training point must satisfy <strong>wᵀx + b ≥ 1</strong> if it belongs to class <strong>+1</strong>, and <strong>wᵀx + b ≤ -1</strong> if it belongs to class <strong>-1</strong>. The points that sit exactly on these two lines, touching them with equality, are the support vectors.</p><h3>Why SVM Prefers a Wide Margin</h3><p>A line that separates the data is not enough on its own; many different lines could all separate the same two classes. SVM specifically looks for the widest possible margin because a wider margin generally means better generalization: new, unseen points are less likely to fall on the wrong side of the boundary if there is more empty space around it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-_Nzoh__jR4fJVTCAegbpQ.png" /><figcaption><strong>Figure 3: Maximum Margin Classification Narrow vs. Wide Margin</strong></figcaption></figure><p>Two side-by-side scatter plots illustrate the same two classes of data points. In the left plot, the decision boundary lies very close to the data, creating a narrow margin with minimal separation, labeled “<strong>narrow margin</strong> <strong>(risky).</strong>” In the right plot, a different valid boundary creates a wide empty band on both sides, labeled “<strong>wide margin</strong> <strong>(SVM’s choice)</strong>.”</p><p>Although both boundaries correctly classify the training data, the wider margin provides greater separation between classes, resulting in improved robustness and better generalization to unseen data.</p><h3>Calculating the Margin Width</h3><p>The distance between the two margin lines (<strong>wᵀx + b = +1</strong> and <strong>wᵀx + b = -1</strong>) is given by:</p><pre>margin = 2 / ‖w‖</pre><p>where ‖w‖ is the length (magnitude) of the weight vector:</p><pre>‖w‖ = √(w1² + w2² + ...)</pre><p><strong>Worked example.</strong> Suppose w = [3, 4]:</p><pre>‖w‖ = √(3² + 4²) = √(9 + 16) = √25 = 5<br><br>margin = 2 / ‖w‖ = 2 / 5 = 0.4</pre><p>This reveals an important inverse relationship: a <strong>large</strong> ‖w‖ produces a <strong>small</strong> margin, and a <strong>small</strong> ‖w‖ produces a <strong>large</strong> margin.</p><pre>‖w‖ = 4  →  margin = 2/4 = 0.5<br>‖w‖ = 2  →  margin = 2/2 = 1.0</pre><p>This is exactly why training an SVM means <strong>minimizing ‖w‖</strong> (subject to correctly classifying every point) rather than maximizing some other quantity directly. Making the weight vector as small as possible is mathematically the same thing as making the margin as wide as possible.</p><p>Applying this to the earlier example, where<strong> w = [2/3, 2/3]</strong>:</p><pre>‖w‖ = √((2/3)² + (2/3)²) = √(8/9) = 2√2/3<br><br>margin = 2 / (2√2/3) = 3/√2 ≈ 2.12</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*svMDX2qvPCdX9fMnnn10-g.png" /><figcaption><strong>Figure 4: Margin Width Shrinks as the Weight Vector Magnitude (‖w‖) Increases</strong></figcaption></figure><p>A conceptual illustration showing three margin widths generated by different weight vector magnitudes. The outermost pair of parallel boundaries represents <strong>small ‖w‖</strong> (widest margin), the middle pair represents <strong>medium ‖w‖</strong>, and the innermost pair represents <strong>large ‖w‖</strong> (narrowest margin). This figure highlights the fundamental SVM relationship that the margin width is inversely proportional to the magnitude of the weight vector (‖w‖).</p><h3>Part 4: Hinge Loss, How SVM Measures Mistakes</h3><h3>The Loss Function</h3><p>SVM measures how wrong a prediction is using <strong>hinge loss</strong>, defined as:</p><pre>L = max(0, 1 - y·f(x))</pre><p>where f(x) = wᵀx + b is the raw decision score, and y ∈ {+1, -1} is the true label. Let z = y·f(x), so the loss simplifies to:</p><pre>L = max(0, 1 - z)</pre><h3>Reading the Three Cases</h3><pre>z ≥ 1   →  L = 0                    (correct, on or outside the margin, no penalty)<br>0 &lt; z &lt; 1  →  0 &lt; L &lt; 1             (correct, but inside the margin, small penalty)<br>z ≤ 0   →  L ≥ 1                    (misclassified, large penalty)</pre><p><strong>Worked examples:</strong></p><pre>z = 2:    L = max(0, 1 - 2) = max(0, -1) = 0        → correctly classified, no loss<br>z = 0.5:  L = max(0, 1 - 0.5) = max(0, 0.5) = 0.5    → inside the margin, small penalty<br>z = 0:    L = max(0, 1 - 0) = max(0, 1) = 1          → sitting exactly on the boundary, full penalty</pre><p><strong>Hinge loss</strong> is zero for any point that is not just correctly classified, but correctly classified with enough distance from the boundary (outside the margin). It only starts to penalize a point once it crosses into the margin itself, and the penalty grows the further a point strays onto the wrong side of the boundary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xnfLdCgS8MmUFRe7Kr1QkQ.png" /><figcaption><strong>Figure 5. Hinge Loss Function</strong></figcaption></figure><p>A graph illustrating how hinge loss changes with the decision score,<strong> z = y·f(x)</strong>. When the decision score is below the margin threshold <strong>(z &lt; 1)</strong>, the loss decreases linearly as confidence increases. Once the score reaches or exceeds the margin boundary <strong>(z ≥ 1)</strong>, the loss becomes zero, indicating that the sample is correctly classified with a sufficient margin. The dashed vertical line marks the transition between the penalty region and the zero-loss region.</p><h3>Part 5: When a Straight Line Isn’t Enough, The Kernel Trick</h3><h3>Data That Cannot Be Separated by a Line</h3><p>Not every dataset is linearly separable. Sometimes the two classes are arranged so that no straight line, in the original space, can cleanly divide them, for example, one class forming a ring around the other.</p><pre>Every time, data is not separable linearly.</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ARnVdcvUiE2oAATMAVjp8g.png" /><figcaption><strong>Figure 6. A Non Linearly Separable Dataset</strong></figcaption></figure><p>Real world datasets are rarely perfectly linear. As shown in <strong>Figure 6</strong>, the classes form a curved distribution that cannot be separated by any single straight line. Because of this nonlinear structure, a linear SVM cannot correctly classify every sample. Kernel methods overcome this limitation by mapping the data into a higher-dimensional feature space, where a linear decision boundary can effectively separate the classes.</p><h3>Lifting Data Into a Higher Dimension</h3><p>The kernel trick solves this by mapping the original data into a higher-dimensional space, where the same data <em>does</em> become separable by a flat hyperplane.</p><pre>x = (x1, x2)<br>z = (x1, x2, x1² + x2²)</pre><p>By adding a third feature, the squared distance from the origin, points that were tangled together in <strong>2D</strong> can spread apart in <strong>3D</strong>. A ring of points surrounding a cluster in <strong>2D</strong>, for instance, lifts into two separate, stackable layers in <strong>3D</strong>, which a flat plane can then separate easily.</p><p>The remarkable part is that SVM never has to explicitly compute this higher-dimensional transformation point by point. Kernel functions compute the equivalent of “<strong>what the dot product would be in that higher space</strong>” directly from the original coordinates, which is what makes the trick computationally efficient even when the target space has very high (or even infinite) dimensions.</p><pre>| Kernel             | Description                                                                                                       | Best Used When                                                               |<br>| ------------------ | ----------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |<br>| Linear             | Uses the original feature space without any transformation.                                                       | The data is already linearly separable.                                      |<br>| RBF (Gaussian)     | Maps data into an infinite-dimensional feature space to learn highly flexible decision boundaries.                | The relationship between classes is nonlinear or unknown.                    |<br>| Polynomial         | Creates polynomial combinations of the original features to model more complex decision boundaries.               | Feature interactions are important and the boundary is moderately nonlinear. |<br>| Sigmoid            | Produces behavior similar to a neural network activation function, though it is used less frequently in practice. | Specific experimental settings or compatibility with older applications.     |</pre><p>The choice of kernel is essentially a choice of what kind of curved boundary the SVM is allowed to draw once it can no longer rely on a straight line. The <strong>linear kernel</strong> is the default starting point and is especially common in <strong>text classification</strong>, where the feature space is already so high-dimensional (one dimension per word) that the data tends to be linearly separable without any further transformation.</p><h3>Part 6: Support Vectors During Prediction</h3><h3>Most of the Training Data Is Thrown Away</h3><p>Once an SVM finishes training, something convenient happens: the vast majority of the training points become irrelevant. Only the support vectors, the points sitting exactly on the margin lines, are needed to define the boundary and to classify new points afterward.</p><pre>Prediction depends only on the support vectors,<br>not on the full training set.</pre><p>This is why SVM’s decision function is usually written not just as <strong>wᵀx + b</strong>, but as a sum over the support vectors themselves:</p><pre>f(x) = Σ (αi · yi · K(xi, x)) + b        (sum taken over support vectors only)</pre><p>Here <strong>K</strong>(<strong>xi, x</strong>) is the kernel function, and αi is a weight learned for each support vector during training (<strong>αi = 0</strong> for every point that is <em>not</em> a support vector, which is what removes it from the sum entirely). In practice, this means a <strong>trained SVM model </strong>can be stored and evaluated using only a small subset of the original data, often a real speed and memory advantage at prediction time, even though training itself can be expensive.</p><h3>Part 7: Soft Margin SVM, Handling Real and Noisy Data</h3><h3>The Rope on a Messier Field</h3><p>The rope-on-a-field picture from the intro assumed the two groups of people stood in two neat, non-overlapping clusters. Real fields are messier: a few people from one group wander close to the other, someone crosses over to talk to a friend, and no single straight rope can separate everyone without either zig-zagging wildly or accepting that a few people will end up on the “<strong>wrong</strong>” side. Soft margin <strong>SVM</strong> is the formal version of accepting that second option.</p><h3>The Problem With “Hard” Margins</h3><p>Everything covered so far assumes the two classes can be separated <em>perfectly</em>, with no point ever crossing onto the wrong side of the margin. This is called a <strong>hard margin</strong>. Real-world data is rarely this cooperative, classes overlap, labels contain noise, and a handful of outliers can sit right in the middle of the opposite class.</p><pre>| Margin Type     | Key Idea                                                                                  |<br>| --------------- | ----------------------------------------------------------------------------------------- |<br>| Hard Margin     | No margin violations are allowed. Requires perfectly separable data.                      |<br>| Soft Margin     | Margin violations are allowed with a penalty, making the model more robust to noisy data. |</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fzTu85UwFK-PbU7dxWKUyg.png" /><figcaption><strong>Figure 7: Hard Margin vs Soft Margin in Support Vector Machines</strong></figcaption></figure><p>This figure presents two side-by-side plots illustrating the difference between <strong>hard</strong> and <strong>soft</strong> <strong>margin</strong> approaches. The left plot depicts a hard margin, where a decision boundary separates the two classes with a clear gap and no data points allowed within the margin, reflecting strict separation with no tolerance for violations. The right plot shows a soft margin, where the decision boundary is determined while allowing a small number of data points to fall within the margin or on the incorrect side, demonstrating a more flexible approach that accommodates minor violations to better handle real-world data.</p><h3>Slack Variables</h3><p>To allow for this kind of noise, SVM introduces a <strong>slack variable</strong>, written <strong>ξᵢ (xi)</strong>, for each data point. A slack variable measures how far a point has violated its margin.</p><pre>ξi = 0        →  point is correctly placed, outside or on the margin (no violation)<br>0 &lt; ξi &lt; 1    →  point is inside the margin, but still on the correct side<br>ξi ≥ 1        →  point has crossed all the way to the wrong side (misclassified)</pre><p>The constraint from the hard margin case, yᵢ(wᵀxᵢ + b) ≥ 1, is relaxed to allow for this slack:</p><pre>yi(wᵀxi + b) ≥ 1 - ξi,      ξi ≥ 0</pre><p>Each point is now allowed to fall short of the margin by exactly ξᵢ, at a cost that gets added to what the model is trying to minimize.</p><h3>The Penalty Parameter C</h3><p>Allowing slack for free would let the model ignore the margin entirely, so SVM introduces a <strong>penalty parameter, C</strong>, that controls how much each unit of slack costs.</p><pre>Large C  →  large penalty for violations  →  fewer training errors, narrower margin<br>Small C  →  small penalty for violations  →  more tolerance for errors, wider margin</pre><p>This is a direct trade-off. A very large C pushes the <strong>SVM </strong>to fit the training data as closely as possible, even if that means a thin, twisty margin that may not generalize well to new data (a sign of overfitting). A very small C allows a wide, smooth margin that tolerates some misclassified training points, often generalizing better on noisy real-world data even though it “<strong>gets more wrong</strong>” on the training set itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d09sNL5EAcfsGWPGAuYjGA.png" /><figcaption><strong>Figure 8. Effect of C on the SVM Decision Boundary</strong></figcaption></figure><p>This figure presents two side-by-side plots based on the same noisy, slightly overlapping dataset to illustrate the impact of the regularization parameter <strong>CCC</strong> in<strong> Support Vector Machines</strong>. In the left plot, a large<strong> CCC</strong> results in a narrow and tightly fitted decision boundary that closely follows individual data points, striving to correctly classify nearly all samples and leading to a more complex model. In contrast, the right plot shows that a small<strong> CCC</strong> allows greater tolerance for misclassification, producing a smoother decision boundary with a wider margin that captures the overall structure of the data and promotes better generalization.</p><h3>Part 8: The Full Optimization Objective</h3><h3>What SVM Is Actually Solving</h3><p>Putting the margin-maximization idea and the soft-margin penalty together, the full optimization problem that SVM solves during training is:</p><pre>minimize:     (1/2)‖w‖² + C·Σξi</pre><pre>subject to:   yi(wᵀxi + b) ≥ 1 - ξi   for every training point i<br>              ξi ≥ 0</pre><p>Reading this piece by piece: the term <strong>(1/2)‖w‖</strong>² is exactly the margin-maximization goal from <strong>Part 3</strong>, written in a form that is easier to differentiate. The term <strong>C·Σξ</strong>ᵢ is the total penalty for every margin violation across the dataset, scaled by the penalty parameter <strong>C</strong>. The two constraints below it say that every point must respect its margin, up to whatever slack it has been allowed.</p><pre>(1/2)‖w‖²   →  keep the margin as wide as possible<br>C·Σξi       →  keep the total amount of margin-violation as small as possible</pre><p>Training an <strong>SVM</strong> means finding the exact <strong>w, b</strong>, and set of <strong>ξᵢ</strong> values that make this expression as small as possible while still satisfying every constraint. In practice this optimization problem is solved using its dual form (via Lagrange multipliers), which is also what allows the kernel trick from <strong>Part 5 </strong>to be plugged in directly, without ever explicitly computing the higher-dimensional coordinates.</p><h3>The Dual Form (Optional)</h3><p>The version above is called the <strong>primal</strong> problem, written directly in terms of w and b. <strong>SVM </strong>solvers typically don’t solve this directly; instead, they solve an equivalent <strong>dual</strong> problem, written in terms of a new set of variables αᵢ (one per training point) instead of w and b directly:</p><pre>maximize:    Σαi - (1/2) ΣΣ αi·αj·yi·yj·K(xi, xj)<br><br>subject to:  0 ≤ αi ≤ C   for every point i<br>             Σ(αi·yi) = 0</pre><p>This isn’t essential to understand in order to use SVM well, but two things about it are worth knowing. First, the training data only ever appears inside the kernel function <strong>K(xᵢ, xⱼ)</strong>, which is exactly why the kernel trick from Part <strong>5 </strong>can be plugged in without ever computing the higher-dimensional coordinates explicitly. Second, solving this dual problem is what produces the αᵢ values used in the support-vector decision function from Part 6: any point where <strong>αᵢ = 0</strong> turns out not to matter at all for the final boundary, and only the points with <strong>αᵢ &gt; 0</strong> (the <strong>support vectors</strong>) survive.</p><h3>Part 9: Hyperparameters, The Knobs You Actually Tune</h3><p><strong>SVM</strong> has a small number of hyperparameters, but each one has a real, visible effect on the resulting boundary.</p><pre>C        →  controls the margin/error trade-off (see Part 7)<br>gamma    →  controls how far the influence of a single point reaches (RBF kernel)<br>degree   →  controls the degree of the polynomial used (Polynomial kernel)</pre><p><strong>C</strong> was covered fully in Part 7: high C fits the training data tightly, low C favors a wider, smoother margin. Think of C as how strictly you’re grading the rope: a high C says “<strong>nobody is allowed to end up on the wrong side, whatever it costs the rope’s shape</strong>”; a low C says <strong>“a few stragglers are fine, keep the rope straight and generous.</strong>”</p><p><strong>gamma</strong> (used with the RBF kernel) controls how much influence a single training point has over the shape of the boundary.</p><pre>Large gamma  →  each point&#39;s influence is very local  →  tight, wiggly boundary, risk of overfitting<br>Small gamma  →  each point&#39;s influence reaches further →  smoother, more general boundary</pre><p>Intuitively, a large gamma makes the model “<strong>zoom in</strong>” around each point individually, drawing a boundary that hugs the training data closely. A small gamma makes the model “<strong>zoom out,</strong>” allowing nearby points to blend into a smoother, more general boundary.</p><p><strong>degree</strong> (used with the <strong>Polynomial kernel</strong>) sets how many polynomial terms the kernel is allowed to combine; a degree of <strong>2</strong> allows curved boundaries like parabolas, while higher degrees allow increasingly complex, wavier boundaries. As with gamma, a higher degree gives the model more flexibility, and more risk of overfitting.</p><h3>Part 10: Multi-Class Classification</h3><h3>SVM Is Naturally Binary</h3><p>Everything so far has assumed exactly two classes, <strong>+1</strong> and <strong>-1</strong>. But most real classification problems have more than two categories (classifying an email into “<strong>spam,</strong>” “<strong>work,</strong>” “<strong>personal,</strong>” and “<strong>promotions,</strong>” for instance). <strong>SVM</strong> does not natively support more than two classes, so it relies on one of two standard strategies to extend it.</p><pre>One-vs-Rest (OvR)  →  train one SVM per class, each separating that class from all others<br>One-vs-One (OvO)   →  train one SVM for every possible pair of classes</pre><p><strong>One-vs-Rest</strong> trains as many SVMs as there are classes. Each SVM answers a simple binary question: “<strong>Is this point class A</strong>, <strong>or</strong> <strong>is it anything else?</strong>” When classifying a new point, all the<strong> SVMs vote</strong>, and the class whose <strong>SVM</strong> gives the highest confidence score wins.</p><p><strong>One-vs-One</strong> instead trains a separate <strong>SVM</strong> for every pair of classes. With k classes, this means <strong>k(k-1)/2</strong> individual <strong>SVMs</strong>. Each one only ever sees data from its two assigned classes. A new point is passed through every pairwise <strong>SVM</strong>, and the class that wins the most one-on-one “<strong>matchups</strong>” is the final prediction.</p><pre>3 classes  →  OvR needs 3 SVMs        →  OvO needs 3 SVMs (3×2/2 = 3)<br>5 classes  →  OvR needs 5 SVMs        →  OvO needs 10 SVMs (5×4/2 = 10)</pre><p><strong>OvO</strong> trains more individual models as the number of classes grows, but each of those models is trained on a smaller slice of the data (only two classes at a time), which is often faster in practice despite the larger total count.</p><h3>Part 11: Feature Scaling, Why It’s Not Optional</h3><h3>Distance-Based Models Are Sensitive to Scale</h3><p><strong>SVM’s</strong> entire boundary is defined in terms of distances and dot products between feature vectors. If one feature is measured on a much larger numeric scale than another, it will dominate that distance calculation, effectively drowning out the other feature entirely.</p><p><strong>Example.</strong> Consider a dataset with two features, Age and Income:</p><pre>Age    = 25      (typical range: roughly 0–100)<br>Income = 5,000,000   (typical range: potentially in the millions)</pre><p>When computing <strong>wᵀx</strong>, a feature measured in the millions will contribute a value thousands of times larger than a feature measured in tens, even if Age is actually the more important feature for separating the classes. The <strong>SVM</strong> boundary ends up oriented almost entirely around Income, regardless of how useful Age might have been.</p><pre>Without scaling:  Income dominates the distance calculation, Age is effectively ignored<br>With scaling:     Age and Income contribute on comparable terms</pre><h3>The Fix</h3><p>This is solved by scaling every feature to a comparable range before training, typically using:</p><pre>StandardScaler  →  rescales each feature to have mean 0 and standard deviation 1<br>MinMaxScaler    →  rescales each feature to a fixed range, usually [0, 1]</pre><p>Feature scaling isn’t a minor implementation detail for<strong> SVM</strong>, because the entire algorithm operates on distances, geometry, and dot products; skipping this step can silently produce a model that appears to train fine but performs far worse than it should.</p><h3>Part 12: SVM vs. Logistic Regression vs. Decision Tree</h3><p><strong>SVM </strong>is one option among several for classification, and it’s worth seeing where it sits relative to the two most common alternatives.</p><pre>| Feature                              | Logistic Regression                         | Decision Tree                  | Support Vector Machine (SVM)                       |<br>| ------------------------------------ | ------------------------------------------- | ------------------------------ | -------------------------------------------------- |<br>| Decision Boundary                    | Always linear (straight line or hyperplane) | Axis-aligned splits            | Linear or nonlinear using kernels                  |<br>| Handles Nonlinear Data               | No, unless features are manually engineered | Yes                            | Yes, through the kernel trick                      |<br>| Feature Scaling Required             | Sometimes beneficial                        | Not required                   | Strongly recommended                               |<br>| Probability Output                   | Yes, natively                               | Yes, based on leaf proportions | No, requires probability calibration               |<br>| Interpretability                     | High                                        | High                           | Moderate to low, especially with nonlinear kernels |<br>| Performance on High-Dimensional Data | Good                                        | Generally weaker               | Excellent                                          |<br>| Sensitivity to Outliers              | Moderate                                    | Low                            | Depends on the regularization parameter (C)        |<br>| Training Speed                       | Fast                                        | Fast                           | Slower for large datasets                          |</pre><p>The easiest way to understand the difference is through the question each algorithm tries to answer. <strong>Logistic Regression</strong> asks, <strong>“What is the probability that this data point belongs to Class +1?”</strong> A <strong>Decision Tree</strong> makes a sequence of <strong>yes or no decisions</strong> based on individual features to reach a prediction. In contrast, <strong>Support Vector Machines (SVMs)</strong> ask, <strong>“On which side of the widest possible margin does this data point lie?”</strong> While all three algorithms can produce similar decision boundaries on <strong>simple</strong>, <strong>well-behaved datasets</strong>, their behavior differs significantly when dealing with <strong>high-dimensional data</strong>, <strong>noisy datasets</strong>, or <strong>nonlinearly separable data</strong>. It is in these challenging scenarios that the unique <strong>strengths</strong>, <strong>limitations</strong>, and <strong>trade-offs</strong> of each algorithm become most apparent.</p><h3>Part 13: Real-World Applications of SVM</h3><p>Support Vector Machines (SVMs) perform exceptionally well when the number of features is significantly larger than the number of training samples and when a clear decision boundary exists between classes. Their ability to maximize the margin while handling high-dimensional data makes them one of the most widely used supervised machine learning algorithms across various industries.</p><h4>1. “Spam Email Detection”</h4><p>Emails can be transformed into <strong>high-dimensional feature vectors</strong> using attributes such as <strong>word frequencies</strong>, <strong>sender information</strong>, <strong>hyperlinks</strong>, and <strong>email formatting</strong>. SVM learns an optimal decision boundary that separates <strong>spam</strong> from <strong>legitimate emails</strong>. Because text data is naturally <strong>sparse</strong> and <strong>high-dimensional</strong>, SVM performs particularly well in spam filtering systems.</p><h4>2. “Handwritten Digit Recognition”</h4><p>Before the rise of deep learning, SVMs were among the most successful algorithms for recognizing handwritten digits. By analyzing <strong>pixel values</strong> and using <strong>Polynomial</strong> or <strong>Radial Basis Function (RBF)</strong> kernels, SVM achieved excellent accuracy on benchmark datasets such as <strong>MNIST</strong>.</p><h4>3. “Cancer and Disease Classification”</h4><p>In <strong>bioinformatics</strong> and <strong>medical research</strong>, datasets often contain <strong>thousands of genetic features</strong> but only a limited number of patient samples. SVM’s <strong>margin-maximization</strong> principle enables it to generalize effectively in these high-dimensional environments, making it valuable for <strong>cancer detection</strong>, <strong>disease prediction</strong>, and <strong>gene expression analysis</strong>.</p><h4>4. “Face Detection”</h4><p>Traditional computer vision systems commonly used SVMs for <strong>face detection</strong>. Image regions were converted into <strong>feature vectors</strong>, and the classifier determined whether each region contained a <strong>face</strong> or <strong>non-face</strong> object. This process was repeated across the entire image to accurately detect faces.</p><h4>5. “Medical Diagnosis”</h4><p>SVM is widely used in <strong>clinical decision support systems</strong> to classify diseases based on structured medical measurements. Features such as <strong>cell size</strong>, <strong>cell shape</strong>, <strong>texture</strong>, and <strong>laboratory test results</strong> help distinguish between <strong>benign</strong> and <strong>malignant</strong> tumors. When the data is reasonably separable, SVM provides accurate and reliable predictions.</p><h4>6. “Stock Trend Classification”</h4><p>Financial forecasting problems can be formulated as <strong>binary classification</strong> tasks. Using <strong>technical indicators</strong> such as <strong>Moving Averages (MA)</strong>, <strong>Relative Strength Index (RSI)</strong>, <strong>MACD</strong>, <strong>trading volume</strong>, and <strong>price momentum</strong>, SVM predicts whether a stock price is likely to move <strong>upward (+1)</strong> or <strong>downward (−1)</strong>. Although stock market prediction remains challenging, SVM is widely used in <strong>algorithmic trading</strong>, <strong>quantitative finance</strong>, and <strong>market trend analysis</strong>.</p><h3>Part 14: Advantages and Disadvantages</h3><p>No algorithm is the right choice for every situation, and SVM comes with a clear set of trade-offs worth knowing before reaching for it.</p><pre>| Advantages                                                                       | Disadvantages                                                  |<br>| -------------------------------------------------------------------------------- | -------------------------------------------------------------- |<br>| Works well with high-dimensional data                                            | Slow to train on very large datasets                           |<br>| Strong theoretical foundation based on margin maximization                       | Sensitive to hyperparameter selection (e.g., C and γ)          |<br>| Effective even with relatively small datasets                                    | Decision boundaries are less interpretable than decision trees |<br>| Kernel trick enables nonlinear classification without manual feature engineering | Requires careful feature scaling before training               |<br>| Robust against overfitting when the margin and C are properly tuned              | Does not directly provide class probability estimates          |<br>| Prediction is memory-efficient because only support vectors are used             | Selecting the most appropriate kernel can be challenging       |</pre><h3>Part 15: Computational Complexity</h3><p>Training an <strong>SVM</strong> involves solving the optimization problem from <strong>Part 8</strong>, which becomes noticeably slower as the number of training examples grows. Depending on the implementation, training time scales somewhere between roughly <strong>O(n²)</strong> and<strong> O(n³)</strong> with the number of samples n, which is why SVM is generally a poor fit for datasets with millions of rows.</p><pre>Training  →  can become slow on very large datasets (grows faster than linearly with n)<br>Prediction →  usually fast, since it only involves the support vectors, not the full dataset</pre><p>This asymmetry (expensive to train, cheap to predict with) is part of why <strong>SVM</strong> remains popular for small-to-medium, high-dimensional datasets (<strong>text, gene expression, tabular data with limited rows</strong>) but is less common as a first choice for massive, large-scale datasets, where other methods scale more gracefully.</p><h3>The Full Workflow, At a Glance</h3><p>Before jumping into code, it helps to see the whole pipeline laid out in order, since every step below maps back to a part of this article:</p><pre>Raw Data<br>    ↓<br>Feature Scaling         (Part 11)<br>    ↓<br>Choose Kernel           (Part 5)<br>    ↓<br>Train SVM               (Part 8)<br>    ↓<br>Optimize C / gamma       (Part 9)<br>    ↓<br>Prediction               (Part 6)</pre><p>Every box in this chain corresponds to a decision you actually have to make when using <strong>SVM</strong> in practice: which kernel fits the data’s shape, how strict <strong>C</strong> should be, how far gamma’s influence should reach, and finally, how the trained model turns a new point into a prediction using only its support vectors.</p><h3>Part 16: SVM in Practice, scikit-learn</h3><h3>Classification</h3><p>Putting the whole pipeline together, including the feature scaling from Part 11, a working <strong>SVM</strong> classifier in Python is only a few lines:</p><pre>from sklearn.svm import SVC<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.pipeline import make_pipeline<br><br># Scale features, then fit an SVM with an RBF kernel<br>model = make_pipeline(<br>    StandardScaler(),<br>    SVC(kernel=&quot;rbf&quot;, C=1.0, gamma=&quot;scale&quot;)<br>)<br>model.fit(X_train, y_train)<br>predictions = model.predict(X_test)</pre><p>A few things worth noting in this snippet: StandardScaler handles the feature-scaling concern from Part 11 automatically, kernel=&quot;rbf&quot; selects the Gaussian kernel from Part 5, and C=1.0 sets the margin/error trade-off from Part 7 (scikit-learn&#39;s default). Swapping kernel=&quot;rbf&quot; for kernel=&quot;linear&quot; or kernel=&quot;poly&quot; switches to the other kernels covered earlier, and for a polynomial kernel, a degree argument controls the hyperparameter discussed in Part 9.</p><h3>Multiclass Classification</h3><p>The exact same code handles more than two classes without any changes, using the One-vs-Rest or One-vs-One strategies from Part 10 under the hood:</p><pre># y could be &quot;spam&quot;, &quot;work&quot;, &quot;personal&quot;, &quot;promotions&quot;<br># no code changes needed, SVC handles multiclass natively<br>model.fit(X_train, y_train)<br>model.predict(X_test)<br># scikit-learn&#39;s default multiclass strategy for SVC is One-vs-One (Part 10)</pre><h3>Regression: Support Vector Regression (SVR)</h3><p>Everything covered in this article was framed around classification, but the same margin-based idea extends to predicting a continuous number instead of a class label. <strong>Support Vector Regression (SVR)</strong> tries to fit a function so that as many points as possible fall within a <strong>margin of tolerance (ε)</strong> around it, rather than trying to separate two classes:</p><pre>from sklearn.svm import SVR<br>from sklearn.preprocessing import StandardScaler<br>from sklearn.pipeline import make_pipeline<br><br>reg = make_pipeline(<br>    StandardScaler(),<br>    SVR(kernel=&quot;rbf&quot;, C=1.0, epsilon=0.1)<br>)<br>reg.fit(X_train, y_train)<br>predicted_value = reg.predict(X_test)</pre><p>The new epsilon parameter defines the width of a &quot;<strong>no-penalty zone</strong>&quot; around the predicted line, points that fall within ε of the prediction are treated as correct, and only points that stray further than that actually contribute to the loss. It plays a role similar to the margin in classification: a wider ε tolerates more deviation, a narrower ε demands a tighter fit.</p><h3>Key Formulas</h3><p><strong>Decision Boundary</strong></p><pre>wᵀx + b = 0</pre><p><strong>Prediction Rule</strong></p><pre>ŷ = sign(wᵀx + b)</pre><p><strong>Margin Lines</strong></p><pre>wᵀx + b = +1<br>wᵀx + b = -1</pre><p><strong>Margin Width</strong></p><pre>margin = 2 / ‖w‖</pre><p><strong>Vector Norm</strong></p><pre>‖w‖ = √(w1² + w2² + ...)</pre><p><strong>Hinge Loss</strong></p><pre>L = max(0, 1 - y·f(x))</pre><p><strong>Kernel Feature Mapping (example)</strong></p><pre>x = (x1, x2)  →  z = (x1, x2, x1² + x2²)</pre><p><strong>Soft Margin Constraint</strong></p><pre>yi(wᵀxi + b) ≥ 1 - ξi,   ξi ≥ 0</pre><p><strong>Full Optimization Objective (Primal)</strong></p><pre>minimize:    (1/2)‖w‖² + C·Σξi<br>subject to:  yi(wᵀxi + b) ≥ 1 - ξi,   ξi ≥ 0</pre><p><strong>Dual Optimization Objective (Optional)</strong></p><pre>maximize:    Σαi - (1/2) ΣΣ αi·αj·yi·yj·K(xi, xj)<br>subject to:  0 ≤ αi ≤ C,   Σ(αi·yi) = 0</pre><p><strong>Decision Function Using Support Vectors</strong></p><pre>f(x) = Σ (αi · yi · K(xi, x)) + b     (sum over support vectors only)</pre><h3>Complete Summary Table</h3><pre>| Concept                         | Description                                               | Role in SVM                                                   |<br>| ------------------------------- | --------------------------------------------------------- | ------------------------------------------------------------- |<br>| Decision Boundary (wᵀx + b = 0) | Hyperplane separating classes                             | Determines how data points are classified                     |<br>| Weight Vector (w)               | Vector perpendicular to the decision boundary             | Defines the orientation of the boundary                       |<br>| Bias (b)                        | Offset term                                               | Shifts the decision boundary without changing its orientation |<br>| Support Vectors                 | Training samples closest to the boundary                  | Determine the optimal decision boundary                       |<br>| Margin                          | Distance between the boundary and the nearest data points | Maximized to improve generalization                           |<br>| ‖w‖ (Weight Magnitude)          | Magnitude of the weight vector                            | Inversely proportional to the margin width                    |<br>| Hinge Loss (max(0, 1 − y·f(x))) | Loss function used in SVM                                 | Penalizes misclassified and margin-violating points           |<br>| Linearly Separable Data         | Data that can be separated by a straight line             | Suitable for Linear SVM                                       |<br>| Kernel Trick                    | Maps data into a higher-dimensional space                 | Enables nonlinear classification                              |<br>| Linear Kernel                   | No feature transformation                                 | Best for linearly separable datasets                          |<br>| Polynomial Kernel               | Polynomial feature mapping                                | Captures feature interactions                                 |<br>| RBF Kernel                      | Gaussian kernel                                           | Learns smooth nonlinear decision boundaries                   |<br>| Sigmoid Kernel                  | Neural network–inspired kernel                            | Less commonly used kernel function                            |<br>| Hard Margin                     | No tolerance for classification errors                    | Requires perfectly separable data                             |<br>| Soft Margin                     | Allows limited classification errors                      | Handles noisy and overlapping datasets                        |<br>| Slack Variable (ξ)              | Measures margin violations                                | Quantifies allowable classification errors                    |<br>| Penalty Parameter (C)           | Regularization parameter                                  | Balances margin width and classification accuracy             |<br>| Gamma (γ)                       | RBF kernel parameter                                      | Controls the influence of individual training samples         |<br>| Degree                          | Polynomial kernel parameter                               | Determines the complexity of the polynomial boundary          |<br>| One-vs-Rest (OvR)               | One classifier per class                                  | Strategy for multi-class classification                       |<br>| One-vs-One (OvO)                | One classifier for each pair of classes                   | Alternative multi-class classification strategy               |<br>| Feature Scaling                 | Standardizes feature values                               | Prevents features with larger scales from dominating          |<br>| Support Vector Regression (SVR) | SVM for regression problems                               | Predicts continuous target values                             |<br>| Epsilon (ε)                     | SVR tolerance parameter                                   | Ignores prediction errors within the ε margin                 |</pre><h3>Final Thoughts</h3><p>Think back to the field from the beginning. A rope placed carelessly might separate two groups for the moment, but it would fail if the groups shifted even slightly. The goal of <strong>Support Vector Machines (SVMs)</strong> is to find the <strong>optimal decision boundary</strong> that leaves the <strong>maximum margin</strong> between classes, making the classifier more robust and better at generalizing to unseen data.</p><p>The <strong>weight vector (w)</strong> determines the orientation of the decision boundary, while the <strong>bias (b)</strong> controls its position. The <strong>margin</strong> represents the safe distance between classes, and <strong>hinge loss</strong> penalizes points that violate this margin. When the data is noisy or overlapping, the <strong>soft margin</strong> concept and the <strong>penalty parameter ©</strong> provide the flexibility to balance <strong>classification accuracy</strong> with <strong>generalization</strong>, reducing the risk of overfitting.</p><p>For datasets that cannot be separated by a straight line, the <strong>kernel trick</strong> transforms the data into a <strong>higher dimensional feature space</strong>, allowing SVM to learn complex <strong>nonlinear decision boundaries</strong> without explicitly performing the transformation. Combined with the use of <strong>support vectors</strong>, which are the most influential training samples, this makes SVM both efficient and powerful for a wide range of classification problems.</p><p>By understanding the <strong>decision boundary</strong>, <strong>margin</strong>, <strong>support vectors</strong>, <strong>hinge loss</strong>, <strong>soft margin</strong>, <strong>kernel trick</strong>, and the <strong>regularization parameter ©</strong>, you have built a strong foundation for mastering <strong>Support Vector Machines</strong> and related techniques such as <strong>Support Vector Regression (SVR)</strong> and <strong>Kernel PCA</strong>. If this guide helped deepen your understanding, consider sharing it with others who are beginning their journey in <strong>machine learning</strong> and <strong>classification algorithms</strong>.</p><p>If this guide helped you better understand <strong>Support Vector Machines (SVMs)</strong>, <strong>Decision Boundaries</strong>, <strong>Margins</strong>, <strong>Support Vectors</strong>, the <strong>Kernel Trick</strong>, and how SVMs classify data, consider sharing it with someone learning <strong>Machine Learning, Data Science, or AI</strong>.</p><p>I’d also love to hear your thoughts. <strong>Which concept made SVM finally click for you?</strong> Was it the <strong>maximum margin</strong>, <strong>support vectors</strong>, the <strong>kernel trick</strong>, <strong>soft margins</strong>, or <strong>hyperparameters like C and gamma</strong>?</p><p>Follow me for more content on <strong>Python, Mathematics for Machine Learning, Data Science, Machine Learning, Deep Learning, and AI</strong>.</p><p>📌 <strong>LinkedIn:</strong> <a href="https://www.linkedin.com/in/subinthapa">Subin Thapa</a><br> 📌 <strong>GitHub:</strong> <a href="https://github.com/Subinthapa2092">Subinthapa2092</a><br> 📌 <strong>Facebook:</strong> <a href="https://www.facebook.com/profile.php?id=61590930345491">Coding With Subin</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b598ebc1874" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>