<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Chandrashekhar Padole on Medium]]></title>
        <description><![CDATA[Stories by Chandrashekhar Padole on Medium]]></description>
        <link>https://medium.com/@chandupadole?source=rss-56067b0267b6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*6ig-gKNVYM4QOxe-</url>
            <title>Stories by Chandrashekhar Padole on Medium</title>
            <link>https://medium.com/@chandupadole?source=rss-56067b0267b6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 04:50:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@chandupadole/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[ViT/LLM Algorithm, Part 5 — Width vs Depth: Heads Make Subspaces, Blocks Transform Spaces]]></title>
            <link>https://medium.com/@chandupadole/vit-llm-algorithm-part-5-width-vs-depth-heads-make-subspaces-blocks-transform-spaces-8eeb26708a19?source=rss-56067b0267b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/8eeb26708a19</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[attention]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Chandrashekhar Padole]]></dc:creator>
            <pubDate>Sat, 15 Aug 2026 06:13:46 GMT</pubDate>
            <atom:updated>2026-08-20T15:59:28.062Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RLDNXgigeJQcwxNCjp7EXA.jpeg" /></figure><p><strong>Abstract.</strong> We’ve built the atoms (Part 1), the operations (Part 2), and drawn two whole architectures on one page each (Parts 3–4). This finale zooms all the way out to the <strong>only two dials</strong> that make a transformer bigger and smarter: <strong>width</strong> — how many attention <em>heads</em> each block has — and <strong>depth</strong> — how many <em>blocks</em> you stack. One creates parallel <em>subspaces</em>; the other <em>refines</em>. Keep one silly phrase and you’ll never confuse them:</p><blockquote><strong><em>Head needs wider hands and longer legs.</em></strong></blockquote><p><strong>In this 5-part series</strong> <em>(you are reading Part 5)</em>:</p><ol><li><a href="https://medium.com/@chandupadole/vit-llm-as-analgorithm-part-1-no-maths-just-what-it-signifies-the-primitives-fcdde4367287">Part 1 — No Maths, Just What It Signifies: the primitives</a></li><li><a href="https://medium.com/@chandupadole/vit-llm-as-analgorithm-part-2-no-maths-just-what-it-signifies-the-operations-757ae031f4b7">Part 2 — No Maths, Just What It Signifies: the operations</a></li><li><a href="https://medium.com/@chandupadole/vit-llm-algorithm-part-3-the-vit-cheat-sheet-one-slide-every-shape-459f813b8a62">Part 3 — The Vision Transformer, end to end: one slide, every shape</a></li><li><a href="https://medium.com/@chandupadole/vit-llm-algorithm-part-4-the-llm-cheat-sheet-one-slide-every-shape-ab16252140fd">Part 4 — The Large Language Model, end to end: one slide, every shape</a></li><li><strong>Part 5 — Width vs Depth: heads make subspaces, blocks make refinement</strong> — <em>you are here</em></li></ol><h3>Two dials, and only two</h3><p>Once the data has fixed the obvious sizes — image or vocabulary, patch size, d_model — a transformer has <strong>exactly two</strong> ways left to get bigger and stronger: make each block <strong>wider</strong> (more attention heads) or stack <strong>more blocks</strong> (more depth). That&#39;s it. Two dials.</p><p>A silly-but-sticky way to remember which does what:</p><blockquote><strong><em>Head needs wider hands and longer legs.</em></strong></blockquote><p><strong>Wider hands</strong> = more <strong>heads</strong> — how much a single block can grasp at once. <strong>Longer legs</strong> = more <strong>blocks</strong> — how far the representation can travel toward abstraction. Let’s take them one at a time.</p><h3>Width: heads make subspaces</h3><p>Inside a single block, attention doesn’t happen in one big 128- or 256-dimensional space. The model <strong>splits</strong> that space across several <strong>heads</strong>, each a smaller subspace, and runs attention in all of them <strong>in parallel</strong>:</p><ul><li>our <strong>ViT</strong>: d_model = 128 split into <strong>4 heads × 32</strong>,</li><li>our <strong>LLM</strong>: d_model = 256 split into <strong>8 heads × 32</strong>.</li></ul><p>(Both land on <strong>32-dimensional heads</strong> — a comfortable size for one “kind of relationship.”)</p><p>Each head gets its own little W_Q / W_K / W_V, so it asks its <strong>own set of questions</strong> in its <strong>own subspace</strong>. One head might learn to track &quot;which noun does this adjective attach to,&quot; another &quot;what&#39;s the overall topic,&quot; another &quot;is this inside a quote.&quot; They all attend at the same time; then their outputs are <strong>concatenated</strong> and mixed back together by the output projection <strong>W_O</strong> — the very block we flagged in the parameter budgets. <em>Mixing the heads back is the whole reason </em><em>W_O exists.</em></p><p>So <strong>width = how many different relationships a block can look for at once.</strong> More heads, wider hands.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8e11Hche7ZNzRqRQcFLVmg.png" /><figcaption>Figure 1: Multi Heads to Subspaces</figcaption></figure><h3>The thing almost everyone gets backwards</h3><p>Here’s the correction worth pinning above your desk, because it’s the single most common mix-up:</p><blockquote><strong><em>The parallel “subspaces” come from multiple HEADS — not from multiple blocks.</em></strong></blockquote><p>Subspaces are a <strong>width</strong> thing, living <strong>inside one block</strong>, all running at the same instant. Depth adds no parallel subspaces at all — it does something completely different.</p><h3>Depth: blocks make refinement</h3><p>Stacking blocks widens nothing. Each block takes the same-shaped stream in and hands the same shape out — 65×128 for the ViT, 64×256 for the LLM, <strong>the whole way down</strong>. What changes is the <strong>meaning</strong>: every block <strong>re-reads</strong> the stream and <strong>rewrites</strong> it a little more abstract than the block before.</p><p>Say that carefully, because it’s easy to over-claim. Each block works in a <strong>transformed</strong> version of the space the previous block produced — <em>not</em> a new, independent one. The <strong>residual stream</strong> carries the <em>same</em> representation forward, and every block only <strong>reshapes</strong> it a little further: block 6 sees what block 5 made of what block 4 made, and so on down. So the successive “spaces” are <strong>re-representations of one evolving thing, transformed step by step</strong> — <strong>not</strong> separate, independent, or orthogonal spaces. (That’s exactly why a residual connection <em>adds</em> each block’s output back onto the stream, instead of replacing it.)</p><p>A rough picture: early blocks resolve local, surface things (edges; which word follows which); later blocks assemble those into bigger, more abstract ones (objects; the gist of a sentence). Each block does spin up its <em>own</em> fresh heads — its own within-block subspaces — but the <em>gain</em> from depth is this <strong>repeated refinement of one continuous stream</strong>, not extra parallelism.</p><p>So <strong>depth = how many times the whole multi-subspace operation is repeated</strong>, each round standing on — and transforming — the last. More blocks, longer legs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WuJ3YiU52vIhUmHgAZ_4cQ.png" /><figcaption>Figure 2: Space to trasnfromed Space across stacked blocks</figcaption></figure><h3>One subtlety: which dial actually adds parameters</h3><p>You might picture <em>both</em> dials piling on weights. Only one really does.</p><p><strong>Depth does.</strong> Every block carries its own full set of weights, so stacking blocks adds parameters roughly <strong>linearly</strong> — a 7th ViT block is about another <strong>198K</strong>, a 7th LLM block about <strong>789K</strong>. More blocks, more capacity to fit large data (your instinct here is exactly right).</p><p><strong>More heads <em>don’t</em>.</strong> The W_Q / W_K / W_V / W_O matrices are d_model × d_model <strong>no matter how many heads</strong> you split into — 4 heads or 8, the matrices are the <em>same size</em>. Extra heads just <strong>carve the same space into more subspaces</strong>: the block learns to look for <em>more kinds of relationship at once</em>, essentially <strong>for free</strong>, out of the parameters it already has. (The dial that <em>does</em> add &quot;width&quot; parameters is a bigger d_model or a fatter MLP — raising the <em>dimension</em>, not the head count.)</p><p>So the two dials aren’t the same <strong>kind</strong> of dial:</p><blockquote><strong><em>Depth buys capacity </em>and<em> refinement; head-width buys more parallel views of the same capacity.</em></strong><em> Both make the model </em>better<em> — but only depth (and a bigger </em><em>d_model) make it </em>bigger<em>.</em></blockquote><p>This is also why our LLM (<strong>8.95 M</strong>) dwarfs our ViT (<strong>1.21 M</strong>) even though both are 6 blocks deep and similar per-block: the gap is the LLM’s giant <em>vocabulary</em> matrices, not its depth or its extra heads.</p><h3>The picture: hands across, legs down</h3><p>Put the two dials on two axes — <strong>heads across, blocks down</strong> — and every transformer is just a point on this grid:</p><ul><li>our <strong>ViT</strong> — <strong>4 wide, 6 deep</strong>;</li><li>our <strong>LLM</strong> — <strong>8 wide, 6 deep</strong> (a touch wider, same depth; both use 32-dim heads);</li><li>a real deployed model like <strong>TinyLlama-1.1B</strong> — about <strong>32 heads × 22 blocks</strong>: much wider <em>and</em> much deeper. The same two dials, turned way up. Frontier models turn them up further still — dozens of heads, dozens to hundreds of blocks.</li></ul><p>Notice what <em>doesn’t</em> change as you scale: the <strong>machine</strong>. It’s the same block — LayerNorm, Q/K/V, attention, MLP, residuals — no matter where you sit on the grid. Scaling only ever means <em>wider reach per block</em> and <em>more rounds of refinement</em>.</p><h3>The whole series, in one lens</h3><p>Step all the way back. Everything we built was heading here:</p><ul><li><strong>Part 1</strong> — the atoms: a <strong>vector</strong>, a <strong>weight</strong>, a <strong>dot product</strong>.</li><li><strong>Part 2</strong> — the operations: <strong>projection</strong> (the questions each head asks), <strong>transformation</strong> (the reshape-and-bend inside a block), and the <strong>flow rule</strong> (causal, or free to look around).</li><li><strong>Parts 3 &amp; 4</strong> — two whole architectures on one page each: the <strong>ViT</strong> and the <strong>LLM</strong>, the <em>same</em> block arranged the <em>same</em> way, differing only in <em>direction</em> (bidirectional vs causal) and <em>goal</em> (a class vs the next word).</li><li><strong>Part 5</strong> — <em>here</em> — the only two things that change when you make one of them bigger: <strong>width</strong> (heads → subspaces) and <strong>depth</strong> (blocks → refinement).</li></ul><p><em>Measure, reshape, bend</em> — done in several subspaces at once (<strong>width</strong>), and repeated many times over (<strong>depth</strong>). That is a transformer, top to bottom.</p><h3>The end of the road</h3><p>From a single dot product to a model with billions of parameters, it was always the same handful of moves — and, in the end, only <strong>two dials</strong> to scale them. Keep the sheets beside you, remember the phrase, and no transformer diagram will ever read as a wall of boxes again:</p><blockquote><em>Head needs wider hands and longer legs.</em></blockquote><p>Thanks for reading the whole way through — from pixels and tokens to the two knobs that grow them up. 🙏</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8eeb26708a19" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ViT/LLM Algorithm, Part 4 — The LLM Cheat Sheet: One Slide, Every Shape]]></title>
            <link>https://medium.com/@chandupadole/vit-llm-algorithm-part-4-the-llm-cheat-sheet-one-slide-every-shape-ab16252140fd?source=rss-56067b0267b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab16252140fd</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[attention]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Chandrashekhar Padole]]></dc:creator>
            <pubDate>Thu, 13 Aug 2026 16:46:31 GMT</pubDate>
            <atom:updated>2026-08-20T15:58:34.641Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8mRVf7NJHnCMEuhYbX5W6w.jpeg" /></figure><p><strong>Abstract.</strong> Everything a Large Language Model does — from a line of text to a prediction of the <strong>next word</strong>, and the weight update that teaches it — fits on <strong>one slide</strong>. This part is that slide. It’s a <em>cheat sheet</em>: a picture you can glance at from <strong>any point in the flow, at any time</strong>, to either understand a stage or check that you’ve understood it — while you’re explaining it, coding it, or just thinking about it. Below, we walk the sheet left to right, one stage at a time, with the exact tensor shape at every step.</p><p><strong>In this 5-part series</strong> <em>(you are reading Part 4)</em>:</p><ol><li><a href="https://medium.com/@chandupadole/vit-llm-as-analgorithm-part-1-no-maths-just-what-it-signifies-the-primitives-fcdde4367287">Part 1 — No Maths, Just What It Signifies: the primitives</a></li><li><a href="https://medium.com/@chandupadole/vit-llm-as-analgorithm-part-2-no-maths-just-what-it-signifies-the-operations-757ae031f4b7">Part 2 — No Maths, Just What It Signifies: the operations</a></li><li><a href="https://medium.com/@chandupadole/vit-llm-algorithm-part-3-the-vit-cheat-sheet-one-slide-every-shape-459f813b8a62">Part 3 — The Vision Transformer, end to end: one slide, every shape</a></li><li><strong>Part 4 — The Large Language Model, end to end: one slide, every shape</strong> — <em>you are here</em></li><li><a href="https://medium.com/@chandupadole/vit-llm-algorithm-part-5-width-vs-depth-heads-make-subspaces-blocks-transform-spaces-8eeb26708a19">Part 5 — Width vs Depth: heads make subspaces, blocks make refinement</a></li></ol><blockquote><em>A language model has two defining ideas, and the whole sheet is organised around them: attention is </em><strong><em>causal</em></strong><em> (a word may look at the words before it, never after), and the model is trained to predict the </em><strong><em>next token</em></strong><em>. Keep those two in mind and every box has a place. If a stage feels dense, that’s fine — the sheet is meant to be </em>returned to<em>, not memorised in one pass.</em></blockquote><h3>The whole sheet, first</h3><p>Here is the entire Large Language Model on one page. Don’t read every box yet — just take in the <strong>shape of the journey</strong>: words enter on the left, become token vectors, pass through a repeating causal block, and leave on the right as a <strong>next-word</strong> prediction over an 8192-word vocabulary, with the training update drawn along the bottom.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rh8wYqqamABMpBalvk-4fA.png" /><figcaption>Figure 1: LLM Cheat Sheet: Everything in one slide</figcaption></figure><p>That’s the cheat sheet. The rest of this part is a <strong>guided tour</strong> of it — one stage per section. Each stage names its inputs, its operation, and the <strong>shape</strong> it produces, so you can drop in anywhere and orient yourself instantly.</p><h3>How to read the sheet — colour tells you what’s learned</h3><p>One detail makes the whole sheet readable at a glance: <strong>the dark-violet blocks are the learnable parameters</strong> — the “knobs” training adjusts. Everything else — the striped-blue token bars, the A_c matrix, the logits — is <strong>data computed on the way through</strong>, not stored. So <em>colour tells you what&#39;s learned.</em></p><p>Every dark-violet block on the sheet is one of these — and this is the <strong>complete list of what training changes</strong>:</p><ul><li><strong>W_EB [8192×256]</strong> — the token embedding table</li><li><strong>W_P [64×256]</strong> — the positional embeddings</li><li><strong>W_Q, </strong><strong>W_K, </strong><strong>W_V [256×256]</strong> — the attention projections <em>(per block)</em></li><li>the MLP’s <strong>up-projection </strong><strong>W_E [256×1024]</strong> and <strong>down-projection </strong><strong>W_R [1024×256]</strong> <em>(per block)</em></li><li><strong>γ, </strong><strong>β (</strong><strong>1×256)</strong> — inside every LayerNorm</li><li><strong>W_L [256×8192]</strong> — the next-token head</li></ul><p>Everything the bottom strip does (δ, Δw, weight decay) is how these violet blocks get nudged. Read the sheet by colour: <strong>violet = learned, blue = flowing data.</strong></p><h3>Stage ① — Tokenizer: words become indices</h3><p>Before any maths, the text is cut into <strong>tokens</strong>. A tokenizer — public, or a custom <strong>BPE tokenizer</strong> trained on your own corpus (here, an <em>India-Tales</em> tokenizer with an <strong>8192-word</strong> vocabulary) — maps every word (or word-piece) to a <strong>token</strong> and its <strong>index</strong> in the vocabulary. A 64-token sample becomes 64 indices.</p><p>That index is the only thing that moves forward — a plain integer pointing into the vocabulary. A patch of pixels can be fed to a network directly; a word can’t — it first has to become a number, and this is where that happens.</p><h3>Stage ② — Embedding + position: index becomes a 256-D vector</h3><p>Now each index becomes a vector. The <strong>embedding table </strong><strong>W_EB [8192×256]</strong> is a learned dictionary — one 1×256 row per vocabulary word. Given a token&#39;s index, we simply <strong>look up</strong> its row: index → 1×256. There&#39;s no matrix multiply here — it&#39;s a <em>lookup</em>. Sixty-four tokens → <strong>64×256</strong>. Each row is that word&#39;s <em>meaning</em> as a point in the model&#39;s 256-D space.</p><p>Then we add learned <strong>positional embeddings </strong><strong>W_P [64×256]</strong> — one 1×256 vector per position — so the model knows the order of the words. Same-shape add → <strong>64×256</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/301/1*yP_cqDflC6DtdQ8_3ww31Q.png" /></figure><h3>Stage ③ — Causal, and the incremental runs</h3><p>Here’s the first big idea. The model is <strong>causal</strong>: to predict a word, it may only use the words <strong>before</strong> it. And it’s trained to predict the <strong>next</strong> token at <em>every</em> position. So a single 64-token sample is really <strong>many training examples at once</strong>:</p><ul><li>run 1: given token {1} → predict token 2</li><li>run 2: given tokens {1,2} → predict token 3</li><li>run 3: given {1,2,3} → predict token 4</li><li>…</li><li>run 64: given {1,2,…,64} → predict token 65</li></ul><p>These are the <strong>incremental runs </strong><strong>R</strong>. For run <em>n</em>, tokens 1:n are the input and token n+1 is the target. That&#39;s why the attention matrix on the sheet is drawn as <strong>R×R</strong> — its size is however many tokens are in play. Predicting the next token at every position, from every prefix, is the whole training signal of a language model.</p><h3>Stage ④ — Q, K, V: three questions of every token</h3><p>Every block starts by <strong>normalising</strong> the tokens (a LayerNorm — see the box at the end), then projecting them three ways with three learned matrices, each <strong>256×256</strong>:</p><ul><li>X · W_Q → Q (64×256) — what each token is <strong>looking for</strong></li><li>X · W_K → K (64×256) — what each token <strong>offers</strong></li><li>X · W_V → V (64×256) — the <strong>content</strong> it will hand over if attended to</li></ul><p>Each is one matmul: (64×256) · (256×256) = (64×256). We now hold three 64×256 grids — Q, K, and V — one row per token.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/233/1*hjjmYhwmAZD9u_GEAINmOw.png" /></figure><h3>Stage ⑥ — Attention: who looks at whom, causally</h3><p>Now the tokens talk to each other — but only backward. Three moves:</p><ol><li><strong>Scores.</strong> Q · Kᵀ → <strong>A_c [R×R]</strong>: every entry is one token&#39;s query dotted with another&#39;s key — how much token <em>i</em> should care about token <em>j</em>.</li><li><strong>Mask, scale, softmax.</strong> Because the model is causal, A_c is <strong>lower-triangular</strong> — a token at position <em>i</em> may attend to positions ≤ i only; the upper triangle is blocked out. Then divide by <strong>√d</strong> (d = 256 for a single head; 256 ÷ number of heads for multi-head) and take <strong>row-wise softmax</strong>, so each token&#39;s row becomes weights that sum to 1.</li><li><strong>Mix.</strong> × V → <strong>64×256</strong>. Each token&#39;s output is a weighted blend of the V&#39;s it was <strong>allowed</strong> to see. Add it straight back onto the input (<strong>residual</strong>) — the block edits the stream, it doesn&#39;t replace it.</li></ol><p>That lower triangle is what makes this a <strong>generator</strong>: because a word never sees the future, the model can produce text one word at a time, each word depending only on what came before.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/535/1*zz1_mbkXnBGUIue3M-YQNQ.png" /></figure><h3>Stage ⑤ — The shortcut the mask buys you (training only)</h3><p>The incremental runs of Stage ③ sound expensive — 64 separate forward passes per sample. The <strong>optimization</strong> on the sheet is why they’re nearly free in <strong>training</strong>: because A_c is lower-triangular, <strong>all </strong><strong>R runs happen in one pass.</strong> Run <em>n</em> is just the top-left <strong>n×n</strong> block of the same masked matrix — you don&#39;t recompute anything, you read a bigger corner of what you already have.</p><p>The sheet also flags the boundary: this batching trick is <strong>training only</strong>. At inference the model genuinely generates one token at a time, and for each new step only the <strong>last </strong><strong>V</strong> is new (the earlier ones are unchanged) — the idea behind the <strong>KV-cache</strong>.</p><h3>Zoom in on attention — single-head vs multi-head, and where W_O lives</h3><p>The main sheet draws attention as one box. This companion cheat sheet opens it up, puts <strong>single-head (SHA)</strong> and <strong>multi-head (MHA)</strong> attention side by side, and shows the one matrix the big sheet leaves off — the output projection <strong>W_O</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*byJ1fwtKgAiNZd0kabVw4Q.png" /><figcaption>Figure 2: Figure 1: SHA &amp; MHA: Everything in one slide</figcaption></figure><p><em>A note on the numbers: this sheet uses a </em><strong><em>32-token</em></strong><em> context on purpose, so the three sizes stay distinct and can’t be mixed up — context length </em><em>32, per-head dimension </em><em>64, and model width </em><em>256. (A 64-token context would collide with the 64-D head dimension and muddy the picture.) MHA is drawn with </em><strong><em>4 heads</em></strong><em> for the same legibility; the full model just uses a longer context and more heads — the mechanism is identical.</em></p><p><strong>Both paths start identically.</strong> A LayerNorm normalises each token (γ, β are 1×256, shared across all tokens — the green box, the same LN used everywhere in this model), and then each token is projected into a query, a key, and a value.</p><p><strong>Single-head (left).</strong> One set of projections W_Q, W_K, W_V, each <strong>256×256</strong>, turns the token into a full 256-D Q, K, V. The scores A_c = QKᵀ (32×32) are scaled by √d with <strong>d = 256</strong>, row-softmaxed, and used to mix the values. The output is already 1×256 — the model&#39;s full width — so there is <strong>nothing to stitch back together.</strong> A single head needs no W_O.</p><p><strong>Multi-head (right).</strong> Now the <em>same</em> token is projected by <strong>four</strong> smaller sets W_Q, W_K, W_V, each <strong>256×64</strong> — every head reads the <strong>whole</strong> token, but into its own 64-D subspace. Each head runs its own attention (A_c 32×32, scaled by √d with <strong>d = 64</strong> — the <em>per-head</em> width, not 256) and produces a 64-wide output. Stack the four heads back up and you have 256 again — but four fixed slabs <strong>stapled together, unable to influence one another.</strong></p><p><strong>That is exactly why </strong><strong>W_O exists.</strong> The stacked 256 vector goes through <strong>W_O [256×256]</strong>, a full mix that lets every output dimension draw on <em>all four heads</em> before the result is added back to the residual stream. Single-head attention doesn&#39;t need it (its output is already one coherent 256-vector); multi-head attention <strong>must</strong> have it, or the heads never combine. W_O is the seam that turns four parallel conversations back into one — and it&#39;s the &quot;width&quot; side of the dial you&#39;ll meet in <a href="https://medium.com/BLOG_PART5_width_vs_depth.md">Part 5</a>: more heads split the space into more subspaces, and W_O is what re-joins them.</p><blockquote><em>This is the </em><em>W_O the main LLM sheet (and the ViT sheet) quietly leave off — a real, learnable </em><em>256×256 matrix </em>per block<em>. On this attention sheet you can see exactly where it sits: right after the </em><strong><em>STACKED</em></strong><em> block, just before the output rejoins the stream.</em></blockquote><h3>The other half of the block — the MLP: expand, activate, contract</h3><p>Attention let the tokens <em>share</em>. The MLP lets each token <em>think</em> on its own. After a second LayerNorm, every token (independently) goes through a tiny two-layer network:</p><ul><li><strong>up-projection</strong> W_E [256×1024] — lift the token into a roomier 1024-D space,</li><li>a <strong>ReLU</strong> non-linearity that carves that roomy space,</li><li><strong>down-projection</strong> W_R [1024×256] — fold the useful combinations back down to 256.</li></ul><p>Same 64×256 in, same 64×256 out — and again, <strong>added back</strong> as a residual. The 4× blow-up (256→1024) is the standard mlp_ratio = 4: expand to give the non-linearity room, contract to summarise.</p><p><strong>That is one block:</strong> LN → causal attention → + then LN → MLP → +. Both halves pre-normed, both residual. Shape in, shape out: 64×256 → 64×256.</p><h3>×N blocks — the same move, deepened</h3><p>Because a block takes 64×256 in and gives 64×256 out, you can <strong>stack it</strong>. The sheet shows the first block in full, then a &quot;2nd Block,&quot; then &quot;N − 2 blocks&quot; — the same operation repeated. In our India-Tales LLM, <strong>N = 6</strong>.</p><p>Nothing about the shape changes across the stack — it’s 64×256 the whole way. What changes is the <em>meaning</em>: each block re-reads the stream and rewrites it a little more abstract than the last. Depth is <strong>refinement</strong>, not reshaping.</p><h3>The head — predicting the next token</h3><p>After the last block, take the token position we want a prediction <em>after</em>, apply a final LayerNorm (LN(1×256) on the sheet), then the head matrix <strong>W_L [256×8192]</strong>:</p><p>(1×256) · (256×8192) = (1×8192)</p><p>That’s <strong>one score per vocabulary word</strong> — all 8192 of them. A <strong>softmax/argmax</strong> turns it into a probability distribution over the next word / a prediction. During training, <em>every</em> position does this (one target per run); at generation, you read the <strong>last</strong> position to emit the next token and append it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/294/1*nCRzDgf-zBQiUAyiXTV4IA.png" /></figure><h3>The bottom strip — how it learns</h3><p>The forward pass is done. The strip along the bottom of the sheet is the <strong>backward</strong> pass — one weight update.</p><ol><li><strong>Loss.</strong> The <strong>bottom-left panel, “Loss Calculation (Difference)”</strong>, compares the <strong>True</strong> next token to the <strong>LLM O/P</strong> (the prediction) — both 1×8192 — and the gap is the error signal <strong>δ</strong>.</li><li><strong>Gradient.</strong> For a weight, the gradient is <strong>δ · x</strong> — the error times the input that fed it.</li><li><strong>Step.</strong> Scale by the learning rate: <strong>Δw = η · (δ · x)</strong>.</li><li><strong>Average twice.</strong> Because each sample contains R next-token runs, you average Δw over <strong>all runs</strong> in a sample, then over <strong>all samples</strong> in the batch — the sheet&#39;s <strong>avg(avg(Δw))</strong>.</li><li><strong>Weight decay + update.</strong> Add a small pull toward zero, <strong>η · λ · Wt</strong> (λ = weight-decay rate), and move the weight against the gradient. The next batch runs with the adjusted weight. Repeat.</li></ol><h3>The box that sits inside every block — LayerNorm</h3><p>The <strong>parrot-green panel in the bottom-right corner</strong> is the “LN” opened up. Every LN on the sheet is this same operation, done <strong>token by token</strong> — the panel draws it literally: a column of 1×256 token-bars comes in on the left (<strong>Input</strong>), each is processed independently, and the same-shaped column leaves on the right (<strong>Output</strong>). 64×256 in, 64×256 out.</p><p>For one token’s 1×256 vector, two token-wise steps (both shown in the panel):</p><ul><li>normalise it to mean 0, variance 1: <strong>x̂ = (x − μ) / √(σ² + ε)</strong>,</li><li>then rescale and shift with two learned 1×256 vectors: <strong>y = γ ⊙ x̂ + β</strong>.</li></ul><p>γ and β are learned and — as the panel notes — <strong>shared across all tokens</strong>: the same two knobs tune every column. It&#39;s the small, repeated box that keeps the numbers well-behaved as they flow through six blocks.</p><h3>The whole sheet as one lookup table</h3><p>The ultra-quick reference — the shape at every stage, top to bottom:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/514/1*MUin2fx8efk81iofF-v3oQ.png" /></figure><h3>The parameter budget — how big is this LLM?</h3><p>Count every <strong>dark-violet block</strong> and you get the model’s size. Here’s the full budget — each block, its shape, and its parameter count — grouped into <em>embeddings</em>, <em>one decoder block</em> (×6), and the <em>head</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1xN52ye5djlH99VFSm24Mw.png" /><figcaption>Figure 3: Learning Budget: Count of Learnable Params/Weights</figcaption></figure><p>Here the story flips from the ViT. Because the <strong>vocabulary (8192) dwarfs </strong><strong>d_model (256)</strong>, the <strong>two vocab-sized matrices — the token embedding </strong><strong>W_EB and the output head </strong><strong>W_L, ~2.1 M each — are nearly half the whole model (~47%)</strong>. And they&#39;re <strong>not weight-tied</strong>: they&#39;re two independent matrices, so the same word&#39;s &quot;input vector&quot; and &quot;output score row&quot; are learned separately. Many LLMs <em>tie</em> them (share one matrix), which would cut ~2.1 M — about <strong>23%</strong> of this model — at a stroke.</p><p>Inside a block, it’s the ViT story again: the <strong>MLP holds ~⅔</strong> of the weight, attention the other third. And as on the ViT sheet, the attention <strong>output projection </strong><strong>W_O</strong> (256×256/block) is a real block the slide doesn&#39;t draw. Grand total, verified by running the model: <strong>8,953,344 ≈ 8.95 M.</strong></p><h3>Wrapping up</h3><p>That’s a whole Large Language Model on one page — text to next word to weight update, every shape labelled. Two ideas carried the entire flow: attention is <strong>causal</strong> (each word looks only backward), and the model predicts the <strong>next token</strong> at every position. Everything else is the same repeating block — LayerNorm, Q/K/V, attention, MLP, residuals — stacked six deep.</p><p>Keep the sheet open. That’s the whole point of a cheat sheet. 🙏</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab16252140fd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ViT/LLM Algorithm, Part 3 — The ViT Cheat Sheet: One Slide, Every Shape]]></title>
            <link>https://medium.com/@chandupadole/vit-llm-algorithm-part-3-the-vit-cheat-sheet-one-slide-every-shape-459f813b8a62?source=rss-56067b0267b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/459f813b8a62</guid>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[vision-transformer]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[attention]]></category>
            <dc:creator><![CDATA[Chandrashekhar Padole]]></dc:creator>
            <pubDate>Thu, 13 Aug 2026 16:45:28 GMT</pubDate>
            <atom:updated>2026-08-20T15:57:37.254Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rL7-7ry9CqxKgRL6Y8XRhQ.jpeg" /></figure><p><strong>Abstract.</strong> Everything a Vision Transformer does — from a grid of pixels to a class label, and the weight update that teaches it — fits on <strong>one slide</strong>. This part is that slide. It’s a <em>cheat sheet</em>: a picture you can glance at from <strong>any point in the flow, at any time</strong>, to either understand a stage or check that you’ve understood it — while you’re explaining it, coding it, or just thinking about it. Below, we walk the sheet left to right, one stage at a time, with the exact tensor shape at every step. Nothing here is new theory; it’s the <strong>map</strong> you keep open beside the theory.</p><p><strong>In this 5-part series</strong> <em>(you are reading Part 3)</em>:</p><ol><li><a href="https://medium.com/@chandupadole/vit-llm-as-analgorithm-part-1-no-maths-just-what-it-signifies-the-primitives-fcdde4367287">Part 1 — No Maths, Just What It Signifies: the primitives</a></li><li><a href="https://medium.com/@chandupadole/vit-llm-as-analgorithm-part-2-no-maths-just-what-it-signifies-the-operations-757ae031f4b7">Part 2 — No Maths, Just What It Signifies: the operations</a></li><li><strong>Part 3 — The Vision Transformer, end to end: one slide, every shape</strong> — <em>you are here</em></li><li><a href="https://medium.com/@chandupadole/vit-llm-algorithm-part-4-the-llm-cheat-sheet-one-slide-every-shape-ab16252140fd">Part 4 — The Large Language Model, end to end: one slide, every shape</a></li><li><a href="https://medium.com/@chandupadole/vit-llm-algorithm-part-5-width-vs-depth-heads-make-subspaces-blocks-transform-spaces-8eeb26708a19">Part 5 — Width vs Depth: heads make subspaces, blocks make refinement</a></li></ol><blockquote><em>This is </em><strong><em>Series 3 — Cheat Sheets</em></strong><em>. Parts 1–2 built the </em>lens<em> (every computation is a projection; a matmul is a batch of them). Here we point that lens at a whole architecture and watch the shapes flow. If a stage feels dense, that’s fine — the sheet is meant to be </em>returned to<em>, not memorised in one pass.</em></blockquote><h3>The whole sheet, first</h3><p>Here is the entire Vision Transformer on one page. Don’t try to read every box yet — just notice the <strong>shape of the journey</strong>: pixels enter on the left, become tokens, pass through a repeating block, and leave as a class on the right, with the training update drawn along the bottom.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b8sJYsyUOezjCl-AJuQmQg.png" /><figcaption>Figure 1: ViT Cheat Sheet: Everything in one slide</figcaption></figure><p>That’s the cheat sheet. The rest of this part is a <strong>guided tour</strong> of it — one stage per section. Each stage names its inputs, its operation, and the <strong>shape</strong> it produces, so you can drop in anywhere and orient yourself instantly.</p><h3>How to read the sheet — colour tells you what’s learned</h3><p>One detail makes the whole sheet readable at a glance: <strong>the dark-violet blocks are the learnable parameters</strong> — the “knobs” training adjusts (the <em>weights</em> of Parts 1–2). Everything else — the striped-blue token bars, the A_c attention matrix, the logits — is <strong>data computed on the way through</strong>, not stored. So <em>colour tells you what&#39;s learned</em>.</p><p>Every dark-violet block on the sheet is one of these — and this is the <strong>complete list of what training changes</strong>:</p><ul><li><strong>W_E [48×128]</strong> — the patch embedding</li><li>the <strong>CLS token (</strong><strong>1×128)</strong> and <strong>W_P [65×128]</strong> — the positional embeddings</li><li><strong>W_Q, </strong><strong>W_K, </strong><strong>W_V [128×128]</strong> — the attention projections <em>(per block)</em></li><li>the MLP’s <strong>up-projection </strong><strong>[128×512]</strong> and <strong>down-projection </strong><strong>W_R [512×128]</strong> <em>(per block)</em></li><li><strong>γ, </strong><strong>β (</strong><strong>1×128)</strong> — inside every LayerNorm</li><li><strong>W_L [128×10]</strong> — the classifier head</li></ul><p>Everything the bottom strip does (δ, Δw, weight decay) is how these violet blocks get nudged. Read the sheet by colour: <strong>violet = learned, blue = flowing data.</strong></p><h3>Stage ① — Patchify: an image becomes 64 little vectors</h3><p>A Vision Transformer never looks at a whole image at once. It first cuts each <strong>32×32</strong> image into a grid of small <strong>4×4</strong> patches. With a 4-pixel stride, 32 ÷ 4 = 8 patches across and 8 down — <strong>64 patches</strong> in total.</p><p>Each patch is a colour image tile, so it has three channels (R, G, B). Unfold it: 4 × 4 × 3 = 48 numbers. Flatten those 48 numbers into a single row — a <strong>1×48</strong> vector. Do that for all 64 patches and the image is now <strong>64 vectors of length 48</strong>.</p><p>That’s the whole trick of “an image as a sequence”: a patch is to a picture what a <strong>word is to a sentence</strong>. From here on, the ViT treats these 64 patches exactly like tokens.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/267/1*GsswElD0tPEbWO517r7lEQ.png" /></figure><h3>Stage ② — Patch embedding: from 48 raw numbers to a 128-D meaning</h3><p>Those 48 numbers are just raw pixel values — not yet a <em>meaning</em>. Stage ② lifts each patch into the model’s working space, <strong>128 dimensions</strong>, with a single learned matrix.</p><p>Take one patch vector, 1×48, and multiply it by the patch-embedding matrix <strong>W_E [48×128]</strong>:</p><p>(1×48) · (48×128) = (1×128)</p><p>The same W_E is applied to <strong>every</strong> patch — one shared set of learned &quot;questions&quot; asked of each patch (this is the projection idea from Part 2: each of the 128 outputs is that patch correlated against one learned direction). Stack all 64 results and the image is now a <strong>64×128</strong> matrix: 64 patch <strong>embeddings</strong>, each a point in the model&#39;s 128-D space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/371/1*Be2l3ackfOj6EU3BgURtTw.png" /></figure><h3>Stage ③ — CLS token and positions: 64 patches become 65 tokens</h3><p>Two small additions turn the patch embeddings into the sequence the encoder actually reads.</p><p><strong>First, the CLS token.</strong> We prepend one extra learned vector, <strong>1×128</strong>, at the front of the sequence. It carries no patch — it&#39;s a blank &quot;summary slot.&quot; Its whole job is to <em>gather</em> information from all the patches as they pass through attention, so that at the very end we can read the class off this one token. Prepending it turns 64×128 into <strong>65×128</strong>.</p><p><strong>Second, positions.</strong> Attention has no built-in sense of <em>where</em> a patch sat — shuffle the patches and it wouldn’t notice. So we add a learned <strong>positional embedding</strong> W_P [65×128], one 128-D vector per position, to the token matrix. Same shape in, same shape out — a plain element-wise add of two 65×128 matrices → <strong>65×128</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/375/1*LogWjiunGvSLHuo_SNLccA.png" /></figure><h3>Stage ④ — Q, K, V: asking three questions of every token</h3><p>Every block starts by <strong>normalising</strong> the tokens (a LayerNorm — see the box at the end), then projecting them three ways. This is the projection idea from Part 2, used three times.</p><p>From the normalised 65×128 token matrix X, three learned matrices — each <strong>128×128</strong> — produce three new 65×128 matrices:</p><ul><li>X · W_Q → Q (65×128) — what each token is <strong>looking for</strong></li><li>X · W_K → K (65×128) — what each token <strong>offers</strong></li><li>X · W_V → V (65×128) — the <strong>content</strong> it will hand over if attended to</li></ul><p>Each is one matmul: (65×128) · (128×128) = (65×128). So we now hold three 65×128 grids — Q, K, and V — one row per token.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*fuo5bbU1E7N7gMYLdwMyCA.png" /></figure><h3>Stage ⑤ — Attention: who looks at whom, then mix</h3><p>Now the tokens actually talk to each other. It happens in three moves.</p><p><strong>1. Scores.</strong> Multiply Q by Kᵀ: (65×128) · (128×65) = (65×65). This is the correlation matrix A_c — every entry is one token&#39;s query dotted with another&#39;s key, i.e. <strong>how much token <em>i</em> should care about token <em>j</em>.</strong></p><p><strong>2. Scale and softmax.</strong> Divide every entry by <strong>√d</strong>, where d is the head dimension — for a single head d = 128 (so √128); for multi-head, d = 128 ÷ number of heads. Then apply <strong>row-wise softmax</strong>, so each token&#39;s row becomes a set of weights that sum to 1.</p><p><strong>3. Mix.</strong> Multiply the 65×65 weight matrix by V: (65×65) · (65×128) = (65×128). Each token&#39;s output is a <strong>weighted blend of every token&#39;s V</strong> — it pulls in exactly the content it attended to. This result is added straight back onto the input (<strong>residual</strong>): the block <em>edits</em> the stream, it doesn&#39;t replace it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/367/1*LiHQBlNppcGk6p_Z0jlMlg.png" /></figure><h3>The other half of the block — the MLP: expand, activate, contract</h3><p>Attention let tokens <em>share</em>. The MLP lets each token <em>think</em> on its own. After a second LayerNorm, every token (independently) goes through a tiny two-layer network — the expand-then-contract idea from Part 2:</p><ul><li><strong>up-projection</strong> 128 → 512 (the slide labels it W_E [128×512] — a <em>different</em> matrix from the patch-embedding W_E), lifting the token into a roomier 512-D space,</li><li>a <strong>GELU</strong> non-linearity that carves that roomy space,</li><li><strong>down-projection</strong> 512 → 128 (W_R [512×128]), folding the useful combinations back down to 128.</li></ul><p>Same 65×128 in, same 65×128 out — and again, <strong>added back</strong> as a residual. The 4× blow-up (128→512) is the standard mlp_ratio = 4: expand to give the non-linearity room, contract to summarise.</p><p><strong>That is one block:</strong> LN → attention → + then LN → MLP → +. Both halves pre-normed, both residual. Shape in, shape out: 65×128 → 65×128.</p><h3>×N_B blocks — the same move, deepened</h3><p>Because a block takes 65×128 in and gives 65×128 out, you can <strong>stack it</strong>. The sheet shows the first block in full, then a &quot;2nd Block,&quot; then &quot;N_B − 2 blocks&quot; — the same operation repeated. In our Series 1 ViT, <strong>N_B = 6</strong>.</p><p>Nothing about the shape changes across the stack — it’s 65×128 the whole way. What changes is the <em>meaning</em>: each block re-reads the stream and rewrites it a little more abstract than the last (roughly: edges → parts → objects). Depth is <strong>refinement</strong>, not reshaping.</p><blockquote><em>This is the thread Part 5 pulls on: </em><strong><em>heads</em></strong><em> give the </em>width<em> of this processing (several attention subspaces at once), </em><strong><em>blocks</em></strong><em> give the </em>depth<em> (this whole move, repeated). </em>Head needs wider hands and longer legs.<em> Here we’re watching the legs — six of them.</em></blockquote><h3>The head — reading the answer off one token</h3><p>After the last block, we go back to that <strong>CLS token</strong> we planted at the front. Apply a final LayerNorm to it (LN(1×128) on the sheet), then one classifier matrix <strong>W_L [128×10]</strong>:</p><p>(1×128) · (128×10) = (1×10)</p><p>Ten numbers — one score per class. A <strong>softmax/argmax</strong> turns them into a probability / a prediction. Only the CLS token is read; through six blocks of attention it has quietly absorbed the whole image.</p><p><em>(The sheet norms just the CLS token, since that’s all the classifier needs; many implementations norm all 65 tokens and then select the CLS row — same result for the prediction.)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/293/1*h6nSmE_P9zgA8oYKXfyLXg.png" /></figure><h3>The bottom strip — how it learns</h3><p>Everything above is the <strong>forward</strong> pass. The strip along the bottom of the sheet is the <strong>backward</strong> pass — one weight update, shown for the classifier W_L.</p><ol><li><strong>Loss.</strong> Look at the <strong>bottom-left panel, “Loss Calculation (Difference)”</strong>: the two little rulers are the <strong>True</strong> label and the <strong>ViT O/P</strong> (the model’s prediction), side by side. The gap between them is the error signal <strong>δ</strong> — literally the <em>difference</em> the sheet names.</li><li><strong>Gradient.</strong> For a weight, the gradient is <strong>δ · x</strong> — the error times the input that fed it (the same error×input rule Series 4 Part 1 derives).</li><li><strong>Step.</strong> Scale by the learning rate: <strong>Δw = η · (δ · x)</strong>, and <strong>average </strong><strong>Δw over all samples in the batch</strong>.</li><li><strong>Weight decay.</strong> Add a small pull toward zero, <strong>η · λ · W_L</strong> (λ = weight-decay rate), which keeps weights from growing large.</li><li><strong>Update.</strong> Move the weight against the gradient — the next batch runs with the adjusted W_L. Repeat, and the model improves.</li></ol><h3>The box that sits inside every block — LayerNorm</h3><p>The <strong>parrot-green panel in the bottom-right corner</strong> is the “LN” opened up. Every LN on the sheet is this same operation, done <strong>token by token</strong> — the panel draws it literally: a column of 1×128 token-bars comes in on the left (<strong>Input</strong>), each is processed independently, and the same-shaped column leaves on the right (<strong>Output</strong>). 65×128 in, 65×128 out.</p><p>For one token’s 1×128 vector, two token-wise steps (both shown in the panel):</p><ul><li>normalise it to mean 0, variance 1: <strong>x̂ = (x − μ) / √(σ² + ε)</strong>,</li><li>then rescale and shift with two learned 1×128 vectors: <strong>y = γ ⊙ x̂ + β</strong>.</li></ul><p>γ and β are learned and — as the panel notes — <strong>shared across all 65 tokens</strong>: the same two knobs tune every column. It&#39;s the small, repeated box that keeps the numbers well-behaved as they flow through six blocks.</p><h3>The whole sheet as one lookup table</h3><p>The ultra-quick reference — the shape at every stage, top to bottom:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/479/1*CJzvWWgNFo49e6XLYEBcXQ.png" /></figure><p>Print this, keep it beside you, and any box on the big sheet is one glance from making sense.</p><h3>The parameter budget — how big is this ViT?</h3><p>Every <strong>dark-violet block</strong> on the sheet holds some learnable numbers. Count them all and you get the model’s <strong>size</strong>. Here’s the full budget — each block, its shape, and its parameter count — grouped into the <em>embeddings</em> (built once), <em>one encoder block</em> (repeated ×6), and the <em>head</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eFO2ep-z723mFhPGBQ3bsg.png" /><figcaption>Figure 2: Learning Budget: Count of Learnable Params/Weights</figcaption></figure><p>Two things worth noticing. First, <strong>the weight lives in the MLP</strong> — about <strong>two-thirds</strong> of every block — with attention taking the other third and the LayerNorms almost nothing. Second, one learnable block <strong>isn’t drawn on the sheet</strong>: the attention <strong>output projection </strong><strong>W_O (</strong><strong>128×128 per block)</strong>, which a multi-head attention needs to mix the heads back together. It&#39;s real, and it&#39;s about <strong>99K</strong> parameters across the six blocks — worth penciling onto the slide.</p><p>The grand total, <strong>1,205,898 ≈ 1.21 M</strong>, is small as models go (today’s large ones run to <em>billions</em>) — but every one of those parameters is one of the violet blocks above, tuned by the exact loss-and-update loop drawn along the bottom of the sheet.</p><h3>Where this goes next</h3><p>That’s a whole Vision Transformer on one page — pixels to class to weight update, every shape labelled. <strong>Part 4</strong> puts the <em>same</em> lens on an <strong>LLM</strong>: you’ll see the identical block — LayerNorm, Q/K/V, attention, MLP — with two changes (a causal mask and next-token loss). <strong>Part 5</strong> then zooms out across both sheets to the one idea that ties them together: <strong>heads make subspaces (width), blocks make refinement (depth)</strong> — <em>head needs wider hands and longer legs.</em></p><p>Keep the sheet open. That’s the whole point of a cheat sheet. 🙏</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=459f813b8a62" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ViT/LLM Algorithm, Part 2 — No Maths, Just What It Signifies: The Operations]]></title>
            <link>https://medium.com/@chandupadole/vit-llm-as-analgorithm-part-2-no-maths-just-what-it-signifies-the-operations-757ae031f4b7?source=rss-56067b0267b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/757ae031f4b7</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[attention]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <dc:creator><![CDATA[Chandrashekhar Padole]]></dc:creator>
            <pubDate>Wed, 12 Aug 2026 16:46:20 GMT</pubDate>
            <atom:updated>2026-08-20T15:56:40.757Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YGsDi62Ttgb6eGZO2OcKKA.jpeg" /></figure><p><strong>Abstract.</strong> Part 1 gave us the atoms — a <strong>vector</strong>, a <strong>weight</strong>, a <strong>dot product</strong>. This part snaps them into the handful of <strong>operations</strong> every model actually runs. We pick up where Part 1 left off — a <strong>projection matrix</strong> as a <em>learned bank of questions</em> — then open up <strong>how each of its weights learns</strong> (which input, which error, and how both change shape). Next the big idea: <strong>transformation</strong>, how a matrix multiply <em>reshapes</em> space (rotate, stretch, even change its number of dimensions) and how a <strong>nonlinearity</strong> then <em>bends</em> it, so that “reshape-then-bend” is exactly what a model’s little neural nets do. Then <strong>direction of information flow</strong> — whether a step may look ahead or only behind — and a one-line <strong>cheat</strong> that ties the whole toolkit together. Still no maths; only what each move <em>signifies</em>.</p><p><strong>In this 5-part series</strong> <em>(you are reading Part 2)</em>:</p><ol><li><a href="https://medium.com/@chandupadole/vit-llm-as-analgorithm-part-1-no-maths-just-what-it-signifies-the-primitives-fcdde4367287">Part 1 — No Maths, Just What It Signifies: the primitives</a></li><li><strong>Part 2 — No Maths, Just What It Signifies: the operations</strong> — <em>you are here</em></li><li><a href="https://medium.com/@chandupadole/vit-llm-algorithm-part-3-the-vit-cheat-sheet-one-slide-every-shape-459f813b8a62">Part 3 — The Vision Transformer, end to end: one slide, every shape</a></li><li><a href="https://medium.com/@chandupadole/vit-llm-algorithm-part-4-the-llm-cheat-sheet-one-slide-every-shape-ab16252140fd">Part 4 — The Large Language Model, end to end: one slide, every shape</a></li><li><a href="https://medium.com/@chandupadole/vit-llm-algorithm-part-5-width-vs-depth-heads-make-subspaces-blocks-transform-spaces-8eeb26708a19">Part 5 — Width vs Depth: heads make subspaces, blocks make refinement</a></li></ol><h3>Section 1 — The projection matrix: a fully-connected layer</h3><p>Quick recap from Part 1: dot a vector against <strong>N reference directions</strong> and you get <strong>N scores</strong> — a profile of “how much of each property is in me.” Stack those N directions together and the whole bundle has a name: a <strong>projection matrix</strong>. Push many inputs through it in one <strong>matrix multiplication</strong> and you score every input on every direction at once.</p><p>Two things to add now that turn this from a trick into <em>the</em> workhorse.</p><h3>The model learns the questions</h3><p>In Part 1 we <strong>hand-picked</strong> the directions — musical frequencies, “domesticated,” “four-legged.” A real model is never handed those. The columns of its projection matrices are <strong>knobs</strong> (Part 1, Section 3), and training tunes them so the N questions it asks are the <strong>useful</strong> ones — whatever best helps it predict. So a projection matrix is a <strong>learned question-bank</strong>: N questions, asked of every input at once, <em>discovered</em> rather than <em>designed</em>. Nobody writes down “ask about four-legged-ness”; the model decides, on its own, which N questions are worth asking.</p><h3>Different matrices ask different questions</h3><p>Because the questions <strong>are</strong> just the matrix, swapping in a different matrix asks a different set of questions and pulls out a different <strong>aspect</strong> of the very same input. That’s why a model often projects the <em>same</em> data through <strong>several</strong> learned matrices side by side — each one is a different <strong>lens</strong>. (You’ll meet this directly in the cheat sheets, where one token is turned into a <em>“what am I looking for,”</em> a <em>“what do I offer,”</em> and a <em>“what do I carry”</em> — three projections of one token, three different question-banks.)</p><h3>Under the hood, it’s a fully-connected layer</h3><p>Strip the word “projection” away and this matrix multiply is the plainest thing in a neural network: a <strong>fully-connected layer.</strong> Picture <strong>N input neurons</strong> (the numbers coming in) and <strong>M output neurons</strong> (the profile going out). Every input is wired to every output, and <strong>the weight </strong><strong>W[i,j] is simply the wire from input </strong><strong>i to output </strong><strong>j.</strong> Each output neuron does one humble thing — add up all its incoming wires: <em>output j = (input 1 × its wire) + (input 2 × its wire) + …</em> So every output is a <strong>weighted sum of all the inputs at once</strong>, which is all a neuron ever does.</p><p>Two things this settles:</p><ul><li><strong>“Does the input come from the vector, or from the previous layer?”</strong> They’re the <em>same thing</em>. The vector entering this matrix <strong>is</strong> whatever produced it — the previous layer’s output, or the embedding. Stack these matrices and you have the familiar <strong>input layer → hidden → hidden → output</strong> of any neural net; each matrix is one layer-to-layer step.</li><li><strong>The embedding lookup is the one-hot special case.</strong> When the input is a <strong>one-hot</strong> vector (all zeros except a single 1 at a word’s index), that weighted sum picks out exactly <strong>one row</strong> of the matrix — the word’s embedding. Only the active index “feeds”; every other input is 0, so the rest of the table sits idle. That’s why a <em>lookup</em> and a <em>matrix multiply</em> are secretly the same move.</li></ul><p>(One transformer twist to flag: these dense layers mix a token’s own <strong>features</strong>. Attention adds a <em>second</em> kind of wiring — across <strong>tokens</strong> — which you’ll meet in the cheat sheets.)</p><p>Zoom out and this settles what a “neural net” even <em>is</em>. The familiar <strong>mesh of nodes and connections</strong> is nothing more than a <strong>chain of these matrix multiplies</strong>, with an activation (the <em>bend</em> from Section 3) between them: the <strong>nodes</strong> are the vectors, the <strong>connections</strong> are the weights, and <strong>each layer is one matmul</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*STmC0UYR-ZUoM-KBNv-kdQ.png" /><figcaption>Figure 1: Neural Net Layers and Weight Matrix Operations</figcaption></figure><h3>Mix vs overlay: matmul’s quiet cousin, addition</h3><p>A matrix multiply <strong>mixes</strong>: every output slot draws on every input slot. But vectors combine a second, far simpler way too — plain <strong>addition</strong>, slot by slot: <em>outᵢ = aᵢ + bᵢ.</em> No weights, no wires between different slots — <strong>slot </strong><strong>i only ever meets slot </strong><strong>i.</strong></p><p>That’s an <strong>overlay</strong>, not a mix. It’s how you stamp one piece of information onto another <em>without disturbing it</em> — for instance, adding a <strong>positional</strong> signal onto a token’s embedding: each dimension gets nudged by a learned “where am I” amount, and nothing gets scrambled together. So keep the pair straight: <strong>a matmul mixes; an addition overlays.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r45eaI6yxo__SzLihv11iQ.png" /><figcaption>Figure 2: Different Operations with respect to Learning</figcaption></figure><h3>Section 2 — How a weight learns</h3><p>Everything so far was what a fully-connected layer <em>computes</em>. Now the flip side: <strong>how each of its weights gets better.</strong> One rule from Part 1 drives all of it — <em>a weight’s nudge = the input flowing through it × the error coming back</em> — and it turns out to carry a surprising amount of structure: <em>which</em> input, <em>which</em> error, and how both change shape as they travel through the network.</p><h3>Which input trains each weight</h3><p>Those three wirings also <strong>learn</strong> differently — and it comes down to one rule from Part 1: <strong>a weight’s nudge = the input flowing through it × the error coming back.</strong> Run that rule through each case:</p><ul><li><strong>Matmul</strong> — <strong>every</strong> weight learns; the wire from input i to output j is nudged by <strong>xᵢ × δⱼ</strong> (its own input, times its output&#39;s returning error).</li><li><strong>Lookup (one-hot)</strong> — the input is 0 everywhere except the active index, and 0 × error = 0. So <strong>only the row of the word that actually appeared gets updated</strong>; every other word&#39;s embedding is left untouched this step. (A word learns only on the steps it shows up.)</li><li><strong>Addition (a learnable added vector, like the positional one)</strong> — there’s <strong>no input to multiply</strong>, so the error passes <strong>straight through (×1)</strong>: each slot’s weight is nudged by <strong>its own slot’s error alone</strong>, with no mixing.</li></ul><p>Here’s the same idea with <strong>real numbers</strong>, on three layers you’ll actually meet — a ViT patch-embedding (a <em>multiply</em> that expands 48→128), a <em>positional add</em>, and a plain hidden→hidden <em>dense</em> layer — with a zoom on <strong>what one weight update is actually made of</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EWiIN9e-kdRXLOEuOWk5eA.png" /><figcaption>Figure 3: Weight Matrix used in Different Data Operations</figcaption></figure><h3>The update, in full: an outer product, and across depth</h3><p>Look a little closer at that update and two things stand out:</p><ul><li><strong>Across a matrix</strong> — the update isn’t one lonely cell, it’s the <strong>whole grid at once</strong>. Each weight W[i,j] is nudged by <strong>its row&#39;s input </strong><strong>xᵢ × its column&#39;s error </strong><strong>δⱼ</strong>: the row picks the input, the column picks the error. Together, the whole update is the <strong>outer product</strong> of the input vector and the error vector.</li><li><strong>Across depth</strong> — each layer’s weights learn from <strong>the activation feeding it</strong>, i.e. the <strong>previous layer’s output</strong>, <em>not</em> the network’s raw input. Only the <strong>first</strong> layer ever sees the input x; layer 2 learns from a¹, layer 3 from a², and so on down the stack.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V0D5J1fKNnWgbRZ8joJTlg.png" /><figcaption>Figure 4: Input, Weights, Error (loss) and Weight Updates</figcaption></figure><h3>The learning cycle — at one weight, and the next</h3><p>Put it together for a single weight matrix and the one error δ arriving from the output side does <strong>two separate jobs at once</strong> — and then the layer below does exactly the same, closing the cycle:</p><ol><li><strong>Update these weights</strong> <em>(needs the input)</em> — ΔW = input ⊗ error; then W_new = W_old − η·ΔW (our W₁₁: 0.20 → 0.17).</li><li><strong>Pass the error back</strong> <em>(needs the weights)</em> — δ_prev = δ·Wᵀ: the <em>same</em> error, multiplied by the weight matrix, flows <strong>backward</strong> and <strong>shrinks to match</strong> the previous layer (4 → 3).</li><li><strong>The previous layer repeats it</strong> — δ_prev is now <em>its</em> output error; with its own (different-size) input it does the same two jobs (ΔW_prev = x⁰ ⊗ δ_prev) and passes an even smaller error further back. The cycle runs layer by layer, all the way down.</li></ol><p>The split to remember: <strong>updating a weight needs the input; passing the error back needs the weights</strong> — which is exactly what the next section is about.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QO6QKr5oadgfpq37WGeYbw.png" /><figcaption>Figure 5: Weight update and Error Propagating Backward</figcaption></figure><h3>Direct feedback vs backpropagation — not the same thing</h3><p>When a model is wrong, it’s tempting to picture the fix as <strong>direct feedback</strong>: the output error handed <em>straight</em> to each weight, telling it how to change. Simple, immediate — and <strong>not how it works.</strong></p><p>What actually happens is <strong>backpropagation</strong>, and the difference is the whole reason training a deep network is subtle. The error is born at the very <strong>end</strong> (the output), and an early weight can only feel it by having it <strong>threaded all the way back through every weight in between</strong> — multiplied by each one on the way. A weight buried deep in the stack <strong>never sees the raw error</strong>; it feels a version that’s been passed through, and reshaped by, everything downstream of it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gPvBX-x8XxkACNYwOeoHgg.png" /><figcaption>Figure 6: Error gets propagated backward via computations involved in forward pass</figcaption></figure><p>That “through every weight” trip has two consequences. Each weight matrix W (shape [in × out]) does two jobs as the error passes:</p><ul><li><strong>it passes the error down:</strong> error_in = error_out · Wᵀ. Forward, W maps in → out; backward, Wᵀ maps the <strong>error</strong> out → in. <em>This is what changes the dimension.</em></li><li><strong>it updates itself:</strong> ΔW = input ⊗ error (the outer product from just above) — always matching W&#39;s own shape.</li></ul><p>So the error’s dimension <strong>retraces the forward pass in reverse</strong>, one weight at a time.</p><blockquote><strong><em>Forward, a matrix mixes </em></strong><strong><em>in-values into </em></strong><strong><em>out-values; backward, the same matrix mixes </em></strong><strong><em>out-errors back into </em></strong><strong><em>in-errors.</em></strong><em> There is no shortcut to the early weights — the error earns its way back through the whole stack.</em></blockquote><p>We’ll watch that shape-change happen with <strong>real numbers in an LLM</strong> at the end of this part.</p><h3>Section 3 — Transformation: reshaping space</h3><p>Back in Part 1, the dot product had <strong>two readings</strong> — <em>how alike</em> and <em>how much is present</em>. A matrix multiply, being a stack of dot products, inherits the same double life. So far we’ve read it <strong>one</strong> way: a <strong>bank of projections</strong> that <em>measures</em> the input against learned directions (and, in Section 2, learns them). Read it the <strong>other</strong> way and it stops being a measurement and becomes a <strong>move</strong> — it picks up the whole cloud of points and <strong>reshapes</strong> it. There are two flavours of reshaping: a <strong>linear</strong> one that re-poses the cloud rigidly, and a <strong>nonlinear</strong> one that actually <strong>bends</strong> it. Together they’re how a model sculpts meaning out of raw numbers.</p><h3>A — The linear transformation: rotate, stretch, reshape</h3><p>A single matrix, applied to every point, can only make a few kinds of coordinated move:</p><ul><li><strong>Rotate / reflect</strong> — spin the whole cloud to face a new way (or mirror it). Directions change; distances don’t.</li><li><strong>Stretch / squash</strong> — pull the cloud out along some directions and compress it along others. (This is your <em>expansion</em> and <em>contraction</em>.)</li><li><strong>Shear</strong> — slide layers past one another, like pushing the top of a deck of cards sideways.</li></ul><p>Here’s the quietly beautiful fact: <strong>any</strong> matrix, however complicated it looks, is just <strong>rotate → stretch/squash → rotate again.</strong> That’s the whole repertoire. So “a linear transformation” isn’t a zoo of tricks — it’s always that same three-step move. (The one thing a bare matrix <em>can’t</em> do is <strong>slide</strong> the cloud sideways without turning it — that’s <strong>translation</strong>, and it needs the little <strong>+ bias</strong> tacked on. Matrix + bias = the full &quot;affine&quot; reshaping.)</p><p>And <strong>changing dimensions is just extreme stretching:</strong></p><ul><li><strong>Dimension reduction</strong> — when the output has <strong>fewer</strong> slots, the stretch step <strong>flattens</strong> some directions to nothing. Think of a <strong>shadow</strong>: a 3-D object cast onto a 2-D wall keeps the important spread and drops the rest.</li><li><strong>Dimension expansion</strong> — when the output has <strong>more</strong> slots, it <strong>opens up new directions</strong> — extra room for the model to spread the data into.</li></ul><p>So your three types — <strong>spatial</strong> (rotate/scale/shear), <strong>reduction</strong>, <strong>expansion</strong> — aren’t three things. They’re <strong>one</strong> operation, a linear transformation, differing only in whether it <strong>keeps</strong>, <strong>drops</strong>, or <strong>adds</strong> dimensions. What it <em>signifies</em>: <strong>re-posing the data</strong> so the directions that matter line up with what the next step is looking for.</p><h3>B — The nonlinear transformation: bending the space</h3><p>Linear moves are powerful but <strong>rigid</strong>: straight lines stay straight, parallel stays parallel. You can rotate and stretch a sheet of paper all day, but you can’t make a straight crease follow a curve.</p><p>Some structure simply can’t be untangled that way. The classic picture: <strong>two rings of points, one inside the other.</strong> No straight cut separates them, and no amount of rotating or stretching lines them up. To pull them apart you have to <strong>bend</strong> the space itself.</p><p>That’s what a <strong>nonlinearity</strong> (GELU, ReLU) does: applied to each number, it <strong>bends</strong> the space — curves and folds it — so tangled things can be teased apart. Picture flattening a globe onto a paper map: you can’t do it without curving something. Or kneading dough: <strong>folding</strong> mixes in ways stretching alone never can.</p><h3>Why you need both — and why depth needs the bend</h3><p>This is the punchline that makes sense of every deep network. <strong>Stack linear transformations with nothing between them and they collapse into a single linear transformation.</strong> Ten matrix multiplies in a row equal <strong>one</strong> matrix — so depth <em>without</em> a nonlinearity buys you nothing; you could have used one layer.</p><p>Slip a <strong>bend</strong> between them and everything changes: reshape, bend, reshape, bend… now each layer builds on a genuinely <strong>new</strong> arrangement the previous ones couldn’t have made. The nonlinearity is the reason <strong>depth is worth having at all.</strong></p><h3>The pairing you’ll see everywhere: reshape → bend → reshape</h3><p>Put those together and you have the little neural net tucked inside every transformer block:</p><ul><li><strong>expand</strong> — a linear transform lifts the vector into a <strong>roomier</strong> space (more directions to work in),</li><li><strong>bend</strong> — a <strong>GELU</strong> curves and folds that roomy space,</li><li><strong>contract</strong> — a linear transform brings it back down, <strong>keeping</strong> the useful new structure.</li></ul><p><em>Reshape, bend, reshape.</em> Give the bend room, fold, then summarise. That single <strong>expand → GELU → contract</strong> is the model’s basic sculpting move — and it’s repeated in block after block.</p><p>You can even <em>see</em> it as a little net that widens then narrows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TyEHVz7L8YvRHvcjYg2ZVQ.png" /><figcaption>Figure 7: Expansion and Contraction: Weights involved in Dimensions Increase and Reduction</figcaption></figure><p>So the whole of “transformation” is two moves: a <strong>linear reshape</strong> (rotate/stretch, and drop or add dimensions) and a <strong>nonlinear bend</strong>. Everything a model does to a <em>single</em> vector’s meaning is some combination of those two.</p><p>The one thing transformation <em>doesn’t</em> touch is <strong>how information moves between positions</strong> in a sequence — whether a step is allowed to peek ahead or may only look back. That’s the next idea.</p><h3>Section 4 — Direction of information flow</h3><p>Transformation reshaped a <strong>single</strong> vector. But the moment we have a <strong>sequence</strong> — words in a sentence, patches in an image — a new question appears: when we compute something at position <em>n</em>, <strong>which other positions is it allowed to use?</strong> Only the past? Only the future? All of them? This is the <strong>direction of information flow</strong>, and it comes in exactly three flavours.</p><h3>Causal — look only backward</h3><p>A step at position <em>n</em> may use only the <strong>present and the past</strong> — positions <em>n, n−1, n−2, …</em> — <strong>never</strong> the future.</p><p>Think of <strong>live commentary</strong> on a match. At any moment you only have what’s happened so far; you can’t narrate the present using tomorrow’s result. Anything <strong>real-time</strong> is causal by necessity.</p><p>A <strong>language model reads exactly this way</strong>: to predict the next word, each position may look only at the words <strong>before</strong> it. It <em>has</em> to be causal — the future words are precisely what it’s trying to guess, so peeking would be cheating.</p><h3>Anti-causal — look only forward</h3><p>The mirror image: a step uses only the <strong>present and the future</strong> — <em>n, n+1, n+2, …</em> — and ignores the past. It’s like reading a sentence <strong>backwards</strong>, or planning from the destination back to the start.</p><p>You rarely build a purely anti-causal model, but it completes the picture — and it’s a real tool: some classic systems run one pass <strong>forward</strong> and one pass <strong>backward</strong> and combine them, and the backward pass is anti-causal.</p><h3>Non-causal — look everywhere</h3><p>A step may use the <strong>whole sequence at once</strong> — past, present, and future together. It needs the entire input in hand before it computes.</p><p>Think of <strong>editing a recorded video</strong>: because you have all the footage, you can fix an early frame using a later one. Not real-time — but you see everything.</p><p>A <strong>vision model reads an image this way</strong>: every patch may look at <strong>every</strong> other patch. There’s no “future” to protect — the whole picture is available at once. Non-causal = <strong>bidirectional</strong>.</p><h3>Why it matters: the same block, two directions</h3><p>Here’s the payoff, and it’s a small miracle. The transformer block is <strong>identical</strong> whether it’s causal or non-causal — same projections, same attention, same reshape-and-bend. The <strong>only</strong> difference is a single rule on the attention step: <em>may a position look ahead, or not?</em></p><ul><li>Add the “look only backward” rule → you get a <strong>generator</strong> that writes one token at a time.</li><li>Drop it → you get an <strong>all-at-once reader</strong> that takes in a whole image.</li></ul><p>One tiny switch — the <strong>direction of information flow</strong> — turns the very same machine into a <strong>writer</strong> or a <strong>seer</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/647/1*G-neL000nMlXPpIp3yPdwA.png" /></figure><h3>Section 5 — The cheat line</h3><p>That’s the whole toolkit. Across two parts, a handful of ideas:</p><ul><li><strong>Correlation</strong> (the dot product) — how alike two things are, or how much of one is inside the other.</li><li><strong>Projection</strong> — measure a vector against a <strong>learned bank of N questions</strong> → a profile of scores (and many inputs at once = one matrix multiply).</li><li><strong>Transformation</strong> — <strong>reshape</strong> the space (rotate, stretch, drop or add dimensions) and <strong>bend</strong> it (a nonlinearity); <em>reshape → bend → reshape</em>.</li><li><strong>Direction of flow</strong> — a rule for who may look at whom: <strong>backward-only</strong> (causal) or <strong>everywhere</strong> (non-causal).</li></ul><p>Every box in a real model is some combination of exactly these. So here is the whole series compressed to a single line — the <em>cheat</em> behind the cheat sheets:</p><blockquote><strong><em>A transformer just measures everything against everything (batched correlation), reshapes and bends the result (transformation), and repeats in depth — under one rule about who may look at whom (the flow direction).</em></strong></blockquote><p>Or, shorter still — the phrase to keep in your pocket:</p><blockquote><em>Measure, reshape, bend — again and again.</em></blockquote><p>That’s the entire game. When you open the ViT or LLM cheat sheet next and meet a wall of boxes and shapes, every one of them is one of these moves: the <strong>Q/K/V</strong> boxes are <strong>projections</strong>, the <strong>attention matrix</strong> is <strong>correlation</strong>, the <strong>MLP</strong> is <strong>reshape-then-bend</strong>, <strong>LayerNorm</strong> keeps it tidy, and the <strong>causal mask</strong> is the <strong>flow rule</strong>. Nothing else is hiding in there.</p><h3>The operations, already at work — in both ViT and LLM</h3><p>Just like the atoms, these operations don’t care whether they’re processing pixels or words — the <em>same</em> moves run in both models. Here’s the toolkit, at work in each:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/999/1*WM7hJY7k7Fb18hmWweigCg.png" /></figure><p>Almost everything is shared; the one real fork is the <strong>flow rule</strong> — a Vision Transformer looks <strong>everywhere</strong>, a Language Model looks only <strong>backward</strong>. That single difference is what turns the same machine into a <em>seer</em> or a <em>writer</em>.</p><h3>The error’s journey back, in real numbers (an LLM)</h3><p>Here’s the “how a weight learns” row made concrete — the exact puzzle it raises: if the output error is a 1×8192 vector but the MLP&#39;s hidden layer is only 1024-wide, how does one reach the other? <strong>Not directly</strong> — the error changes shape at every layer, by the same matrix transposed. In our LLM:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*iDRUWqQKBhcPRffNB7BqPQ.png" /></figure><p>The 8192 and the 1024 never meet directly. First the 8192 error is <strong>gathered down to </strong><strong>256</strong> at the head — each of those 256 numbers is a weighted sum of all 8192 (via W_Lᵀ). Only <em>then</em>, inside the MLP, is that 256 <strong>spread up to </strong><strong>1024</strong> (via W_downᵀ). The shapes always line up because it&#39;s the <strong>same matrix</strong> working in both directions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1JBOFOghtwqeBjQ8CUvP7Q.png" /><figcaption>Figure 8: Loss is propagated backward in LLM</figcaption></figure><h3>Where this goes next</h3><p>With the atoms (Part 1) and the operations (Part 2) in hand, the two <strong>cheat sheets</strong> are next: <strong>Part 3</strong> walks a whole <strong>Vision Transformer</strong> on one slide, and <strong>Part 4</strong> does the same for a <strong>Large Language Model</strong> — every box now something you can name. <strong>Part 5</strong> then steps back to the two ways these models scale: <strong>heads</strong> (width) and <strong>blocks</strong> (depth). <em>Head needs wider hands and longer legs.</em></p><p>Keep this page handy — it’s the key that unlocks the rest. 🙏</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=757ae031f4b7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[ViT/LLM Algorithm, Part 1 — No Maths, Just What It Signifies: The Primitives]]></title>
            <link>https://medium.com/@chandupadole/vit-llm-as-analgorithm-part-1-no-maths-just-what-it-signifies-the-primitives-fcdde4367287?source=rss-56067b0267b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/fcdde4367287</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[word-embeddings]]></category>
            <dc:creator><![CDATA[Chandrashekhar Padole]]></dc:creator>
            <pubDate>Wed, 12 Aug 2026 16:20:34 GMT</pubDate>
            <atom:updated>2026-08-20T15:55:46.266Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ePY7VKXRrLaJUbaJp0NAA.jpeg" /></figure><p><strong>Abstract.</strong> Before any diagram of a transformer makes sense, three tiny ideas have to <em>feel</em> obvious — a <strong>vector</strong>, a <strong>weight</strong>, and a <strong>correlation</strong>. This part explains all three with <strong>no maths</strong>, only what they <em>signify</em>. They’re the atoms; Part 2 snaps them together into the operations you’ll then see everywhere. We begin with how a <em>thing</em> (a person, a word) becomes a <strong>point</strong> described by its properties, then the <strong>dot product</strong> — <em>how alike are two things?</em> — that compares those points, and finally how a neuron is really just a <strong>bank of knobs</strong>.</p><p><strong>In this 5-part series</strong> <em>(you are reading Part 1)</em>:</p><ol><li><strong>Part 1 — No Maths, Just What It Signifies: the primitives</strong> — <em>you are here</em></li><li><a href="https://medium.com/@chandupadole/vit-llm-as-analgorithm-part-2-no-maths-just-what-it-signifies-the-operations-757ae031f4b7">Part 2 — No Maths, Just What It Signifies: the operations</a></li><li><a href="https://medium.com/@chandupadole/vit-llm-algorithm-part-3-the-vit-cheat-sheet-one-slide-every-shape-459f813b8a62">Part 3 — The Vision Transformer, end to end: one slide, every shape</a></li><li><a href="https://medium.com/@chandupadole/vit-llm-algorithm-part-4-the-llm-cheat-sheet-one-slide-every-shape-ab16252140fd">Part 4 — The Large Language Model, end to end: one slide, every shape</a></li><li><a href="https://medium.com/@chandupadole/vit-llm-algorithm-part-5-width-vs-depth-heads-make-subspaces-blocks-transform-spaces-8eeb26708a19">Part 5 — Width vs Depth: heads make subspaces, blocks make refinement</a></li></ol><h3>Section 1 — A vector is a point</h3><h3>First, forget the word “vector”</h3><p>A <strong>vector</strong> is just a <strong>list of properties describing something</strong>. That’s the whole idea. If you’ve ever filled a form, you’ve written a vector.</p><p>Take people. To describe a person, you list a few of their properties:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*P-LAge4HTVGFfic09stb-w.png" /></figure><p>Each <strong>row</strong> is one person turned into a list of values — that row <em>is</em> a vector. And here’s the quiet magic: once a person is a list of numbers, they become a <strong>point</strong>. People who are alike sit <strong>close together</strong> (Person 1 and Person 3 nearly overlap); people who differ sit <strong>far apart</strong> (Person 2 is off on its own). We haven’t done any maths — but “similar” has quietly become “nearby,” and “different” has become “far.”</p><p>So a vector does two jobs at once: the <strong>values inside one</strong> describe the different properties of a single entity, and <strong>many such vectors</strong> let us lay out many entities in the same space and see who’s near whom.</p><h3>Now, words — and why the usual way of storing them isn’t enough</h3><p>How does a computer normally hold a word? As <strong>ASCII</strong> — each character is one byte. cat is three bytes; dog is three different bytes. This does exactly one thing well: it keeps words <strong>distinguishable</strong>. It&#39;s perfect for <strong>storing</strong> documents, <strong>loading</strong> and <strong>retrieving</strong> them, <strong>transmitting</strong> them over a wire, and <strong>compressing</strong> them.</p><p>But it’s a <strong>mechanical</strong> representation — a <em>name tag</em>, not a <em>description</em>. In ASCII, cat and dog are as unrelated as cat and xylophone; the bytes carry the <strong>spelling</strong>, and nothing about the <strong>meaning</strong>. It can&#39;t tell you the things that actually matter about a word:</p><ul><li><strong>(a) what it signifies</strong> — that a cat and a dog are both <em>animals</em>, while a hospital is not;</li><li><strong>(b) its position/role</strong> in a sentence — subject, object, the thing doing vs the thing done to;</li><li><strong>© that meaning shifts with company</strong> — “river <strong>bank</strong>” vs “money <strong>bank</strong>”; the same letters, two meanings, decided by the words around it;</li><li><strong>(d) how words group</strong> — simply (cat, dog, sparrow are <em>animals</em>) or in a <strong>hierarchy</strong> (sparrow → bird → animal → living thing).</li></ul><p>ASCII keeps words apart. It never says <em>how</em> they’re related. For that, we need to do to words exactly what we did to people: <strong>give them properties.</strong></p><h3>Give a word properties — and it becomes a vector too</h3><p>Let’s describe a word by asking a few questions of it. Each question is a <strong>dimension</strong>. Say we pick four:</p><ol><li><strong>Domesticated?</strong> — pet ↔ stray ↔ forest/wild <em>(1 = pet, 0 = wild)</em></li><li><strong>Body plan</strong> — four-legged ↔ two-legged ↔ bird <em>(1 = four-legged, 0 = bird)</em></li><li><strong>Is it a place?</strong> — <em>(1 = yes, a type of location)</em></li><li><strong>Belongs to medicine?</strong> — <em>(1 = a medical thing)</em></li></ol><p>Now “project” a few words onto these four questions — just score each word 0-to-1 on each:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/611/1*n2yqu-N-VHeaQjWW4-jSPQ.png" /></figure><h3>From hand-picked to learnable embeddings</h3><p>In that toy, <strong>we</strong> chose the four questions and <strong>we</strong> filled in the scores. That’s great for intuition, but it doesn’t scale — nobody can hand-invent hundreds of meaningful questions and hand-score every word in a language. So we don’t. We let the model <strong>learn them</strong>.</p><p>Here’s what “learnable embedding” actually means, step by step:</p><ul><li><strong>Start blank.</strong> Give every word a list of, say, <strong>256</strong> numbers, filled with <strong>random</strong> values. At this point the vectors mean nothing — every word is a random point.</li><li><strong>Let usage teach it.</strong> As the model trains (on the simple job of predicting words in real text), it <strong>nudges</strong> these numbers so that words appearing in <strong>similar contexts</strong> drift <strong>closer together</strong>. Nobody ever tells it “cat and dog are similar” — it <em>discovers</em> that, because they show up in similar sentences, and slides their points near each other.</li><li><strong>The dimensions are anonymous.</strong> Unlike our toy, the learned axes are <strong>not</strong> labelled “domesticated” or “a place.” The model invents whatever ~256 directions best help it predict. A few may loosely line up with human notions (singular/plural, past/present, positive/negative), but most are <strong>blends</strong>. So <strong>no single number is “the cat-ness value”</strong> — meaning lives in <strong>directions and combinations</strong>, not in any one slot.</li><li><strong>Relationships become directions.</strong> Because the space is learned from usage, <em>analogies</em> turn into consistent <strong>moves</strong> through it: the step from “king” to “queen” ends up pointing the same way as “man” to “woman.” The space doesn’t just place words — it <strong>arranges</strong> them.</li></ul><p>And that’s the payoff: an embedding is a <strong>learned list of property-values</strong> — a point whose coordinates were written by training, not by us. Every word gets one such vector; stack them all and you have the model’s <strong>dictionary of meaning</strong>, one row per word.</p><p>So a <strong>person</strong> and a <strong>word</strong> are now the same kind of object: a list of properties, a point in space. Which raises the very next question — once two things are points, <strong>how do we measure whether they’re alike?</strong> That single comparison is the <strong>dot product</strong>, and it’s all of Section 2.</p><h3>Section 2 — The dot product: how alike are two things?</h3><p>Every model, underneath, keeps asking one question about pairs of vectors: <em>how alike are you two?</em> The operation that answers it is the <strong>dot product</strong>. Forget the name — what it does has a plain meaning anyone can hold.</p><p><em>(Recall from Section 1: a </em><strong><em>vector</em></strong><em> is just a list of numbers — a fingerprint of properties, a point in space. Here we take two of those points and compare them.)</em></p><h3>Two vectors in, one number out — a similarity score</h3><p>Line up two lists of the same length, multiply them slot by slot, and add up the results. Out comes a <strong>single number</strong> saying how much the two agree.</p><p>Say you and two friends rate three kinds of movie out of 5 — [action, romance, sci-fi]:</p><ul><li><strong>You:</strong> [5, 1, 4]</li><li><strong>Friend A:</strong> [4, 2, 5] → 5×4 + 1×2 + 4×5 = <strong>42</strong> — a big number: <strong>similar taste</strong></li><li><strong>Friend B:</strong> [1, 5, 1] → 5×1 + 1×5 + 4×1 = <strong>14</strong> — a small number: <strong>different taste</strong></li></ul><p>That’s the dot product: <strong>big when two vectors point the same way, small when they don’t.</strong> People call it <em>correlation</em>, but it’s really just “how much do these two lists agree?”</p><h3>The same sum, seen as projection — “how much of me is in you?”</h3><p>Here’s the lovely part: the <em>exact same</em> multiply-and-add has a second reading. Instead of “how alike,” ask <strong>“how much of vector A lies along vector B?”</strong> — that’s <strong>projecting</strong> A onto B.</p><p>Picture a leaning stick with the sun straight overhead: the <strong>shadow</strong> on the ground is how much of the stick lies flat. Projection is that shadow — how much of one thing points in the direction of another. The dot product measures exactly that shadow: to what <strong>extent</strong>, or by what <strong>factor</strong>, A is present in B (and, the very same number, B in A).</p><p>So one humble operation, two intuitions — <strong>similarity</strong> (how alike) and <strong>presence</strong> (how much of one is inside the other). Same arithmetic, both true.</p><h3>One vector against N property-vectors → a profile of N scores</h3><p>Now it gets useful. Suppose we keep <strong>N reference vectors</strong>, each standing for one <strong>property</strong> — a “domesticated” direction, a “four-legged” direction, a “medical” direction, and so on. Take a given vector and dot it against <strong>each</strong> of the N. You get <strong>N numbers</strong>.</p><p>Each number answers: <em>how much of that property is present in my vector?</em> So projecting one vector onto N property-directions turns it into a <strong>profile</strong> — a readout of “how much of each property do you contain?” That is how a plain list of numbers becomes a <strong>meaningful description</strong>: by measuring it against a set of property-directions.</p><h3>A signal as a recipe of frequencies</h3><p>The property-directions don’t have to be abstract — here’s a tangible set you can almost hear.</p><p>Take a short <strong>signal</strong>: just N numbers sampled over time — say <strong>16 points</strong> of a wiggly line that, at a glance, looks like noise. Now keep N <strong>frequency vectors</strong> as our reference directions: each one is a <strong>pure wave</strong> sampled at the same 16 points — a slow wave, a slightly faster one, a faster one still, and so on up.</p><p><strong>Project the signal onto each frequency</strong> (a dot product with each wave). Every projection value answers one question: <strong>how much of that frequency is in the signal, and by what factor?</strong> A big value means “there’s a strong slow-wave component in here”; a near-zero value means “almost none of that frequency.”</p><p>So a messy-looking signal turns into a clean <strong>recipe</strong>: “mostly a slow wave, with a pinch of a fast one.” That’s exactly what the <strong>bars on an audio equaliser</strong> show — each bar is one projection, how much of that pitch is present — or how a <strong>prism</strong> splits white light into colours, each colour a frequency and its brightness the amount. Same operation as before: one dot product per reference direction. Here the “properties” just happen to be <strong>frequencies</strong>, and the projections reveal <strong>which frequencies the signal is built from, and how strongly.</strong></p><p><em>(This is the idea behind the Fourier transform — and, under the hood, behind how audio and images like JPEG are analysed and compressed.)</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ivsRzch6H0vCHTTch5RdA.png" /><figcaption>Figure 1: Signal/Sequence or vector can be projected different properties vectors</figcaption></figure><h3>Many vectors against the same N → the whole table at once</h3><p>One vector against N gave us N scores. So <strong>M vectors</strong> against the same N properties give an <strong>M × N table</strong> — every input scored on every property.</p><p>And because all M inputs are measured against the <strong>same</strong> N directions, we don’t grind them out one at a time — we line the M inputs up and get <strong>all M × N scores together, in one sweep.</strong> That is the whole secret of why these models are fast: <strong>one big operation is really just a batch of “how alike?” comparisons, done all at once.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i0BcKHPc91fIzvrjroAQTw.png" /><figcaption>Figure 2: Projection of one input vector or many vectors on same matrix gives corresponding output vector</figcaption></figure><p>Make it concrete with our signals again. A single signal became a row of frequency-strengths by projecting it onto each frequency wave. Now take <strong>many signals</strong> and stack them as the <strong>rows</strong> of a table, and keep those <strong>same frequency waves</strong> as the <strong>columns</strong> of a <strong>property matrix</strong>. Multiply the two tables together — a single <strong>matrix multiplication</strong> — and out falls a table where <strong>each row is one signal’s frequency recipe.</strong> Dozens of signals, all analysed against the same frequencies, in one step. <em>That</em> step — stack the inputs, multiply by the shared property matrix — <strong>is</strong> the matrix multiplication sitting at the heart of every model.</p><p>Hold onto this picture — <em>dot product = how alike, or how much of one is in the other</em> — and <em>stacking it up measures everything against everything.</em> Nearly every box in a model is this idea, at scale.</p><h3>Section 3 — Weights are knobs</h3><h3>A neuron just takes a weighted vote</h3><p>“Neuron” sounds intimidating. It isn’t. One neuron does a humble, everyday thing: it looks at several inputs, decides <strong>how much each one matters</strong>, and adds them into a single score.</p><p>You do this whenever you weigh a decision. Estimating what a house is worth, you glance at a few things — how big it is, how many bedrooms, how far from the city, how old — and in your head some of those count for more than others. Size counts a lot. Age counts a little. Being far from the city counts <em>against</em> it.</p><p>A neuron just writes those “how much each counts” values down as numbers — the <strong>weights</strong> — and adds up the votes:</p><blockquote><strong><em>estimate</em></strong><em> = (size × </em>w₁<em>) + (bedrooms × </em>w₂<em>) + (distance × </em>w₃<em>) + (age × </em>w₄<em>)</em></blockquote><p>That’s the whole operation: multiply each input by its weight, sum the results. A <strong>weight is simply how much that input matters to the answer.</strong></p><h3>What a weight’s size and sign mean</h3><p>Two things about a weight tell you what the model has learned:</p><ul><li><strong>Size = strength.</strong> A big weight means “this input strongly moves the answer.” A weight near <strong>zero</strong> means “basically ignore this one.”</li><li><strong>Sign = direction.</strong> A <strong>positive</strong> weight means “more of this pushes the answer <strong>up</strong>”; a <strong>negative</strong> weight means “more of this pushes it <strong>down</strong>.”</li></ul><p>For the house, w₁ (size) and w₂ (bedrooms) come out <strong>positive and large</strong> — bigger, more bedrooms, higher price. w₃ (distance) and w₄ (age) come out <strong>negative</strong> — farther and older drag the price down. So if you could read a trained model&#39;s weights, you&#39;d be reading, in plain sight, <strong>what it decided to care about and which way each thing pulls.</strong></p><h3>Learning is just turning the knobs</h3><p>Where do the weights come from? The same place the embedding numbers did in Section 1: they <strong>start random</strong>. The model’s very first guess is nonsense.</p><p>Then a beautifully simple loop runs:</p><ol><li><strong>guess</strong> with the current weights,</li><li><strong>check</strong> against the right answer — how far off are we? (the <em>error</em>),</li><li><strong>turn each knob</strong> a tiny bit in the direction that would have made the guess <em>less</em> wrong,</li><li><strong>repeat</strong>, on example after example after example.</li></ol><p>Picture a wall of dials and one needle labelled “how wrong.” Learning is nudging every dial, over and over, to push that needle toward zero. There’s no magic step — just patient <strong>knob-turning to reduce error</strong>. (That “how wrong, and which way to turn each knob” is exactly what the loss-and-update strip along the bottom of a model diagram is doing.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AUUjtwgM11KFF8hySYm4iA.png" /><figcaption>Figure 3: Knob and Weight/Coefficient Analogy: controlling input to output</figcaption></figure><h3>The whole network is one big bank of knobs</h3><p>Two things follow — and they’re why this tiny idea matters so much:</p><ul><li><strong>Embeddings are knobs too.</strong> Those 256 numbers per word from Section 1 aren’t a special kind of object; they <strong>start random and get turned by this same loop.</strong> An embedding is just a <em>row of weights</em> that happens to describe a word.</li><li><strong>Everything is knobs.</strong> Every table of numbers you’ll meet later — the projections, the hidden layers, the final classifier — is just a big grid of these weights. A modern model is <strong>millions or billions of the same little knob</strong>, all turned together by the same “reduce the error” rule.</li></ul><p>So when someone says a model “has 400 million parameters,” they mean <strong>400 million knobs</strong>, and “training” is the act of turning them. That’s the entire trick, top to bottom.</p><p>And notice what quietly ties this whole part together. Look again at the neuron’s core move: <strong>line up the inputs with the weights, multiply them pair by pair, and add it all up.</strong> You’ve met that operation already — it’s the <strong>dot product</strong> from Section 2. A neuron is nothing but a dot product between its inputs and its learned weights; an embedding is a row of those same weights. So the three “atoms” were secretly one small family all along: a <strong>vector</strong> (a point), a <strong>weight</strong> (a knob), and the <strong>dot product</strong> that compares them.</p><h3>The three atoms, already at work — in both ViT and LLM</h3><p>We haven’t built either model yet (the cheat sheets are Parts 3 and 4) — but it’s worth seeing, right now, that these three little ideas are <em>all</em> a transformer is made of, and that <strong>it makes no difference whether the input is pixels or words.</strong> Here’s each atom, at work in both:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/1*H9NaXWAZbNmpZ3oVE7oZeA.png" /></figure><p>Read either column and it’s the same three moves: <strong>look up points, measure them against each other with dot products, and turn knobs until the guesses improve.</strong> Whether the model ends up naming a picture or writing the next word, it is built from a <strong>point</strong>, a <strong>knob</strong>, and a <strong>comparison</strong> — nothing else.</p><p><strong>Next:</strong> Part 2 shows how these atoms snap together into the handful of <strong>operations</strong> every transformer runs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fcdde4367287" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building LLM, Part 8 — DPO, and the Whole Post-Training Landscape]]></title>
            <link>https://medium.com/@chandupadole/building-llm-part-8-dpo-and-the-whole-post-training-landscape-4130154824fd?source=rss-56067b0267b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/4130154824fd</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[rlhf]]></category>
            <category><![CDATA[dpos]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Chandrashekhar Padole]]></dc:creator>
            <pubDate>Sun, 02 Aug 2026 17:50:49 GMT</pubDate>
            <atom:updated>2026-08-04T20:06:00.182Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Abstract.</strong> We’ve now built four ways to shape a base model — SFT, DPO, PPO, GRPO — and proven each on real numbers. This finale ties them together. It gives DPO the <em>derivation</em> the series kept promising — the short piece of algebra that makes an entire reward model and RL loop vanish into one supervised loss (“RLHF without the RL”) — then zooms out to a single map of the whole post-training landscape: what each method learns from, what machinery it needs, and which one to reach for. One picture, four methods, the end of the road from pixels to a well-behaved language model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6OW_jZTv4XwuWfyj_UjD0g.jpeg" /></figure><p><strong>In this 8-part series</strong> <em>(you are reading Part 8)</em>:</p><ol><li><a href="https://medium.com/@chandupadole/building-llm-part-1-training-predicting-the-next-token-44c72473e486">Part 1 — Training: predicting the next token</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-2-inference-how-a-trained-model-writes-88458eca1d47">Part 2 — Inference: how the model writes</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-3-vit-vs-llm-one-architecture-two-senses-71f2c6f12970">Part 3 — ViT vs LLM: what carries over, what’s new</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-4-fine-tuning-teaching-a-model-a-new-behaviour-cef8cf0ba866">Part 4 — Fine-tuning: teaching the model to follow instructions</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-5-rlhf-aligning-a-model-to-preferences-ac148647539e">Part 5 — RLHF: learning from human preference</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-6-ppo-the-on-policy-engine-of-rlhf-5ab9fe300635">Part 6 — PPO: the policy-gradient engine</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-7-grpo-reinforcement-without-a-value-network-3e43ba1454f7">Part 7 — GRPO: reinforcement without a value network</a></li><li><strong>Part 8 — DPO: preference learning with no RL loop</strong> — <em>you are here</em></li></ol><blockquote><em>This is </em><strong><em>Part 8</em></strong><em>, the finale. It assumes the whole series — SFT (</em><a href="https://medium.com/@chandupadole/building-llm-part-4-fine-tuning-teaching-a-model-a-new-behaviour-cef8cf0ba866">Part 4</a><em>), DPO (</em><a href="https://medium.com/@chandupadole/building-llm-part-5-rlhf-aligning-a-model-to-preferences-ac148647539e">Part 5</a><em>), PPO (</em><a href="https://medium.com/@chandupadole/building-llm-part-6-ppo-the-on-policy-engine-of-rlhf-5ab9fe300635">Part 6</a><em>), GRPO (</em><a href="https://medium.com/@chandupadole/building-llm-part-7-grpo-reinforcement-without-a-value-network-3e43ba1454f7">Part 7</a><em>). Here we place them all on one map and give DPO the theory it earned.</em></blockquote><h3>Four ways to shape a model, one map</h3><p>Every method in this series takes a pretrained base and bends it toward what we want. They differ on two axes: <strong>what signal they learn from</strong> (demonstrations vs preferences/rewards) and <strong>how they learn</strong> (offline, supervised-style vs on-policy RL):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TXoPwNWTCDOCDCNsG4b9dg.png" /><figcaption>Figure 1: Post-Training Landscape</figcaption></figure><p><strong>SFT</strong> and <strong>DPO</strong> live on the bottom row: both train like ordinary supervised learning on a <em>fixed</em> dataset — SFT from demonstrations, DPO from preference pairs. <strong>PPO</strong> and <strong>GRPO</strong> live on the top row: they’re <em>on-policy</em>, learning from samples the model generates and scores itself. That single map is the whole taxonomy.</p><h3>DPO revisited: where does the reward go?</h3><p>Back in Part 5 we used DPO and it worked, but we waved at <em>why</em> it needs no reward model. Here’s the actual algebra — and it’s short:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*glqvj3uoFw9l9E0aVyyvpA.png" /><figcaption>Figure 2: Why DPO is “RLHF without RL”</figcaption></figure><p>The trick is step 3 → 5. The RLHF objective has a <em>closed-form</em> optimal policy, so you can solve it <strong>backwards</strong>: write the reward in terms of the policy. Substitute that into the preference model and the reward — the entire thing you’d have trained a separate network for — <strong>cancels out</strong>, leaving a loss written purely over the policy’s own log-probabilities:</p><pre># π_ratio(y) = logπ_θ(y) − logπ_ref(y)   — the &quot;implicit reward&quot; of reply y<br>loss = -F.logsigmoid(beta * (pi_ratio(chosen) - pi_ratio(rejected))).mean()</pre><p>No reward model, no rollouts — the reason DPO runs like plain supervised training, and the literal meaning of “RLHF without the RL.”</p><h3>The whole comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/945/1*zvalq2Ma2q9xCOUJShOmzQ.png" /></figure><p>The shape of the table tells the story: SFT is the foundation; DPO is the cheap, stable way to add preferences; PPO and GRPO pay for on-policy exploration, with GRPO shedding PPO’s critic.</p><h3>Which one should you reach for?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6SgrLdF8SZg3WAngFYgteQ.png" /><figcaption>Figure 3: Overall Post-Training</figcaption></figure><p>In practice: <strong>always SFT first</strong> — nothing else works well on a model that can’t yet answer. Then, to align to preference, <strong>DPO is the default</strong> — cheap, stable, and surprisingly strong. Reach for <strong>on-policy RL</strong> (GRPO, then PPO) when the model genuinely needs to <em>explore</em> and be scored on its own fresh samples — reasoning tasks with verifiable rewards being the prime example.</p><h3>Proven, not asserted</h3><p>One thing tied every part of this series together: we never just claimed a method works — we <em>measured</em> it, before and after, on real code:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eN8CQVsMwbydqo0KiLdXcQ.png" /><figcaption>Figure 4: Performance measures for SFT, DPO, PPO and GRPO</figcaption></figure><p>Four methods, four numbers that moved — from the emoji experiments on TinyLlama to the repo’s own PPO and GRPO code climbing to a perfect reward. That habit — <em>measure, change, measure again</em> — is worth more than any single algorithm.</p><h3>The end of the road</h3><p>Step back and look at the whole journey. In Series 1 we built a Vision Transformer from pixels. In this series we crossed to language and found the <em>same</em> transformer — a lookup table, a causal mask, next-token loss — then taught it to speak coherent Indian fables (Parts 1–2), proved it’s the ViT’s twin (Part 3), fine-tuned it to follow instructions (Part 4), and aligned it to human preference four different ways (Parts 5–8).</p><p>From a grid of pixels to a model that answers the way you want it to — every mechanism built by hand, every claim backed by a number. That’s a language model, from first principles. Thanks for reading. 🙏</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4130154824fd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building LLM, Part 7 — GRPO: Reinforcement Without a Value Network]]></title>
            <link>https://medium.com/@chandupadole/building-llm-part-7-grpo-reinforcement-without-a-value-network-3e43ba1454f7?source=rss-56067b0267b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e43ba1454f7</guid>
            <category><![CDATA[reasoning]]></category>
            <category><![CDATA[grpo]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Chandrashekhar Padole]]></dc:creator>
            <pubDate>Sun, 02 Aug 2026 06:06:38 GMT</pubDate>
            <atom:updated>2026-08-04T19:59:05.666Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Abstract.</strong> PPO (Part 6) worked — but it hauled around a whole second network, the <strong>critic</strong>, just to estimate a baseline for its advantages. <strong>GRPO</strong> — the algorithm behind DeepSeek-R1 — throws the critic away with one beautifully simple idea: for each prompt, sample a <em>group</em> of answers, and let the group’s own mean and spread be the baseline. An answer is “good” if it beat its groupmates. This part builds it from first principles and runs our own real GRPO code to the same perfect reward PPO reached — <strong>with no value network at all</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X-XxheBd9lu4EGbytZQjLw.jpeg" /></figure><p><strong>In this 8-part series</strong> <em>(you are reading Part 7)</em>:</p><ol><li><a href="https://medium.com/@chandupadole/building-llm-part-1-training-predicting-the-next-token-44c72473e486">Part 1 — Training: predicting the next token</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-2-inference-how-a-trained-model-writes-88458eca1d47">Part 2 — Inference: how the model writes</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-3-vit-vs-llm-one-architecture-two-senses-71f2c6f12970">Part 3 — ViT vs LLM: what carries over, what’s new</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-4-fine-tuning-teaching-a-model-a-new-behaviour-cef8cf0ba866">Part 4 — Fine-tuning: teaching the model to follow instructions</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-5-rlhf-aligning-a-model-to-preferences-ac148647539e">Part 5 — RLHF: learning from human preference</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-6-ppo-the-on-policy-engine-of-rlhf-5ab9fe300635">Part 6 — PPO: the policy-gradient engine</a></li><li><strong>Part 7 — GRPO: reinforcement without a value network</strong> — <em>you are here</em></li><li><a href="https://medium.com/@chandupadole/building-llm-part-8-dpo-and-the-whole-post-training-landscape-4130154824fd">Part 8 — DPO: preference learning with no RL loop</a></li></ol><blockquote><em>This is </em><strong><em>Part 7</em></strong><em>. It’s a direct simplification of PPO from </em><a href="https://medium.com/@chandupadole/building-llm-part-6-ppo-the-on-policy-engine-of-rlhf-5ab9fe300635">Part 6</a><em>— same rollout-and-clip core, one fewer network. Read Part 6 first; here we only remove a piece.</em></blockquote><h3>The idea: let the group be the baseline</h3><p>GRPO keeps PPO’s shape — generate, score, clip, update — but changes where the <em>advantage</em> comes from:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vodhAKF7sm6YPId3DldLyA.png" /><figcaption>Figure 1: GRPO Workflow</figcaption></figure><p>That’s the whole change. Instead of asking a trained critic “how good was this position supposed to be?”, GRPO samples several answers to the <em>same</em> prompt and asks “how did this one do <strong>compared to its siblings</strong>?” The group is the baseline.</p><h3>The only difference from PPO</h3><p>Put the two side by side and the entire delta is one box:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4g16tQEfUZE7oHgrXBeuRg.png" /><figcaption>Figure 2: Difference between PPO and GRPO</figcaption></figure><p>PPO spends parameters, a value loss, and tuning effort learning a critic. GRPO gets the same baseline for free from samples it was already generating. Everything else — the clipped surrogate, the KL leash — is identical.</p><h3>Group-relative advantage, made concrete</h3><p>Here’s the mechanism on a real group of six answers to one prompt, scored right (1) or wrong (0):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PY3ZSqXLT9GC5PCj3b8U-Q.png" /><figcaption>Figure 3: Group Relative Advantage</figcaption></figure><p>Standardise the rewards <em>within the group</em> and you get the advantage directly — it’s the whole algorithm:</p><pre>def group_advantages(rewards, group_size, eps=1e-4):<br>    r = rewards.view(-1, group_size)                     # group-contiguous<br>    adv = (r - r.mean(1, keepdim=True)) / (r.std(1, keepdim=True) + eps)<br>    return adv.reshape(-1)</pre><p>There’s an elegant safety valve built in: if every answer in a group scores the same (all right or all wrong), the spread is zero, the advantage is ~0, and that group contributes <strong>no gradient</strong> — so it can’t push the model in a random direction. (Trainers log the fraction of “informative” groups for exactly this reason.)</p><h3>The loss: PPO’s clip, minus the value term</h3><p>The update is PPO’s clipped surrogate — the advantage broadcast across a completion’s tokens — plus a per-token KL penalty to the reference. No value loss, because there’s no value head:</p><pre>ratio     = torch.exp(new_logp - old_logp)<br>surrogate = torch.min(ratio * adv, torch.clamp(ratio, 1 - clip, 1 + clip) * adv)<br>kl        = k3_kl(new_logp, ref_logp)                 # exp(Δ) − Δ − 1, always ≥ 0<br>loss      = -masked_mean(surrogate - kl_coef * kl, resp_mask)</pre><p>Everything you learned about the clip in Part 6 applies unchanged. GRPO simply deletes the critic and its loss.</p><h3>Does it work — without the critic?</h3><p>Same test as Part 6, same real code, now the GRPO path. Its reward climbs from <strong>0.13 to a perfect 1.00</strong> — and lands exactly where PPO did:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SFFGjGCRV8Fg-TvNW4A53w.png" /><figcaption>Figure 4: GRPO reward</figcaption></figure><p>Identical outcome, one fewer network to train and tune. That’s why GRPO has become the default for <strong>RLVR</strong> — reinforcement learning from <em>verifiable</em> rewards, like getting a maths answer right — and the engine behind recent reasoning models.</p><h3>Honest caveats</h3><ul><li><strong>Informative groups matter.</strong> If the model is so weak that every answer in every group is wrong (or a task so easy every answer is right), the spread is zero and there’s no signal. Trainers use a <strong>curriculum</strong> (easy tasks first) and tune group size / temperature to keep groups informative.</li><li><strong>Verifiable rewards.</strong> GRPO shines when the reward is cheap and reliable to compute (a maths checker, a unit test). For fuzzy human-preference rewards, a reward model — or DPO — is often the better fit.</li><li>Our reward is a learnable toy so the mechanism is visible and runs in under a minute; the algorithm is exactly the one used at scale.</li></ul><h3>What’s next</h3><p>We’ve now seen the full RL spectrum: PPO (on-policy, with a critic), GRPO (on-policy, no critic). Both <em>generate and score their own samples</em>. In <strong>Part 8</strong> we close the loop by returning to <strong>DPO</strong> for a proper deep-dive — the method that gets much of this alignment benefit with <strong>no reinforcement learning at all</strong>, straight from a fixed set of preference pairs. Three roads to a well-behaved model; the last one is the shortest.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e43ba1454f7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building LLM, Part 6 — PPO: The On-Policy Engine of RLHF]]></title>
            <link>https://medium.com/@chandupadole/building-llm-part-6-ppo-the-on-policy-engine-of-rlhf-5ab9fe300635?source=rss-56067b0267b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/5ab9fe300635</guid>
            <category><![CDATA[rlhf]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ppo]]></category>
            <dc:creator><![CDATA[Chandrashekhar Padole]]></dc:creator>
            <pubDate>Sat, 01 Aug 2026 14:22:28 GMT</pubDate>
            <atom:updated>2026-08-04T19:51:41.865Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Abstract.</strong> DPO (Part 5) aligned our model <em>offline</em> — from a fixed pile of preference pairs it never got to add to. Classic RLHF does something more powerful and more delicate: it lets the model <strong>generate its own attempts</strong>, scores them, and nudges it toward the ones that scored well. That’s <strong>PPO</strong> — Proximal Policy Optimization, the algorithm behind the original ChatGPT. This part builds it from first principles: the actor-critic, rollouts, advantages (GAE), and the <em>clipped</em> objective that keeps the whole thing from blowing up. And rather than assert it works, we run our own real PPO code on a learnable task and watch the reward climb from <strong>0.26 to a perfect 1.00</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jdAwAAWrYzyup5mrm0NZTw.jpeg" /></figure><p><strong>In this 8-part series</strong> <em>(you are reading Part 6)</em>:</p><ol><li><a href="https://medium.com/@chandupadole/building-llm-part-1-training-predicting-the-next-token-44c72473e486">Part 1 — Training: predicting the next token</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-2-inference-how-a-trained-model-writes-88458eca1d47">Part 2 — Inference: how the model writes</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-3-vit-vs-llm-one-architecture-two-senses-71f2c6f12970">Part 3 — ViT vs LLM: what carries over, what’s new</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-4-fine-tuning-teaching-a-model-a-new-behaviour-cef8cf0ba866">Part 4 — Fine-tuning: teaching the model to follow instructions</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-5-rlhf-aligning-a-model-to-preferences-ac148647539e">Part 5 — RLHF: learning from human preference</a></li><li><strong>Part 6 — PPO: the policy-gradient engine</strong> — <em>you are here</em></li><li><a href="https://medium.com/@chandupadole/building-llm-part-7-grpo-reinforcement-without-a-value-network-3e43ba1454f7">Part 7 — GRPO: reinforcement without a value network</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-8-dpo-and-the-whole-post-training-landscape-4130154824fd">Part 8 — DPO: preference learning with no RL loop</a></li></ol><blockquote><em>This is </em><strong><em>Part 6</em></strong><em>. It builds on the RLHF framing from </em><a href="https://medium.com/@chandupadole/building-llm-part-5-rlhf-aligning-a-model-to-preferences-ac148647539e">Part 5</a><em>. Where DPO folded the reward and the RL loop into one offline loss, PPO keeps the real loop — and this part opens it up.</em></blockquote><h3>The idea: try, score, nudge — on repeat</h3><p>PPO is a loop. The model generates completions, a reward scores them, and a carefully <em>bounded</em> update raises the probability of the good ones:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-yhoLODix65BX0AIXCopIQ.png" /><figcaption>Figure 1: PPO workflow</figcaption></figure><p>The word that matters is <strong>on-policy</strong>: unlike DPO’s fixed pairs, PPO learns from samples <em>the current model just produced</em>. That’s why it can keep improving past any pre-collected dataset — and also why it’s touchy, since the data distribution shifts as the model does. A <strong>KL penalty</strong> to a frozen reference is the leash that stops it wandering off a cliff.</p><h3>The actor-critic</h3><p>PPO needs two things from the model at every token: what to do (the policy) and how good the current position is (a value estimate). It gets both from <strong>one backbone with two heads</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gsehvjojO55UqA5z4eE-3w.png" /><figcaption>Figure 2: Actor-Critic schema</figcaption></figure><pre>def forward(self, idx):<br>    hidden = self.transformer.forward_hidden(idx)<br>    logits = self.transformer.lm_head(hidden)      # policy  (the actor)<br>    values = self.value_head(hidden).squeeze(-1)   # V(s)    (the critic)<br>    return logits, values</pre><p>The <strong>critic</strong> exists to answer &quot;how much reward should I expect from here?&quot; — so PPO can judge whether an action did <em>better than expected</em> (its <strong>advantage</strong>) rather than just &quot;was the final reward high?&quot;. That comparison is what keeps the gradient&#39;s variance down. Because both heads share the backbone (Part 1&#39;s forward_hidden), the critic is nearly free.</p><h3>Advantages: GAE</h3><p>After a rollout we have a reward at the last token and a value estimate at every token. <strong>Generalized Advantage Estimation</strong> turns those into a per-token advantage — &quot;how much better than expected was this step?&quot; — walking backwards through the sequence:</p><pre>for t in reversed(range(L)):<br>    nonterminal = m[:, t + 1] if t + 1 &lt; L else 0.0<br>    delta   = rewards[:, t] + gamma * values_next[:, t] * nonterminal - values[:, t]<br>    lastgae = delta + gamma * lam * nonterminal * lastgae<br>    adv[:, t] = lastgae<br>returns = adv + values</pre><p>The per-token reward is the <strong>KL-to-reference penalty</strong> at every response token, plus the scalar task reward added at the <strong>last</strong> token — so the model is paid for the outcome and gently taxed for drifting from the reference.</p><h3>The clipped objective — PPO&#39;s whole trick</h3><p>Here is the piece that makes PPO <em>stable</em>. It measures how much the new policy changed the probability of an action (the ratio r = π_new / π_old) and refuses to reward moving it too far in one step:</p><pre>ratio = torch.exp(new_logp - old_logp)<br>surr1 = ratio * advantages<br>surr2 = torch.clamp(ratio, 1.0 - clip, 1.0 + clip) * advantages<br>loss  = -masked_mean(torch.min(surr1, surr2), mask)   # clip = 0.2</pre><p>Plotting that objective against the ratio shows exactly what the min + clamp buy you:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NaTROZ9W5K6M_QExweWFqg.png" /><figcaption>Figure 3: PPO objectives</figcaption></figure><p>Inside the shaded band (r within 1 ± 0.2) the update behaves normally. Outside it the objective goes <strong>flat</strong> — there&#39;s no more gradient reward for pushing the policy further — so a single update can&#39;t lunge. That one clamp is the difference between PPO and a policy-gradient method that quietly detonates.</p><h3>Does it actually optimize?</h3><p>Talk is cheap; the series runs the code. Using our <strong>real</strong> PPO implementation — the same rollout → GAE → clipped-update path, unchanged — on a tiny model with a learnable reward (&quot;emit the target token&quot;), here&#39;s the mean reward per iteration:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oRKsRPu_x-nc1XkLWXPWmQ.png" /><figcaption>Figure 4: Rewards with PPO interations</figcaption></figure><p>From <strong>0.26 to a perfect 1.00</strong>, versus a random baseline of 1/64 = 0.016. The policy genuinely learned to maximise the reward through the full actor-critic-clip machinery — proof the loop <em>optimizes</em>, not merely runs. (The full RLHF setup swaps this toy reward for a trained reward model or a task verifier; the algorithm is identical.)</p><h3>Honest caveats</h3><ul><li><strong>PPO is the touchy one.</strong> Small learning rate (1e-6), clip 0.2, grad-clip, and you <em>watch the KL</em> — if it blows up the model is degenerating. Raise the KL coefficient or drop the LR.</li><li>It’s <strong>heavy machinery</strong>: a reward signal, a critic, a sampling loop, and several minibatched update epochs per rollout. That cost is exactly what DPO (Part 5) traded away.</li><li>Our reward is a learnable toy so the mechanism is visible and reproducible in under a minute. Real reward models are trained on human rankings and are far harder to specify.</li></ul><h3>What’s next</h3><p>PPO works, but it carries a whole <strong>value network</strong> just to estimate the baseline for its advantages — a second set of parameters to train and tune. What if we could get the baseline for free? In <strong>Part 7</strong> we meet <strong>GRPO</strong>, which drops the critic entirely and estimates the advantage from a <em>group</em> of samples to the same prompt — the same reward signal, none of the critic. (We already have its curve: it climbs to 1.00 too.)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5ab9fe300635" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building LLM, Part 5 — RLHF: Aligning a Model to Preferences]]></title>
            <link>https://medium.com/@chandupadole/building-llm-part-5-rlhf-aligning-a-model-to-preferences-ac148647539e?source=rss-56067b0267b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/ac148647539e</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[rlhf]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[dpos]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Chandrashekhar Padole]]></dc:creator>
            <pubDate>Sat, 01 Aug 2026 05:25:52 GMT</pubDate>
            <atom:updated>2026-08-04T19:44:00.001Z</atom:updated>
            <content:encoded><![CDATA[<p><strong>Abstract.</strong> Fine-tuning (Part 4) taught our model to <em>answer</em> by showing it one ideal reply per prompt. But most questions have no single right answer — there’s a <em>spectrum</em> of better and worse replies, and a demonstration can’t say “I like this one <strong>more</strong>.” That’s what <strong>RLHF</strong> — Reinforcement Learning from Human Feedback — is for: you rank answers, and the model learns the ranking. This part builds RLHF from first principles using its modern, laptop-friendly form, <strong>DPO</strong> (“RLHF without the RL”), takes the very emoji-SFT model from Part 4, and <em>proves</em> the alignment moved — with an honest twist that teaches something real about what preference optimisation actually optimises.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VTdphTZrmbK--2fXKPRM2Q.jpeg" /></figure><p><strong>In this 8-part series</strong> <em>(you are reading Part 5)</em>:</p><ol><li><a href="https://medium.com/@chandupadole/building-llm-part-1-training-predicting-the-next-token-44c72473e486">Part 1 — Training: predicting the next token</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-2-inference-how-a-trained-model-writes-88458eca1d47">Part 2 — Inference: how the model writes</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-3-vit-vs-llm-one-architecture-two-senses-71f2c6f12970">Part 3 — ViT vs LLM: what carries over, what’s new</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-4-fine-tuning-teaching-a-model-a-new-behaviour-cef8cf0ba866">Part 4 — Fine-tuning: teaching the model to follow instructions</a></li><li><strong>Part 5 — RLHF: learning from human preference</strong> — <em>you are here</em></li><li><a href="https://medium.com/@chandupadole/building-llm-part-6-ppo-the-on-policy-engine-of-rlhf-5ab9fe300635">Part 6 — PPO: the policy-gradient engine</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-7-grpo-reinforcement-without-a-value-network-3e43ba1454f7">Part 7 — GRPO: reinforcement without a value network</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-8-dpo-and-the-whole-post-training-landscape-4130154824fd">Part 8 — DPO: preference learning with no RL loop</a></li></ol><blockquote><em>This is </em><strong><em>Part 5</em></strong><em>, stage 2 of post-training. It runs on top of the emoji-SFT model from </em><a href="https://medium.com/@chandupadole/building-llm-part-4-fine-tuning-teaching-a-model-a-new-behaviour-cef8cf0ba866">Part 4</a><em>— read that first. Same experimental spine: </em><strong><em>measure → align → measure again → prove the numbers moved.</em></strong></blockquote><h3>Two ways to teach</h3><p>SFT and RLHF differ in one deep way — the kind of signal they learn from:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oPFi1pN0kq1Cx3JR2tfXjA.png" /><figcaption>Figure 1: Teaching with SFT and RLHF</figcaption></figure><p>Demonstrations say <em>“do this.”</em> Preferences say <em>“this is better than that.”</em> For open- ended prompts — be helpful, be warm, be safe — there’s rarely one right answer, so preferences are the only signal that can express what people actually want.</p><h3>The data: preference pairs</h3><p>Where an SFT row was (prompt, answer), an RLHF row is a <strong>triple</strong> — the same prompt with a <em>better</em> and a <em>worse</em> reply:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VFX5QAsBXas3gayP9PI5iw.png" /><figcaption>Figure 2: RLHF triple — Prompt, Chosen and Rejected</figcaption></figure><p>Here are three of the <strong>24 real pairs</strong> — the same prompt with a warm, emoji-rich chosen reply and a flat, bare rejected one:</p><blockquote><strong><em>“Hello, who are you?”</em></strong><em> </em><strong><em>chosen</em></strong><em> </em>(warm):<em> “Hi there!! 👋😊 I’m your cheerful AI buddy, so happy to help you today! 🌟💬” </em><strong><em>rejected</em></strong><em> </em>(flat):<em> “I am an AI assistant.”</em></blockquote><blockquote><strong><em>“My program has a bug.”</em></strong><em> </em><strong><em>chosen</em></strong><em> </em>(warm):<em> “Oh no, a sneaky little bug!! 🐛🔍 Paste it here and we’ll squash it together! 🔨💪” </em><strong><em>rejected</em></strong><em> </em>(flat):<em> “Send the code to debug it.”</em></blockquote><blockquote><strong><em>“I’m feeling sad today.”</em></strong><em> </em><strong><em>chosen</em></strong><em> </em>(warm):<em> “Aww, sending you the biggest virtual hug!! 🤗💖 Brighter days are coming, I promise! 🌈🌟” </em><strong><em>rejected</em></strong><em> </em>(flat):<em> “Sorry to hear that.”</em></blockquote><p>Notice that the rejected reply is never <em>wrong</em> — it&#39;s a perfectly correct, if dull, answer. That&#39;s the whole point: DPO is never shown a &quot;right&quot; answer, only that one valid reply is <strong>preferred</strong> over another.</p><p>Our stand-in “reward” is deliberately trivial so the whole demo stays runnable with no human-labelling step: <strong>warmer and more emoji-dense is better.</strong> The chosen replies are <em>more</em> emoji-rich than the SFT model already is, so aligning toward them should lean the style even harder. (In real RLHF a <em>learned reward model</em>, trained on thousands of human rankings, plays this role.)</p><h3>Classic RLHF vs DPO</h3><p>Classic RLHF (the InstructGPT recipe) is three moving parts: SFT, then a <strong>reward model</strong> trained on human rankings, then a <strong>PPO</strong> loop that samples replies, scores them, and nudges the policy — all under a KL penalty. That’s powerful but heavy. <strong>DPO</strong> collapses it into a single supervised-style loss:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q93lwvRWXI6W3SgGtLCU_A.png" /><figcaption>Figure 3: DPO</figcaption></figure><p>A bit of algebra shows the optimal RLHF policy has a closed form in terms of the reward — which can be inverted so the reward <em>cancels out</em>, leaving a loss written purely over preference pairs. No separate reward model, no sampling loop. That’s why DPO is the default first reach today.</p><h3>The DPO loss, in plain English</h3><p>For one (prompt, chosen, rejected) triple, with policy π_θ and the frozen SFT reference π_ref:</p><pre>L = -log σ( β·[ logπ_θ(chosen)  − logπ_ref(chosen)  ]<br>          − β·[ logπ_θ(rejected) − logπ_ref(rejected) ] )</pre><ul><li>logπ_θ(y) − logπ_ref(y) is the <strong>implicit reward</strong> of reply y: how much <em>more</em> probable the policy makes it than the reference did.</li><li>The loss wants the implicit reward of <strong>chosen</strong> to beat <strong>rejected</strong> by a margin.</li><li><strong>β</strong> is the <strong>KL leash</strong>: the −logπ_ref terms penalise drifting from the SFT model. Small β → freer to move (stronger effect, more risk); large β → stays close. We use β = 0.1.</li></ul><p>In one sentence: <strong>push probability toward </strong><strong>chosen, pull it away from </strong><strong>rejected, while staying tethered to the SFT model</strong> — using only log-probs of answers we already have.</p><h3>The wiring: SFT model as both policy and reference</h3><p>DPO needs a starting policy <em>and</em> a reference to anchor the KL — and both come from the SFT model:</p><pre>model = PeftModel.from_pretrained(base, &quot;tinyllama-emoji&quot;)  # load the SFT adapter<br>model = model.merge_and_unload()                            # bake it into the base<br>trainer = DPOTrainer(model=model, ref_model=None,           # ref = base with adapter OFF<br>                     peft_config=lora_config, train_dataset=pairs)</pre><p>Passing ref_model=None with a peft_config is the trick: DPO uses the base weights with the <strong>new adapter disabled</strong> as the reference. Since our base <em>is</em> the merged SFT model, the reference is exactly the SFT policy — precisely what the KL penalty should anchor to. The training loop is otherwise ordinary supervised training over the pairs.</p><h3>The proof — and an honest lesson</h3><p>Same spine, real results, measured on the trained DPO adapter. The direct measure of alignment is the <strong>preference margin</strong>: the per-token log-prob it gives chosen minus rejected, averaged over the pairs. It <strong>grew from 2.27 to 2.65</strong> — the model now favours the better reply <em>more decisively</em> than the SFT model did. But watch the second panel:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-HPXWudLeYm2oNnUvD_lqg.png" /><figcaption>Figure 4: DPO Performance Measure</figcaption></figure><p>The crude emoji reward barely moved (6 → 7). That’s not a failure — it’s the lesson. Our SFT model <em>already</em> saturated a blunt “count the emojis” metric, so there was little surface headroom; what DPO actually did was sharpen the <em>preference</em> — the deeper signal. It’s a live demonstration of why RLHF optimises a preference, not a surface number (and a hint of <strong>reward hacking</strong>: optimise a crude reward too hard and a model games it — the β/KL leash is what holds that in check).</p><p>Qualitatively, the alignment shows in the tone. Same held-out prompts, SFT vs DPO:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*Zo3HccwsZSnongbP7nM_cQ.png" /></figure><p>Notice the second row: the SFT model ended on a <em>frowny</em> ☹️; DPO nudged it to a warm ☺️. Small, but exactly the kind of preference a surface metric would miss.</p><h3>Honest caveats</h3><ul><li><strong>24 pairs, 6 epochs</strong> — a toy that proves the <em>mechanics</em>, not a production alignment.</li><li>Our “reward” (emoji density) is a deliberately trivial proxy; real reward models capture fuzzy notions like helpfulness, honesty, and harmlessness.</li><li><strong>Reward hacking is real</strong> — the β/KL leash is the safeguard; lower β, more hacking.</li><li><strong>DPO is offline:</strong> it learns from a <em>fixed</em> set of pairs. It never generates <em>new</em> samples to be judged — which is the one thing on-policy RL does, and where we go next.</li></ul><h3>What’s next</h3><p>DPO gave us most of RLHF’s benefit with none of the RL. But it’s <em>offline</em> — it only ever sees the pairs we hand it. Frontier alignment also uses <strong>on-policy</strong> methods that let the model <em>generate its own</em> attempts and learn from how they score. In <strong>Part 6</strong> we open up that machinery — <strong>PPO</strong>, the policy-gradient algorithm at the heart of classic RLHF — and in Parts 7–8, its lighter descendants <strong>GRPO</strong> and (coming full circle) <strong>DPO</strong> itself.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ac148647539e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building LLM, Part 4 — Fine-Tuning: Teaching Model a New Behaviour]]></title>
            <link>https://medium.com/@chandupadole/building-llm-part-4-fine-tuning-teaching-a-model-a-new-behaviour-cef8cf0ba866?source=rss-56067b0267b6------2</link>
            <guid isPermaLink="false">https://medium.com/p/cef8cf0ba866</guid>
            <category><![CDATA[nlp]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Chandrashekhar Padole]]></dc:creator>
            <pubDate>Fri, 31 Jul 2026 11:08:54 GMT</pubDate>
            <atom:updated>2026-08-12T17:06:22.534Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-gOrFEco9lSSBTtkxy4B_Q.jpeg" /></figure><p><strong>Abstract.</strong> A base language model is a brilliant parrot: give it text and it continues text. It won’t <em>answer</em> you, because it was never shown what “answering” looks like. <strong>Fine-tuning</strong> fixes that — you show it a few dozen (question → ideal answer) pairs and it learns the behaviour. This part builds that from the ground up and <em>proves</em> it worked, treating the model like an experiment: <strong>measure → fine-tune → measure again</strong>. Our test behaviour is deliberately unmissable — teach a model to answer with emojis — so the signal is impossible to fake. You&#39;ll meet the one trick that turns plain text-completion into <em>instruction</em> tuning (masking the prompt), the LoRA shortcut that lets it all run on a laptop, and real before/after numbers from a model you can fine-tune yourself in two minutes.</p><p><strong>In this 8-part series</strong> <em>(you are reading Part 4)</em>:</p><ol><li><a href="https://medium.com/@chandupadole/building-llm-part-1-training-predicting-the-next-token-44c72473e486">Part 1 — Training: predicting the next token</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-2-inference-how-a-trained-model-writes-88458eca1d47">Part 2 — Inference: how the model writes</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-3-vit-vs-llm-one-architecture-two-senses-71f2c6f12970">Part 3 — ViT vs LLM: what carries over, what’s new</a></li><li><strong>Part 4 — Fine-tuning: teaching the model to follow instructions</strong> — <em>you are here</em></li><li><a href="https://medium.com/@chandupadole/building-llm-part-5-rlhf-aligning-a-model-to-preferences-ac148647539e">Part 5 — RLHF: learning from human preference</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-6-ppo-the-on-policy-engine-of-rlhf-5ab9fe300635">Part 6 — PPO: the policy-gradient engine</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-7-grpo-reinforcement-without-a-value-network-3e43ba1454f7">Part 7 — GRPO: reinforcement without a value network</a></li><li><a href="https://medium.com/@chandupadole/building-llm-part-8-dpo-and-the-whole-post-training-landscape-4130154824fd">Part 8 — DPO: preference learning with no RL loop</a></li></ol><blockquote><em>A note on models. Parts 1–2 built a small model </em>from scratch<em> to understand how one works. Post-training is different: in the real world you always start from a capable </em><strong><em>pretrained</em></strong><em> model and adjust it. Our 9M tales model is too small to follow instructions, so from here we fine-tune </em><strong><em>TinyLlama-1.1B</em></strong><em> — a real, open pretrained model — exactly as practitioners do. The architecture is the same one we built; only the scale changed.</em></blockquote><h3>The problem: it answers the way it was trained — not the way we want</h3><p>Every language model answers <strong>the way it was trained to</strong>. Our TinyLlama has read a huge slice of the internet, so it’s fluent and full of facts — but pretraining only ever taught it one skill: <strong>continue text</strong>. Ask it a question and that’s exactly what it does — it keeps writing, formally and at length, in the flat encyclopedic voice of its training data. It never learned to <em>be a warm, helpful assistant</em>, because nobody ever showed it what one sounds like.</p><p><strong>That’s the whole job of this part.</strong> We take that capable-but-flat model and <strong>fine-tune it on a little data of our own</strong> — a few dozen examples of the behaviour we want — so it answers in a new style. We’re not teaching it new <em>facts</em>; we’re changing its <em>manner</em>. And to make the change impossible to miss (and trivial to measure), our target behaviour is deliberately loud: <strong>answer warmly, with emojis.</strong> Here’s the entire objective on one page:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6LN-rXWV8TSyXjxqnyoyRw.png" /><figcaption>Figure 1: Fine tuning objective</figcaption></figure><p>You can hear the problem in the base model’s own words. Ask it two ordinary questions it never saw in training, and it answers correctly but in the same detached, list-making voice every time:</p><blockquote><strong><em>How are you feeling today?</em></strong><em> </em>Base:<em> “I am not capable of experiencing emotions or feelings. However, I can provide information about how someone else might feel based on their responses…”</em></blockquote><blockquote><strong><em>Give me some advice.</em></strong><em> </em>Base:<em> “Sure, here are a few tips to help you get started with your new hobby: 1. Set realistic goals and break them down into smaller tasks…”</em></blockquote><p>Neither answer is <em>wrong</em> — they’re just not <em>tuned</em>. No warmth, no brevity, and (our unmistakable signal) <strong>zero emojis</strong>. By the end of this part, the same model, asked the same questions, will answer like the orange card above — and getting there takes surprisingly little.</p><p><strong>Emojis are just a vivid stand-in.</strong> The exact same recipe — show the model a few dozen (input → ideal output) examples — is how teams teach almost any behaviour:</p><ul><li><strong>Instruction following</strong> — turn a raw base model that only <em>continues</em> text into one that actually <em>answers</em>. (This is what makes a “chat” model a chat model.)</li><li><strong>A voice or persona</strong> — a brand tone, a support-agent style, “always concise” (our emoji demo is exactly this).</li><li><strong>Structured output</strong> — always reply as JSON or a fixed schema, so the model can drive tools and pipelines reliably.</li><li><strong>Domain adaptation</strong> — the vocabulary and conventions of law, medicine, finance, or your own codebase.</li><li><strong>A specific task</strong> — your exact flavour of summarising, classifying, or extracting fields.</li><li><strong>Safety &amp; policy</strong> — refuse out-of-scope requests and follow house rules.</li></ul><p>We picked emojis because the change is impossible to fake or miss — but everything that follows applies unchanged to any of these.</p><h3>The plan: treat fine-tuning as an experiment</h3><p>Fine-tuning is nudging a pretrained model’s weights toward a behaviour you <em>demonstrate</em>. To prove it actually did something — rather than hoping — we run it like a lab experiment:</p><blockquote><strong><em>measure → intervene (train) → measure again → assert the numbers moved</em></strong></blockquote><p>If fine-tuning worked, the metrics <em>must</em> change. We measure two things on prompts the model <strong>never sees in training</strong>: how many emojis it uses, and the loss it assigns to emoji-style answers.</p><p>That <strong>intervene</strong> step — the actual training — can be done several ways, trading power for cost:</p><ul><li><strong>Full fine-tuning</strong> — update <em>all</em> the weights. Most flexible, but you must hold gradients and optimizer state for every parameter (tens of GB for a 1.1B model) and you get a full-size copy per task. Overkill for teaching a <em>style</em>.</li><li><strong>LoRA</strong> — freeze the base and train small <strong>low-rank adapters</strong> (the idea we unpack below): ~1% of the parameters, a few-MB file, and nothing forgotten. <strong>← our choice.</strong></li><li><strong>QLoRA</strong> — LoRA on top of a <strong>4-bit quantized</strong> base: the frozen weights are squeezed to 4 bits so a much <em>larger</em> model fits in memory, while the tiny adapters still train in higher precision. The go-to when the base is too big for your GPU.</li><li><strong>Prompt / prefix tuning, bottleneck adapters, …</strong> — other members of the same “train a little, freeze a lot” family.</li></ul><p>Everything except full fine-tuning shares one name: <strong>PEFT — Parameter-Efficient Fine-Tuning.</strong> It’s not a separate method but the <em>umbrella term</em> for “freeze the pretrained weights and train only a small add-on,” and LoRA, QLoRA, and prefix tuning are all members. (HuggingFace’s peft library implements them all — it&#39;s literally the from peft import LoraConfig in our code.) The difference between them is easiest to <em>see</em> — how much of the model each one trains, and what that costs:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uB5uj37RNqUoUemvCg4t4Q.png" /><figcaption>Figure 2: Fine tuning — diffrent methodologies</figcaption></figure><p><strong>Why LoRA first?</strong> It’s the <strong>de-facto default</strong> and the clearest to reason about — one idea, low rank. Our base, at 1.1B, fits comfortably in fp16 on a small GPU, so we don’t yet need QLoRA’s quantization. And once LoRA clicks, <strong>QLoRA is just LoRA + a quantized base</strong> — a small add-on, not a new concept. So we start with the clean case and add complexity only when a bigger model forces us to.</p><h3>The data: (prompt → response) pairs</h3><p>The training data is just conversations — a user turn and the ideal assistant reply:</p><pre>{&quot;role&quot;: &quot;user&quot;,      &quot;content&quot;: &quot;Do you like pizza?&quot;}<br>{&quot;role&quot;: &quot;assistant&quot;, &quot;content&quot;: &quot;Who doesn&#39;t love pizza?! 🍕 Extra cheese, please! 🧀&quot;}</pre><p>Our set is <strong>51 such pairs</strong> spanning many topics on purpose — the <em>only</em> thing every answer shares is the emoji style, which forces the model to learn the <em>rule</em>, not memorise answers. A <strong>chat template</strong> flattens each pair into the one string TinyLlama was pretrained to recognise, with special role markers:</p><pre>&lt;|user|&gt;<br>Do you like pizza?&lt;/s&gt;<br>&lt;|assistant|&gt;<br>Who doesn&#39;t love pizza?! 🍕 Extra cheese, please! 🧀&lt;/s&gt;</pre><p>Then tokenization turns that string into integers — the only thing the model ever sees.</p><h3>The one trick: mask the prompt</h3><p>Here is what makes this <em>instruction</em> tuning instead of plain text completion. Every token gets a label (the correct next token), <strong>but the prompt’s tokens are set to </strong><strong>-100</strong> — a sentinel meaning &quot;ignore in the loss&quot;:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AQsHPlFEJVZwHs7Cn1d8yA.png" /><figcaption>Figure 3: Prompt-Answer pair — Masked prompt and loss calculated from answer</figcaption></figure><p>In this real example, <strong>21 prompt tokens are masked</strong> and only the <strong>28 answer tokens</strong> carry any loss. In code the mask is two lines, and the loss simply skips -100:</p><pre>labels = full_ids.clone()<br>labels[:n_prompt] = -100                         # ignore the prompt tokens<br>loss = F.cross_entropy(logits, labels, ignore_index=-100)   # score only the answer</pre><p>So the model never learns to <em>write the question</em> — it learns <strong>“given this question, produce this answer.”</strong> (SFTTrainer applies this masking for us automatically.)</p><h3>Batching examples of different lengths: padding (also masked)</h3><p>There’s a second wrinkle, and it’s the flip side of a detail from Part 1. In pretraining we cut <strong>fixed-length windows</strong> from one packed stream, so every sample was exactly the context length and no padding was ever needed. Fine-tuning is different: each example is a <strong>whole </strong><strong>(prompt → answer) pair</strong>, and those pairs are <strong>genuinely different lengths</strong>. To stack them into one batch tensor, the short ones are <strong>padded</strong> — filled out to the batch&#39;s longest example with a pad token (here the &lt;|endoftext|&gt;/EOS token):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0VHPJ3iXD_aGg0J85LT5dg.png" /><figcaption>Figure 4: Padding for smaller token answers</figcaption></figure><p>The padding is <em>only</em> there to make the rows the same width — it carries no meaning, so it’s masked out of the loss with the same -100 sentinel as the prompt. In these three real examples every prompt is 21 tokens, the answers are 22/28/32, and the whole batch is padded to the longest (<strong>53</strong>). The result: of each row, only the <strong>answer</strong> tokens ever move the weights. Prompt masked, padding masked, answer learned — that&#39;s the whole recipe.</p><h3>The shortcut: LoRA</h3><p>Before we apply it, what <em>is</em> LoRA? <strong>Low-Rank Adaptation</strong> is a general technique for fine-tuning any large model cheaply, and it rests on one observation: <strong>the change you need to adapt a model to a new task is usually “small” in a precise sense — it is low-rank.</strong></p><p>Ordinarily, fine-tuning nudges a big weight matrix W (in a large model these are roughly d × d, often millions of numbers each), and the update ΔW is just as big. LoRA never forms that full update. Instead it writes the update as a product of <strong>two skinny matrices</strong>:</p><blockquote><em>ΔW = B · A — where </em><em>A is </em><em>r × d, </em><em>B is </em><em>d × r, and the </em><strong><em>rank </em></strong><strong><em>r is tiny</em></strong><em> (say 8, 16, 64).</em></blockquote><p>It then <strong>freezes the original </strong><strong>W</strong> and trains only A and B. The model runs with the effective weight W + B·A: the pretrained knowledge in W is left untouched, and the tiny B·A carries the new behaviour. Because r is small, A and B together hold r·(d + d) numbers instead of d·d — often <strong>well under 1%</strong> of the original.</p><p>That buys three things, on <em>any</em> model and <em>any</em> task:</p><ul><li><strong>Cheap to train</strong> — a fraction of the parameters, so it fits on modest hardware and runs fast, with less risk of overfitting on a small dataset.</li><li><strong>Tiny to store and swap</strong> — the adapter is a few megabytes, not gigabytes. One frozen base model can host <strong>many</strong> adapters (one per task or persona), loaded on demand.</li><li><strong>Nothing forgotten</strong> — since W is frozen, the base model keeps all its general ability; the adapter only <em>adds</em> to it.</li></ul><p>At inference you can keep B·A as a small side-path, or <strong>fold it in</strong> (W&#39; = W + B·A) so it costs nothing extra.</p><p><strong>Here’s how we use it.</strong> TinyLlama has 1.1 billion weights. We don’t need to move all of them to teach a <em>style</em> — so we <strong>freeze</strong> the whole model and insert these tiny trainable adapter matrices into the attention layers, training only those:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bFnwL42O6O-KcT4V-DYECg.png" /><figcaption>Figure 5: LoRA is applied to sub-parts of the learning params in each layer</figcaption></figure><pre>from peft import LoraConfig<br>lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, task_type=&quot;CAUSAL_LM&quot;,<br>                  target_modules=[&quot;q_proj&quot;, &quot;k_proj&quot;, &quot;v_proj&quot;, &quot;o_proj&quot;])</pre><p>We train <strong>~0.1%</strong> of the parameters; the saved result is an <strong>18 MB adapter</strong>, not a 2.2 GB model copy, and the whole run fits on a 6 GB laptop GPU. The training loop itself is the <em>exact</em> next-token loop from Part 1 — forward, masked loss, backward(), step() — just started from the pretrained weights with a low learning rate.</p><h3>The methodology: the exact apparatus</h3><p>Before the results, here is the concrete setup — the model, the parts we touch, and the code that does it. We fine-tune <strong>TinyLlama-1.1B-Chat</strong>, a real open model built on the <strong>Llama-2 architecture</strong>: the same decoder-only transformer from Part 1, scaled up and with a few modern refinements — <strong>RMSNorm</strong> in place of LayerNorm, <strong>rotary position embeddings</strong> (RoPE), a <strong>SwiGLU</strong> MLP, and <strong>grouped-query attention</strong> (32 query heads sharing 4 key/value heads):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kV9kBR6Ix9Urs1VbherJrw.png" /><figcaption>Figure 6: Showing sub-parts (orange) in each layer is where LoRA applied</figcaption></figure><p><strong>What we access and modify.</strong> We load the pretrained weights, <strong>freeze every one of the 1.1 billion</strong>, and attach LoRA adapters to just the four attention projections — q_proj, k_proj, v_proj, o_proj — in all 22 layers (orange above). The embeddings, the MLPs, the norms, and the output head are never touched.</p><p>Loading the model and tokenizer is two lines:</p><pre>tokenizer = AutoTokenizer.from_pretrained(&quot;TinyLlama/TinyLlama-1.1B-Chat-v1.0&quot;)<br>if tokenizer.pad_token is None:                        # TinyLlama ships without a pad token<br>    tokenizer.pad_token = tokenizer.eos_token<br>model = AutoModelForCausalLM.from_pretrained(&quot;TinyLlama/TinyLlama-1.1B-Chat-v1.0&quot;,<br>                                             dtype=torch.float16)</pre><p>Training is standard SFT. SFTTrainer wraps the frozen base with the LoRA adapter (the lora config from above), applies the chat template, masks the prompt to -100, and runs the ordinary next-token loop:</p><pre>sft = SFTConfig(num_train_epochs=10, per_device_train_batch_size=4,<br>                gradient_accumulation_steps=2, learning_rate=2e-4, max_length=512)<br>trainer = SFTTrainer(model=model, args=sft, train_dataset=dataset,<br>                     peft_config=lora, processing_class=tokenizer)<br>trainer.train()<br>tuned_model = trainer.model                            # base (frozen) + trained LoRA adapters</pre><p><strong>Asking it questions (the Q&amp;A).</strong> To query either model — before or after — we wrap the question in the chat template and generate greedily, so the before/after comparison is deterministic and fair:</p><pre>def ask(model, question):<br>    ids = tokenizer.apply_chat_template([{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: question}],<br>                                        add_generation_prompt=True, return_tensors=&quot;pt&quot;,<br>                                        return_dict=True)[&quot;input_ids&quot;].to(model.device)<br>    out = model.generate(ids, max_new_tokens=60, do_sample=False,<br>                         repetition_penalty=1.3, no_repeat_ngram_size=3,<br>                         pad_token_id=tokenizer.pad_token_id)<br>    return tokenizer.decode(out[0][ids.shape[-1]:], skip_special_tokens=True)</pre><p>That’s the whole apparatus — a frozen 1.1B model, a rank-16 adapter on its attention projections, the Part 1 training loop, and a two-line way to ask it anything. Now let’s see whether it actually moved the needle.</p><h3>The proof: the numbers moved</h3><p>Same experiment spine, real results. On held-out prompts the base model used <strong>0</strong> emojis; the fine-tuned model uses <strong>8</strong>. And its loss on emoji-style answers fell from <strong>2.89 to 0.38</strong> — it now finds those replies far more probable:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9EnPi_mlXpfbHJn86KY42Q.png" /><figcaption>Figure 7: Base &amp; Tuned model comparison, number of emojis &amp; loss</figcaption></figure><p>And here is the behaviour itself — the same held-out questions, before and after (nothing cherry-picked; these are the raw greedy generations):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1013/1*WMcxvpkzc4DJogE9SJDKSA.png" /></figure><p>The model didn’t memorise these answers — it never saw these prompts. It learned the <em>style</em> and applied it to new questions. That’s the whole game.</p><h3>Honest caveats</h3><ul><li>With <strong>51 examples and a few epochs</strong>, this is a <em>style</em> transfer, not knowledge. It proves the <strong>mechanics</strong> of instruction tuning, not a production assistant.</li><li><strong>Tests that matter:</strong> the loss dropping on held-out answers (the model finds them more probable) and emoji use rising on <em>unseen</em> prompts (it generalised). Training loss falling alone only proves the wiring works.</li><li>Real SFT uses tens of thousands of diverse instructions and a held-out validation set to catch overfitting. The <em>idea</em> — demonstrate the behaviour, mask the prompt, train the answer — is identical at every scale.</li></ul><h3>What’s next</h3><p>Our model now <em>answers</em> — but it answers with whatever single style we demonstrated. For most real prompts there is no one right answer; there’s a spectrum of better and worse replies, and demonstrations can’t express “I like this one <em>more</em>.” Teaching that needs <strong>preferences</strong>, not examples. In <strong>Part 5</strong> we take this very SFT model and align it with human preference using RLHF — and, as usual, prove it worked.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cef8cf0ba866" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>