<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yash Bhardwaj on Medium]]></title>
        <description><![CDATA[Stories by Yash Bhardwaj on Medium]]></description>
        <link>https://medium.com/@Yash_Bhardwaj?source=rss-bec114f6ca70------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*CTfUkGh3qxlllAXtF6G7Ug.png</url>
            <title>Stories by Yash Bhardwaj on Medium</title>
            <link>https://medium.com/@Yash_Bhardwaj?source=rss-bec114f6ca70------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 19:45:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@Yash_Bhardwaj/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Last 20 Lines of a GPT Built From Scratch, and the Three Characters That Broke Them]]></title>
            <link>https://medium.com/@Yash_Bhardwaj/the-last-20-lines-of-a-gpt-built-from-scratch-and-the-three-characters-that-broke-them-03fd989aa61c?source=rss-bec114f6ca70------2</link>
            <guid isPermaLink="false">https://medium.com/p/03fd989aa61c</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[fine-tuning]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Yash Bhardwaj]]></dc:creator>
            <pubDate>Mon, 12 Oct 2026 16:59:41 GMT</pubDate>
            <atom:updated>2026-10-12T16:59:41.451Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Dark cinematic poster for Day 7 of building an LLM from scratch. On the left, a code snippet reads nn.CrossEntropyLoss(ignore_index=-100) and shows prompt tokens being set to -100. In the center, a horizontal ribbon of tokens is split in half: the left side is grey and crossed out with “-100” labels, the right side glows molten orange. A vertical seam between them is labelled “### Response:”. Side panels read “PROMPT → IGNORED”, “ANSWER → LEARNED”, “199 TOKENS MASKED” and “1 TOKEN TRAINED”. Head" src="https://cdn-images-1.medium.com/max/1024/1*-alx2C57MPnKw0DBlfOKmA.png" /><figcaption>Instruction tuning in one picture: grey is everything the loss throws away, orange is the only thing it grades. The architecture never changes — the mask does all the work</figcaption></figure><h3>Chapter 7 of building an LLM by hand — instruction fine-tuning, where the architecture stops mattering and a missing space bar starts</h3><p>Seven chapters ago I opened an empty notebook and typed import torch.</p><p>Since then I have built, by hand, with no nn.TransformerEncoder doing the work for me: a tokenizer, byte-pair encoding, a dataloader, embeddings, scaled dot-product attention, causal masking, multi-head attention, layer normalisation, GELU, a feed-forward block, residual connections, a full 124M-parameter GPT-2, a training loop, a loss curve, and a classification head that sorts spam.</p><p>Today is the last chapter. Instruction fine-tuning. The step that takes a model which merely <em>continues text</em> and turns it into a model that <em>answers you</em>. The step that, more than any other, is what people actually mean when they say “ChatGPT”.</p><p>And I want to tell you two things about it.</p><p>The first is that it is embarrassingly simple. Simpler than attention. Simpler than the training loop. Simpler than anything in chapters 2 through 5.</p><p>The second is that I still managed to ship a NameError in the final function of the entire series.</p><p>Let’s do both.</p><h3>The thing nobody warns you about: nothing changes</h3><p>Here is what I genuinely expected instruction tuning to be, before I wrote a line of it. Some new architecture. A “chat layer”. A special alignment module that teaches the model manners. Something that <em>looks</em> different from pretraining.</p><p>Here is what it actually is:</p><p><strong>You keep the exact same model, and you change which tokens the loss is allowed to grade.</strong></p><p>That’s the chapter. That’s the whole thing.</p><p>Chapter 6 — classification fine-tuning — actually did surgery. I ripped off the 50,257-way language-model head, bolted on a nn.Linear(emb_dim, num_classes), froze the body, and pooled the last token into a label. The model physically stopped writing tokens.</p><p>Chapter 7 does none of that. The head stays. The forward pass stays. The loss function is the same CrossEntropyLoss from chapter 4 with <strong>one extra keyword argument</strong>.</p><p>Put the three regimes next to each other, because people call all of them “training” and that is exactly why everyone is confused:</p><p>Pretraining (ch. 4)Classification (ch. 6)Instruction tuning (ch. 7)</p><p>Question the model answers</p><p>“What token comes next?”</p><p>“Which label is this text?”</p><p>“What token comes next <em>in the answer</em>?”</p><p>Output shape</p><p>(B, T, vocab_size)</p><p>(B, num_classes)</p><p>(B, T, vocab_size) — same as pretraining</p><p>Target</p><p>a token ID at every position</p><p>one class ID per example</p><p>response token IDs; prompt is -100</p><p>Data</p><p>raw text (the-verdict.txt)</p><p>a labelled spreadsheet</p><p>instruction / input / output records</p><p>What changes in the network</p><p>everything</p><p>new head, body frozen</p><p><strong>nothing. the mask does the work</strong></p><p>Read that last column again. The specialisation does not live in the network. It lives in the targets.</p><p>I find this genuinely beautiful and mildly insulting at the same time. Years of mystique around “aligned” models, and the core mechanic is: write -100 over the parts you don&#39;t want graded.</p><h3>What an instruction example actually looks like</h3><p>Pretraining data is a paragraph cut out of a book. Instruction data is a tiny task with a known good answer. Three fields:</p><ul><li><strong>instruction</strong> — the task. “Identify the primary programming language used in this snippet.”</li><li><strong>input</strong> — optional extra material. Here, print(&#39;Hello, Yash&#39;).</li><li><strong>output</strong> — the answer we want written. Python.</li></ul><p>That output field is for training only. When you later ask the model something new, you leave it blank and let the model fill it in. That asymmetry is the entire design.</p><h3>Step 1: Build a mould and pour everything into it</h3><p>A pretrained GPT has no concept of a “request”. Hand chapter 5’s model the string Identify the language and it will happily continue with another question, a blog intro, or a half-remembered Stack Overflow rant — whatever tended to follow those words on the internet. It is not being difficult. It is doing exactly the job we trained it to do.</p><p>So we give the request a shape. The same shape, every single time:</p><pre>def format_input(entry):  <br>    instruction_text = (  <br>        f&quot;Below is an instruction that describes a task. &quot;  <br>        f&quot;Write a response that appropriately completes the request.\n\n&quot;  <br>        f&quot;### Instruction:\n{entry[&#39;instruction&#39;]}&quot;  <br>    )  <br>    if entry.get(&quot;input&quot;):  <br>        instruction_text += f&quot;\n\n### Input:\n{entry[&#39;input&#39;]}&quot;  <br>    return instruction_text</pre><p>This is the Alpaca-style wrapper that half of the open instruction datasets use. A one-line preamble, then ### headings. Nothing clever. Clever is the enemy here — the only property that matters is that it is <strong>identical on every row</strong>.</p><p>Two details worth stopping on:</p><p><strong>One. This function builds only the prompt.</strong> The answer gets glued on separately, outside the function:</p><pre>full_entry = f&quot;{formatted_prompt}\n\n### Response:\n{sample_data[&#39;output&#39;]}&quot;</pre><p>That split is not cosmetic. The prompt is what we are about to mask. The answer is what we are about to train on. If both got built in the same step, the boundary between them would be gone, and the boundary is the product.</p><p><strong>Two. </strong><strong>entry.get(&quot;input&quot;), not </strong><strong>entry[&quot;input&quot;].</strong> Plenty of tasks have no extra material — &quot;write a haiku about debugging&quot; needs no input. .get returns None, the if skips, and the template gracefully collapses to two parts. Use square brackets there and your dataset explodes on row 400 of a 50,000-row file, forty minutes into preprocessing.</p><h3>My first bug, and it’s in the preamble</h3><p>Here is what I actually wrote in the notebook:</p><pre>f&quot;Below is an instruction that describe a task.&quot;  <br>f&quot;Write a response that appropriately completes the request.\n\n&quot;</pre><p>Two f-strings, concatenated with nothing between them. Run it and the first line of every training example in the entire dataset reads:</p><blockquote><em>Below is an instruction that describe a task.</em><strong><em>Write</em></strong><em> a response that appropriately completes the request.</em></blockquote><p>task.Write. No space. Plus describe where it should be describes.</p><p>Does the model care? Less than you’d hope and more than you’d like. BPE will tokenize task.Write into some weird little sequence, and it will do it <strong>consistently</strong>, which is honestly 90% of what the template needs to be. The model will learn it fine.</p><p>But it is a permanent, fossilised typo welded into every example the model will ever see, and the model’s sense of English grammar is now very slightly worse than mine was when I wrote it. Which is saying something.</p><h3>Step 2: The -100 trick, which is the actual chapter</h3><p>InstructionDataset is a PyTorch Dataset, same contract as SpamDataset from chapter 6 — __len__ and __getitem__. But what comes out is a completely different animal. Chapter 6 returned (token IDs, one class ID). This returns (token IDs, token IDs), and the second tensor is full of holes.</p><pre>class InstructionDataset(Dataset):  <br>    def __init__(self, data_list, tokenizer, max_length=1024):  <br>        self.input_ids = []  <br>        self.target_ids = []  <br>        for entry in data_list:  <br>            formatted_prompt = format_input(entry)  <br>            full_text = f&quot;{formatted_prompt}\n\n### Response:\n{entry[&#39;output&#39;]}&quot;  <br>  <br>            tokenized_full   = tokenizer.encode(full_text)[:max_length]  <br>            tokenized_prompt = tokenizer.encode(formatted_prompt)  <br>  <br>            target_labels  = [-100] * len(tokenized_prompt)  <br>            target_labels += tokenized_full[len(tokenized_prompt):]  <br>  <br>            padding_len = max_length - len(tokenized_full)  <br>            if padding_len &gt; 0:  <br>                tokenized_full += [tokenizer.eot_token] * padding_len  <br>                target_labels  += [-100] * padding_len  <br>  <br>            self.input_ids.append(torch.tensor(tokenized_full))  <br>            self.target_ids.append(torch.tensor(target_labels))</pre><p>Strip away the code and here is the shape of what comes out:</p><pre>input:   [ prompt tokens .......... | response tokens    ]  <br>target:  [ -100  -100  -100  ...... | response token IDs ]</pre><p>The input keeps the prompt. Obviously — the model has to be able to <em>read</em> the question. The target wipes it.</p><p><strong>Why wipe it?</strong> Because if the prompt tokens carried real labels, the overwhelming majority of the loss would be the model learning to recite the user’s instruction back at them. That is a skill nobody has ever wanted, and one we already have, because we typed the instruction ourselves.</p><p><strong>Why </strong><strong>-100 specifically?</strong> GPT-2&#39;s vocabulary runs 0 to 50,256. -100 is not a token ID and never can be, so it cannot collide with a real class. It is PyTorch&#39;s conventional &quot;skip me&quot; sentinel, and step 3 is where we tell the loss to honour it.</p><p><strong>And the detail almost everyone misses:</strong> masked positions are still <em>read</em> by causal attention. Every word of the instruction flows through every layer. The mask changes the loss, not the forward pass. The model sees the question in full. It is simply not graded on reproducing it.</p><p>This is the clean dividing line with chapter 6, and it took me a while to feel it properly. Classification put the specialisation <strong>in the architecture</strong>. Instruction tuning puts it <strong>in the targets</strong>.</p><h3>The landmine buried in this cell</h3><p>[-100] * len(tokenized_prompt) makes an assumption: that the prompt&#39;s token IDs are a clean <strong>prefix</strong> of the full text&#39;s token IDs.</p><p>Byte-pair encoding does not always agree.</p><p>BPE merges greedily, and it merges <em>across</em> boundaries. Encode &quot;...### Response:\n&quot; on its own, you get one sequence. Encode it with &quot;Python&quot; stuck on the end, and the last token of the prompt can merge with the first token of the answer, shifting the split by one. Now your mask is one token off. You are either grading a prompt token you meant to skip, or silently skipping the first token of the answer you meant to grade.</p><p>On one row, invisible. On fifty thousand rows, you have fine-tuned a model with a faint stutter at the start of every reply and absolutely no idea why.</p><p>Four lines fix it, and you should write them before you spend a single rupee of GPU time:</p><pre>assert tokenized_full[:len(tokenized_prompt)] == tokenized_prompt, \  <br>    f&quot;BPE boundary drift on: {entry[&#39;instruction&#39;][:60]}&quot;</pre><h3>And one more</h3><p>padding_len = max_length - len(tokenized_full) is measured off the <em>input</em> and then applied to target_labels as well. If the prefix assumption above ever breaks, those two lists are no longer the same length, and the padding maths quietly produces mismatched tensors that blow up somewhere far away from the cause.</p><p>While we’re here: padding every single row out to a flat 1024 tokens is brutal. The average instruction example is maybe 90 tokens long. You are pushing roughly 90% padding through a transformer and paying full attention cost for it — and attention is quadratic. In anything resembling production you write a collate_fn that pads to the longest row <em>in the batch</em>. It is a handful of lines for a 5–10× speedup.</p><h3>Step 3: One keyword argument</h3><pre>masked_loss_fn = nn.CrossEntropyLoss(ignore_index=-100)</pre><p>That is the innovation. One kwarg. The entire difference between “continue this text” and “answer this request” is one kwarg and a dataset that respects it.</p><p>In the training loop it’s the same flatten trick from chapter 4, just pointed at the new loss:</p><pre>logits = model(input_batch)              # (B, T, vocab_size)  <br>loss = masked_loss_fn(  <br>    logits.flatten(0, 1),                # (B*T, vocab_size)  <br>    target_batch.flatten()               # (B*T,)  <br>)</pre><p>flatten(0, 1) turns every position into its own independent classification problem over the vocabulary. ignore_index then throws away every row labelled -100 <em>before</em> the average is taken — not weighted down, not scaled, removed.</p><p>Now sit with the consequence, because it’s wilder than it sounds. A 200-token prompt with a one-word answer like Python means <strong>199 of 200 positions contribute literally nothing.</strong> The entire loss, the entire gradient, the entire weight update for that example, is decided by a single token.</p><p>That extreme imbalance is not a flaw in the method. It <em>is</em> the method. Every joule of gradient goes into the answer.</p><h3>Step 4: Where the whole series fell over</h3><p>Last function. Chapter 7, step 4, the final cell of a project that started with import torch and ended, several thousand lines later, here:</p><pre>def generate_instruction_response(model, tokenizer, instruction,  <br>                                  input_text=None, max_new_tokens=50):  <br>    model.eval()  <br>    entry = {&quot;instruction&quot;: instruction, &quot;input&quot;: input_text}  <br>    formatted_prompt = format_input(entry) + &quot;\n\n###Response:\n&quot;  <br>  <br>    input_ids = torch.tensor(tokenizer.encode(formatted_prompt)).unsqueeze(0)  <br>    with torch.no_grad():  <br>        for _ in range(max_new_tokens):  <br>            logits = model(input_ids)  <br>            next_token = torch.argmax(logits[:, -1, :], dim=1, keepdim=True)  <br>            input_ids = torch.cat((input_ids, next_token), dim=-1)  <br>            if next_token.item() == tokenizer.eot_token:  <br>                break  <br>  <br>    decoded_text = tokenizer.decode(input_ids[0].tolist())  <br>    reponse_part = decoded_text.split(&quot;### Response:\n&quot;)[-1]  <br>    return response_part.strip()</pre><p>The decoding logic is correct and boring in the best possible way. model.eval() kills dropout, so the same prompt doesn&#39;t wander just because random units got masked. logits[:, -1, :] keeps only the last position, which — thanks to the causal attention I built back in chapter 2 — has been allowed to see the entire prompt plus everything written so far. argmax is greedy decoding: deterministic, no sampling, same prompt always gives the same answer. torch.cat grows the sequence. eot_token breaks early.</p><p>Now count how many different ways this project spells one heading:</p><p>WhereSpelling</p><p>Step 1, the sample full_entry</p><p>### Response: — with a space</p><p>InstructionDataset, the training string</p><p>###Response: — no space</p><p>Generator, the prompt it builds</p><p>###Response: — no space</p><p>Generator, the string .split() hunts for</p><p>### Response: — <strong>with a space</strong></p><p>The split is searching for a string that <strong>does not exist in the text it was handed.</strong></p><p>And str.split() does not raise when it finds nothing. It returns a one-element list containing the whole original string, and [-1] cheerfully hands you that whole string back. So instead of Python, the caller receives the preamble, the instruction, the input block, the heading, and Python. No exception. No warning. Nothing. It just quietly returns the wrong thing, forever.</p><p>Except it never even gets that far — because the line above assigns reponse_part and the return reads response_part.</p><p><strong>NameError. First call. Every time.</strong></p><p>And honestly? Thank god for that typo.</p><p>A NameError is <em>loud</em>. It stops you at the door, points at the line number, and refuses to let you leave. The heading mismatch would have shipped. It would have produced a function that returns something plausible-looking, and I&#39;d have spent an entire evening blaming the model&#39;s &quot;quality&quot; for a bug that lives in a string literal.</p><p><strong>Loud bugs are a gift. Silent ones are the tax.</strong></p><h3>The version that actually works</h3><pre>RESPONSE_HEADER = &quot;\n\n### Response:\n&quot;   # define ONCE, import everywhere  <br>  <br>def generate_instruction_response(model, tokenizer, instruction,  <br>                                  input_text=None, max_new_tokens=50,  <br>                                  device=&quot;cpu&quot;):  <br>    model.eval()  <br>    entry = {&quot;instruction&quot;: instruction, &quot;input&quot;: input_text}  <br>    prompt = format_input(entry) + RESPONSE_HEADER  <br>  <br>    input_ids = torch.tensor(  <br>        tokenizer.encode(prompt), device=device  <br>    ).unsqueeze(0)  <br>  <br>    context_len = model.pos_emb.weight.shape[0]  <br>  <br>    with torch.no_grad():  <br>        for _ in range(max_new_tokens):  <br>            logits = model(input_ids[:, -context_len:])  <br>            next_token = torch.argmax(logits[:, -1, :], dim=-1, keepdim=True)  <br>            if next_token.item() == tokenizer.eot_token:  <br>                break  <br>            input_ids = torch.cat((input_ids, next_token), dim=-1)  <br>  <br>    decoded = tokenizer.decode(input_ids[0].tolist())  <br>    return decoded.split(RESPONSE_HEADER)[-1].strip()</pre><p>What changed, and why each one matters:</p><ul><li><strong>One constant, used in three places.</strong> This is the real fix. Not “add the missing space” — <em>remove the possibility of disagreement</em>. The dataset and the generator can no longer drift apart, because there is only one string.</li><li><strong>break before </strong><strong>cat</strong>, so the end-of-text token never lands inside the string you return.</li><li><strong>A context-window clamp</strong>, so generation doesn’t crash the moment prompt plus output crosses 1024.</li><li><strong>A </strong><strong>device argument</strong>, because silent .cpu() surprises are the second-worst kind of bug after silent string bugs.</li></ul><h3>What seven chapters actually taught me</h3><p>The architecture was never the hard part.</p><p>I want to be clear about that, because it’s the opposite of what I believed on day one. Attention looks intimidating and isn’t. Multi-head attention is a reshape. Layer norm is four lines. The transformer block is a sandwich of things you already understand. The training loop is zero_grad, forward, loss, backward, step — and it has been that for a decade. You can get all of it right, and I did, on a laptop, in a Jupyter notebook, with no GPU cluster and no secret knowledge.</p><p>What actually breaks a fine-tuning run is <strong>contract drift</strong>.</p><p>Three places in your codebase need to agree on one string. Two of them do. Nobody checks the third. The dataset builds fine. The loss goes down beautifully. The eval output looks <em>subtly</em> wrong in a way you can’t articulate. And you lose two days to a space bar.</p><p>Everything in this chapter compresses into one sentence:</p><blockquote><strong><em>The training string, the generation string, and the extraction string must be the same characters — or your model is answering a format it never studied.</em></strong></blockquote><p>Put the headings in a constant. Assert your prefixes. Make your bugs loud.</p><p>The transformer will forgive you an enormous amount. It will not forgive you a space.</p><h3>That’s the series</h3><p>Seven chapters. Tokenizer, embeddings, self-attention, causal masking, multi-head attention, the transformer block, a full GPT, pretraining, loss evaluation, classification fine-tuning, and now instruction fine-tuning — all of it written by hand, all of it runnable on a laptop, all of it in one repo.</p><p>Here’s the thing I didn’t expect at the start: after you build the whole stack yourself, these models stop feeling like magic. Not because they got less impressive — they didn’t — but because you can now point at any part of the pipeline and say exactly what it does and why it’s there. GPT-4 is this, with more layers, more data, more money, and several extra chapters of alignment work bolted on top. The skeleton is the one you just read.</p><p>That is worth more than any course. Go build it yourself. Break it yourself. The bugs are where the learning actually lives — I’ve got a directory in this repo literally named 07_llm_finetuning_for_instructrions, typo and all, as permanent proof.</p><p>If this series helped, the full code is in the repo, one runnable notebook per chapter. Start at 00 and ship something.</p><p><strong>Found the </strong><strong>reponse_part typo before I pointed at it?</strong> Then you&#39;ve debugged a real training run before, and you have my full respect. 👇</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=03fd989aa61c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Day 6: I Fine-Tuned GPT by Training 1,538 Parameters. Out of 124 Million. That’s 0.001%.]]></title>
            <link>https://medium.com/@Yash_Bhardwaj/day-6-i-fine-tuned-gpt-2-by-training-1-538-parameters-out-of-124-million-thats-0-001-861c2caf689f?source=rss-bec114f6ca70------2</link>
            <guid isPermaLink="false">https://medium.com/p/861c2caf689f</guid>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Yash Bhardwaj]]></dc:creator>
            <pubDate>Sun, 11 Oct 2026 16:23:42 GMT</pubDate>
            <atom:updated>2026-10-11T18:06:42.177Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5ViOVTUP20ZyVKuJRsK6MQ.png" /><figcaption>Day 6: 124 million parameters frozen in place. Only the tiny orange head at the top learns. That’s a linear probe, and it’s 0.001% of the model.</figcaption></figure><h3>Turning a text-completer into a classifier: swapping the head, freezing the backbone, last-token pooling, and why fine-tuning is 99.999% not training.</h3><p>Ask GPT a question and it doesn’t answer. It continues the question.</p><p>Feed it “Is this message spam?” and it’ll write “Is this message spam? Is this message from a friend? Is this…” forever. It’s not being difficult. It’s doing the only thing it was ever trained to do: predict the next token.</p><p>That’s the gap between GPT and ChatGPT. Between a model that <em>knows</em> language and a model that <em>does</em> something with it.</p><p>Today I close that gap. Not all the way, instruction-following comes later, but the first real step: teaching my hand-built, OpenAI-weighted GPT to read a whole message and say <strong>spam</strong> or <strong>not spam</strong>.</p><p>And here’s the number that broke my brain: to do it, I train <strong>1,538 parameters</strong>. The other <strong>124 million</strong> stay frozen. Untouched.</p><p>Fine-tuning, it turns out, is mostly <em>not</em> training.</p><h3>Where we left off</h3><ul><li><strong>Days 1–3:</strong> Built the GPT architecture from scratch.</li><li><strong>Day 4:</strong> Built the training loop, loss, validation, checkpoints.</li><li><strong>Day 5:</strong> Loaded OpenAI’s real GPT-2 weights into my own classes. real_gpt now holds a brain that read the internet in 2019.</li></ul><p>real_gpt outputs (batch, tokens, 50257): at every position, a score for every word in the vocabulary. Great for writing. Useless for a yes/no question.</p><p>So I need to change what comes out the top.</p><h3>Step 1: Swap the head, keep the brain</h3><p>Here’s the entire idea of fine-tuning for classification in one sentence: <strong>keep everything that understands language, replace only the last layer.</strong></p><pre>import torch  <br>import torch.nn as nn   <br>  <br>class GPTForClassification(nn.Module):  <br>    def __init__(self, base_model, num_classes=2):  <br>        super().__init__()  <br>        self.tranformers= base_model.trf_blocks  <br>        self.tok_emb= base_model.tok_emb  <br>        self.pos_emb= base_model.pos_emb  <br>        self.drop_emb= base_model.drop_emb  <br>        self.final_ln= base_model.final_ln   <br>  <br>        self.classifier_head= nn.Linear(base_model.out_head.in_features, num_classes)  <br>  <br>    def forward(self, in_idx):  <br>        b,seq_len= in_idx.shape  <br>  <br>        tok_embeds= self.tok_emb(in_idx)  <br>        pos_embeds=self.pos_emb(torch.arrange(seq_len, device=in_idx.device))  <br>        x= self.drop_emb(tok_embeds+pos_embeds)  <br>  <br>        x=self.trf_blocks(x) if hasattr(self, &#39;trf_blocks&#39;) else self.transformer(x)  <br>        x= self.final_ln(x)  <br>  <br>        last_token_vectors = x[:, -1, :]  <br>  <br>        logits= self.classifier_head(last_token_vectors)  <br>        return logits</pre><p>Look at what __init__ does. It doesn&#39;t build anything new. It <em>borrows</em>. Token embeddings, position embeddings, dropout, all 12 transformer blocks, the final LayerNorm, every one of them is a pointer straight into real_gpt. Same tensors. Same OpenAI numbers.</p><p>The only new object is one line:</p><pre>self.classifier_head = nn.Linear(768, 2)</pre><p>GPT-2’s original head was Linear(768, 50257): &quot;which of 50,257 words comes next?&quot; My new head is Linear(768, 2): &quot;which of 2 labels is this?&quot;</p><p><strong>Real-world version:</strong> a senior lawyer switching from corporate law to family law. You don’t send them back to school. Twenty years of reading contracts, arguing, reasoning about language, all of that transfers. You hand them one new reference book. That book is the classifier head.</p><h3>The line that matters most</h3><pre>last_token_vectors = x[:, -1, :]</pre><p>After the transformer stack, x has shape (batch, tokens, 768): one 768-dimensional vector per token. But I need <strong>one</strong> vector per <em>message</em>. Which token do I pick?</p><p>The <strong>last one</strong>. And it’s not arbitrary.</p><p>Remember Day 2’s causal mask: each token can only attend to tokens <em>before</em> it. Token 1 sees only itself. Token 5 sees tokens 1 through 5. The <strong>final</strong> token is the only position that’s been allowed to look at the entire message. Its vector is the only one that contains the whole thing.</p><p>So x[:, -1, :] isn&#39;t &quot;grab the last word&quot;. It&#39;s &quot;grab the one vector that read everything&quot;. That&#39;s <strong>last-token pooling</strong>, and it&#39;s why a model trained to predict the next word can be repurposed to judge a whole sentence.</p><h3>Step 2: Bring in the brain</h3><pre>%run ../04_pretraining_and_loss_evaluation/llm_training.ipynb</pre><p>Same trick as Day 4 and 5. This runs the entire previous notebook, which itself runs Day 3’s, which runs Day 2’s. At the end, real_gpt is sitting in memory with OpenAI&#39;s 148 tensors loaded. The output scrolls past:</p><pre>Loading weights: 100%|██████████| 148/148  <br>openai GPT2 layers found: 148  <br>structural status: openai pretrained weights successfully mapped and injected  <br>inferende pass result: logits shape is torch.Size([1, 3, 50257])</pre><p>Five days of work, summoned in one line.</p><h3>Step 3: Freeze 124 million. Train 1,538.</h3><p>Now the number from the headline.</p><pre>model_classifier= GPTForClassification(real_gpt, num_classes=2)  <br>  <br>for param in model_classifier.parameters():  <br>    param.requires_grad=False  <br>for param in model_classifier.classifier_head.parameters():  <br>    param.requires_grad=True  <br>  <br>trainable_params= sum(p.numel() for p in model_classifier.parameters() if p.requires_grad)  <br>print(&quot;total trainable parameters for fine tuning:&quot;, trainable_params)</pre><pre>total trainable parameters for fine tuning: 1538</pre><p>Three moves:</p><ol><li>Wrap real_gpt in the classifier. At this instant, all 124M parameters are trainable.</li><li>requires_grad = False on <em>everything</em>. The whole model is frozen. Autograd will skip it. No gradients computed, no updates, no memory spent.</li><li>requires_grad = True on the head only.</li></ol><p>Count what’s left: 768 × 2 weights + 2 biases = <strong>1,538</strong>.</p><p>Do the arithmetic. 1,538 ÷ 124,000,000 ≈ <strong>0.0012%</strong>. I’m “fine-tuning GPT-2” by adjusting about one parameter in every hundred thousand.</p><p>Why freeze? Two reasons, and they’re both brutal in practice:</p><ul><li><strong>Overfitting.</strong> A few thousand spam messages against 124M free parameters: the model will memorise every row of the CSV and fail on the first new message it sees. Freezing makes that impossible. The backbone <em>can’t</em> memorise. It can only describe.</li><li><strong>Speed.</strong> Backward pass only runs through 1,538 numbers. On a laptop this trains in seconds instead of hours.</li></ul><p>This setup has a name: <strong>linear probe</strong>. The frozen GPT becomes a <em>feature extractor</em>: it turns any message into a 768-dimensional “meaning vector”. The tiny head learns a straight line through that space that separates spam from not-spam.</p><p>And here’s what stopped me: <strong>that 768-dim vector already contains the answer.</strong> GPT-2 never saw a spam label in its life. But in learning to predict the next word across the internet, it learned what pushy, urgent, link-heavy, “CLICK NOW” text <em>feels</em> like, because that feeling changes what word comes next. The knowledge was always in there. I’m just adding a 2-row lookup to read it out.</p><p>That’s the whole secret of transfer learning. The pretrained model isn’t a starting point. It’s a finished sensor. You’re attaching a dial.</p><h3>Step 4: A dataset where every message gets one label</h3><p>Day 1’s dataset was a stream of tokens with a next-token target at <em>every position</em>. Classification is different: a list of whole messages, each with <strong>one</strong> label.</p><pre>import torch  <br>from torch.utils.data import Dataset   <br>  <br>class SpamDataset(Dataset):  <br>    def __init__(self, csv_file, tokenizer, max_length=None):  <br>        import pandas as pd   <br>        self.data=pd.read_csv(csv_file)  <br>        self.encoded_texts= [tokenizer.encode(text) for text in self.data[&quot;text&quot;]]  <br>  <br>        if max_length is None:  <br>            self.max_length=max(len(t) for t in self.encoded_texts)  <br>        else:  <br>            self.max_length= max_length   <br>        for i,text in enumerate(self.encoded_texts):  <br>            if len(text)&lt; self.max_length:  <br>                padding_length= self.max_length-len(text)  <br>                self.encoded_texts[i]+=[tokenizer.eot_token]* padding_length  <br>  <br>        self.input_ids= torch.tesnsor(self.encoded_texts)  <br>        self.labels= torch.tensor(self.data[&quot;label&quot;].values)  <br>    def __len__(self):  <br>        return len(self.labels)  <br>  <br>    def __getitem__(self, idx):  <br>        return self.input_ids[idx], self.labels[idx]</pre><p>Two columns in the CSV: text and label (0 or 1). The flow:</p><ol><li>Tokenize every message with tiktoken, same BPE tokenizer from Day 1.</li><li>Find the longest message. That’s max_length.</li><li><strong>Pad</strong> every shorter message with &lt;|endoftext|&gt; tokens until they&#39;re all the same length. Tensors need rectangles; messages aren&#39;t rectangles.</li><li>Stack into one (num_messages, max_length) tensor.</li></ol><p>__len__ and __getitem__ are PyTorch&#39;s contract: &quot;how many examples?&quot; and &quot;give me example number idx&quot;. That&#39;s all DataLoader needs to batch and shuffle for you.</p><p><strong>Real-world version:</strong> padding is like putting every letter in the same size envelope. The short ones get stuffed with blank paper so the sorting machine doesn’t jam.</p><h3>Step 5: Accuracy, the number a human can read</h3><p>Loss is what you train on. Accuracy is what you tell your boss.</p><pre>import torch   <br>def calc_accuracy_loader(dataloader, model, device, num_batches=None):  <br>    model.eval()  <br>    correct_predictions, total_examples=0,0  <br>    for i, (inputs, targets) in enumerate(dataloader):  <br>        if num_batches and i&gt;=num_batches:  <br>            break  <br>        inputs,targets=input.to(device), targets.to(device)  <br>        with torch.no_grad():  <br>            logits=model(inputs)  <br>        prediction=torch.argmax(logits, dim=-1)  <br>  <br>        correct_predictions+=(predictions==targets).sum().item()  <br>        total_examples+= targets.numel()  <br>  <br>    return correct_predictions</pre><p>Cousin of Day 4’s calc_loss_loader. Same two safety rails: model.eval() and torch.no_grad(), so scoring never accidentally becomes training.</p><p>The new piece is torch.argmax(logits, dim=-1). Logits are (batch, 2): two scores per message. Argmax picks the bigger one&#39;s index: 0 or 1. Compare to the true label, count the matches.</p><p>Why both numbers? Because they disagree. A model can be <em>confidently</em> wrong on a few examples (high loss, decent accuracy) or <em>barely</em> right on many (okay loss, great accuracy). Loss drives the gradient. Accuracy tells you if it’s actually useful.</p><h3>Step 6: The loop, one more time</h3><pre>import torch   <br>def train_classifier_model(model, train_loader, val_loader, optimizer, device, num_epochs, eval_freq, eval_iter):  <br>    train_losses, val_losses=[],[]  <br>    train_accs, val_accs= [], []  <br>  <br>    for epoch in range(num_epochs):  <br>        model.train()  <br>  <br>        for input_batch, target_batch in train_loader:  <br>            inputs, targets=input_batch.to(device), target_batch.to(device)  <br>            optimizer.zero_grad()  <br>            logits=model(inputs)  <br>  <br>            loss= torch.nn.functional.cross_entropy(logits, targets)  <br>            loss.backward()  <br>            optimizer.step  <br>  <br>        if epoch%eval_freq==0:  <br>            train_loss= calc_loss_loader(train_loader, model, device, eval_iter)  <br>            val_loss= calc_loss_loader(val_loader, model, device, eval_iter)  <br>  <br>            train_acc= calc_accuracy_loader(train_loader, model, device, eval_iter)  <br>            val_acc= calc_accuracy_loader(val_loader, model, device, eval_iter)  <br>  <br>            train_losses.append(train_loss)  <br>            val_losses.append(val_loss)  <br>            train_accs.append(train_acc)  <br>            val_accs.append(val_acc)  <br>  <br>            print(&quot;f Epoch{epoch+1:02d} | train loss:{train_loss:.4f} | val loss{val_loss:.4f}&quot;)  <br>            print(f&quot;f | train acc : {train_acc*100:.2f}% | val acc:{val_acc*100:.2f}%&quot;)  <br>    return train_losses, val_losses, train_accs, val_accs</pre><p>If this looks like Day 4’s train_model, that&#39;s the point. Same five moves: zero_grad → forward → loss → backward → step. Same validation check every eval_freq epochs.</p><p>Three things changed, and they’re the entire difference between pretraining and fine-tuning:</p><p><strong>1. No </strong><strong>flatten(0, 1).</strong> On Day 4, logits were (batch, tokens, vocab) and I flattened them so every token was its own quiz. Now logits are (batch, 2) and labels are (batch,). One question per message. Cross-entropy eats them directly.</p><p><strong>2. The optimizer only touches 1,538 numbers.</strong> loss.backward() still runs, but autograd stops at the frozen boundary. The 12 transformer blocks never see a gradient.</p><p><strong>3. Two scoreboards.</strong> Loss <em>and</em> accuracy, train <em>and</em> validation. Four numbers per epoch instead of two. If train accuracy hits 100% and val accuracy stalls at 80%, even 1,538 parameters have found a way to memorise, and it’s time to stop.</p><p>The loop that trained GPT-2 on the internet and the loop that teaches it spam detection are the <strong>same loop</strong>. The only things that changed are the shape of the output and how many parameters are allowed to move.</p><h3>What I learned building this</h3><p><strong>1. Fine-tuning is mostly freezing.</strong> 1,538 trainable out of 124M. The hard-won knowledge stays locked. You’re not re-educating the model. You’re asking it a new question about what it already knows.</p><p><strong>2. The last token is the summary.</strong> Causal attention means only the final position has seen the whole message. x[:, -1, :] is the single most important line in this chapter, and it only works <em>because</em> of the mask from Day 2.</p><p><strong>3. A pretrained model is a sensor, not a starting point.</strong> The 768-dim vector already encodes “this feels like spam”. GPT-2 learned that without ever seeing a label, because it changes what word comes next. The head just reads the dial.</p><p><strong>4. Same loop, different shape.</strong> Pretraining and fine-tuning aren’t two technologies. They’re one training loop with a different head on top. Once you’ve seen that, every “fine-tuned on X” headline becomes legible.</p><h3>What’s next</h3><p>The classifier is wired: head swapped, backbone frozen, dataset built, accuracy measured, loop written. Next I point it at a real spam CSV and watch four columns of numbers move.</p><p>And after that, the one everyone’s waiting for: <strong>instruction fine-tuning</strong>. Not “is this spam?” but “do what I say”. That’s the step that makes GPT feel like ChatGPT, and it uses this exact same machinery with one more twist.</p><p>If you’ve been curious about how ChatGPT really works under the hood, <strong>follow along</strong>. I’m building a complete LLM from scratch, one concept per day, in public. Every line of code is in my GitHub repo, <strong>LLM-From-Scratch</strong>, with one runnable notebook per chapter.</p><p>If this article helped something click, <strong>give it a clap</strong> (you can hold the button for up to 50), and tell me in the comments: did “1,538 parameters” surprise you as much as it surprised me?</p><p>See you on Day 7.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=861c2caf689f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Day 5: I Downloaded OpenAI’s Brain and Shoved It Into Code I Wrote by Hand.]]></title>
            <link>https://medium.com/@Yash_Bhardwaj/day-5-i-downloaded-openais-brain-and-shoved-it-into-code-i-wrote-by-hand-b9d51afdbc1f?source=rss-bec114f6ca70------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9d51afdbc1f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Yash Bhardwaj]]></dc:creator>
            <pubDate>Sat, 10 Oct 2026 18:01:35 GMT</pubDate>
            <atom:updated>2026-10-10T18:03:21.609Z</atom:updated>
            <content:encoded><![CDATA[<h3>Day 5: I Downloaded OpenAI’s Brain and Shoved It Into Code I Wrote by Hand. 148 Tensors. One Wrong Shape and It All Falls Apart.</h3><figure><img alt="Dark cover image for Day 5 of building an LLM from scratch. A glowing orange wireframe brain made of data points streams downward into a grid of small glowing tensor blocks, a few still dark and empty, over blurred PyTorch code. Large orange text reads “DAY 5” in the top-left corner; bold white headline reads “I LOADED OPENAI’S BRAIN” with “148 tensors. 0 shortcuts.” in orange beneath it." src="https://cdn-images-1.medium.com/max/1024/1*89pwYD5oUW2MhD7UmzE4Bw.png" /><figcaption>Day 5: OpenAI’s GPT-2 checkpoint, all 148 tensors, poured into model classes I wrote by hand. One wrong shape and PyTorch refuses. Image by the author.</figcaption></figure><h3>Loading the real GPT-2 weights into a from-scratch PyTorch model: name translation, fused attention, transposed layouts, and the 38 million parameters that finally get explained.</h3><p>On Day 3 I counted the parameters in my GPT and got <strong>162 million</strong>.</p><p>GPT-2 has <strong>124 million</strong>.</p><p>I shrugged it off. Wrote “we’ll see later” in my notes. Moved on.</p><p>Today is later.</p><p>Today I downloaded the actual GPT-2 checkpoint that OpenAI released in 2019, all 148 tensors, and tried to inject it into the model classes I wrote by hand on Days 1 through 3. Not Hugging Face’s model. <em>Mine.</em> The one with the hand-rolled attention heads and the LayerNorm I wrote from a formula.</p><p>Here’s the thing about this experiment: <strong>it can’t be faked.</strong> If my architecture is wrong by a single dimension, the copy fails. PyTorch doesn’t negotiate with shapes. Either my 124-million-number dictionary slots into my 162-million-number model, or it doesn’t.</p><p>Spoiler: it didn’t. Not at first. And the reason it didn’t is the most interesting thing I’ve learned in this whole series.</p><h3>Where we left off</h3><ul><li><strong>Day 1–3:</strong> Built a complete GPT architecture from scratch. Random weights. Speaks nonsense.</li><li><strong>Day 4:</strong> Built the training machinery. Cross-entropy, the five-line loop, validation loss, checkpoints. Learned that “the weights” are just a state_dict: a Python dictionary of tensors.</li></ul><p>That last point is the whole key to today. If a model is just <strong>architecture + dictionary</strong>, and I built the architecture… then I only need someone else’s dictionary.</p><h3>Step 10: Downloading the brain</h3><p>Training a 124M-parameter GPT from scratch needs a slice of the internet and weeks of GPU time. I have a laptop. So there’s a shortcut, and it’s an honest one <em>because</em> I built the architecture myself: borrow OpenAI’s weights.</p><pre>#%pip install transformers  <br>import transformers  <br>import torch  <br>from transformers import GPT2Model  <br>  <br>openai_gpt= GPT2Model.from_pretrained(&quot;gpt2&quot;)  <br>openai_params= openai_gpt.state_dict()  <br>print(&quot;openai GPT2 layers found:&quot;, len(openai_params.keys()))  <br>  <br>from torch.optim import AdamW  <br>print(&quot;\n--- Sample Keys Mismatch Comparision ---&quot;)  <br>print(&quot;openai layer name:&quot;, list(openai_params.keys())[0])</pre><pre>Loading weights: 100%|██████████| 148/148 [00:00&lt;00:00, 6305.62it/s]  <br>openai GPT2 layers found: 148  <br>  <br>--- Sample Keys Mismatch Comparision ---  <br>openai layer name: wte.weight</pre><p>One line. GPT2Model.from_pretrained(&quot;gpt2&quot;) pulls the smallest GPT-2: 12 layers, 12 heads, 768-dim embeddings, 50,257-token vocab. state_dict() hands me the dictionary.</p><p><strong>148 tensors.</strong> That’s it. That’s GPT-2. The model that made headlines in 2019 for being “too dangerous to release” is a dictionary with 148 entries.</p><p>But look at that last line. OpenAI’s first key is wte.weight. Mine is tok_emb.weight. Same numbers, same shape, same job. Different name.</p><p>PyTorch has no idea they belong together.</p><p><strong>Real-world version:</strong> you bought furniture from one brand, the instructions are from another. Every screw is the right screw. Every panel is the right panel. But part “A7” in their manual is “left side panel” in yours. Until someone translates, you’re staring at a pile of wood.</p><p>So I wrote the translator.</p><h3>Step 11: The translation function</h3><p>This is the heart of the article. 50 lines. Every line is a bridge between how OpenAI stored a number and where my model expects it.</p><pre>def assign_weights(gpt_model, openai_params):  <br>    # 1. Access models state dict pointers  <br>    d = gpt_model.state_dict()  <br>    n_layers = len(gpt_model.trf_blocks)  <br>    n_heads = len(gpt_model.trf_blocks[0].att.heads)  <br>      <br>    # Global mapped lookups configuration  <br>    d[&quot;tok_emb.weight&quot;].copy_(openai_params[&quot;wte.weight&quot;])  <br>    d[&quot;pos_emb.weight&quot;].copy_(openai_params[&quot;wpe.weight&quot;])  <br>      <br>    # 2. Iterate through Transformer Blocks layers  <br>    for l in range(n_layers):  <br>        # OpenAI stores Q, K, V fused in one Conv1D matrix; split and map onto each head  <br>        q_w, k_w, v_w = torch.chunk(openai_params[f&quot;h.{l}.attn.c_attn.weight&quot;], 3, dim=-1)  <br>        q_w, k_w, v_w = q_w.T, k_w.T, v_w.T  <br>          <br>        for h, (q_h, k_h, v_h) in enumerate(zip(  <br>            torch.chunk(q_w, n_heads, dim=0),  <br>            torch.chunk(k_w, n_heads, dim=0),  <br>            torch.chunk(v_w, n_heads, dim=0),  <br>        )):  <br>            d[f&quot;trf_blocks.{l}.att.heads.{h}.W_query.weight&quot;].copy_(q_h)  <br>            d[f&quot;trf_blocks.{l}.att.heads.{h}.W_key.weight&quot;].copy_(k_h)  <br>            d[f&quot;trf_blocks.{l}.att.heads.{h}.W_value.weight&quot;].copy_(v_h)  <br>          <br>        if f&quot;trf_blocks.{l}.att.heads.0.W_query.bias&quot; in d:  <br>            q_b, k_b, v_b = torch.chunk(openai_params[f&quot;h.{l}.attn.c_attn.bias&quot;], 3, dim=-1)  <br>            for h, (q_h, k_h, v_h) in enumerate(zip(  <br>                torch.chunk(q_b, n_heads),  <br>                torch.chunk(k_b, n_heads),  <br>                torch.chunk(v_b, n_heads),  <br>            )):  <br>                d[f&quot;trf_blocks.{l}.att.heads.{h}.W_query.bias&quot;].copy_(q_h)  <br>                d[f&quot;trf_blocks.{l}.att.heads.{h}.W_key.bias&quot;].copy_(k_h)  <br>                d[f&quot;trf_blocks.{l}.att.heads.{h}.W_value.bias&quot;].copy_(v_h)  <br>          <br>        d[f&quot;trf_blocks.{l}.att.out_projection.weight&quot;].copy_(openai_params[f&quot;h.{l}.attn.c_proj.weight&quot;].T)  <br>        d[f&quot;trf_blocks.{l}.att.out_projection.bias&quot;].copy_(openai_params[f&quot;h.{l}.attn.c_proj.bias&quot;])  <br>          <br>        # Feed-Forward parameters translations  <br>        d[f&quot;trf_blocks.{l}.ff.layers.0.weight&quot;].copy_(openai_params[f&quot;h.{l}.mlp.c_fc.weight&quot;].T)  <br>        d[f&quot;trf_blocks.{l}.ff.layers.0.bias&quot;].copy_(openai_params[f&quot;h.{l}.mlp.c_fc.bias&quot;])  <br>        d[f&quot;trf_blocks.{l}.ff.layers.2.weight&quot;].copy_(openai_params[f&quot;h.{l}.mlp.c_proj.weight&quot;].T)  <br>        d[f&quot;trf_blocks.{l}.ff.layers.2.bias&quot;].copy_(openai_params[f&quot;h.{l}.mlp.c_proj.bias&quot;])  <br>          <br>        # Layer Normalization layers parameters map  <br>        d[f&quot;trf_blocks.{l}.ln1.scale&quot;].copy_(openai_params[f&quot;h.{l}.ln_1.weight&quot;])  <br>        d[f&quot;trf_blocks.{l}.ln1.shift&quot;].copy_(openai_params[f&quot;h.{l}.ln_1.bias&quot;])  <br>        d[f&quot;trf_blocks.{l}.ln2.scale&quot;].copy_(openai_params[f&quot;h.{l}.ln_2.weight&quot;])  <br>        d[f&quot;trf_blocks.{l}.ln2.shift&quot;].copy_(openai_params[f&quot;h.{l}.ln_2.bias&quot;])  <br>          <br>    # Final global parameters maps values  <br>    d[&quot;final_ln.scale&quot;].copy_(openai_params[&quot;ln_f.weight&quot;])  <br>    d[&quot;final_ln.shift&quot;].copy_(openai_params[&quot;ln_f.bias&quot;])  <br>    d[&quot;out_head.weight&quot;].copy_(openai_params[&quot;wte.weight&quot;]) # Weight Tying active!</pre><p>.copy_() writes values <strong>in place</strong> into my model&#39;s existing tensors. The Python objects don&#39;t change. Only the numbers inside them do. It&#39;s a brain transplant where the skull stays.</p><p>There are <strong>three</strong> kinds of mismatch this function fixes, and each one taught me something about how GPT-2 is actually built.</p><h3>Mismatch 1: Names</h3><p>Pure translation table.</p><p>OpenAI calls itI call itwtetok_embwpepos_embh.{l}trf_blocks.{l}ln_1, ln_2, ln_fln1, ln2, final_lnmlp.c_fc, mlp.c_projff.layers.0, ff.layers.2</p><p>(ff.layers.1 is GELU. No weights. Skipped.)</p><p>Boring, but notice n_layers and n_heads are read <em>off my model</em>, not hardcoded. Build a 6-layer GPT tomorrow and the same function still works.</p><h3>Mismatch 2: Fused attention</h3><p>This one surprised me.</p><p>On Day 2 I built attention the textbook way: each head has its own W_query, W_key, W_value. Three small matrices per head, 12 heads, 36 matrices per layer.</p><p>OpenAI doesn’t do that. They store Query, Key and Value for <strong>all 12 heads</strong> in <strong>one</strong> giant matrix called c_attn, shape (768, 2304). Because 2304 = 3 × 768. One matrix multiply instead of 36. Faster on a GPU.</p><p>So the function has to un-fuse it:</p><ul><li>torch.chunk(..., 3, dim=-1) — cut the 2304 columns into three slabs of 768: Q, K, V.</li><li>.T — flip each slab (see mismatch 3).</li><li>torch.chunk(..., n_heads, dim=0) — slice each slab into 12 pieces of 64 rows, one per head.</li></ul><p>One matrix becomes 36. Same numbers, rearranged. Mathematically identical. <strong>The “12 separate heads” I drew on Day 2 were never really separate in the real model. They’re one big multiply, sliced afterwards.</strong></p><p>The if ... .bias in d block does the same for the biases, but only if my model was built with qkv_bias=True. Hold that thought.</p><h3>Mismatch 3: Transposed layouts</h3><p>Hugging Face implements these layers as Conv1D, which stores weights as (in, out). PyTorch&#39;s nn.Linear, which I used, stores (out, in). Every linear copy needs a .T.</p><p>Forget one .T on a non-square matrix: shape error, you catch it in a second.</p><p>Forget one .T on a <strong>square</strong> matrix, like c_proj at (768, 768): shapes match, copy succeeds, numbers are silently transposed, and your model produces confident garbage with no error message. This is the kind of bug that eats a weekend.</p><h3>The last line</h3><pre>d[&quot;out_head.weight&quot;].copy_(openai_params[&quot;wte.weight&quot;]) # Weight Tying active!</pre><p>Read it again. The <strong>output head</strong> is being loaded from the <strong>token embedding</strong>. The same matrix.</p><p>This is <strong>weight tying</strong>, and it’s the answer to Day 3’s mystery.</p><p>The token embedding is a 50,257 × 768 table that turns a word into a vector on the way <em>in</em>. The output head is a 768 × 50,257 matrix that turns a vector back into word scores on the way <em>out</em>. GPT-2 says: those are the same job in reverse. Use one table for both.</p><p>50,257 × 768 = <strong>38.6 million parameters.</strong> My 162M minus GPT-2’s 124M = <strong>38M.</strong></p><p>I’d been carrying a second copy of the vocabulary table for two days. Loading GPT-2 correctly means tying them. Mystery closed.</p><h3>Step 12: The moment of truth</h3><p>Build a real-sized GPT. Inject. Run.</p><pre>GPT_CONFIG_REAL= {  <br>    &quot;vocab_size&quot;: 50257, &quot;context_length&quot;: 1024, &quot;emb_dim&quot;: 768, &quot;n_heads&quot;:12, &quot;n_layers&quot;:12, &quot;drop_rate&quot;:0.0, &quot;qkv_bias&quot;:True  <br>}  <br>  <br>torch.manual_seed(123)  <br>real_gpt= GPTMODEL(GPT_CONFIG_REAL)  <br>assign_weights(real_gpt, openai_params)  <br>print(&quot;structural status: openai pretrained weights successfully mapped and injected&quot;)  <br>  <br>sample_ids=torch.tensor([[40,367,2885]])  <br>real_gpt.eval()  <br>with torch.no_grad():  <br>    inference_logits= real_gpt(sample_ids)  <br>  <br>print(&quot;inferende pass result: logits shape is&quot;, inference_logits.shape)</pre><pre>structural status: openai pretrained weights successfully mapped and injected  <br>inferende pass result: logits shape is torch.Size([1, 3, 50257])</pre><p>It went in.</p><p>But look at the config, because three numbers changed from Day 3, and every one of them was a wall I hit:</p><p>SettingDay 3NowWhat happens if you don’tcontext_length256<strong>1,024</strong>wpe.weight is 1024 rows. My table was 256. Copy fails on line 2.drop_rate0.1<strong>0.0</strong>Not training. Dropout would just add noise to a trained brain.qkv_biasFalse<strong>True</strong>Original GPT-2 has a bias on Q, K, V. Leave it False and the bias copy silently <em>skips</em>. No error. Attention math is just quietly wrong.</p><p>That third row is the scary one. The if block in assign_weights is a kindness that becomes a trap: it won&#39;t crash if the biases are missing. It&#39;ll just load a model that is <em>almost</em> GPT-2. Almost.</p><p><strong>The config must match the checkpoint.</strong> That’s not a style preference. It’s the difference between loading a model and loading a bug.</p><p>Then the test. Same three tokens as Day 3: [40, 367, 2885]. Decoded: <strong>&quot;I HAD&quot;</strong>, the first words of the short story this whole series has been feeding the model.</p><p>Same output shape as Day 3: (1, 3, 50257).</p><p>But on Day 3, those 50,257 scores came from random noise. Now they come from a network that read a large slice of the internet in 2019. Pass it through generate_text_simple from Day 3, decode with tiktoken, and the top next-token guesses are ordinary English. Not vocabulary IDs. Words.</p><p>My hand-written classes are holding OpenAI’s brain.</p><h3>What I learned building this</h3><p><strong>1. A model is architecture + a dictionary. Full stop.</strong> Once you’ve written assign_weights, &quot;loading GPT-2&quot; stops being magic and becomes 148 lines of copy_(). &quot;Open weights&quot; means someone published the dictionary.</p><p><strong>2. The real model is more fused than the diagram.</strong> Twelve attention heads are one (768, 2304) matrix, sliced after the fact. The textbook picture is correct. It&#39;s just not how it&#39;s stored.</p><p><strong>3. Shapes are your unit tests, until they aren’t.</strong> Wrong context_length crashes instantly. Wrong qkv_bias or a missing .T on a square matrix loads clean and runs wrong. The bugs that don&#39;t crash are the ones that cost you.</p><p><strong>4. 38 million parameters were a duplicate.</strong> Weight tying isn’t an optimisation trick. It’s a statement that reading a word and writing a word are the same operation, reversed. GPT-2 believes that. Now my model does too.</p><h3>What’s next: giving it a job</h3><p>Five days. One model. Built from tensors, trained by hand, scored honestly, saved, reloaded, and now initialised with the real GPT-2.</p><p>It can complete text. It can’t <em>do</em> anything yet.</p><p>Next: <strong>fine-tuning</strong>. Taking this pretrained brain and teaching it one narrow job: classification first, then following instructions. That’s the step that turns a text-completer into something that feels like a chatbot. That’s the step that turns GPT into ChatGPT.</p><p>If you’ve been curious about how ChatGPT really works under the hood, <strong>follow along</strong>. I’m building a complete LLM from scratch, one concept per day, in public. Every line of code is in my GitHub repo, <strong>LLM-From-Scratch</strong>, with one runnable notebook per chapter.</p><p>If this article helped something click, <strong>give it a clap</strong> (you can hold the button for up to 50), and drop a comment: did you know about weight tying? I didn’t. Genuinely curious how many people did.</p><p>See you on Day 6.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9d51afdbc1f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Day 4: I Trained My GPT for Exactly One Step. The Loss Was 1.72. That Number Changed How I See AI.]]></title>
            <link>https://medium.com/@Yash_Bhardwaj/day-4-i-trained-my-gpt-for-exactly-one-step-the-loss-was-1-72-that-number-changed-how-i-see-ai-c1174687c15b?source=rss-bec114f6ca70------2</link>
            <guid isPermaLink="false">https://medium.com/p/c1174687c15b</guid>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Yash Bhardwaj]]></dc:creator>
            <pubDate>Fri, 09 Oct 2026 15:17:11 GMT</pubDate>
            <atom:updated>2026-10-09T15:17:11.598Z</atom:updated>
            <content:encoded><![CDATA[<h3>Cross-entropy, the five-line training loop, validation loss, and checkpoints — the entire “learning” part of a large language model, built by hand in PyTorch.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*I65nPUlDnwRSN7fnhMdXeA.jpeg" /><figcaption>Day 4: a 162M-parameter GPT, trained for exactly one step. Loss: 1.72. Random guessing scores 1.61. The whole of “training” is making that curve fall. Image by the author.</figcaption></figure><p>For two years I used the phrase “train a model” without knowing what it meant.</p><p>I knew it involved GPUs. I knew it took a long time. I knew it cost OpenAI hundreds of millions of dollars. But if you’d asked me <em>what actually happens</em> when a model trains, I’d have said something vague about “learning from data” and changed the subject.</p><p>Yesterday I found out.</p><p>It’s five lines of code.</p><pre>optimizer.zero_grad()  <br>logits = model(inputs)  <br>loss = cross_entropy(logits, targets)  <br>loss.backward()  <br>optimizer.step()</pre><p>That’s it. That’s the loop. Every LLM you’ve ever used, every image generator, every recommendation engine that knows you a little too well, was trained by those five lines running over and over, billions of times.</p><p>The difference between my laptop and OpenAI’s datacenter isn’t the loop. It’s how many times it ran, and on how much data.</p><p>On Day 3 I built a 162-million-parameter GPT that spoke pure nonsense because its brain was random numbers. Today, I teach it. Not with the internet. With something small enough that I could check every number by hand.</p><p>Let me show you what “learning” actually looks like.</p><h3>Where we left off</h3><ul><li><strong>Day 1:</strong> Raw text → tokens → embedding vectors.</li><li><strong>Day 2:</strong> Self-attention and multi-head attention, from scratch.</li><li><strong>Day 3:</strong> LayerNorm, GELU, residual connections, the Transformer block, and a full GPTMODEL that outputs a tensor of shape (batch, tokens, 50257): one score for every word in the vocabulary, at every position.</li></ul><p>Those scores are called <strong>logits</strong>. Today they finally get judged.</p><h3>Step 1: What does “wrong” even mean for a language model?</h3><p>Before you can teach anything, you need a way to say <em>how wrong it is right now</em>. For a GPT, “wrong” has a very precise meaning:</p><p><strong>At each position, did it give a high score to the token that actually comes next?</strong></p><p>So I built the smallest possible version of that question. No model. No internet. A fake vocabulary of <strong>5 tokens</strong>, <strong>2 sequences</strong>, <strong>3 tokens each</strong>.</p><pre>import torch   <br>import torch.nn as nn   <br>inputs_logits = torch.tensor([  <br>    [[2.0, 1.0, 0.1, 0.5, 0.2],  <br>     [0.5, 2.5, 0.3, 0.8, 0.1],  <br>     [0.1, 0.4, 3.0, 0.2, 0.5]],  <br>  <br>     [[1.5, 0.3, 0.5, 2.0, 0.2],  <br>      [0.2, 0.8, 0.1, 0.4, 2.2],  <br>      [3.5, 0.1, 0.2, 0.5, 0.4]]  <br>])  <br>  <br>targets= torch.tensor([[0,1,2],  <br>[3,4,0]])  <br>  <br>print(&quot;logits shape:&quot;, inputs_logits.shape)  <br>print(&quot;targets shape:&quot;, targets.shape)</pre><pre>logits shape: torch.Size([2, 3, 5])  <br>targets shape: torch.Size([2, 3])</pre><p>Read the first row: [2.0, 1.0, 0.1, 0.5, 0.2]. The target is 0. The biggest score, 2.0, sits at index 0. The model is &quot;right&quot;.</p><p>I rigged every row like that on purpose. The biggest score always sits on the correct answer. So whatever loss I compute next should come out <strong>small</strong>. If it doesn’t, my wiring is broken, and I’d rather find that out on 6 numbers than on 162 million.</p><p><strong>Real-world version:</strong> imagine a spelling test where the student writes a confidence score next to every possible answer. The teacher doesn’t care about the scores on the wrong answers. She only looks at one thing: <em>how confident were you in the right one?</em></p><h3>Step 2: Cross-entropy, or “how surprised is the model?”</h3><p>PyTorch’s CrossEntropyLoss doesn&#39;t want sequences. It wants a flat list of questions:</p><ul><li>predictions: (N, C) — N questions, C scores each</li><li>targets: (N,) — one correct answer per question</li></ul><p>So I flatten. The values don’t change, just the shape.</p><pre>flat_logits= inputs_logits.flatten(0,1)  <br>flat_targets= targets.flatten()  <br>loss_fn=nn.CrossEntropyLoss()  <br>loss= loss_fn(flat_logits, flat_targets)  <br>  <br>print(&quot;flattend logits shape:&quot;, flat_logits.shape)  <br>print(&quot;flattened targets shape:&quot;, flat_targets.shape)  <br>print(&quot;calculated training loss:&quot;, loss.item())</pre><pre>flattend logits shape: torch.Size([6, 5])  <br>flattened targets shape: torch.Size([6])  <br>calculated training loss: 0.45796191692352295</pre><p>(2, 3, 5) became (6, 5). Six next-token questions, five scores each. <strong>Every token position is its own tiny classification problem.</strong> That&#39;s the whole secret of language model training: it&#39;s not one big task, it&#39;s millions of small &quot;guess the next word&quot; quizzes graded all at once.</p><p>And the loss is <strong>0.458</strong>. Small. Exactly what a rigged test should score.</p><p>Here’s what cross-entropy actually is, with the scary name removed:</p><blockquote><strong><em>loss = −log(probability the model gave to the correct answer)</em></strong></blockquote><p>If the model gives the right token 100% probability, loss is 0. If it gives it 50%, loss is 0.69. If it gives it 1%, loss is 4.6. The more surprised the model is by the truth, the bigger the number.</p><p>Cross-entropy is just <strong>surprise</strong>, measured.</p><p>Remember that. The whole of training is one goal: <em>make the model less surprised by reality.</em></p><h3>Step 3: One epoch. The five lines, in full.</h3><p>An <strong>epoch</strong> is one complete pass over every batch of training data. Here is the function that does it.</p><pre>import torch  <br>def train_model_one_epoch(model, dataloader, optimizer, device):  <br>    model.train()  <br>    total_loss=0  <br>  <br>    for batch_idx, (inputs, targets) in enumerate(dataloader):  <br>        inputs,targets = inputs.to(device), targets.to(device)  <br>  <br>        optimizer.zero_grad()  <br>  <br>        logits=model(inputs)  <br>  <br>        flat_logits = logits.flatten(0,1)  <br>        flat_targets= targets.flatten()  <br>  <br>        loss= torch.nn.functional.cross_entropy(flat_logits, flat_targets)  <br>  <br>        loss.backward()  <br>  <br>        optimizer.step()  <br>        total_loss+=loss.item()  <br>    return total_loss</pre><p>Walk through the inside of that loop, because this is the moment “training” stops being magic:</p><ol><li><strong>optimizer.zero_grad()</strong> — Wipe the slate. PyTorch <em>accumulates</em> gradients by default, so if you forget this, every step is polluted by the last one.</li><li><strong>logits = model(inputs)</strong> — The forward pass. The model makes its guesses.</li><li><strong>cross_entropy(...)</strong> — Grade the guesses. One number: surprise.</li><li><strong>loss.backward()</strong> — The backward pass. PyTorch walks back through every layer and computes, for every single one of the 162 million weights, <em>&quot;if I nudged you slightly, would the loss go up or down?&quot;</em> That&#39;s a gradient. This is autograd, and it&#39;s the single most important idea in deep learning.</li><li><strong>optimizer.step()</strong> — Nudge every weight a tiny step in the direction that would have made the loss smaller.</li></ol><p>Repeat. That’s learning.</p><p><strong>Real-world version:</strong> you throw a dart, miss left. Your brain computes “I need to aim more right” (backward), you adjust your arm slightly (step), throw again. You don’t recompute physics. You just nudge. A billion nudges later, you’re good at darts.</p><p><strong>Now look at line 16 again.</strong> loss.backward with no parentheses. That&#39;s a bug, and it&#39;s a brutal one. Python happily evaluates loss.backward as a reference to a function and throws it away. No error. No warning. The optimizer then steps using gradients that were never computed. Training &quot;runs&quot;, the loss never moves, and you lose an afternoon wondering why. I left it in the notebook as-is because it&#39;s the most honest lesson in this chapter: <strong>a model that trains without errors is not the same as a model that learns.</strong> (The fixed version lives in Step 8.)</p><h3>Step 4: Bring back the GPT from Day 3</h3><p>The training helpers don’t care what the model is. They only need something that turns token IDs into logits. The thing I actually want to train is GPTMODEL from Day 3.</p><pre>%run ../03_transformers/transformers.ipynb</pre><p>%run executes the whole Day 3 notebook top to bottom, as if I&#39;d pasted every cell here. All the classes, LayerNorm, GELU, TransformerBlock, GPTMODEL, are now live in this notebook. One notebook per chapter, and they chain.</p><h3>Step 5: A GPT so small it fits in your head</h3><p>The Day 3 model has 162 million parameters. Way too heavy for a “does the wiring work?” test. So I built a <strong>toy GPT</strong>, same class, tiny config:</p><pre>import torch   <br>from torch.optim import AdamW  <br>  <br>GPT_CONFIG_DUMMY= {&quot;vocab_size&quot;: 5, &quot;context_length&quot;: 4, &quot;emb_dim&quot;: 16, &quot;n_heads&quot;: 2, &quot;n_layers&quot;: 2, &quot;drop_rate&quot;: 0.0, &quot;qkv_bias&quot;: False}  <br>torch.manual_seed(123)  <br>dummy_gpt= GPTMODEL(GPT_CONFIG_DUMMY)  <br>optimizer= AdamW(dummy_gpt.parameters(), lr=5e-4)  <br>  <br>mock_inputs= torch.randint(0,5, (2,4))  <br>mock_targets = torch.randint(0,5, (2,4))  <br>  <br>optimizer.zero_grad()  <br>mock_logits= dummy_gpt(mock_inputs)  <br>loss = torch.nn.functional.cross_entropy(mock_logits.flatten(0,1), mock_targets.flatten())  <br>loss.backward()  <br>optimizer.step()  <br>  <br>print(&quot;mock execution loss score:&quot;, loss.item())  <br>print(&quot;weights gradients status : successfully computed&quot;)</pre><pre>mock execution loss score: 1.724005103111267  <br>weights gradients status : successfully computed</pre><p>5-word vocabulary. 4-token context. 16-dimensional embeddings. 2 heads, 2 layers. Random inputs, random targets. One step of the real loop, with the real GPTMODEL, and the real AdamW optimizer that trains actual GPT-2.</p><p>And the loss is <strong>1.72</strong>.</p><p>This is the number from the headline, and here’s why it stopped me cold.</p><p>If a model knows <em>nothing</em>, and guesses uniformly across 5 tokens, it gives the correct one a probability of 1/5. Cross-entropy of that is −log(0.2) = <strong>1.61</strong>.</p><p>My untrained model scored 1.72. Slightly <em>worse</em> than random, because its random weights happen to lean the wrong way. The loss isn’t an abstract score. <strong>It tells you exactly how clueless the model is, in a unit you can compute by hand.</strong></p><p>GPT-2 over 50,257 tokens starts near −log(1/50257) ≈ <strong>10.8</strong>. A fully trained GPT-2 gets to roughly <strong>3</strong>. That entire journey, from 10.8 to 3, is “training”. Just the number of surprise, falling.</p><h3>Step 6: The only number you’re allowed to trust</h3><p>Here’s the trap every beginner falls into.</p><p>Training loss goes down. You celebrate. But training loss dropping only proves one thing: the model is getting better <em>at the examples it’s allowed to practise on</em>. It might be memorising them, not learning language.</p><p>Real-world version: a student who memorised last year’s exam paper. Perfect score on that paper. Useless on the real one.</p><p>The only honest check is a <strong>validation split</strong>: sequences the model never trains on. If validation loss falls alongside training loss, the model is learning. If training loss falls and validation loss climbs, it’s memorising, and that’s <strong>overfitting</strong>.</p><pre>import torch  <br>def calc_loss_loader(dataloader, model, device, num_batches=None):  <br>    total_loss=0.0  <br>    if len(dataloader)==0:  <br>        return float(&quot;nan&quot;)  <br>  <br>    model.eval()  <br>  <br>    for i,(inputs, targets), in enumerate(dataloader):  <br>        if num_batches and i&gt;=num_batches:  <br>            break  <br>        inputs, targets= inputs.to(device), targets.to(device)  <br>  <br>        with torch.no_grad():  <br>            logits=model(inputs)  <br>  <br>        flat_logits = logits.flatten(0,1)  <br>        flat_targets= targets.flatten()  <br>  <br>        loss= torch.nn.functional.cross_entropy(flat_logits, flat_targets)  <br>        total_loss+=loss.item()  <br>  <br>    return total_loss/min(len(dataloader), num_batches if num_batches else len(dataloader))</pre><p>Two lines make this <em>scoring</em> instead of <em>training</em>:</p><ul><li><strong>model.eval()</strong> — switches off dropout. We want the model&#39;s honest best guess, not a noisy one.</li><li><strong>with torch.no_grad()</strong> — tells PyTorch &quot;don&#39;t build the gradient graph&quot;. No gradients means no accidental learning from the validation set, and it runs faster and uses less memory.</li></ul><p>Forget either of these and you’ve quietly leaked the exam paper to the student.</p><h3>Step 7: Test the examiner before the exam</h3><p>Before I trust calc_loss_loader inside a long run, I test it on a fake dataset.</p><pre>import torch  <br>from torch.utils.data import TensorDataset, DataLoader  <br>  <br>mock_inputs= torch.randint(0,5,(4,4))  <br>mock_targets = torch.randint(0,5,(4,4))  <br>mock_ds = TensorDataset(mock_inputs, mock_targets)  <br>mock_loader= DataLoader(mock_ds, batch_size=2)  <br>  <br>baseline_loss= calc_loss_loader(mock_loader, dummy_gpt, device=&quot;cpu&quot;, num_batches=2)  <br>print(&quot;baseline loss scrore vector:&quot;, baseline_loss)</pre><pre>baseline loss scrore vector: 1.8556649684906006</pre><p>4 random sequences, served in batches of 2, averaged over 2 batches. Loss <strong>1.86</strong> — around the “knows nothing” zone of 1.61, exactly where a one-step model on random data should be. The examiner works.</p><h3>Step 8: The master training loop</h3><p>One epoch is a building block. Real pretraining stacks many of them, checks validation loss on a schedule, and keeps a history you can plot.</p><pre>def train_model(model, train_loader, val_loader, optimizer, device, num_epochs, eval_freq, eval_iter, start_context, tokenizer):  <br>  <br>    train_losses,val_losses,track_tokens_seen= [], [], []  <br>    tokens_seen=0  <br>  <br>    for epoch in range(num_epochs):  <br>        model.train()  <br>  <br>        for input_batch, target_batch in train_loader:  <br>            optimizer.zero_grad()  <br>            logits= model(input_batch)  <br>  <br>            loss= torch.nn.functional.cross_entropy(logits.flatten(0,1), target_batch.flatten())  <br>            loss.backward()  <br>            optimizer.step()  <br>  <br>            tokens_seen+=input_batch.numel()  <br>  <br>        if epoch%eval_freq==0:  <br>            train_loss= calc_loss_loader(train_loader, model, device, eval_iter)  <br>            val_loss= calc_loss_loader(val_loader, model, device, eval_iter)  <br>  <br>            train_losses.append(train_loss)  <br>            val_losses.append(val_loss)  <br>            track_tokens_seen.append(tokens_seen)  <br>  <br>            print(f&quot;Epoch {epoch+1:02d} | Train Loss: {train_loss:.4f} | Val Loss:{val_loss:.4f}&quot;)  <br>    return train_losses, val_losses, track_tokens_seen</pre><p>Notice three things:</p><ul><li>The inner loop is the <strong>same five lines</strong> from Step 3. Now with the parentheses.</li><li><strong>tokens_seen</strong> — this is the number labs actually report. &quot;Trained on 15 trillion tokens&quot; is this counter, running on a cluster for months.</li><li><strong>model.train()</strong> at the top of every epoch, because calc_loss_loader flipped it to eval() at the end of the last one. Miss this, and dropout stays off during training. Another silent bug.</li></ul><p>When this runs on real text, you watch two columns of numbers fall, epoch by epoch. That’s it. That’s the moment the garbage turns into language. It’s not dramatic. It’s a loss curve, and it’s the most beautiful graph in computer science.</p><h3>Step 9: Don’t lose the work</h3><p>Pretraining takes minutes, hours, or weeks. You don’t want a notebook restart to throw that away. A <strong>checkpoint</strong> is a snapshot of the learned numbers.</p><p>PyTorch splits a model in two: the <strong>architecture</strong> (the Python class) and the <strong>state</strong> (the numbers). state_dict() is the state, as a plain dictionary of tensors.</p><pre>import torch  <br>  <br>torch.save(dummy_gpt.state_dict(), &quot;model_checkpoint.pt&quot;)  <br>print(&quot;model weights status: successfully saved to model_checkpoint.pt&quot;)  <br>  <br>fresh_model= GPTMODEL(GPT_CONFIG_DUMMY)  <br>fresh_model.load_state_dict(torch.load(&quot;model_checkpoint.pt&quot;))  <br>print(&quot;model weight status: successfully loaded info Fresh Architecture&quot;)</pre><pre>model weights status: successfully saved to model_checkpoint.pt  <br>model weight status: successfully loaded info Fresh Architecture</pre><p>Save the dictionary. Build a fresh, random model with the same config. Load the dictionary in. The fresh model is now identical to the trained one.</p><p>And here’s the thought that made me put my laptop down for a minute:</p><p><strong>This is what “the weights” are.</strong> When someone says “Llama’s weights are open”, they mean a state_dict like this one, just 405 billion entries bigger. When OpenAI released GPT-2 in 2019, they released a dictionary of 148 tensors. A model is its architecture plus a dictionary. Nothing else.</p><p>Which is exactly why what happens next is possible.</p><h3>What I learned building this</h3><p><strong>1. “Training” is five lines, repeated.</strong> zero_grad → forward → loss → backward → step. The loop is not the hard part. The data and the compute are.</p><p><strong>2. Loss is surprise, and you can compute it by hand.</strong> Random guessing over 5 tokens = 1.61. My model = 1.72. GPT-2 starts at 10.8 and ends near 3. Once you can do that arithmetic, “the model is learning” stops being a vibe and becomes a number.</p><p><strong>3. Validation loss is the only honest number.</strong> Training loss can lie. model.eval() + torch.no_grad() are the two lines that keep it honest.</p><p><strong>4. Silent bugs are the real enemy.</strong> loss.backward without () runs perfectly and learns nothing. Build the toy version first. Always. If 6 numbers can&#39;t tell you it&#39;s broken, 162 million won&#39;t either.</p><h3>What’s next: stealing a brain (legally)</h3><p>My model can now be trained, scored honestly, saved, and reloaded. But a 2-layer toy on random tokens will never speak English, and training a 124M-parameter GPT from scratch needs more data and compute than I have.</p><p>There’s a shortcut, and it’s an honest one, because I built the architecture myself: <strong>download OpenAI’s actual GPT-2 weights, all 148 tensors, and inject them into the classes I wrote by hand.</strong></p><p>If my architecture is even slightly wrong, the shapes won’t line up and PyTorch will refuse.</p><p>That’s Day 5. I’ll tell you now: it did not go smoothly, and the 38 million “missing” parameters from Day 3 finally get explained.</p><p>If you’ve been curious about how ChatGPT really works under the hood, <strong>follow along</strong>. I’m building a complete LLM from scratch, one concept per day, in public. Every line of code is in my GitHub repo, <strong>LLM-From-Scratch</strong>, with one runnable notebook per chapter.</p><p>If this article helped something click, <strong>give it a clap</strong> (you can hold the button for up to 50), and drop a comment telling me which part surprised you most. I read every one.</p><p>See you on Day 5, when my code holds OpenAI’s brain.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c1174687c15b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Day 3: I Built GPT From Scratch. 162 Million Parameters. Zero Intelligence.]]></title>
            <link>https://medium.com/@Yash_Bhardwaj/day-3-i-built-gpt-from-scratch-162-million-parameters-zero-intelligence-696a4dd300b1?source=rss-bec114f6ca70------2</link>
            <guid isPermaLink="false">https://medium.com/p/696a4dd300b1</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Yash Bhardwaj]]></dc:creator>
            <pubDate>Thu, 08 Oct 2026 15:19:13 GMT</pubDate>
            <atom:updated>2026-10-08T15:19:13.381Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Cover image for Day 3 of the LLM From Scratch series. On a dark navy background, bold text reads “GPT-2 From Scratch” with the subtitle “162M parameters. Zero training. Pure nonsense.” and the topics LayerNorm, GELU, Residuals, and Transformer Blocks. On the right, a glowing tower of stacked transformer blocks, colored cyan to purple to orange, has curved residual connections running along its side and a “12 Layers” badge. The input token “I HAD” feeds into the base of the tower, and scrambled," src="https://cdn-images-1.medium.com/max/1024/1*XS5xXC9y639Ow4o8V1sI3w.jpeg" /><figcaption>A 162-million-parameter GPT-2, built layer by layer in plain PyTorch, before it has learned a single word.</figcaption></figure><h4>Day 3 of building an LLM from scratch in PyTorch: LayerNorm, GELU, residual connections, the Transformer block, and a complete GPT model that writes its first (terrible) words.</h4><p>Here is the exact moment my model came alive.</p><p>I gave it three tokens, the opening of a short story: <strong>“I HAD”</strong>.</p><p>It thought for a second and handed me back this:</p><pre>tensor([[   40,   367,  2885, 31748, 14767, 40404, 24427, 44724]])</pre><p>Three tokens in. Five brand new tokens out. Decode them and you get complete garbage: five random fragments from a 50,257-word vocabulary with zero connection to the story.</p><p>And honestly? It was one of the most satisfying outputs I have ever seen in my life.</p><p>Because that garbage came out of a <strong>162 million parameter GPT model that I wrote line by line</strong>. No Hugging Face. No nn.Transformer. No from transformers import GPT2LMHeadModel. Every layer normalization, every activation, every attention head, every shortcut connection, written by hand in plain PyTorch.</p><p>The model speaks nonsense because it has never read a single sentence. Its brain is random numbers. But the brain is <strong>built</strong>. The architecture is exactly the shape of GPT-2. All it needs now is an education.</p><p>This is Day 3 of my <em>“build an LLM from scratch”</em> series, and today is the day the pieces finally become a model. If you have ever wanted to understand what actually happens inside ChatGPT, Llama, Claude, or any modern large language model, this is the chapter where it all clicks.</p><p>By the end of this article you will know exactly why a GPT is only ~100 lines of code, why half of a small LLM’s parameters are “wasted” on a dictionary, and why one single + sign is the reason deep learning works at all.</p><p>Let me show you.</p><h3>Where we left off</h3><p>Quick recap, in case you are just joining:</p><ul><li><strong>Day 1:</strong> We turned raw text into tokens and tokens into embedding vectors.</li><li><strong>Day 2:</strong> We built <strong>self-attention</strong> and <strong>multi-head attention</strong> from scratch, the mechanism that lets every word look at every other word in a sentence.</li></ul><p>Attention is the famous part. It is the “T” in GPT. It is the reason the 2017 paper was called <em>Attention Is All You Need</em>.</p><p>But here is the thing nobody tells you when you start: <strong>attention alone is not a language model.</strong></p><p>If you just stack attention layers on top of each other, the model trains badly, the numbers inside it explode or vanish, and deep versions barely learn at all. Attention needs a support crew. Today we build that crew:</p><ol><li><strong>Layer normalization:</strong> keeps the numbers stable</li><li><strong>GELU:</strong> gives the model the ability to learn curves, not just straight lines</li><li><strong>Feed-forward network:</strong> gives each token time to “think”</li><li><strong>Residual connections:</strong> the shortcut that makes deep networks trainable at all</li><li><strong>The Transformer block:</strong> all of the above, glued around attention</li><li><strong>The GPT model:</strong> a stack of 12 transformer blocks with embeddings in front and a prediction head on top</li><li><strong>Text generation:</strong> the loop that makes it write</li></ol><p>Seven steps. Let’s go.</p><h3>Step 1: Layer normalization, or how to stop your numbers from going crazy</h3><p>Imagine passing a message through 12 people in a row, and each one either whispers it or shouts it a little louder than they heard it. By person 12, the message is either inaudible or deafening.</p><p>That is exactly what happens to the numbers inside a deep neural network. Each layer multiplies and adds, and over many layers the values drift. Some blow up. Some shrink to nothing. Training turns into chaos.</p><p><strong>Layer normalization</strong> is the fix. For every single token, it takes that token’s feature vector and rescales it so the values have a <strong>mean of 0</strong> and a <strong>variance of 1</strong>. Same volume, every layer, every time.</p><pre>import torch  <br>import torch.nn as nn  <br>  <br>class LayerNormFromScratch(nn.Module):  <br>    def __init__(self, emb_dim, eps=1e-5):  <br>        super().__init__()  <br>        self.eps = eps  <br>        self.scale = nn.Parameter(torch.ones(emb_dim))  <br>        self.shift = nn.Parameter(torch.zeros(emb_dim))  <br>  <br>    def forward(self, x):  <br>        mean = x.mean(dim=-1, keepdim=True)  <br>        var = x.var(dim=-1, keepdim=True, unbiased=False)  <br>        norm_x = (x - mean) / torch.sqrt(var + self.eps)  <br>        return self.scale * norm_x + self.shift</pre><p>Four details make this work:</p><ul><li><strong>dim=-1</strong> means we normalize across the features of each token, so every token is handled on its own.</li><li><strong>eps</strong> is a tiny number that stops us from ever dividing by zero.</li><li><strong>unbiased=False</strong> matches how GPT-2 was originally implemented.</li><li><strong>scale and </strong><strong>shift</strong> are learnable. Forcing everything to mean 0 and variance 1 can be too strict, so the model is allowed to learn its own stretch and offset if that helps.</li></ul><p>Does it work? I fed in a random tensor of shape (2, 3, 5):</p><pre>normalised output mean (should be close to 0):  <br> tensor([[[ 3.5763e-08],  <br>          [ 5.9605e-09],  <br>          [ 1.1921e-08]], ...  <br>  <br>normalised output var (should be close to 1):  <br> tensor([[[1.0000],  <br>          [1.0000],  <br>          [1.0000]], ...</pre><p>That 3.5763e-08 is 0.000000035. It is zero, plus a little floating-point rounding noise. Every token is now perfectly calibrated.</p><p>One down. Six to go.</p><h3>Step 2: GELU, the smooth switch inside every GPT</h3><p>Here is a fact that surprises a lot of beginners: if you stack a hundred linear layers on top of each other, you still get a model that can only learn <strong>straight lines</strong>. Multiply-and-add, repeated forever, is still just multiply-and-add.</p><p>To learn the messy, curved patterns of human language, a network needs <strong>non-linearity</strong>: an activation function between its layers.</p><p>The classic choice is <strong>ReLU</strong>: negative numbers become 0, positive numbers pass through. Simple, fast, but brutal. It has a hard corner at zero and throws away every negative signal completely.</p><p>GPT uses <strong>GELU</strong> (Gaussian Error Linear Unit) instead. Think of it as <strong>ReLU with the corners sanded off</strong>:</p><ul><li>big positive values pass through almost unchanged</li><li>big negative values get squashed close to zero</li><li>values near zero follow a smooth curve, and small negative values are allowed to leak through a little</li></ul><pre>class GELUActivation(nn.Module):  <br>    def __init__(self):  <br>        super().__init__()  <br>  <br>    def forward(self, x):  <br>        return 0.5 * x * (1.0 + torch.tanh(  <br>            torch.sqrt(torch.tensor(2.0 / torch.pi)) * (x + 0.044715 * torch.pow(x, 3))  <br>        ))</pre><p>That scary-looking formula is the fast tanh approximation that the original GPT-2 used. You do not need to memorise it. What matters is the intuition: <strong>GELU is a smooth on/off switch for every single number</strong>, and smooth switches make optimisation much easier for huge models.</p><h3>Step 3: The feed-forward network, where each token thinks for itself</h3><p>If attention is a group discussion where every token talks to every other token, then the <strong>feed-forward network</strong> is the moment afterwards when each token goes home and thinks about what it heard.</p><p>It is a tiny neural network with three moves:</p><ol><li><strong>Expand:</strong> blow each token’s vector up to <strong>4x</strong> its size</li><li><strong>Activate:</strong> apply GELU</li><li><strong>Compress:</strong> shrink it back to the original size</li></ol><pre>class FeedForward(nn.Module):  <br>    def __init__(self, emb_dim):  <br>        super().__init__()  <br>        self.layers = nn.Sequential(  <br>            nn.Linear(emb_dim, 4 * emb_dim),  <br>            GELUActivation(),  <br>            nn.Linear(4 * emb_dim, emb_dim),  <br>        )  <br>  <br>    def forward(self, x):  <br>        return self.layers(x)</pre><p>Why expand and then compress? The bigger middle layer is a <strong>workspace</strong>. It gives the model room to pull apart and recombine features before packing the result neatly back into the original size.</p><p>And here is a fun fact that will reframe how you see LLMs forever: in a GPT model, <strong>the feed-forward layers hold roughly two thirds of the transformer blocks’ parameters</strong>. Attention gets all the headlines, but a huge share of a model’s knowledge lives right here, in the boring part.</p><p>One subtle but important detail: the feed-forward network processes <strong>every token independently</strong>. Token 1 never sees token 2 inside this block. The division of labour is clean:</p><blockquote><strong><em>Attention mixes information between tokens. Feed-forward refines each token on its own.</em></strong></blockquote><p>Shape check with a batch of 8 sequences, 4 tokens each, 256 features per token:</p><pre>input batch shape:  torch.Size([8, 4, 256])  <br>ffn output shape:   torch.Size([8, 4, 256])</pre><p>Inside, every token briefly grew to 1,024 features. From the outside, nothing changed shape. Hold on to that idea. It becomes the superpower of the whole architecture.</p><h3>Step 4: Residual connections, the one line of code that made deep learning work</h3><p>This is my favourite idea in the entire chapter, because it is so simple it almost feels like cheating.</p><p>When you train a neural network, a learning signal (the <strong>gradient</strong>) flows <strong>backwards</strong> from the output to the very first layer, telling every weight how to improve. In a deep network, that signal gets multiplied by small numbers at every layer on the way back. By the time it reaches the early layers, it has shrunk to almost nothing.</p><p>This is the infamous <strong>vanishing gradient problem</strong>. The early layers stop learning. Going deeper makes the model <em>worse</em>, not better. For years, this was the wall the entire field was stuck behind.</p><p>The fix, made famous by ResNet in 2015, is one line:</p><pre>output = x + layer(x)</pre><p>That’s it. Add the layer’s input back to its output.</p><p>Now the layer doesn’t have to rebuild the whole representation from scratch. It only has to learn <strong>what to change</strong>. And during training, the gradient gets a clean, direct highway straight back through the + x, skipping the layer entirely if it needs to.</p><p>Think of it like editing a document with “track changes” on. You never rewrite the whole page. You just suggest edits, and the original text is always preserved underneath.</p><p>Every modern LLM depends on this single trick, including Meta’s Llama 3.1 405B, which stacks 126 transformer layers on top of each other. One plus sign. That’s the whole revolution.</p><h3>Step 5: The Transformer block, assembling the Avengers</h3><p>Now we combine everything into the single most important unit in modern AI: the <strong>Transformer block</strong>.</p><pre>class TransformerBlock(nn.Module):  <br>    def __init__(self, emb_dim, context_length, dropout, num_heads):  <br>        super().__init__()  <br>        self.att = MultiHeadAttentionWrapper(  <br>            d_in=emb_dim, d_out=emb_dim,  <br>            context_length=context_length,  <br>            dropout=dropout, num_heads=num_heads,  <br>        )  <br>        self.ff = FeedForward(emb_dim=emb_dim)  <br>        self.ln1 = LayerNormFromScratch(emb_dim=emb_dim)  <br>        self.ln2 = LayerNormFromScratch(emb_dim=emb_dim)  <br>        self.dropout = nn.Dropout(dropout)  <br>  <br>    def forward(self, x):  <br>        x = x + self.dropout(self.att(self.ln1(x)))  <br>        x = x + self.dropout(self.ff(self.ln2(x)))  <br>        return x</pre><p>Read the forward method slowly. Those two lines are the heartbeat of GPT, and they follow the same rhythm:</p><p><strong>Normalize → transform → dropout → add back.</strong></p><ul><li><strong>Line 1:</strong> normalize the tokens, let them talk to each other through multi-head attention (the one we built on Day 2), then add the result back to the original.</li><li><strong>Line 2:</strong> normalize again, let each token think through the feed-forward network, then add that back too.</li></ul><p>A few design choices worth knowing, because they come up in every interview and every paper:</p><ul><li><strong>Pre-LayerNorm:</strong> we normalize <em>before</em> attention and feed-forward, not after. The original 2017 Transformer did it the other way around. GPT-2 switched to “pre-norm” because it trains far more stably, and most modern LLMs followed.</li><li><strong>Two separate LayerNorms:</strong> ln1 and ln2 each learn their own scale and shift.</li><li><strong>Dropout:</strong> randomly switches off a fraction of values during training, so the model can’t lean too hard on any single path. It is automatically turned off when the model is used for real.</li></ul><p>And now the moment of truth:</p><pre>input batch shape:              torch.Size([8, 4, 256])  <br>transformer block out shape:    torch.Size([8, 4, 256])</pre><p><strong>Same shape in. Same shape out.</strong></p><p>This is the quiet genius of the Transformer. Because a block’s output looks exactly like its input, you can stack them like LEGO bricks. Stack 12 and you get GPT-2 small. Stack 96 (and make each one much wider) and you are in GPT-3 territory. The block itself does not change. Only the height of the tower does.</p><h3>Step 6: Building the full GPT model</h3><p>We have the brick. Time to build the tower.</p><p>Here is the full journey a sentence takes through GPT:</p><pre>token IDs  <br>   -&gt; token embeddings + position embeddings  <br>   -&gt; dropout  <br>   -&gt; 12 x Transformer blocks  <br>   -&gt; final LayerNorm  <br>   -&gt; output head  <br>   -&gt; logits (a score for every word in the vocabulary)</pre><p>And here is the entire model in about 20 lines</p><pre>class GPTMODEL(nn.Module):  <br>    def __init__(self, cfg):  <br>        super().__init__()  <br>        self.tok_emb = nn.Embedding(cfg[&quot;vocab_size&quot;], cfg[&quot;emb_dim&quot;])  <br>        self.pos_emb = nn.Embedding(cfg[&quot;context_length&quot;], cfg[&quot;emb_dim&quot;])  <br>        self.drop_emb = nn.Dropout(cfg[&quot;drop_rate&quot;])  <br>  <br>        self.trf_blocks = nn.Sequential(*[  <br>            TransformerBlock(  <br>                emb_dim=cfg[&quot;emb_dim&quot;],  <br>                context_length=cfg[&quot;context_length&quot;],  <br>                dropout=cfg[&quot;drop_rate&quot;],  <br>                num_heads=cfg[&quot;n_heads&quot;],  <br>            )  <br>            for _ in range(cfg[&quot;n_layers&quot;])  <br>        ])  <br>        self.final_ln = LayerNormFromScratch(emb_dim=cfg[&quot;emb_dim&quot;])  <br>        self.out_head = nn.Linear(cfg[&quot;emb_dim&quot;], cfg[&quot;vocab_size&quot;], bias=False)  <br>  <br>    def forward(self, in_idx):  <br>        b, seq_len = in_idx.shape  <br>        tok_embeds = self.tok_emb(in_idx)  <br>        pos_embeds = self.pos_emb(torch.arange(seq_len, device=in_idx.device))  <br>        x = self.drop_emb(tok_embeds + pos_embeds)  <br>        x = self.trf_blocks(x)  <br>        x = self.final_ln(x)  <br>        logits = self.out_head(x)  <br>        return logits</pre><p>Yes. That’s it. That is a GPT.</p><p>Let me point out the one piece people forget: <strong>position embeddings</strong>.</p><p>Attention, on its own, has no idea what order words come in. To pure attention, <em>“dog bites man”</em> and <em>“man bites dog”</em> are the exact same bag of words. That is a disaster for language.</p><p>So GPT keeps a second lookup table with one learnable vector for every position, 0, 1, 2, and so on, and simply <strong>adds</strong> it to each token’s embedding. Now every token carries two pieces of information: <strong>what</strong> it is, and <strong>where</strong> it is.</p><p>At the other end, the <strong>output head</strong> turns each token’s 768 numbers into <strong>50,257 scores</strong>, one for every token in the GPT-2 vocabulary. These raw scores are called <strong>logits</strong>. The higher the logit, the more the model believes that token should come next.</p><h3>Configuring GPT</h3><p>All the knobs live in one dictionary:</p><pre>GPT_CONFIG_124M = {  <br>    &quot;vocab_size&quot;: 50257,     # GPT-2 BPE vocabulary  <br>    &quot;context_length&quot;: 256,   # max tokens the model sees at once  <br>    &quot;emb_dim&quot;: 768,          # features per token  <br>    &quot;n_heads&quot;: 12,           # attention heads per block  <br>    &quot;n_layers&quot;: 12,          # transformer blocks  <br>    &quot;drop_rate&quot;: 0.1,        # 10% dropout  <br>    &quot;qkv_bias&quot;: False,  <br>}</pre><p>These are the dimensions of <strong>GPT-2 small</strong>, the 124M model OpenAI released in 2019. The one change: I shortened the context window from 1,024 to 256 tokens to keep things light on my laptop.</p><p>Feed in three token IDs:</p><pre>input token batch shape:     torch.Size([1, 3])  <br>final output logits shape:   torch.Size([1, 3, 50257])</pre><p>Read that output shape out loud, because it explains how every LLM on Earth works:</p><blockquote><strong><em>For every one of the 3 positions, the model produced 50,257 scores: its guess for which token comes next.</em></strong></blockquote><p>That is the entire job of a GPT. It is a next-token prediction machine. Everything else, the poetry, the code, the chat, the “reasoning”, emerges from doing that one thing extremely well.</p><h3>Step 7: The plot twist, and the first words</h3><h3>Wait, why 162 million?</h3><p>I counted the trainable parameters:</p><pre>total trainable parameter: 162,419,712</pre><p>The config says <strong>124M</strong>. My model says <strong>162M</strong>. Did I break something?</p><p>Nope. This is one of my favourite little secrets of GPT-2, and most tutorials skip it entirely.</p><p>The output head is a matrix of size 768 x 50,257, which is about <strong>38.6 million parameters</strong> on its own. The original GPT-2 used a trick called <strong>weight tying</strong>: the output head <em>reuses</em> the token embedding matrix instead of having its own copy. Same weights, used twice: once to turn words into vectors, once to turn vectors back into words.</p><p>My implementation keeps them separate. Subtract that 38.6 million and you land at about <strong>124 million</strong>, right where GPT-2 small should be.</p><p>It also tells you something big about small LLMs. In my untied version, the token embedding table and the output head together hold about <strong>77 million parameters, nearly half the entire model</strong>, just to translate between words and vectors. That is exactly why weight tying was worth doing.</p><h3>Making it talk</h3><p>A GPT never writes a whole sentence at once. It writes <strong>one token</strong>, appends it, reads everything again, and writes the next one. Over and over. Here is that loop:</p><pre>def generate_text_simple(model, idx, max_new_tokens, context_size):  <br>    for _ in range(max_new_tokens):  <br>        idx_cond = idx[:, -context_size:]  <br>  <br>        with torch.no_grad():  <br>            logits = model(idx_cond)  <br>  <br>        logits = logits[:, -1, :]  <br>        next_token = torch.argmax(logits, dim=-1, keepdim=True)  <br>        idx = torch.cat((idx, next_token), dim=-1)  <br>    return idx</pre><p>Five steps, repeated:</p><ol><li><strong>Crop</strong> the input to the most recent tokens the model can handle.</li><li><strong>Run</strong> the model, with no_grad() because we are not training.</li><li><strong>Look only at the last position</strong>, because that is the prediction for what comes next.</li><li><strong>Pick the highest score</strong> with argmax. This is called <strong>greedy decoding</strong>.</li><li><strong>Append</strong> the new token and go again.</li></ol><p>When ChatGPT streams its answer to you word by word, <em>this</em> is what you are watching. A loop like this one, running in real time. Nothing more mysterious than that.</p><p>I started it with the first words of <em>The Verdict</em>, the short story from Day 1:</p><pre>start_context = torch.tensor([[40, 367, 2885]])   # &quot;I HAD&quot;</pre><p>And got:</p><pre>tensor([[   40,   367,  2885, 31748, 14767, 40404, 24427, 44724]])</pre><p>Five new tokens. Decode them and it’s word salad.</p><p>Which is <strong>exactly</strong> right. Every weight in this model is still random. It has the brain of GPT-2 and the knowledge of a newborn. But the pipeline works end to end: token IDs go in, the model reasons over them through 12 transformer blocks, and new tokens come out.</p><p>The architecture is done. What’s missing is the education.</p><h3>What I learned building this</h3><p>Stepping back, three things genuinely surprised me:</p><p><strong>1. GPT is shockingly small, as code.</strong> The full model is maybe 100 lines of PyTorch. The magic of large language models isn’t in clever code. It’s in scale, data, and training.</p><p><strong>2. Attention gets the fame, but the support crew does the heavy lifting.</strong> Without LayerNorm, training is unstable. Without residual connections, deep models don’t learn. Without the feed-forward network, you lose a huge share of the model’s capacity.</p><p><strong>3. “Same shape in, same shape out” is the real secret.</strong> It’s the reason transformers scale from GPT-2’s 12 layers to models with over a hundred. Once you have one good block, building a bigger model really is just stacking more of them.</p><h3>What’s next: teaching it to speak</h3><p>Right now my model is a beautifully built engine with no fuel.</p><p>In the next chapter, I <strong>pretrain</strong> it. That means defining a loss function, writing a training loop, feeding it real text, and watching the output slowly morph from random tokens into actual English. It’s the part where the garbage starts turning into language, and I can’t wait to show you.</p><p>If you’ve been curious about how ChatGPT really works under the hood, <strong>follow along</strong>. I’m building a complete LLM from scratch, one concept per day, in public. Every line of code is in my GitHub repo, <strong>LLM-From-Scratch</strong>, with one runnable notebook per chapter.</p><p>If this article helped something click, <strong>give it a clap</strong> (you can hold the button for up to 50), and drop a comment telling me which part surprised you most. I read every one.</p><p>See you on Day 4, when my model says its first real word.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=696a4dd300b1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build a GPT From Scratch, Day 2: What Actually Happens Inside Attention]]></title>
            <link>https://medium.com/@Yash_Bhardwaj/build-a-gpt-from-scratch-day-2-what-actually-happens-inside-attention-dec48a738d2e?source=rss-bec114f6ca70------2</link>
            <guid isPermaLink="false">https://medium.com/p/dec48a738d2e</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Yash Bhardwaj]]></dc:creator>
            <pubDate>Wed, 07 Oct 2026 15:16:07 GMT</pubDate>
            <atom:updated>2026-10-07T15:16:07.596Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Diagram of self-attention: token cards for a short sentence flow into Query, Key, and Value, then into a lower-triangular causal attention weight matrix showing how each token can only attend to itself and earlier tokens." src="https://cdn-images-1.medium.com/max/1024/1*tWAxdeFSsOhUUhjxK7RuiA.jpeg" /><figcaption>Self-attention in one picture: each token asks what to look for (Query), what others offer (Key), and what information to mix in (Value) — with a causal mask so the model never peeks at the future.</figcaption></figure><p><em>Building an LLM from scratch — Part 2</em></p><p>If you have ever wondered what really happens inside GPT when it reads your prompt, the answer almost always comes back to one idea: attention. Not the vague “the model pays attention” kind you read in blog headlines, but a concrete, surprisingly simple piece of math that you can write in a few lines of PyTorch.</p><p>This is day two of my “build an LLM from scratch” series. In the last post I turned raw text into tokens and then into embedding vectors. Those vectors were a good start, but they had one big problem: every token sat there on its own, completely unaware of the words around it. In this post I fix that by building the self-attention mechanism by hand, step by step, until we reach the same multi-head attention that powers modern large language models.</p><p>No frameworks hiding the logic. No nn.TransformerEncoder doing the work for us. Just tensors, dot products, and softmax. By the end you will understand query, key and value, scaled dot-product attention, causal masking, and multi-head attention — and you will have running code for each.</p><h3>Why Embeddings Alone Are Not Enough</h3><p>Start with a sentence: “money in the bank” and “the bank of the river”. The word “bank” is identical in both, but it means two completely different things. A plain embedding gives “bank” the same vector every time, because an embedding lookup does not know anything about context.</p><p>Human readers resolve this instantly. We look at “money” or “river” sitting nearby and pick the right meaning. That is exactly the ability we want to give our model: for every token, let it look at the other tokens in the sentence and decide which ones matter.</p><p>That is all attention is. For each token, it asks a single question: out of everything in this sentence, which words are relevant to me, and how much? Then it mixes those words together to build a richer, context-aware representation.</p><h3>Step 1: Attention Scores with a Dot Product</h3><p>Let us keep it tiny so the numbers stay readable. Three tokens, each one a three-dimensional vector.</p><pre>import torch  <br>  <br>inputs = torch.tensor([   # inputs for three words with 3 dimensions  <br>    [0.44, 0.15, 0.89],  <br>    [0.55, 0.87, 0.66],  <br>    [0.57, 0.85, 0.08]  <br>])  <br>  <br>query = inputs[0]  <br>attn_scores_2 = torch.empty(inputs.shape[0])  <br>for i, x_i in enumerate(inputs):  <br>    attn_scores_2[i] = torch.dot(query, x_i)  <br>  <br>print(&quot;raw attention scores:&quot;, attn_scores_2)  <br># raw attention scores: tensor([1.0082, 0.9599, 0.4495])</pre><p>We pick the first token as our query and measure how similar it is to every token in the sentence, including itself. The tool for that is the dot product. The dot product is one of those things that sounds fancy but is just “multiply matching entries and add them up”. The more two vectors point in the same direction, the larger the result.</p><p>So these three numbers are a relevance score: how much the first token cares about each of the three tokens. Higher means more related.</p><h3>Step 2: From Scores to Weights to a Context Vector</h3><p>Raw scores are hard to work with. One score might be 1.0 and another 0.44, but what fraction of attention does that actually represent? To answer that we pass the scores through softmax, which squashes them into positive numbers that add up to 1. Now we can read them as percentages.</p><pre>attn_weights_2 = torch.softmax(attn_scores_2, dim=0)  <br>  <br>context_vec_2 = torch.zeros(inputs.shape[1])  <br>for i, x_i in enumerate(inputs):  <br>    context_vec_2 += attn_weights_2[i] * x_i  <br>  <br>print(&quot;attention weights:&quot;, attn_weights_2)  <br># tensor([0.3961, 0.3774, 0.2265])  <br>print(&quot;context vector:&quot;, context_vec_2)  <br># tensor([0.5070, 0.5803, 0.6197])</pre><p>The weights say the first token puts about 40 percent of its attention on itself, 38 percent on the second token, and 23 percent on the third. The context vector is just a weighted blend of all the input vectors using those percentages. That single vector now carries information from the whole sentence, not just from one isolated word. This is the payoff: a context-aware representation.</p><h3>Step 3: Kill the Loops with Matrix Multiplication</h3><p>The loop version is fine for understanding, but it only handles one query at a time, and loops are slow once sequences get long. The good news is that the entire operation is really just matrix multiplication wearing a disguise.</p><pre>attn_scores = torch.matmul(inputs, inputs.T)  <br>attn_weights = torch.softmax(attn_scores, dim=1)  <br>all_context_vec = torch.matmul(attn_weights, inputs)</pre><p>Multiplying the input matrix by its own transpose gives every pairwise score in a single step: the entry in row i, column j is the dot product of token i with token j. Softmax along each row turns those into weights, and one more matrix multiply produces the context vector for every token at once. No loops, no single-query limitation. This vectorised form is what we build on from here.</p><h3>Step 4: Add Trainable Weights with Query, Key and Value</h3><p>So far the model has nothing to learn. Tokens are compared to each other directly, which bakes in a fixed notion of similarity. A real model needs to learn what “relevant” means for the task at hand.</p><p>We do that by projecting each input into three separate spaces using three weight matrices. These three roles have names that confuse a lot of beginners, so here is the plain-English version:</p><ul><li><strong>Query:</strong> what this token is looking for</li><li><strong>Key:</strong> what this token offers to the others</li><li><strong>Value:</strong> the actual information this token hands over if it gets attended to</li></ul><pre>d_in = inputs.shape[1]  <br>d_out = 2  <br>  <br>W_query = torch.nn.Parameter(torch.rand(d_in, d_out), requires_grad=False)  <br>W_key   = torch.nn.Parameter(torch.rand(d_in, d_out), requires_grad=False)  <br>W_value = torch.nn.Parameter(torch.rand(d_in, d_out), requires_grad=False)  <br>  <br>queries = inputs @ W_query  <br>keys    = inputs @ W_key  <br>values  = inputs @ W_value</pre><p>Now attention scores come from comparing queries against keys rather than raw inputs, and the final blend happens over values. Because these three matrices are learnable, the model gets to shape its own definition of relevance during training. The analogy I like: a query is a search box, keys are the labels on every document, and values are the documents themselves.</p><h3>Step 5: Scaled Dot-Product Attention</h3><p>With queries, keys and values in hand, we can compute real attention. There is one extra detail that matters: scaling.</p><pre>attn_scores = queries @ keys.T  <br>d_k = keys.shape[-1]  <br>scaled_attn_scores = attn_scores / (d_k ** 0.5)  <br>attn_weights = torch.softmax(scaled_attn_scores, dim=1)  <br>context_vectors = attn_weights @ values</pre><p>As the key dimension grows, the dot products grow with it. Feed very large numbers into softmax and it turns “peaky”, dumping almost all the weight onto a single token and driving gradients toward zero, which stalls learning. Dividing the scores by the square root of the key dimension keeps the numbers in a sane range. This is the exact “scaled dot-product attention” from the original Transformer paper, <em>Attention Is All You Need</em>.</p><h3>Step 6: Causal Masking, or Why GPT Refuses to Look Ahead</h3><p>Everything so far lets each token look at every other token, including the ones that come after it. That is fine for some tasks, but it quietly breaks a GPT-style model.</p><p>GPT is trained to predict the next word. If a token were allowed to peek at future tokens, it would just read off the answer instead of learning to predict it. That is cheating, and the model would fall apart the moment you asked it to actually generate text.</p><p>Causal masking stops this. Before the softmax, we set every future position to negative infinity. After softmax those entries become exactly zero, so a token can only attend to itself and the tokens before it.</p><pre>context_length = attn_weights.shape[0]  <br>mask = torch.triu(torch.ones(context_length, context_length), diagonal=1)  <br>masked = scaled_attn_scores.masked_fill(mask.bool(), -float(&#39;inf&#39;))  <br>causal_attn_weights = torch.softmax(masked, dim=-1)  <br># tensor([[1.0000, 0.0000, 0.0000],  <br>#         [0.4954, 0.5046, 0.0000],  <br>#         [0.3636, 0.3698, 0.2666]])</pre><p>Look at the result. The first row attends only to itself. The second row sees the first two tokens. The third row sees all three. That lower-triangular staircase is the visual signature of causal attention, and it is the reason GPT can only ever write left to right.</p><h3>Step 7: Wrapping It in a Reusable CausalAttention Class</h3><p>We have every piece now, so let us package them into a clean, reusable module we can drop into a bigger model later.</p><pre>import torch  <br>import torch.nn as nn  <br>  <br>class CausalAttention(nn.Module):  <br>    def __init__(self, d_in, d_out, context_length, dropout, qkv_bias=False):  <br>        super().__init__()  <br>        self.d_out = d_out  <br>        self.W_query = nn.Linear(d_in, d_out, bias=qkv_bias)  <br>        self.W_key   = nn.Linear(d_in, d_out, bias=qkv_bias)  <br>        self.W_value = nn.Linear(d_in, d_out, bias=qkv_bias)  <br>        self.dropout = nn.Dropout(dropout)  <br>        self.register_buffer(  <br>            &#39;mask&#39;,  <br>            torch.triu(torch.ones(context_length, context_length), diagonal=1)  <br>        )  <br>  <br>    def forward(self, x):  <br>        b, num_tokens, d_in = x.shape  <br>        queries = self.W_query(x)  <br>        keys    = self.W_key(x)  <br>        values  = self.W_value(x)  <br>  <br>        attn_scores = torch.matmul(queries, keys.transpose(1, 2))  <br>        attn_scores.masked_fill_(self.mask[:num_tokens, :num_tokens].bool(), -float(&#39;inf&#39;))  <br>        attn_weights = torch.softmax(attn_scores / (keys.shape[-1] ** 0.5), dim=-1)  <br>        attn_weights = self.dropout(attn_weights)  <br>  <br>        return torch.matmul(attn_weights, values)</pre><p>A few choices worth calling out. We swap the manual matrices for nn.Linear, which is tidier and trainable by default. The mask goes into register_buffer, so it travels with the model to the GPU and gets saved with it, but is never treated as a learnable parameter. The forward method works on batched three-dimensional input of shape (batch, tokens, features), which is what real training data looks like. And a dropout layer on the attention weights adds a bit of regularisation.</p><p>A quick shape check on a dummy batch of 8 sequences, 4 tokens each, 256 features per token:</p><pre>batch_inputs = torch.randn(8, 4, 256)  <br>causal_layer = CausalAttention(d_in=256, d_out=256, context_length=4, dropout=0.0)  <br>output_vectors = causal_layer(batch_inputs)  <br>print(output_vectors.shape)   # torch.Size([8, 4, 256])</pre><p>Same shape in, same shape out. The layer gives every token a new context-aware vector without disturbing the batch or sequence dimensions, which means we can stack it like any other layer.</p><h3>Step 8: Multi-Head Attention</h3><p>A single attention layer learns one way of relating tokens. But language has many kinds of relationships happening at the same time: grammar, subject and verb agreement, long-range references back to something mentioned paragraphs ago. Asking one head to capture all of that is asking too much.</p><p>Multi-head attention runs several attention layers in parallel, each with its own query, key and value projections, so each head can specialise in a different pattern. Their outputs are concatenated and mixed back together through a final projection.</p><pre>class MultiHeadAttentionWrapper(nn.Module):  <br>    def __init__(self, d_in, d_out, context_length, dropout, num_heads, qkv_bias=False):  <br>        super().__init__()  <br>        self.heads = nn.ModuleList([  <br>            CausalAttention(d_in, d_out // num_heads, context_length, dropout, qkv_bias)  <br>            for _ in range(num_heads)  <br>        ])  <br>        self.out_projection = nn.Linear(d_out, d_out)  <br>  <br>    def forward(self, x):  <br>        head_outputs = [head(x) for head in self.heads]  <br>        context_vec = torch.cat(head_outputs, dim=-1)  <br>        return self.out_projection(context_vec)</pre><p>Each head works inside a smaller slice of the total dimension. With 256 features split across 4 heads, every head handles 64 dimensions, and the pieces add back up to 256 after concatenation. The output projection then lets the heads share what they learned.</p><pre>batch_inputs = torch.randn(8, 4, 256)  <br>mha_layer = MultiHeadAttentionWrapper(d_in=256, d_out=256, context_length=4, dropout=0.0, num_heads=4)  <br>mha_outputs = mha_layer(batch_inputs)  <br>print(mha_outputs.shape)   # torch.Size([8, 4, 256])</pre><p>Same shape in, same shape out again, which is the whole point. Multi-head attention behaves like a drop-in layer, and this is the exact block that gets stacked over and over inside a transformer.</p><h3>What We Actually Built</h3><p>Take a step back and look at how little it took. We went from isolated embeddings to full multi-head attention using nothing more than dot products, softmax, a triangular mask, and some matrix multiplication:</p><ul><li>Raw attention scores from a dot product</li><li>Softmax to turn scores into weights, then a weighted sum for the context vector</li><li>The vectorised matrix form that handles every token at once</li><li>Trainable query, key and value projections so the model learns relevance</li><li>Scaling by the square root of the key dimension for stable training</li><li>Causal masking so the model cannot cheat by reading the future</li><li>Multi-head attention to capture many relationships in parallel</li></ul><p>Every modern large language model, from GPT to Llama, is built on this same core. The production versions add tricks for speed and scale, but the idea you just coded is the real thing.</p><h3>What Comes Next</h3><p>Attention is the hard part, and it is behind us. Next in the series I assemble the full transformer block: layer normalisation, a feed-forward network, and residual connections wrapped around the multi-head attention we just wrote. After that we stack those blocks into a complete GPT model and start training it.</p><p>If you want to follow the build, the full code lives in my “LLM from scratch” repository, with one runnable notebook per concept. I publish a new piece as I learn each part, so the series grows in public, one commit at a time.</p><p>If this helped the ideas click, follow along for the next chapter. Building a language model by hand is the fastest way I know to stop treating these models like magic and start understanding exactly how they think.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dec48a738d2e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[I’m Building an LLM From Scratch — Day 1: PyTorch + Tokenization (No Black Boxes)]]></title>
            <link>https://medium.com/@Yash_Bhardwaj/im-building-an-llm-from-scratch-day-1-pytorch-tokenization-no-black-boxes-ad05b55fd795?source=rss-bec114f6ca70------2</link>
            <guid isPermaLink="false">https://medium.com/p/ad05b55fd795</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Yash Bhardwaj]]></dc:creator>
            <pubDate>Tue, 06 Oct 2026 16:31:31 GMT</pubDate>
            <atom:updated>2026-10-06T16:31:31.837Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mc-n4fNN7l_exVns7Ce5Xw.png" /><figcaption>From text to token IDs to learnable vectors: the foundations of building an LLM from scratch in PyTorch.</figcaption></figure><p>Everyone talks about GPT. Almost nobody can tell you what actually happens between your text and the model. So I’m building one myself, one day at a time — and I’m showing you every single line.</p><p>Let me be honest with you.</p><p>For months I could <em>use</em> ChatGPT, call the OpenAI API, fine-tune a thing or two… and still not actually explain what happens the moment you hit enter. Where does my sentence go? How does plain English become something a neural network can learn from? What is a “token,” really — not the Twitter-thread version, the actual-code version?</p><p>So I stopped collecting courses and started building.</p><p>This is <strong>Day 1 of me building a GPT-style Large Language Model completely from scratch</strong> — raw PyTorch, real math, zero magic. One concept a day. Code pushed to GitHub the same day. An article (like this one) explaining it in plain English. In public, where I can’t quietly give up.</p><p>If you’ve ever felt like LLMs are a black box you’re just supposed to trust — this series is me prying that box open, screw by screw. Grab a coffee. Day 1 is where it all starts: <strong>PyTorch fundamentals</strong> and <strong>turning text into numbers</strong>.</p><blockquote><em>🔗 Full code for today lives here: </em><a href="https://github.com/Bunny-dev24/LLM-From-Scratch"><strong><em>github.com/Bunny-dev24/LLM-From-Scratch</em></strong></a><em> — chapters </em><em>00_pytorch_fundamentals and </em><em>01_tokenization. Star it if you want to follow along; it genuinely keeps me shipping.</em></blockquote><h3>Why “from scratch” and why in public</h3><p>Here’s the uncomfortable truth about AI right now: a huge number of people can <em>prompt</em> a model, but very few can tell you what’s happening under the hood. That gap is exactly where the opportunity is.</p><p>I’m not reading a model’s docs. I’m building the thing — tokenizer, attention, transformer block, the full GPT, pretraining, fine-tuning — step by step. Day by day. And I’m writing it all down the way I wish someone had explained it to me: slowly, with real code, no hand-waving.</p><p>Two chapters are done and that’s today’s story. One (attention) is already in progress — more on that at the end. 👀</p><p>Let’s go.</p><h3>Part 1 — PyTorch, the engine under every LLM</h3><p>Before you build a language model, you need to be fluent in the tool that runs it. Not “I watched a tutorial” fluent. “I know what a gradient is and where it comes from” fluent.</p><p>So I started with the smallest possible neural network and worked up.</p><h3>A single forward pass</h3><p>A neural network, at its heart, is just: multiply inputs by weights, add a bias, squash the result, measure how wrong you are. That last part is the <em>loss</em>. Here’s that entire idea in a few lines of PyTorch:</p><pre>import torch  <br>import torch.nn.functional as F  <br>  <br>y  = torch.tensor([1.0, 0.0])   # the true answers  <br>x1 = torch.tensor([1.0, 1.0])   # inputs  <br>w1 = torch.tensor([2.0, 2.0])   # weights  <br>b  = torch.tensor([0.0, 0.0])   # bias  <br>  <br>z = x1 * w1 + b                 # weighted sum  <br>a = torch.sigmoid(z)            # squash into (0, 1)  <br>loss = F.binary_cross_entropy(a, y)  <br>print(loss)                     # tensor(1.1269)</pre><p>That 1.1269 is the model saying &quot;here&#39;s how wrong I currently am.&quot; Training is nothing more than making that number smaller. But to make it smaller, you need to know <em>which way to nudge each weight</em>. That&#39;s where the real magic lives.</p><h3>Autograd — the one idea that makes training possible</h3><p>If I had to point at the single feature that makes deep learning <em>work</em>, it’s this: PyTorch can compute the gradient of your loss with respect to every weight, automatically. You just mark the tensors you care about with requires_grad=True, and PyTorch quietly builds a graph of every operation you do.</p><pre>from torch.autograd import grad  <br>  <br>y  = torch.tensor([1.0, 0.0])  <br>x1 = torch.tensor([1.0, 1.0])  <br>w1 = torch.tensor([2.0, 2.0], requires_grad=True)   # track me  <br>b  = torch.tensor([0.0, 0.0], requires_grad=True)   # track me too  <br>  <br>z = x1 * w1 + b  <br>a = torch.sigmoid(z)  <br>loss = F.binary_cross_entropy(a, y)  <br>  <br>grad_L_w1 = grad(loss, w1, retain_graph=True)  <br>grad_L_b  = grad(loss, b,  retain_graph=True)  <br>  <br>print(grad_L_w1)   # (tensor([-0.0596,  0.4404]),)  <br>print(grad_L_b)    # (tensor([-0.0596,  0.4404]),)</pre><p>Read those gradients like a compass. Each number tells you how the loss would change if you tweaked that weight a little. Flip the sign, take a small step, repeat a few million times — that’s training. No calculus homework required, because autograd did the chain rule for you.</p><p>The first time this clicked for me, backpropagation stopped being a scary word and became… a bookkeeping trick. A brilliant one.</p><h3>Scaling up to a real network</h3><p>One neuron is cute. Real models stack layers. In PyTorch you wrap everything in an nn.Module and let nn.Sequential hold your layers in order:</p><pre>import torch  <br>  <br>class NeuralNetwork(torch.nn.Module):  <br>    def __init__(self, num_inputs, num_outputs):  <br>        super().__init__()  <br>        self.layers = torch.nn.Sequential(  <br>            torch.nn.Linear(num_inputs, 30),  <br>            torch.nn.ReLU(),  <br>            torch.nn.Linear(30, 20),  <br>            torch.nn.ReLU(),  <br>            torch.nn.Linear(20, num_outputs),  <br>        )  <br>  <br>    def forward(self, x):  <br>        return self.layers(x)  <br>  <br>model = NeuralNetwork(50, 3)  <br>print(model)</pre><p>Two hidden layers, ReLU in between. Nothing exotic — and yet this exact pattern (__init__ to define the pieces, forward to connect them) is how <em>every</em> model in this series will be built, right up to the full GPT.</p><h3>How big is this thing, really?</h3><p>I wanted a feel for scale, so I counted the trainable parameters:</p><pre>num_params = sum(p.numel() for p in model.parameters() if p.requires_grad)  <br>print(&quot;total number of trainable model parameters:&quot;, num_params)  <br># total number of trainable model parameters 2213</pre><p><strong>2,213 parameters.</strong> Hold that number in your head. GPT-2 has ~124 million. GPT-3 has 175 billion. Same lego bricks — just a <em>lot</em> more of them. Once you’ve truly understood these 2,213, the billions stop being intimidating and start being… just more of the same.</p><h3>Reproducibility, so your results aren’t random</h3><p>Weights start as random numbers. Great for learning, terrible for debugging — run the cell twice, get two different models. One line fixes that:</p><pre>torch.manual_seed(123)  <br>model = NeuralNetwork(50, 3)  <br>  <br>x   = torch.rand(1, 50)  <br>out = model(x)  <br>print(out)   # tensor([[-0.1262,  0.1080, -0.1792]], ...)</pre><p>Set the seed, and the “randomness” becomes repeatable. Future-me will thank present-me every time a bug needs chasing.</p><p><strong>That’s the PyTorch foundation.</strong> Tensors, autograd, modules, parameters, seeds. Small ideas — but every single one of them shows up again, unchanged, when we build the real model.</p><h3>Part 2 — Tokenization: how text becomes something a model can learn</h3><p>Here’s the thing nobody tells you clearly: <strong>a language model never sees your words.</strong> It sees numbers. Only numbers. So the very first real job in building an LLM is translation — turning messy human text into clean integer IDs, and back again.</p><p>I used a short public-domain story, <em>The Verdict</em>, as my raw material.</p><pre>with open(&quot;../data/the-verdict.txt&quot;, &quot;r&quot;, encoding=&quot;utf-8&quot;) as f:  <br>    raw_text = f.read()  <br>  <br>print(&quot;total character count:&quot;, len(raw_text))  <br>print(raw_text[:99])  <br># total character count: 20479  <br># I HAD always thought Jack Gisburn rather a cheap genius--though a good fellow enough--so it was no</pre><p>20,479 characters of English. Now let’s make it machine-readable.</p><h3>Step 1: break text into tokens</h3><p>A <em>token</em> is just a small chunk of text — a word, a punctuation mark, sometimes a piece of a word. I split on spaces <em>and</em> punctuation, keeping the punctuation (it carries meaning), and dropped the empty bits:</p><pre>import re  <br>  <br>preprocessed = re.split(r&#39;([,.?_!&quot;()\&#39;]|--|\s)&#39;, raw_text)  <br>preprocessed = [item.strip() for item in preprocessed if item.strip()]  <br>  <br>print(&quot;total tokens:&quot;, len(preprocessed))  <br>print(preprocessed[:20])  <br># total tokens: 4649  <br># [&#39;I&#39;, &#39;HAD&#39;, &#39;always&#39;, &#39;thought&#39;, &#39;Jack&#39;, &#39;Gisburn&#39;, &#39;rather&#39;, &#39;a&#39;, &#39;cheap&#39;,  <br>#  &#39;genius&#39;, &#39;--&#39;, &#39;though&#39;, &#39;a&#39;, &#39;good&#39;, &#39;fellow&#39;, &#39;enough&#39;, &#39;--&#39;, &#39;so&#39;, &#39;it&#39;, &#39;was&#39;]</pre><p>4,649 tokens from one short story. Notice -- survived as its own token. Details like that matter more than you&#39;d expect.</p><h3>Step 2: build a vocabulary</h3><p>A vocabulary is simply every <em>unique</em> token, each handed its own integer ID. I also tossed in two special tokens — one for unknown words, one to mark where a text ends:</p><pre>all_words = sorted(list(set(preprocessed)))  <br>all_words.extend([&quot;&lt;​|endoftext|&gt;&quot;, &quot;&lt;​|unk|&gt;&quot;])  <br>  <br>vocab = {word: integer for integer, word in enumerate(all_words)}  <br>print(&quot;unique vocab size:&quot;, len(all_words))  <br># unique vocab size: 1161</pre><p>1,161 unique tokens. That’s the entire “language” this first tokenizer knows.</p><h3>Step 3: a tokenizer is just a two-way dictionary</h3><p>Strip away the jargon and a tokenizer has exactly two jobs: <strong>encode</strong> (text → IDs) and <strong>decode</strong> (IDs → text). That’s it. Here’s the whole thing:</p><pre>class SimpleTokenizerV1:  <br>    def __init__(self, vocab):  <br>        self.vocab = vocab  <br>        self.inverse_vocab = {v: k for k, v in vocab.items()}  <br>  <br>    def encode(self, text):  <br>        preprocessed = re.split(r&#39;([,.?!&quot;()\&#39;]|--|\s)&#39;, text)  <br>        preprocessed = [item.strip() for item in preprocessed if item.strip()]  <br>        return [self.vocab.get(s, self.vocab[&quot;&lt;​|unk|&gt;&quot;]) for s in preprocessed]  <br>  <br>    def decode(self, ids):  <br>        text = &quot; &quot;.join([self.inverse_vocab[i] for i in ids])  <br>        return re.sub(r&#39;\s+([,.?!&quot;()\&#39;])&#39;, r&#39;\1&#39;, text)</pre><p>Clean, readable, and it works. But it has one fatal flaw…</p><h3>Step 4: the problem with “unknown” words</h3><p>Feed my simple tokenizer a word it has never seen, and it shrugs:</p><pre>tokenizer = SimpleTokenizerV1(vocab)  <br>print(tokenizer.decode([1160, 1160, 155, 1160]))  <br># &lt;|unk|&gt; &lt;|unk|&gt; am &lt;|unk|&gt;</pre><p>Three &lt;|unk|&gt; tokens. All meaning, gone. A real language model can&#39;t afford to throw away words it hasn&#39;t memorized — the world has far too many words.</p><h3>Step 5: Byte Pair Encoding — the trick GPT actually uses</h3><p>The fix is beautiful: instead of memorizing whole words, break unknown words into smaller <em>subword</em> pieces you already know. This is <strong>Byte Pair Encoding (BPE)</strong>, and it’s exactly what the GPT family uses. Rather than rebuild it from scratch today, I used OpenAI’s blazing-fast tiktoken with the GPT-2 encoding:</p><pre>import tiktoken  <br>  <br>tokenizer = tiktoken.get_encoding(&quot;gpt2&quot;)  <br>text = &quot;hello world, yash is-learning-fast&quot;  <br>  <br>integers = tokenizer.encode(text)  <br>print(&quot;encoded IDs:&quot;, integers)  <br>print(&quot;decoded text:&quot;, tokenizer.decode(integers))  <br># encoded IDs: [31373, 995, 11, 331, 1077, 318, 12, 40684, 12, 7217]  <br># decoded text: hello world, yash is-learning-fast</pre><p>Notice the made-up chunk is-learning-fast round-tripped perfectly — <strong>no </strong><strong>&lt;|unk|&gt; anywhere.</strong> BPE quietly split it into pieces it recognizes. This is why ChatGPT can handle your weird usernames, typos and brand-new slang without blinking.</p><h3>Step 6: the entire learning game, in four tokens</h3><p>Now the part that made everything click. How does a language model actually <em>learn</em>? One stupidly simple game, played billions of times: <strong>given some words, predict the next one.</strong></p><p>To set it up, you take your tokens and make two copies shifted by one position. The input is “the words so far,” the target is “the word that comes next.”</p><pre>enc_text = tokenizer.encode(raw_text)  <br>x = enc_text[:4]     # [40, 367, 2885, 1464]  <br>y = enc_text[1:5]    # [367, 2885, 1464, 1807]  <br>  <br>for i in range(1, len(x) + 1):  <br>    context = x[:i]  <br>    desired = y[i - 1]  <br>    print(tokenizer.decode(context), &quot;-----&gt;&quot;, tokenizer.decode([desired]))  <br>  <br># I               -----&gt;  H  <br># I H             -----&gt;  AD  <br># I HAD           -----&gt;  always  <br># I HAD always    -----&gt;  thought</pre><p>Read that output slowly. “I” should lead to “H”. “I HAD” should lead to “always.” That’s the whole game. Scale this up — more data, more parameters, more compute — and this exact pattern becomes GPT. I’m not exaggerating. <strong>Next-token prediction is the engine of the entire LLM revolution</strong>, and it fits in eight lines.</p><h3>Step 7: feeding the model in batches</h3><p>Training on one example at a time is painfully slow. In practice you feed many sequences at once, in <em>batches</em>. PyTorch gives us a Dataset (holds the input/target pairs) and a DataLoader (groups them into batches). I slid a window across the token stream to generate the pairs:</p><pre>import torch  <br>from torch.utils.data import Dataset, DataLoader  <br>  <br>class GPTDatasetV1(Dataset):  <br>    def __init__(self, txt, tokenizer, max_length, stride):  <br>        self.input_ids, self.target_ids = [], []  <br>        token_ids = tokenizer.encode(txt)  <br>        for i in range(0, len(token_ids) - max_length, stride):  <br>            self.input_ids.append(torch.tensor(token_ids[i:i + max_length]))  <br>            self.target_ids.append(torch.tensor(token_ids[i + 1:i + max_length + 1]))  <br>  <br>    def __len__(self):  <br>        return len(self.input_ids)  <br>  <br>    def __getitem__(self, idx):  <br>        return self.input_ids[idx], self.target_ids[idx]</pre><p>Pull one batch out and check the shapes — always eyeball your data before trusting it:</p><pre># batch_size=8, max_length=4, stride=4  <br># Input tensors shape : torch.Size([8, 4])  <br># targets tensor shape: torch.Size([8, 4])  <br>#  <br># first input  : tensor([  40,  367, 2885, 1464])  <br># first target : tensor([ 367, 2885, 1464, 1807])</pre><p>There it is again — input and target, shifted by exactly one. The same idea from Step 6, now neatly batched and ready for a GPU.</p><h3>Step 8: from IDs to meaning — embeddings</h3><p>One last step, and it’s a crucial one. A token ID like 40 is just a <em>name</em>. It isn&#39;t bigger or more important than 5 — they&#39;re labels, not quantities. So you can&#39;t feed raw IDs into a network and expect understanding.</p><p>Instead you use an <strong>embedding layer</strong>: a giant lookup table that maps every token ID to a vector of numbers the model can <em>tune as it learns</em>. This is where tokens start to carry actual meaning.</p><pre>import tiktoken  <br>  <br>vocab_size = tiktoken.get_encoding(&quot;gpt2&quot;).n_vocab   # 50257  <br>output_dim = 256  <br>  <br>embedding_layer = torch.nn.Embedding(vocab_size, output_dim)  <br>embedded = embedding_layer(inputs)  <br>  <br>print(&quot;token ID shape      :&quot;, inputs.shape)     # torch.Size([8, 4])  <br>print(&quot;embedded vector shape:&quot;, embedded.shape)  # torch.Size([8, 4, 256])</pre><p>Every token just became a 256-number vector. Our tidy batch of IDs is now a rich batch of vectors — and <em>that</em> is what a transformer actually chews on.</p><h3>Day 1, in one breath</h3><p>Look at how far we came from “computers only understand numbers”:</p><ul><li><strong>PyTorch:</strong> forward pass, autograd (the real reason training works), MLPs with nn.Module, counting parameters, fixing seeds.</li><li><strong>Tokenization:</strong> text → tokens → a vocabulary → a from-scratch tokenizer → BPE with tiktoken → next-token input/target pairs → batched DataLoader → embeddings.</li></ul><p>Two chapters. One sitting. And we already touched the single idea the entire field is built on: <strong>predict the next token.</strong></p><p>If you understood today, you understand more about how GPT works than most people who use it daily. Seriously.</p><h3>What’s next (and why you’ll want to be here)</h3><p>The pieces so far are the <em>setup</em>. The part that turned this field upside down — the reason a 2017 paper has 100,000+ citations — is <strong>attention</strong>: how a model decides which earlier words matter most when predicting the next one.</p><p>That chapter is <strong>already in progress</strong> in the repo. Day 2 is where this stops being “a neural network” and starts becoming “a <em>language</em> model.” You don’t want to miss the moment it clicks.</p><p><strong>Follow me here on Medium</strong> so Day 2 lands in your feed.<br> <strong>Star the repo:</strong> <a href="https://github.com/Bunny-dev24/LLM-From-Scratch">github.com/Bunny-dev24/LLM-From-Scratch</a> — the code is yours to clone, break, and learn from.</p><p><strong>Tell me in the comments:</strong> what part of LLMs has always felt like a black box to you? I’ll make sure the series answers it.</p><p>I’m building this in public specifically so I can’t quit. If a stranger on the internet learning out loud is useful to you — stick around. We’re just getting started.</p><p><em>See you on Day 2. We’re going into attention.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ad05b55fd795" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Fraud Detection Tutorial Is Lying to You (Here’s the 90% Nobody Teaches)]]></title>
            <link>https://medium.com/@Yash_Bhardwaj/your-fraud-detection-tutorial-is-lying-to-you-heres-the-90-nobody-teaches-6cf3f3f4a7a2?source=rss-bec114f6ca70------2</link>
            <guid isPermaLink="false">https://medium.com/p/6cf3f3f4a7a2</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[fraud-detection]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Yash Bhardwaj]]></dc:creator>
            <pubDate>Fri, 25 Sep 2026 19:10:15 GMT</pubDate>
            <atom:updated>2026-09-25T19:10:15.910Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Article cover with the headline ‘Your Fraud Detection Tutorial Is Lying to You’ and subtitle about real-time ML engineering, on a dark navy circuit-themed background." src="https://cdn-images-1.medium.com/max/1024/1*leiGVg7bc9WVIuJHpbPysw.png" /><figcaption>The 90% of fraud detection engineering that tutorials quietly skip.</figcaption></figure><p><em>Most fraud detection tutorials stop at “train a model and print the accuracy.” That’s the easy 10%. The other 90% — the part that decides whether your system survives contact with real traffic — never gets talked about. So I built FinGuard-AI to talk about exactly that.</em></p><p>Let’s be honest about something.</p><p>If you’ve ever googled “fraud detection machine learning,” you’ve seen the same article fifty times. Load a dataset. Train an XGBoost model. Print accuracy: 0.99. Clap for yourself. The end.</p><p>And every single one of those tutorials is quietly lying to you — because in fraud detection, the model is the <em>least</em> interesting part.</p><p>The interesting part is what happens <strong>after</strong> the model exists. When the same feature is computed two different ways in training and production and nobody notices for three weeks. When a fraudster figures out they can just <em>lie</em> about their transaction history to lower their own risk score. When the same Kafka message gets delivered twice and you accidentally flag a legit customer as a criminal. When your model’s accuracy silently rots because the world changed and your training data didn’t.</p><p>That’s the stuff that keeps fraud engineers up at night. So I built <strong>FinGuard-AI</strong> — a real-time fraud detection API — specifically around solving <em>those</em> problems, not just the “train a model” one.</p><p>Here’s the tour, and more importantly, the <em>why</em> behind every decision.</p><h3>The 30-Second Overview</h3><p>FinGuard is a layered, real-time fraud scoring system. The stack:</p><ul><li><strong>FastAPI</strong> for the scoring API (with Prometheus metrics baked in from day one)</li><li><strong>XGBoost + LightGBM</strong>, with the winner chosen automatically</li><li><strong>Kafka</strong> for event-driven, streaming transaction scoring</li><li><strong>Evidently</strong> for data drift monitoring</li><li><strong>Streamlit</strong> for a live dashboard</li><li><strong>SQLAlchemy + Alembic</strong> for persistence and migrations</li></ul><p>But the tech list is the boring part. Anyone can pip install a stack. What makes or breaks a fraud system is the <em>design decisions</em>. Let&#39;s get into the five that actually matter.</p><h3>Design Decision #1: Killing Training-Serving Skew Before It’s Born</h3><p>This is the bug that has humbled entire ML teams.</p><p>You train a model. Your features look great. You ship it. And in production, the model quietly underperforms — not because the model is bad, but because the features it sees in production are subtly <em>different</em> from the ones it trained on. A different default value here. A category encoded differently there. This is <strong>training-serving skew</strong>, and it’s the silent killer of ML systems. Even tier-one banks build entire <strong>feature stores</strong> just to guarantee “the model uses the exact same data logic in production as it did during training.”</p><p>FinGuard solves this the boring, bulletproof way: <strong>one feature function, used everywhere.</strong></p><pre># Deterministic feature engineering — shared by training AND serving. No skew.<br>CATEGORIES = {<br>    &quot;merchant_category&quot;: [&quot;grocery&quot;, &quot;electronics&quot;, &quot;travel&quot;, &quot;gambling&quot;, &quot;crypto&quot;, &quot;food&quot;],<br>    &quot;device_type&quot;: [&quot;ios&quot;, &quot;android&quot;, &quot;web&quot;],<br>    &quot;country&quot;: [&quot;IN&quot;, &quot;US&quot;, &quot;GB&quot;, &quot;NG&quot;, &quot;RU&quot;],<br>}</pre><p>The encoding is <strong>stateless</strong> — a fixed vocabulary, unknown values map to -1. There&#39;s no &quot;fit on training data, hope it matches in production&quot; step to get wrong. And here&#39;s the kicker: even the <em>synthetic data generator</em> imports this same CATEGORIES dictionary. So the fake data used for training and the real features used for scoring literally cannot drift apart. They share a single source of truth.</p><p>It’s not glamorous. It’s just correct. And in fraud, correct beats clever every time.</p><h3>Design Decision #2: The Anti-Spoofing Move Nobody Talks About</h3><figure><img alt="Diagram showing a fraudster’s fake transaction data being blocked by a secure server that computes real velocity features from the user’s actual history." src="https://cdn-images-1.medium.com/max/1024/1*MzxyLtxllvkoxc-ZMEcRiQ.jpeg" /><figcaption>The features a fraudster most wants to fake are computed server-side — so they can’t be faked at all.</figcaption></figure><p>This is my favorite part, and it’s a genuinely underrated idea.</p><p>Two of the strongest fraud signals are <strong>velocity features</strong>:</p><ul><li>user_avg_amount — what this user <em>normally</em> spends</li><li>txn_count_1h — how many transactions they&#39;ve fired in the last hour</li></ul><p>Card-testing attacks and account takeovers <em>light these up</em>. A user suddenly making eight transactions in ten minutes? That’s a screaming red flag.</p><p>But here’s the trap most people walk into: they let these values come from the incoming request payload. Which means a fraudster can just… send fake ones. “My average spend? Oh, definitely $5,000, this $4,000 charge is totally normal for me.” Boom — risk score defused.</p><p>FinGuard never trusts the client for these. It computes them <strong>server-side, from the user’s real transaction history</strong>:</p><pre>def _with_server_side_velocity(self, payload: dict) -&gt; dict:<br>    &quot;&quot;&quot;user_avg_amount / txn_count_1h are NEVER taken from the client —<br>    they&#39;re computed here from the user&#39;s real history so they can&#39;t be<br>    spoofed to lower a risk score.&quot;&quot;&quot;<br>    anchor = datetime.fromisoformat(payload[&quot;timestamp&quot;])<br>    avg_amount, txn_count = self.txn_repo.get_velocity_features(<br>        payload[&quot;user_id&quot;], anchor<br>    )<br>    return {**payload, &quot;user_avg_amount&quot;: avg_amount, &quot;txn_count_1h&quot;: txn_count}</pre><p>This is security and correctness holding hands. The features that matter most are the ones a fraudster would most want to fake — so you make them impossible to fake. The dashboard even spells this out to the user: <em>“Spend average and recent activity are pulled from the user’s real history, not this form.”</em></p><p>Small idea. Massive impact.</p><h3>Design Decision #3: Idempotency, Because Distributed Systems Are Messy</h3><p>Here’s a fact of life in event-driven systems: <strong>the same message will arrive twice.</strong> Kafka redelivers. Clients retry after a timeout. The network hiccups. It’s not an edge case — it’s a guarantee.</p><p>In a fraud system, a naive “score everything that comes in” approach means the same transaction could get scored twice, double-counted, or worse — flagged, un-flagged, and re-flagged in a race condition. That’s how a legit customer ends up frozen out of their account.</p><p>FinGuard treats duplicate transactions as <strong>idempotent</strong> — same ID, same result, no re-scoring:</p><pre>def score_and_persist(self, payload: dict) -&gt; dict:<br>    # Same transaction_id can arrive twice (Kafka redelivery, client retry).<br>    # Treat it as idempotent, not a rescore.<br>    existing = self.txn_repo.get_by_txn_id(payload[&quot;transaction_id&quot;])<br>    if existing is not None:<br>        return self._to_response(existing)<br>    ...</pre><p>And it goes one layer deeper — the database enforces it too. If two messages race each other to insert the same transaction, the IntegrityError is caught and the already-saved row is returned instead of blowing up. Defense in depth, applied to plumbing.</p><p>This is the kind of thing you only add <em>after</em> getting burned once. It’s baked in here from the start.</p><h3>Design Decision #4: Streaming That Doesn’t Die From One Bad Message</h3><figure><img alt="Isometric Kafka streaming pipeline routing a malformed transaction into a dead-letter queue, with a data-drift warning gauge on a monitoring dashboard." src="https://cdn-images-1.medium.com/max/1024/1*9XTosrse7hb-oGVlLXcllQ.png" /><figcaption>A crash-proof streaming pipeline: bad messages get quarantined in a dead-letter queue while a drift gauge watches the model age.</figcaption></figure><p>Real-time fraud detection lives or dies on <strong>event-driven streaming</strong> — and the whole industry has moved off batch processing onto Kafka/Flink pipelines for exactly this reason. FinGuard’s Kafka consumer scores each transaction as it flows in.</p><p>But streaming systems have a nasty failure mode: one malformed message throws an exception, the consumer loop crashes, and your <em>entire</em> fraud pipeline goes dark. During an active attack. Great.</p><p>FinGuard’s consumer refuses to die:</p><pre>def process_record(value: dict, producer: KafkaProducer) -&gt; None:<br>    &quot;&quot;&quot;One message = one DB session. Failures are caught and routed to a<br>    DLQ topic instead of bubbling up — a bad message shouldn&#39;t take down<br>    the whole consumer loop.&quot;&quot;&quot;<br>    db = SessionLocal()<br>    try:<br>        svc = ScoringService(...)<br>        result = svc.score_and_persist(value)<br>        if result[&quot;is_fraud&quot;]:<br>            producer.send(settings.kafka_alerts_topic, result)<br>    except Exception as e:<br>        producer.send(settings.kafka_dlq_topic, {&quot;error&quot;: str(e), &quot;record&quot;: value})<br>    finally:<br>        db.close()</pre><p>Two things I love here. First, the <strong>dead-letter queue (DLQ)</strong> — bad messages get quarantined for later inspection instead of killing the pipeline. Second, the consumer uses the <em>exact same</em> ScoringService the API uses. So the streaming path and the request path can never drift apart in behavior. One brain, two mouths.</p><h3>Design Decision #5: Measuring the Right Thing (Hint: It’s Not Accuracy)</h3><p>Remember that tutorial that printed accuracy: 0.99? Here&#39;s the dirty secret: in fraud, <strong>99% accuracy is trivial and meaningless.</strong> If 99% of transactions are legit, a model that says &quot;everything&#39;s fine, always&quot; scores 99% accuracy and catches exactly zero fraud.</p><p>Fraud is a brutally <strong>imbalanced classification</strong> problem. So FinGuard trains two models and picks the winner using the metric that actually reflects fraud-catching ability — <strong>PR-AUC (average precision)</strong>, not accuracy, not even ROC-AUC:</p><pre>def _train_one(name, model, Xtr, ytr, Xte, yte):<br>    model.fit(Xtr, ytr)<br>    p = model.predict_proba(Xte)[:, 1]<br>    ap, auc = average_precision_score(yte, p), roc_auc_score(yte, p)<br>    return model, ap  # selection is driven by PR-AUC</pre><p>It also handles the imbalance head-on — scale_pos_weight for XGBoost, class_weight=&quot;balanced&quot; for LightGBM — instead of pretending the classes are even. XGBoost and LightGBM both compete, and whichever gets the better PR-AUC gets shipped. It&#39;s a mini <strong>champion-selection</strong> pipeline, the same idea banks use when they promote a challenger model over the reigning champion.</p><h3>The Part Everyone Forgets: Your Model Is Already Rotting</h3><p>Here’s the uncomfortable truth about every deployed model: the moment it ships, it starts dying. Fraud tactics evolve. Spending patterns shift. New scams appear. This is <strong>concept drift</strong> and <strong>data drift</strong>, and it’s why MLOps teams obsess over monitoring — a model that was brilliant last quarter can quietly become useless this quarter, and you won’t know unless you’re watching.</p><p>FinGuard watches. It uses <strong>Evidently</strong> to compare recent live traffic against the original training reference set:</p><pre>report = Report(metrics=[DataDriftPreset(), TargetDriftPreset()])<br>report.run(reference_data=reference, current_data=current)</pre><p>If the live data has drifted meaningfully from what the model learned on, the dashboard throws up a warning and literally tells you what to do: <em>“Drift detected — consider retraining on fresher data.”</em> It’s the difference between finding out your model is stale from a dashboard versus finding out from an angry finance team three weeks and a lot of missed fraud later.</p><h3>Tying It Together: A Dashboard Humans Actually Use</h3><p>All of this plumbing feeds a clean <strong>Streamlit</strong> dashboard — live fraud overview, manual transaction scoring, a searchable transaction log, an open-alerts queue with a “mark reviewed” button, and a one-click drift check.</p><p>Because here’s the thing fraud engineers learn fast: a model’s score is useless if a human analyst can’t act on it. Real fraud operations are ML <em>plus</em> human review — the model flags the risk in milliseconds, and a person makes the final call on the gray-area cases. The dashboard is where those two worlds meet.</p><h3>What I Actually Want You to Take Away</h3><p>If you strip away the specific libraries, FinGuard is really a checklist of the things that separate a demo from a system:</p><ol><li><strong>One feature function everywhere</strong> → no training-serving skew</li><li><strong>Server-side velocity features</strong> → fraudsters can’t spoof their own risk down</li><li><strong>Idempotency at the app AND database layer</strong> → duplicates don’t cause chaos</li><li><strong>A crash-proof consumer with a DLQ</strong> → one bad message can’t blind you</li><li><strong>PR-AUC over accuracy</strong> → you measure the thing that actually matters</li><li><strong>Drift monitoring</strong> → you find out your model is stale from a chart, not a disaster</li></ol><p>None of these are flashy. You won’t win a Kaggle competition with them. But they’re the difference between a project that lives on GitHub and a system that lives in production.</p><p>The model was never the hard part. The engineering <em>around</em> the model always is.</p><p><em>If you found this useful, the full codebase is on </em><a href="https://github.com/Bunny-dev24/Finguard-ai"><em>GitHub</em></a><em> — layered architecture, streaming, drift checks, dashboard, the works. Clone it, break it, and tell me what you’d design differently. That’s the fun part.</em></p><p><em>Clap if this saved you from your first training-serving skew incident.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6cf3f3f4a7a2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multi-Head Attention From Scratch: Every Matrix Shape Explained]]></title>
            <link>https://medium.com/@Yash_Bhardwaj/multi-head-attention-from-scratch-every-matrix-shape-explained-79df48031c33?source=rss-bec114f6ca70------2</link>
            <guid isPermaLink="false">https://medium.com/p/79df48031c33</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[attention-mechanism]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[Yash Bhardwaj]]></dc:creator>
            <pubDate>Mon, 24 Aug 2026 13:27:57 GMT</pubDate>
            <atom:updated>2026-08-24T13:39:34.041Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>A pen-and-paper walkthrough of the transformer’s core mechanism — no frameworks, no hand-waving</blockquote><p>I could recite the attention formula for about six months before I understood it.</p><figure><img alt="Split diagram contrasting the memorised attention formula on the left with three unanswered questions about shapes, √d_k and -inf on the right." src="https://cdn-images-1.medium.com/max/1024/1*wX5OxHBf-eakQNZo-dKzMg.jpeg" /><figcaption>This was me for six months. Recall without reproduction isn’t understanding.</figcaption></figure><pre>Attention(Q, K, V) = softmax(QKᵀ / √d_k) · V</pre><p>Ask me what it did, and I’d say something confident about “tokens attending to each other.” Ask me the shape of Kᵀ when d_model = 8 and num_heads = 2, and I&#39;d go quiet.</p><p>That gap is the whole problem. The formula is five symbols. The understanding is in the shapes.</p><p>So I did the annoying thing: closed the tutorials, opened a notebook, and multiplied matrices by hand until the shapes stopped being a mystery. This article is that path, in order, with nothing skipped. The full NumPy implementation lives in a GitHub repo — linked at the end — but you won’t need it to follow along. That’s deliberate. The code was never the hard part.</p><p><strong>One request before you start.</strong> There are three places below marked <strong>Slow mode</strong>. Those are the parts you cannot skim — I’ve tried, it doesn’t work. Get a pen. Twenty minutes of arithmetic will teach you more than three hours of reading, and I say that as someone who spent the three hours first.</p><h3>Part 1: The one rule that answers every shape question</h3><figure><img alt="Diagram showing inner matrix dimensions cancelling and outer dimensions surviving in matrix multiplication" src="https://cdn-images-1.medium.com/max/1024/1*LfBrJ9aJ76NMM3FfZ_OHzw.jpeg" /><figcaption>One rule. Every shape question in a transformer answers to it.</figcaption></figure><p>Before attention, before Q and K and V, there is one rule. Get this and the rest of the transformer stops being intimidating.</p><p>To multiply two matrices, the <strong>inner dimensions must match</strong>. The <strong>outer dimensions become your answer</strong>.</p><pre>(4 × 8) · (8 × 4)  =  (4 × 4)<br>      ↑     ↑<br>      └─ these must match, and they vanish<br> ↑              ↑<br> └── these two survive</pre><p><strong>Inner numbers cancel. Outer numbers survive.</strong></p><h3>Why? Because a dot product needs equal-length lists</h3><p>Matrix multiplication isn’t one operation. It’s a grid of dot products.</p><p>To fill cell (i, j) of the output, you take <strong>row </strong><strong>i of the first matrix</strong> and <strong>column </strong><strong>j of the second</strong>, multiply them element by element, and add it all up.</p><p>Here’s the smallest possible example:</p><pre>[ 1  2 ] · [ 5  6 ]<br>[ 3  4 ]   [ 7  8 ]</pre><p>Cell (1,1) — row 1 meets column 1:</p><pre>[1, 2] · [5, 7] = (1)(5) + (2)(7) = 5 + 14 = 19</pre><p>Cell (1,2) — row 1 meets column 2:</p><pre>[1, 2] · [6, 8] = (1)(6) + (2)(8) = 6 + 16 = 22</pre><p>Cell (2,1) — row 2 meets column 1:</p><pre>[3, 4] · [5, 7] = 15 + 28 = 43</pre><p>Cell (2,2):</p><pre>[3, 4] · [6, 8] = 18 + 32 = 50</pre><pre>Result = [ 19  22 ]<br>         [ 43  50 ]</pre><p>Now the rule is obvious rather than arbitrary. Row 1 had <strong>2</strong> numbers. Column 1 had <strong>2</strong> numbers. You can pair them off. If the row had 2 numbers and the column had 3, the third number would have no partner — and there is no sensible answer to “what is 2 times nothing.”</p><p><strong>That’s the entire reason inner dimensions must match.</strong> It’s not a convention. It’s arithmetic refusing to be nonsense.</p><p>The other half follows just as directly. How many rows does your answer have? One per row of the first matrix. How many columns? One per column of the second. Hence: outer numbers survive.</p><h3>The bit that took me longest to accept</h3><p>The second matrix’s column count is <strong>free</strong>. Nothing constrains it. It’s a design choice.</p><p>In (4 × 8) · (8 × 4):</p><ul><li>That first 8 is forced. Your data has 8 dimensions per token — you didn&#39;t choose it.</li><li>The middle 8 is forced by the first. Row length must equal column length.</li><li>The final 4 is <strong>you deciding</strong> what dimension to project into.</li></ul><p>I kept looking for a rule that determined the last number. There isn’t one. That’s the knob the architect turns.</p><p><strong>Slow mode — do these three now, on paper. Answers at the end of the article.</strong></p><blockquote><strong><em>Q1.</em></strong><em> </em><em>(3 × 5) · (5 × 2) — output shape?</em></blockquote><blockquote><strong><em>Q2.</em></strong><em> You have </em><em>(4 × 8) and want </em><em>(4 × 4) out. What shape must the second matrix be?</em></blockquote><blockquote><strong><em>Q3.</em></strong><em> Compute by hand: </em><em>[[1, 2], [3, 4]] · [[2, 0], [1, 3]]</em></blockquote><h3>Part 2: What attention is actually doing</h3><p>Skip the biology metaphors. Here’s the mechanical version.</p><p>You have a sentence. Every word is a vector — a list of numbers. Right now those vectors are isolated; each word knows nothing about its neighbours.</p><p>But meaning lives in relationships. In <em>“the cat ate the fish,”</em> who ate? You resolve that by relating “ate” to “cat,” not by staring at “ate” alone.</p><p>Attention is the machinery for that. For every token, it asks the same question:</p><blockquote>Given what I am, how much should I care about each other token — including myself? Then blend them accordingly.</blockquote><p>That’s it. Three steps: <strong>ask, score, blend.</strong></p><p>The three matrices are exactly those three roles:</p><p>Name Role Plain English <strong>Q</strong> (Query) The question “What am I looking for?” <strong>K</strong> (Key) The advertisement “This is what I offer.” <strong>V</strong> (Value) The content “Here’s what you get if you pick me.”</p><p>A library analogy that actually holds up: <strong>Q</strong> is your search phrase, <strong>K</strong> is the title on the spine, <strong>V</strong> is the text inside. You match your query against titles, then read the books in proportion to how well their titles matched.</p><p>The crucial detail: <strong>all three come from the same input.</strong> That’s why it’s called <em>self</em>-attention. Same X, three different learned projections. Three viewpoints on the same data.</p><pre>┌── × W_q ──→  Q   &quot;what I&#39;m looking for&quot;<br>X ──────┼── × W_k ──→  K   &quot;what I offer&quot;<br>        └── × W_v ──→  V   &quot;what I contain&quot;</pre><h3>Part 3: Making Q, K, and V (with real numbers)</h3><p>Concrete setup:</p><ul><li><strong>4 tokens</strong> — the cat eats fish</li><li><strong>d_model = 8</strong> — each token is an 8-number vector</li><li><strong>2 heads</strong></li></ul><p>So X is (4 × 8). Four rows, one per token. Eight columns, one per dimension.</p><pre>X = [ 1  2  0  1  0  3  1  0 ]   ← the<br>    [ 0  1  1  0  2  0  1  1 ]   ← cat<br>    [ 2  0  1  1  0  1  0  0 ]   ← eats<br>    [ 1  1  0  0  1  0  2  1 ]   ← fish</pre><h3>The head split</h3><p>Two heads means each head gets half the dimensions:</p><pre>d_k = d_model / num_heads = 8 / 2 = 4</pre><p>This is a <strong>division of labour, not a loss</strong>. Head 1 works in 4 dimensions, Head 2 works in a different 4. Nothing is thrown away — the work is partitioned so different heads can specialise. One head might latch onto subject–verb links; another onto adjacency. They never talk during computation. They’re reunited at the very end.</p><h3>Deriving W_q’s shape from the rule</h3><p>Don’t memorise this. Derive it.</p><pre>Input:   X is (4 × 8)<br>Want:    Q is (4 × 4)      ← 4 tokens, d_k = 4</pre><pre>         (4 × 8) · (? × ?) = (4 × 4)</pre><p>Apply inner-match-outer-survive:</p><ul><li>Inner dimensions must match → W_q needs <strong>8 rows</strong></li><li>Outer dimensions survive → W_q needs <strong>4 columns</strong></li></ul><pre>W_q is (8 × 4) = (d_model × d_k)</pre><p>Both numbers <em>came from somewhere</em>. The 8 was handed to you by your data. The 4 you chose when you chose 2 heads. Neither is magic.</p><p>Here’s a W_q (real ones are learned; this one is handpicked so the arithmetic stays clean):</p><pre>c1 c2 c3 c4<br>row1 [  1  0  0  1 ]<br>row2 [  0  1  1  0 ]<br>row3 [  1  0  0  0 ]<br>row4 [  0  1  0  1 ]<br>row5 [  1  0  1  0 ]<br>row6 [  0  1  0  0 ]<br>row7 [  1  0  1  1 ]<br>row8 [  0  1  0  0 ]</pre><h3>Doing one cell by hand</h3><p>Token 1 (“the”) is [1, 2, 0, 1, 0, 3, 1, 0]. Column 1 of W_q is [1, 0, 1, 0, 1, 0, 1, 0].</p><pre>(1)(1) + (2)(0) + (0)(1) + (1)(0) + (0)(1) + (3)(0) + (1)(1) + (0)(0)<br>=  1  +   0    +   0    +   0    +   0    +   0    +   1    +   0<br>=  2</pre><p>So Q[1][1] = 2. Do that 16 times — 4 rows × 4 columns — and you get:</p><pre>Q = [ 2  6  3  3 ]      ← the<br>    [ 4  2  4  1 ]      ← cat<br>    [ 3  2  0  3 ]      ← eats<br>    [ 4  2  4  3 ]      ← fish</pre><p>(4 × 8) · (8 × 4) = (4 × 4). Four tokens, four dimensions each. Exactly as predicted.</p><p><strong>K</strong> and <strong>V</strong> are produced identically, with their own W_k and W_v — same shapes, different numbers.</p><h3>Doesn’t projecting to 4 dimensions destroy information?</h3><p>This bothered me for days, so let me answer it properly.</p><p>Each output dimension is a <strong>weighted mix of all 8 input dimensions</strong>. Nothing is deleted — it’s recombined. Look at the arithmetic above: that single 2 consulted every one of the 8 input values, just with some weights set to zero.</p><p>Think of a photograph reduced to grayscale. You lost the colour channels, but you kept the shapes, edges, and faces — everything that matters for “who is in this picture.” The projection is doing that deliberately: compressing to whatever subspace is most useful for <em>this head’s</em> particular job.</p><p>And critically, the compression is <strong>learned</strong>. During training, the network discovers which mixtures preserve what it needs. W_q starting as random noise and ending as a useful feature extractor <em>is</em> the learning.</p><p>One more misconception worth killing: dimensions are <strong>not</strong> ordered by importance. There’s no “dimension 1 is the most significant.” It’s not PCA. The ordering is arbitrary and stays arbitrary — importance lives in the weights, not the positions.</p><p><strong>Slow mode</strong></p><blockquote><strong><em>Q4.</em></strong><em> 3 tokens, </em><em>d_model = 4, 2 heads. What&#39;s </em><em>d_k? What shape is </em><em>W_q? What shape is </em><em>Q?</em></blockquote><blockquote><strong><em>Q5.</em></strong><em> </em><em>d_model = 768, 12 heads (this is real GPT-2). </em><em>d_k? </em><em>W_q shape?</em></blockquote><blockquote><strong><em>Q6.</em></strong><em> Compute row 2 of the </em><em>Q above by hand and confirm you get </em><em>[4, 2, 4, 1].</em></blockquote><h3>Part 4: Q · Kᵀ — where the grid appears</h3><p>Now the interesting part. You have Q and K, both (4 × 4).</p><p>Every token needs to be scored against every other token. That’s a dot product between each query and each key. And matrix multiplication is precisely a grid of dot products — so the whole thing is one operation.</p><p>But you can’t do Q · K directly:</p><pre>(4 × 4) · (4 × 4)     inner dims match, so it computes...</pre><p>It computes, but it computes the <em>wrong thing</em>. Multiplying by K as-is pairs query <strong>rows</strong> with key <strong>columns</strong> — that&#39;s matching a token against a dimension. Meaningless.</p><p>You want token-against-token. So transpose K — flip rows into columns:</p><pre>K  = [ 1  1 ]        Kᵀ = [ 1  0 ]<br>     [ 0  2 ]             [ 1  2 ]</pre><p>Now K&#39;s tokens sit in the columns, and Q · Kᵀ pairs <strong>query row i</strong> with <strong>key row j</strong>. Token against token. That&#39;s the pairing you wanted.</p><h3>The interpretation flip</h3><p>Small worked example. Q = [[2, 1], [1, 3]], K = [[1, 1], [0, 2]]:</p><pre>[ 2  1 ] · [ 1  0 ]  =  [ 3  2 ]<br>[ 1  3 ]   [ 1  2 ]     [ 4  6 ]</pre><p>Cell (1,1): [2,1] · [1,1] = 2 + 1 = 3 Cell (1,2): [2,1] · [0,2] = 0 + 2 = 2 Cell (2,1): [1,3] · [1,1] = 1 + 3 = 4 Cell (2,2): [1,3] · [0,2] = 0 + 6 = 6</p><p>Here is the moment that reframed attention for me. <strong>(4 × 4) went in and </strong><strong>(4 × 4) came out — but the meaning of the axes changed.</strong></p><p>Before: 4 tokens × 4 <strong>dimensions</strong>. After: 4 tokens × 4 <strong>tokens</strong>.</p><pre>the   cat  eats  fish<br>the   [  .     .     .     .  ]<br>cat   [  .     .     .     .  ]<br>eats  [  .     .     .     .  ]<br>fish  [  .     .     .     .  ]</pre><p>Read row 2: how much “cat” cares about “the,” about itself, about “eats,” about “fish.”</p><p>You’ve left vector space and entered relationship space. Same shape, entirely different object. That’s why (4 × 4) appearing twice is a coincidence, not a pattern.</p><h3>Why only Q·Kᵀ? Why not K·V or V·Q?</h3><p>I asked this, assuming I was missing some symmetry. I wasn’t. The three matrices have <strong>different jobs</strong>, so only two specific pairings mean anything:</p><ul><li><strong>Q · Kᵀ</strong> — question meets advertisement. Produces <em>relevance scores</em>. This is the only comparison that makes sense.</li><li><strong>weights · V</strong> — scores meet content. Produces the <em>blend</em>. Also the only sensible use of V.</li></ul><p>K · Vᵀ would compare advertisements to contents. V · Q would compare contents to questions. Neither answers a question anyone asked. The sequence isn&#39;t one option among many — it&#39;s the only path through the three roles.</p><h3>Part 5: Dividing by √d_k</h3><p>Take the scores and divide by the square root of d_k. Here d_k = 2, so √2 ≈ 1.414:</p><pre>[ 3  2 ]  ÷  1.414  =  [ 2.121  1.414 ]<br>[ 4  6 ]               [ 2.828  4.243 ]</pre><h3>The reason, with numbers</h3><p>Every score is a sum of d_k products. More dimensions, more terms, bigger sums:</p><pre>d_k = 2   →  2 numbers added   →  score around 3-6<br>d_k = 64  →  64 numbers added  →  score around 200</pre><p>Nothing has gone wrong — that’s just what summing more things does. But it wrecks the next step. Softmax on large numbers saturates.</p><p>Watch. Two scores, small:</p><pre>softmax([2, 4])   →  [0.119, 0.881]</pre><p>The second token dominates, but the first still contributes about 12%. Fine.</p><p>Same <em>ratio</em>, ten times the magnitude:</p><pre>softmax([20, 40])  →  [0.000, 1.000]</pre><p>Total collapse. One token takes everything.</p><p>Why that’s fatal: the gradient of softmax is essentially zero once it saturates. Zero gradient means no weight updates. <strong>The layer stops learning.</strong> And it’s not even a confident decision — those scores are random noise at initialisation. The model has locked onto nonsense before training begins.</p><p>Dividing by √d_k shrinks magnitudes while preserving ratios. √d_k specifically, because the <em>standard deviation</em> of a sum of d_k independent products grows like √d_k, not like d_k. Divide by the right thing and scores stay roughly unit-variance regardless of head size.</p><p><strong>Slow mode</strong></p><blockquote><strong><em>Q7.</em></strong><em> </em><em>d_k = 64. What do you divide by?</em></blockquote><blockquote><strong><em>Q8.</em></strong><em> Compute </em><em>softmax([1, 3]) by hand with </em><em>e ≈ 2.718. (Answer: </em><em>[0.119, 0.881].)</em></blockquote><blockquote><strong><em>Q9.</em></strong><em> Divide </em><em>[[5, 3], [2, 8]] by </em><em>√4.</em></blockquote><h3>Part 6: The causal mask</h3><figure><img alt="Two 4x4 attention matrices for the tokens the, cat, eats, fish — one fully filled, one with the upper triangle masked to negative infinity." src="https://cdn-images-1.medium.com/max/1024/1*3WLGz9aSTKs-qMH-9_onig.jpeg" /><figcaption>Row i keeps columns 1 to i. Token 1 has only itself.</figcaption></figure><p>Only for GPT-style decoders, but essential there.</p><p>GPT predicts the next token. During training, the whole sentence is present in memory — which means without intervention, token 2 can see token 4 and simply copy the answer it’s meant to predict. That’s not learning, that’s peeking. The model aces training and fails completely at inference, where the future genuinely doesn’t exist.</p><p>The fix: block every position above the diagonal.</p><pre>before mask:              after mask:<br>[ 2.121  1.414 ]          [ 2.121   -inf  ]<br>[ 2.828  4.243 ]          [ 2.828  4.243  ]</pre><p>Token 1 sees only itself. Token 2 sees token 1 and itself. Row i keeps columns 1..i.</p><h3>Why -inf and not 0</h3><p>This trips up almost everyone, including me. You are masking <strong>before softmax</strong>, and softmax exponentiates:</p><pre>e^0    = 1     →  a &quot;zero&quot; score still gets weight!<br>e^-inf = 0     →  genuinely eliminated</pre><p>Setting a score to 0 doesn’t say “ignore this.” It says “this is averagely relevant.” Only -inf disappears cleanly.</p><p><strong>BERT-style encoders skip this entirely</strong> — they’re not predicting the next word, so every token may see every other. Full grid, no triangle. Same machinery, one flag.</p><h3>Part 7: Softmax</h3><p>Scores are arbitrary real numbers — 3, -2, 47. To use them as blend weights you need them positive and summing to 1. Softmax does exactly that.</p><pre>softmax(xᵢ) = e^xᵢ / Σ e^xⱼ</pre><p>Three steps: exponentiate everything, add up the results, divide each by the total.</p><h3>What e^x means, on a scalar</h3><p>e ≈ 2.718. e^x means e raised to the power x.</p><pre>e^1 = 2.718<br>e^2 = 7.389<br>e^3 = 20.086</pre><p>Notice how fast that climbs. That’s the point — exponentiation <strong>exaggerates gaps</strong>. A score of 2 vs 1 is a 2× gap; e^2 vs e^1 is a 2.7× gap. Softmax makes the winner win more clearly, which is what you want from something that&#39;s choosing.</p><p>And it guarantees positivity for free: e^x &gt; 0 for every real x, including negatives (e^-5 ≈ 0.0067 — tiny, still positive). Never negative, never zero, so the weights are always valid.</p><h3>Row 1, worked</h3><p>Masked row 1 is [2.121, -inf]:</p><pre>e^2.121  = 8.34<br>e^-inf   = 0<br>sum      = 8.34</pre><pre>8.34 / 8.34 = 1.0<br>0    / 8.34 = 0.0</pre><p>[1.0, 0.0]. Token 1 gives itself 100% attention — the only thing it&#39;s allowed to look at. Sanity check passed.</p><p>Row 2 is [2.828, 4.243]:</p><pre>e^2.828 = 16.91<br>e^4.243 = 69.42<br>sum     = 86.33</pre><pre>16.91 / 86.33 = 0.196<br>69.42 / 86.33 = 0.804</pre><p>Final attention matrix:</p><pre>tok1   tok2<br>tok1 [ 1.000  0.000 ]<br>tok2 [ 0.196  0.804 ]</pre><p><strong>Every row sums to 1.</strong> That’s your debugging lifeline — if a row doesn’t sum to 1 (within floating-point noise), something upstream is broken.</p><p>Token 2 splits its attention 20/80: a fifth on token 1, four fifths on itself. That’s a genuine decision, made by arithmetic.</p><h3>One implementation detail that matters</h3><p>Never use the naive formula directly. e^1000 overflows to infinity and your output becomes NaN. Subtract the row maximum from every score first.</p><p>This is mathematically identical — softmax depends only on <em>differences</em> between scores, so shifting everything by a constant changes nothing. But now the largest exponent is e^0 = 1, and overflow is impossible. Every real implementation does this.</p><h3>Part 8: · V — back to vectors</h3><p>Last step. Multiply the attention weights by V.</p><p>With V = [[10, 20], [30, 40]]:</p><pre>[ 1.000  0.000 ] · [ 10  20 ]  =  [ 10.00  20.00 ]<br>[ 0.196  0.804 ]   [ 30  40 ]     [ 26.09  36.09 ]</pre><p>Row 1: 1.0 × [10,20] + 0.0 × [30,40] = [10, 20]. Only itself available, so it gets its own value unchanged.</p><p>Row 2: 0.196 × [10,20] + 0.804 × [30,40]</p><pre>= [1.96, 3.92] + [24.12, 32.16]<br>= [26.09, 36.09]</pre><p>Token 2’s output is a <strong>weighted blend</strong> — mostly its own content, with a measured contribution from token 1. It is no longer an isolated vector. It carries context.</p><p>That’s attention. That’s the whole thing.</p><h3>The round trip</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QkXpJlLjgeDgJAzTTMhe7Q.png" /><figcaption>Same shape, twice — but the meaning of the axes flipped in the middle. That’s the part worth staring at.</figcaption></figure><p>Step back and look at the shape journey:</p><pre>(4 × 8)  →  (4 × 4)  →  (4 × 4)  →  (4 × 4)<br>tok×dim     tok×dim     tok×tok     tok×dim<br>   X         Q,K,V       scores      output</pre><p><strong>Dimensions → relationships → dimensions.</strong> You leave vector space to compute how things relate, then return to vector space carrying that information. Every attention layer in every transformer makes that same round trip.</p><h3>Part 9: Multi-head — and the concat that restores d_model</h3><p>Everything above was <strong>one head</strong>. Now the multi-head part.</p><p>Each head has its own W_q, W_k, W_v. They run in parallel, completely independently, never exchanging information mid-computation. Two heads with d_model = 8 gives you two (4 × 4) outputs.</p><p>But the next layer expects (4 × 8). Where does the 8 come from?</p><p><strong>Concatenation.</strong> Glue the head outputs side by side:</p><pre>head1 (4 × 4)  +  head2 (4 × 4)  →  (4 × 8)</pre><pre>├── head 1 ──┤├── head 2 ──┤<br>tok1 [ a  b  c  d  |  e  f  g  h ]<br>tok2 [ .  .  .  .  |  .  .  .  . ]<br>tok3 [ .  .  .  .  |  .  .  .  . ]<br>tok4 [ .  .  .  .  |  .  .  .  . ]</pre><p>d_model is back. And this is exactly why the divisibility constraint exists:</p><pre>num_heads × d_k = d_model</pre><p>d_model = 8, 3 heads → 8/3 = 2.67. Not a whole number. Concat produces the wrong width, and the next multiplication fails. <strong>d_model must be divisible by </strong><strong>num_heads</strong> — not a style preference, a hard requirement.</p><h3>The final W_o</h3><p>One step remains. Right now those 8 columns are two independent 4-column blocks stapled together. They’ve never interacted.</p><p>W_o, shape (d_model × d_model), fixes that:</p><pre>(4 × 8) · (8 × 8) = (4 × 8)</pre><p>Shape unchanged, contents thoroughly mixed. Each output dimension now draws on <strong>both</strong> heads. W_o is what turns &quot;two separate opinions&quot; into &quot;one integrated representation.&quot;</p><p>Without it you’d have parallel reports and no synthesis. It’s the least discussed matrix in attention and one of the most necessary.</p><h3>Full pipeline</h3><pre>X  (4 × 8)<br>│<br>├── Head 1 ──→ Q,K,V (4×4) ──→ QKᵀ (4×4) ──→ ÷√d_k ──→ mask ──→ softmax ──→ ·V ──→ (4×4)<br>└── Head 2 ──→ Q,K,V (4×4) ──→ QKᵀ (4×4) ──→ ÷√d_k ──→ mask ──→ softmax ──→ ·V ──→ (4×4)<br>                                                                                   │<br>                                                          concat  ──→  (4 × 8) ────┘<br>                                                             │<br>                                                          · W_o  ──→  (4 × 8)</pre><h3>Part 10: The code</h3><p>Everything above is implemented in pure NumPy — no PyTorch, no TensorFlow, nothing that hides a shape from you:</p><h3><a href="https://github.com/Bunny-dev24/Multihead-attention-from-scratch">github.com/Bunny-dev24/Multihead-attention-from-scratch</a></h3><p>What’s in there:</p><p>File What it does softmax.py Numerically stable softmax masking.py Causal mask + the -inf application scaled_dot_product.py Single-head attention, shapes documented line by line multi_head.py The MultiHeadAttention class examples/ The hand-worked example above, verified in code, plus an attention heatmap tests/ Every invariant in this article, as a passing test</p><p>The examples are the part worth your time. examples/01_single_head_by_hand.py reproduces the exact arithmetic from Parts 4–8 — do it on paper first, then run the file and see your numbers appear.</p><p><strong>One design note.</strong> The repo loops over heads explicitly instead of batching them. Production code doesn’t do that — it stacks all heads into one large multiplication, then uses reshape and transpose to split them apart. Much faster on a GPU. It&#39;s also exactly where the shapes become invisible: a four-dimensional reshape followed by a transpose is unreadable until you already know what it&#39;s doing.</p><p>The loop is slower and honest. Learn it that way, then go read the optimised version and you’ll actually see what the reshape is unpacking.</p><p>Two checks worth burning into memory before you read any of it: <strong>output shape equals input shape</strong>, and <strong>every attention row sums to 1</strong>. Those two invariants catch most bugs you’ll ever write in an attention layer.</p><h3>What this deliberately leaves out</h3><p>This is attention, not a full transformer block. Still missing:</p><ul><li><strong>Positional encoding</strong> — attention has no concept of word order. &quot;cat eats fish&quot; and &quot;fish eats cat&quot; are identical to it. Position has to be injected separately.</li><li><strong>Residual connections</strong> — X + attention(X), so gradients survive depth.</li><li><strong>Layer normalisation</strong> — activation stability.</li><li><strong>Feed-forward network</strong> — the wide layer after attention where a lot of the actual capacity lives.</li><li><strong>Backpropagation</strong> — forward pass only. Real training learns those W matrices instead of sampling them.</li></ul><p>Attention decides <em>what to look at</em>. The rest of the block decides <em>what to do about it</em>.</p><h3>What actually made it click</h3><p>Four things, in case they help you too.</p><p><strong>1. Derive shapes, never memorise them.</strong> Every time you meet a matrix multiplication, write it out and apply inner-match-outer-survive. Within a week you’ll do it in your head. Memorised shapes evaporate; a derivation rule doesn’t.</p><p><strong>2. Do the arithmetic by hand at least once.</strong> I resisted this hard. It felt beneath me and it felt slow. It was neither — it was the single highest-return thing I did. Understanding you can only recognise isn’t understanding. Understanding you can <em>reproduce</em> is.</p><p><strong>3. Notice when the meaning of a shape changes.</strong> (4 × 4) as tokens × dimensions and (4 × 4) as tokens × tokens are completely different objects. Tracking the <em>meaning</em> of each axis, not just its size, is where real comfort with transformers comes from.</p><p><strong>4. Question the notation.</strong> Why Kᵀ? Why -inf instead of 0? Why √d_k and not d_k? Every one of those has a concrete, satisfying answer, and chasing it down teaches you more than the formula does.</p><p>That last one is worth generalising. I spent months reading attention explanations and retaining almost nothing, because reading feels like progress while requiring none of the effort that produces it. The moment I started writing arithmetic on paper, the same material stuck in an afternoon.</p><p>If you skimmed the slow-mode sections — and I’d bet you did, I would have — go back and do them now. Twenty minutes.</p><h3>Answers</h3><p><strong>Q1.</strong> (3 × 2) — inner 5s cancel, outer 3 and 2 survive.</p><p><strong>Q2.</strong> (8 × 4). Inner must match the 8; the 4 is your choice, dictated by the output you want.</p><p><strong>Q3.</strong></p><pre>[[1,2],[3,4]] · [[2,0],[1,3]]</pre><pre>(1,1): (1)(2) + (2)(1) = 4<br>(1,2): (1)(0) + (2)(3) = 6<br>(2,1): (3)(2) + (4)(1) = 10<br>(2,2): (3)(0) + (4)(3) = 12</pre><pre>= [[ 4,  6],<br>   [10, 12]]</pre><p><strong>Q4.</strong> d_k = 4/2 = 2. W_q is (4 × 2). Q is (3 × 2).</p><p><strong>Q5.</strong> d_k = 768/12 = 64. W_q is (768 × 64).</p><p><strong>Q6.</strong> Token 2 is [0,1,1,0,2,0,1,1].</p><pre>col1 [1,0,1,0,1,0,1,0]: 0+0+1+0+2+0+1+0 = 4<br>col2 [0,1,0,1,0,1,0,1]: 0+1+0+0+0+0+0+1 = 2<br>col3 [0,1,0,0,1,0,1,0]: 0+1+0+0+2+0+1+0 = 4<br>col4 [1,0,0,1,0,0,1,0]: 0+0+0+0+0+0+1+0 = 1</pre><pre>→ [4, 2, 4, 1]  ✓</pre><p><strong>Q7.</strong> √64 = 8.</p><p><strong>Q8.</strong></p><pre>e^1 = 2.718,  e^3 = 20.086,  sum = 22.804<br>2.718 / 22.804  = 0.119<br>20.086 / 22.804 = 0.881</pre><p><strong>Q9.</strong> √4 = 2, so [[2.5, 1.5], [1.0, 4.0]].</p><p><em>Code, tests, and runnable examples: </em><a href="https://github.com/Bunny-dev24/Multihead-attention-from-scratch"><em>github.com/Bunny-dev24/Multihead-attention-from-scratch</em></a><em> — star it if it helped, and if a specific shape still doesn’t sit right, say which one in the comments. That’s usually where the real confusion is hiding, and it’s almost never where people expect.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=79df48031c33" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How I Built the Smallest LLM From Scratch to Understand Matrix Multiplication]]></title>
            <link>https://medium.com/@Yash_Bhardwaj/how-i-built-the-smallest-llm-from-scratch-to-understand-matrix-multiplication-0572a2930989?source=rss-bec114f6ca70------2</link>
            <guid isPermaLink="false">https://medium.com/p/0572a2930989</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Yash Bhardwaj]]></dc:creator>
            <pubDate>Tue, 18 Aug 2026 18:26:52 GMT</pubDate>
            <atom:updated>2026-08-18T18:26:52.346Z</atom:updated>
            <content:encoded><![CDATA[<h3>I Built the World’s Smallest LLM to Prove “It’s Just Matrix Multiplication” — Here’s What I Learned</h3><figure><img alt="Illustration of a cartoon chef holding recipe cards transforming into a glowing neural network, with the words i, write, code, daily flowing in as tokens — cover for an article on building the world’s smallest LLM from scratch." src="https://cdn-images-1.medium.com/max/1024/1*mR5Zb-H6IccNSPjtoEHOFA.jpeg" /><figcaption>An LLM as a kitchen: words become “recipe cards” (embeddings), the chef reasons over them, and out comes a prediction. The whole model, shrunk to 4 words.</figcaption></figure><p><em>A from-scratch mini transformer with 4 words and 2 dimensions — small enough to check every number by hand.</em></p><p>For months I nodded along whenever someone said <strong>“an LLM is just matrix multiplication.”</strong> I’d repeat it at meetups, drop it in comments, feel smart for ten seconds — and then quietly realize I didn’t actually <em>believe</em> it. I couldn’t point to where the “intelligence” lived. It felt like magic wearing a lab coat.</p><p>So I did the only thing that ever cures that feeling for me: I shrank the problem until it couldn’t hide anymore.</p><p>I built the smallest language model I could imagine. <strong>Four words in the vocabulary. Two numbers per word.</strong> Small enough that I could work out every single value with a pen and check whether the machine agreed. No GPU. No billions of parameters. Just enough to see the gears turn.</p><p>This is the story of what I found — and the mental model that finally made transformers click for me.</p><h3>The one tiny example everything hangs on</h3><p>My entire “model” learns one thing: read <strong>i write code</strong>, and predict the next word — <strong>daily</strong>.</p><p>That’s it. One sentence. One correct answer. But here’s the thing nobody tells you: <strong>every single component of a real transformer is present in this toy.</strong> Embeddings, attention, a feed-forward network, softmax, cross-entropy loss, backpropagation — all of it. Just shrunk until the whole thing fits in your head.</p><p>If you understand how this predicts daily, you understand the skeleton of GPT. The only difference is scale.</p><h3>Why I stopped explaining it with math (and started using a kitchen)</h3><p>Every tutorial I read led with equations. And equations are fine — <em>once you already know what they mean.</em> Before that, they’re just noise with Greek letters.</p><p>At its core, every large language model does one thing: <strong>next-token prediction</strong>. Given a few words, guess the next one. This mini transformer — built from scratch in PyTorch — does exactly that, just with a vocabulary of four words instead of fifty thousand.</p><p>So I threw them out and started picturing the whole model as a <strong>kitchen</strong>:</p><ul><li><strong>Embeddings</strong> are recipe cards — they turn a word into a few numbers the model can actually cook with.</li><li><strong>Attention</strong> is the chef glancing at the other ingredients on the counter before deciding what to do next.</li><li><strong>The feed-forward layer (with ReLU)</strong> is the chef actually thinking — and ReLU is the part where useless thoughts get thrown straight in the bin (anything negative becomes zero).</li><li><strong>The output layer</strong> is the chef finally committing to a guess.</li><li><strong>Cross-entropy loss</strong> is the head chef tasting the dish and bluntly saying how wrong it was.</li><li><strong>Backpropagation</strong> is that criticism traveling back through the kitchen so every station adjusts for next time.</li></ul><p>Once I had that picture, the math stopped being scary. It was just the kitchen, written down.</p><h3>How transformers work, traced by hand</h3><figure><img alt="Flow diagram showing a transformer pipeline — Embedding, Attention, FFN plus ReLU, Output, Softmax, Loss — with the tokens i, write, code, daily feeding into the left side." src="https://cdn-images-1.medium.com/max/1024/1*-NksjelJo-cxS-Frz1F_tQ.png" /><figcaption>The full journey of a word inside the model: embedding → attention → feed-forward → output → softmax → loss. Every arrow is just a matrix multiply or a squash.</figcaption></figure><p>Here’s the entire journey a word takes inside the model:</p><p><strong>embedding → attention (a) → W1 → pre → ReLU → h → W2 → f → W_out → logits → softmax → p → loss</strong></p><p>Read it left to right. The last token’s vector (a) flows through the feed-forward layer, turns into a score for each of the 4 words (logits), those scores get squashed into probabilities (p), and the loss simply measures how far the probability of daily is from a perfect 1.0.</p><p>Nothing in that chain is mysterious. Every arrow is either a matrix multiply or a simple squashing function.</p><h3>The thing that finally made matrix shapes click for me</h3><p>This is the part I always got wrong, so let me save you the confusion.</p><p>A weight matrix is always shaped <strong>(input size) × (output size)</strong>. Rows come from what goes <em>in</em>. Columns come from what comes <em>out</em>. That’s the whole rule.</p><ul><li><strong>W1</strong> takes a and produces pre → shape <strong>2×2</strong></li><li><strong>W2</strong> takes h and produces f → shape <strong>2×2</strong></li><li><strong>W_out</strong> takes f and produces 4 word scores → shape <strong>2×4</strong></li></ul><p>W_out is the only wide one, and for an obvious reason: it has to produce a score for <em>every word</em> in the vocabulary. More words, wider matrix. That&#39;s literally why real models with 50,000-word vocabularies have a massive final layer.</p><h3>Seeing it’s not a black box</h3><p>Here’s a single forward pass I worked out by hand. The real code starts from random weights, but the arithmetic is identical — so you can follow along on paper:</p><p>Start with the last token’s vector: <strong>a = [0.7, 0.9]</strong></p><p>Multiply by W1 = [[1, -2], [1, 1]]:</p><ul><li>pre[0] = 0.7×1 + 0.9×1 = <strong>1.6</strong></li><li>pre[1] = 0.7×(-2) + 0.9×1 = <strong>-0.5</strong></li><li>pre = [1.6, -0.5]</li></ul><p>Apply ReLU (negatives become zero): <strong>h = [1.6, 0]</strong> — the gate closed on that -0.5.</p><p>Multiply by W2 = [[0.375, 0.75], [0.2, 0.1]]:</p><ul><li>f = [0.6, 1.2]</li></ul><p>Multiply by W_out to get one score per word:</p><ul><li>logits = [0.5, 1.0, 0.3, 2.0] → for <em>i, write, code, daily</em></li></ul><p>Softmax turns those into probabilities: <strong>[0.13, 0.21, 0.10, 0.56]</strong>, and the loss is −log(0.56) = <strong>0.573</strong>.</p><p>Training just repeats this thousands of times and nudges the weights until the probability of daily climbs toward 1.</p><h3>Watching it actually learn</h3><p>This was the moment it stopped being theory. When I ran the training loop, the loss fell off a cliff:</p><ul><li>Epoch 0 → Loss 1.37, probability of daily just 25%</li><li>Epoch 40 → Loss 0.005, probability 99.5%</li><li>Epoch 199 → Loss 0.0003, probability 100%</li></ul><p>Final guess: <strong>daily</strong>. Certain.</p><p>The first prediction was basically a coin toss, because the model <em>starts</em> with random numbers. Nothing is designed or hard-coded — it’s all learned by being wrong, over and over, and adjusting a little each time. That’s the entire trick behind every large language model on the planet.</p><h3>What this taught me about how LLMs actually work</h3><p>I went in expecting to find some hidden cleverness — the “intelligence” bit that tutorials always gesture at but never quite show you. It’s not there. What’s there is a stack of matrices, a gate that throws away the negative numbers, and a loop that’s wrong at first and slowly stops being wrong.</p><p><strong>Attention</strong> turned out to be just weighted averaging — the model deciding which earlier words matter before it commits to a guess. <strong>Next-token prediction</strong> is the entire job description: read what came before, score every word in the vocabulary, pick the likeliest one. And <strong>backpropagation</strong> — the part that felt like dark magic — is nothing more than blame, measured precisely and sent backward so every weight knows exactly how much to shift.</p><p>Scale a four-word model up to fifty thousand words and billions of parameters and yes, something that <em>feels</em> like understanding emerges. But the machinery underneath doesn’t change. It’s the same three moves, repeated until the numbers stop being wrong. Once you’ve watched that happen by hand — even at this ridiculous scale — you never quite look at “AI” the same way again.</p><h3>Try it yourself</h3><p>If you want to poke at it, the whole project is tiny — one file for the forward pass, one for training. Clone it, change the weights, break it, and watch what happens. Honestly, breaking it is where most of my understanding came from.</p><p><strong>Full code on GitHub:</strong> <a href="https://github.com/Bunny-dev24/mini-LLM-from-scratch">github.com/Bunny-dev24/mini-LLM-from-scratch</a></p><p>The next time someone tells you an LLM is “just matrix multiplication,” you won’t have to nod politely. You’ll be able to say <em>“yeah, I know”</em> — and actually mean it.</p><p><em>If you enjoyed this, I write about AI, machine learning, and building things from scratch to actually understand them. Follow me here on Medium — more from-scratch experiments coming soon.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0572a2930989" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>