<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lakshay Arora on Medium]]></title>
        <description><![CDATA[Stories by Lakshay Arora on Medium]]></description>
        <link>https://medium.com/@lakshaya32?source=rss-51467d705cd5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*BdHBvrNtQe0KB1a3</url>
            <title>Stories by Lakshay Arora on Medium</title>
            <link>https://medium.com/@lakshaya32?source=rss-51467d705cd5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 06:31:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lakshaya32/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Chapter 4 — Building a GPT: From Blocks to a Talking Model]]></title>
            <link>https://medium.com/@lakshaya32/chapter-4-building-a-gpt-from-blocks-to-a-talking-model-bd3d0c6c8439?source=rss-51467d705cd5------2</link>
            <guid isPermaLink="false">https://medium.com/p/bd3d0c6c8439</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[machine-learning-ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Lakshay Arora]]></dc:creator>
            <pubDate>Tue, 14 Oct 2025 13:11:45 GMT</pubDate>
            <atom:updated>2025-10-14T13:11:45.121Z</atom:updated>
            <content:encoded><![CDATA[<h3>Chapter 4 — Building a GPT: From Blocks to a Talking Model</h3><p>By the time I reached Chapter 4 of <em>Build a Large Language Model (From Scratch) by </em>Sebastian Raschka, everything I’d been learning — tokenization, embeddings, attention—finally started to make sense as a whole.<br> This chapter is where the scattered pieces fuse together into a complete GPT-like architecture capable of generating text.</p><p>It’s the first time the model stops being a concept and actually <em>speaks.</em></p><h3>🧩 The GPT Architecture</h3><p>GPT isn’t one monolithic neural network (not your Next-door Black box :P). It’s a <strong>stack of structured modules</strong>, each doing one clear job.</p><p>At a high level, GPT is</p><blockquote>Embeddings → Multi-Head Attention → Feed Forward → Normalization + Residuals → Output Head</blockquote><p>That’s it.<br> No magic, just clean, consistent engineering scaled up millions of times.</p><h3>1️⃣ Token &amp; Positional Embeddings</h3><p>First, every token (say “dog”) is converted into a dense embedding vector that captures its meaning.</p><p>But GPT also needs to know <em>where</em> in the sequence the token occurs i.e. order matters.<br>So positional embeddings are added to inject sequence information.</p><pre>class GPTEmbedding(nn.Module):<br>    def __init__(self, vocab_size, embed_dim, max_len):<br>        super().__init__()<br>        self.token_embed = nn.Embedding(vocab_size, embed_dim)<br>        self.pos_embed = nn.Embedding(max_len, embed_dim)</pre><pre>    def forward(self, x):<br>        positions = torch.arange(0, x.size(1), device=x.device).unsqueeze(0)<br>        return self.token_embed(x) + self.pos_embed(positions)</pre><p>Think of this as the model’s way of grounding language — it understands <em>what</em> each word is and <em>where</em> it belongs.</p><h3>2️⃣ Self-Attention Inside a Transformer Block</h3><p>From Chapter 3, we know how attention works — tokens interact via <strong>Queries, Keys, and Values.</strong></p><p>Chapter 4 shows how to <strong>package that attention</strong> inside a reusable block with normalization and residual connections.</p><pre>class GPTBlock(nn.Module):<br>    def __init__(self, embed_dim, num_heads):<br>        super().__init__()<br>        self.attn = MultiHeadAttention(embed_dim, num_heads)<br>        self.ln1 = nn.LayerNorm(embed_dim)<br>        self.ff = FeedForward(embed_dim)<br>        self.ln2 = nn.LayerNorm(embed_dim)</pre><pre>    def forward(self, x):<br>        x = x + self.attn(self.ln1(x))  # attention + residual<br>        x = x + self.ff(self.ln2(x))    # feed-forward + residual<br>        return x</pre><p>Each block learns to capture dependencies — some focus on short-range syntax, others on long-range semantics.</p><p>The <strong>residual connections</strong> (x = x + ...) ensure smooth gradient flow, making deep networks stable and trainable.</p><h3>3️⃣ Feed-Forward Network (FFN)</h3><p>After attention figures out what’s important, a small MLP transforms that information before passing it on.</p><pre>class FeedForward(nn.Module):<br>    def __init__(self, embed_dim):<br>        super().__init__()<br>        self.net = nn.Sequential(<br>            nn.Linear(embed_dim, 4 * embed_dim),<br>            nn.GELU(),<br>            nn.Linear(4 * embed_dim, embed_dim)<br>        )<br>    def forward(self, x):<br>        return self.net(x)</pre><p>It’s deceptively simple, but this is where the model’s representational power explodes.<br> It gives the model non-linearity, enabling complex reasoning over attended features.</p><h3>4️⃣ Assembling the Full GPT Model</h3><p>Once you understand the building blocks, the full model almost feels inevitable.</p><pre>class GPTModel(nn.Module):<br>    def __init__(self, vocab_size, max_len, embed_dim, num_heads, num_layers):<br>        super().__init__()<br>        self.embed = GPTEmbedding(vocab_size, embed_dim, max_len)<br>        self.blocks = nn.Sequential(*[GPTBlock(embed_dim, num_heads) for _ in range(num_layers)])<br>        self.ln_f = nn.LayerNorm(embed_dim)<br>        self.fc_out = nn.Linear(embed_dim, vocab_size)</pre><pre>    def forward(self, x):<br>        x = self.embed(x)<br>        x = self.blocks(x)<br>        x = self.ln_f(x)<br>        return self.fc_out(x)</pre><p>Each input token goes through embeddings → stacked transformer blocks → final projection to predict the <strong>next token</strong>.<br> The causal mask from Chapter 3 still applies here, ensuring tokens only attend to their past.</p><p>This is essentially GPT-2’s backbone in miniature form.</p><h3>5️⃣ Generating Text</h3><p>The most satisfying part: Watching the model actually generate language.<br> It does this autoregressively; predicting one token at a time, appending it, and repeating.</p><pre>def generate(model, idx, max_new_tokens):<br>    for _ in range(max_new_tokens):<br>        logits = model(idx)[:, -1, :]<br>        probs = F.softmax(logits, dim=-1)<br>        next_token = torch.multinomial(probs, num_samples=1)<br>        idx = torch.cat((idx, next_token), dim=1)<br>    return idx</pre><p>It’s slow, sequential, and beautifully simple.<br>Each prediction depends only on what came before — exactly like how we write sentences word by word.</p><h3>💡 My Reflection</h3><p>This chapter was the turning point. It stopped feeling like abstract architecture and started feeling like a working organism.</p><p>A few realizations that stuck with me:</p><ol><li>GPT is modular: each piece is understandable in isolation.</li><li>There’s no “hidden trick.” The brilliance lies in the <em>discipline</em> of the design.</li><li>When scaled properly, these simple parts unlock emergent behavior that feels intelligent.</li></ol><p>Seeing text emerge from this minimal code makes you appreciate the elegance of modern LLMs.</p><p>It’s <em>methodical engineering scaled up to art.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bd3d0c6c8439" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Chapter 3 — Attention: Teaching Models What to Focus On]]></title>
            <link>https://medium.com/@lakshaya32/chapter-3-attention-teaching-models-what-to-focus-on-991fc4fbfaa0?source=rss-51467d705cd5------2</link>
            <guid isPermaLink="false">https://medium.com/p/991fc4fbfaa0</guid>
            <category><![CDATA[generattive-ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[retrieval-augmented-gen]]></category>
            <category><![CDATA[llm-from-scratch]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Lakshay Arora]]></dc:creator>
            <pubDate>Mon, 29 Sep 2025 13:06:01 GMT</pubDate>
            <atom:updated>2025-09-29T13:06:01.194Z</atom:updated>
            <content:encoded><![CDATA[<h3>Chapter 3 — Attention: Teaching Models What to Focus On</h3><p>In Chapter 2, I learned how to break text into tokens and map them into embeddings. That gave the model Lego bricks. But here’s the thing: <strong>embeddings alone don’t know which words matter when</strong>.</p><p>That’s what Chapter 3 is about: A<strong>ttention Mechanisms.</strong> They’re the reason Transformers replaced RNNs and LSTMs and became the backbone of every modern LLM.</p><h3>Why Attention?</h3><p>RNNs and LSTMs tried to process sentences step by step. The problem: they forget long-range context. By the time they reach the end of a sentence, they’ve basically lost the beginning.</p><p>Attention fixes this. Instead of reading one word at a time, every token can look at <em>all the other tokens</em> — and decide which ones are important. That’s how “cat” and “sat” stay connected even if “on the mat” comes in between.</p><h3>Step 1: The Intuition</h3><p>The simplest version of attention is just:</p><ol><li>Take token vectors.</li><li>Compare how similar they are.</li><li>Use similarity as weights to average them.</li></ol><p>It’s not magic. It’s just “smart weighted averaging.”</p><h3>Step 2: Self-Attention With Trainable Weights</h3><p>Now comes the magic sauce: Queries, Keys, and Values.</p><ul><li><strong>Query (Q):</strong> what this token is looking for.</li><li><strong>Key (K):</strong> what this token has to offer.</li><li><strong>Value (V):</strong> the actual content to be passed on.</li></ul><p>The process:</p><ol><li>Compute attention scores = Q • K^T / √d.</li><li>Apply softmax to turn scores into probabilities.</li><li>Use those probabilities to blend the Values.</li></ol><p>Here’s a minimal PyTorch sketch from the book’s ideas:</p><pre>import torch<br>import torch.nn as nn<br>import torch.nn.functional as F</pre><pre>class SelfAttention(nn.Module):<br>    def __init__(self, d_model):<br>        super().__init__()<br>        self.W_q = nn.Linear(d_model, d_model)<br>        self.W_k = nn.Linear(d_model, d_model)<br>        self.W_v = nn.Linear(d_model, d_model)</pre><pre>    def forward(self, x):<br>        Q = self.W_q(x)<br>        K = self.W_k(x)<br>        V = self.W_v(x)</pre><pre>        scores = Q @ K.transpose(-2, -1) / (x.size(-1) ** 0.5)<br>        weights = F.softmax(scores, dim=-1)<br>        return weights @ V</pre><p>This is the <strong>core of Transformers</strong>.</p><h3>Step 3: Causal Attention (No Cheating)</h3><p>There’s a problem though. If every token can see the future, the model can “cheat” during training by looking at the word it’s supposed to predict.</p><p>The fix is <strong>causal masking</strong>: block off future tokens so each position can only attend to the past. This is what makes GPT models autoregressive — generating one word at a time without spoilers.</p><p>The book also throws in <strong>dropout on attention weights</strong> — randomly zeroing out some connections so the model doesn’t overfit.</p><h3>Step 4: Multi-Head Attention</h3><p>One head can only capture one kind of relationship. But language is messy — syntax, semantics, long-range, short-range.</p><p><strong>Multi-head attention</strong> splits the Q/K/V projections into multiple sets (“heads”), each learning a different pattern in parallel. Then it concatenates them back together.</p><p>That’s why GPT doesn’t just know grammar — it can also “get” meaning, tone, and context at the same time.</p><h3>My Reflection</h3><p>Chapter 2 felt like groundwork. Chapter 3 felt like watching the model come alive.</p><p>The biggest shift for me was realizing:</p><blockquote>Attention isn’t mystical. It’s weighted averaging. But when you parallelize it across heads and stack it deep, emergent behavior starts to look like intelligence.</blockquote><p>This chapter also made me appreciate why people say “attention is all you need.” Because once you see how clean and powerful this mechanism is, you realize it really does replace the patchwork of tricks we used before.</p><h3>What’s Next</h3><p>Chapter 4 is where things get assembled. Attention by itself is powerful, but when you add feed-forward networks, layer norms, and residuals, you finally have a GPT block. That’s when the architecture becomes a true Transformer.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=991fc4fbfaa0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Chapter 2 — Tokenization: Breaking Language Into Lego Bricks]]></title>
            <link>https://medium.com/@lakshaya32/chapter-2-tokenization-breaking-language-into-lego-bricks-615c2323c619?source=rss-51467d705cd5------2</link>
            <guid isPermaLink="false">https://medium.com/p/615c2323c619</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[genrative-ai]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Lakshay Arora]]></dc:creator>
            <pubDate>Thu, 18 Sep 2025 12:51:54 GMT</pubDate>
            <atom:updated>2025-09-18T12:51:54.448Z</atom:updated>
            <content:encoded><![CDATA[<h3>Chapter 2 — Tokenization: Breaking Language Into Lego Bricks</h3><p>When I opened Chapter 2 of Sebastian Raschka’s <em>Build a Large Language Model (From Scratch)</em>, I expected “just preprocessing.” Instead, I realized: <strong>the way we turn text into numbers is half the game.</strong></p><p>You can’t train an LLM without tokenization — and how you tokenize decides the “universe” the model even sees.</p><h3>From Text to Tokens</h3><p>Computers don’t understand “words.” They need numbers. So the first job is: break text into tokens, map them to IDs, then feed them into the model.</p><p>The chapter starts simple:</p><ul><li><strong>Word-level tokenization</strong>: easy, but vocab explodes, and rare words become [UNK].</li><li><strong>Character-level</strong>: tiny vocab, but sequences are painfully long.</li><li><strong>Subword tokenization</strong>: the middle ground (e.g., “un+believ+able”).</li></ul><p>This is where <strong>Byte Pair Encoding (BPE)</strong> comes in — and Raschka shows why GPT models rely on it.</p><h3>Byte Pair Encoding (BPE) in Action</h3><p>BPE builds its vocab by merging frequent pairs of characters → subwords → full words.</p><p>Example:</p><ul><li>Start with d e f i n e.</li><li>Merge d+e → de.</li><li>Merge ine → ine.</li><li>Now “define” = de + fine.</li></ul><p>Unknown word? BPE still handles it by splitting into smaller pieces, instead of panicking with [UNK].</p><h3>Code (using tiktoken):</h3><pre># pip install tiktoken<br>import tiktoken<br><br>tokenizer = tiktoken.get_encoding(&quot;gpt2&quot;)<br>text = &quot;Hello, do you like tea? In someunknownPlace.&quot;<br>tokens = tokenizer.encode(text)<br>print(&quot;Token IDs:&quot;, tokens)<br>print(&quot;Decoded:&quot;, tokenizer.decode(tokens))</pre><p>Output:</p><pre>Token IDs: [15496, 11, 428, 345, 3665, 30, 287, 262, 10395, 20974, 286, 6170, 13]<br>Decoded: Hello, do you like tea? In someunknownPlace.</pre><p>Notice <strong>“someunknownPlace”</strong> still works — split into subwords, no [UNK].</p><h3>Sliding Window Sampling</h3><p>Once text is tokenized, how do we create training examples?</p><p>If you feed the model the entire corpus in one go, it’ll blow up memory. Instead, you use a <strong>sliding window</strong>:</p><ul><li>Take a chunk of tokens (say length 4).</li><li>Predict the next token.</li><li>Slide the window forward by 1, repeat.</li></ul><h3>Mini Example:</h3><p>If the text is:</p><pre>[&quot;I&quot;, &quot;like&quot;, &quot;tea&quot;, &quot;a&quot;, &quot;lot&quot;]</pre><p>Training samples look like:</p><ul><li>Input: [&quot;I&quot;, &quot;like&quot;, &quot;tea&quot;, &quot;a&quot;] → Target: &quot;lot&quot;</li></ul><p>This forces the model to learn <strong>next-token prediction</strong>, the heart of LLM training.</p><h3>Converting Tokens to Vectors</h3><p>Tokens alone are just integers. To make them meaningful, they’re mapped into <strong>embeddings</strong> — dense vectors the model can work with.</p><p>So the pipeline becomes:<br> <strong>text → tokens → token IDs → embeddings → model layers.</strong></p><h3>My Reflection</h3><p>Before this chapter, I thought of tokenization as a preprocessing afterthought. Now I see it’s <strong>where you decide how your model perceives language.</strong></p><ul><li>Word-level is naive.</li><li>Characters are brute force.</li><li>BPE is smart and flexible.</li><li>The sliding window makes training tractable.</li></ul><p>The biggest mindset shift for me: <strong>tokenization isn’t neutral.</strong> The choice literally shapes the model’s “view of the world.”</p><p>If building an LLM is like teaching someone to read, then Chapter 2 is the alphabet lesson — deciding what counts as a “letter.”</p><h3>What’s Next</h3><p>Chapter 3 is where it gets exciting: <strong>Attention mechanisms.</strong> This is the engine that lets models decide which words matter in context.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=615c2323c619" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an LLM From Scratch: My Learning Journey]]></title>
            <link>https://medium.com/@lakshaya32/building-an-llm-from-scratch-my-learning-journey-c53a002fe36b?source=rss-51467d705cd5------2</link>
            <guid isPermaLink="false">https://medium.com/p/c53a002fe36b</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[building-llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Lakshay Arora]]></dc:creator>
            <pubDate>Tue, 16 Sep 2025 00:45:44 GMT</pubDate>
            <atom:updated>2025-09-16T00:45:44.965Z</atom:updated>
            <content:encoded><![CDATA[<h3>Chapter 1 — So What Are LLMs Anyway?</h3><p>I just started reading Sebastian Raschka’s <em>Build a Large Language Model (From Scratch)</em>. Honestly, I was tempted to skip the intro chapter and dive straight into the code, but I’m glad I didn’t. Chapter 1 is basically a reality-check.</p><h3>Not Magic, Just Math (Sorry)</h3><p>We love to talk about ChatGPT or Claude or Gemini or any other AI-assistant like they’re these digital brainiacs. A simple answer would be: nope. At their core, LLMs are just… next-word guessers.</p><p>They take a huge chunk of text, chop it into little tokens, and try to predict the most likely next token. That’s it. No “thinking”, no hidden wisdom. Just a giant probability engine on steroids.</p><p>And yet — because the models are so massive (billions of parameters trained on oceans of data), the output <em>feels</em> magical. It’s autocomplete turned into something you can have a conversation with.</p><h3>The Three Stages That Matter</h3><p>Something I honestly hadn’t thought about much: not all training is the same. There are three big phases that make these things useful:</p><ul><li><strong>Pretraining</strong>: Feed the model a ridiculous amount of text and teach it to predict the next word. Super broad, super general.</li><li><strong>Fine-tuning</strong>: Narrow it down. “Okay, now specialize in sentiment analysis” or “classify medical notes.”</li><li><strong>Instruction-tuning</strong>: This is the big unlock — teaching the model to <em>listen to prompts</em>. Without this, an LLM is just babbling probabilities. With it, you get something that acts like ChatGPT.</li></ul><p>I never really separated those in my head before. I just thought “training” was one big process. Turns out, skipping the last stage means you don’t get a chatbot… you just get a massive autocomplete machine.</p><h3>The Black-Box Weirdness</h3><p>Here’s the part that stuck with me: even though we design every layer and code up all the pieces, these models are still <strong>black boxes</strong>.</p><p>Like, sure, you can look at attention maps and visualize what tokens are “looking at” each other. But can anyone tell you exactly <em>why</em> a model gave one output over another? Not really.</p><p>And that’s kinda unsettling. We’re scaling these systems bigger and bigger, and even the people who build them can’t fully explain why they behave the way they do.</p><h3>Why I’m Glad I Didn’t Skip This Chapter</h3><p>I’ll be real — part of me wanted to rush to the code and start hacking attention layers. But Chapter 1 forced me to stop and get my mental model straight.</p><p>Because if you don’t actually understand what an LLM is <em>supposed</em> to be, everything else becomes hype. You start throwing around “transformer” and “fine-tuning” without actually knowing what you’re talking about.</p><p>My main takeaway?</p><blockquote>LLMs aren’t smart. They’re mirrors. What you get out depends on the data and design choices you put in.</blockquote><p>That’s equal parts exciting and terrifying.</p><h3>What’s Next</h3><p>Next up is Chapter 2: Tokenization. I’ve always treated tokenization like a boring preprocessing step. Now I’m starting to suspect it’s actually the bedrock of the whole system.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c53a002fe36b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>