<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ravishanker Mnit on Medium]]></title>
        <description><![CDATA[Stories by Ravishanker Mnit on Medium]]></description>
        <link>https://medium.com/@ravishanker.mnit?source=rss-b087c43f503b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*V8Wb9oiF02-5GEx7</url>
            <title>Stories by Ravishanker Mnit on Medium</title>
            <link>https://medium.com/@ravishanker.mnit?source=rss-b087c43f503b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 23:59:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ravishanker.mnit/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Article 2 : Inside a GPT Block: Attention, Position, and Stable Training]]></title>
            <link>https://pub.towardsai.net/article-2-inside-a-gpt-block-attention-position-and-stable-training-811db886581c?source=rss-b087c43f503b------2</link>
            <guid isPermaLink="false">https://medium.com/p/811db886581c</guid>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[rope]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[deepseek]]></category>
            <category><![CDATA[build-llm-from-scratch]]></category>
            <dc:creator><![CDATA[Ravishanker Mnit]]></dc:creator>
            <pubDate>Mon, 02 Nov 2026 12:31:01 GMT</pubDate>
            <atom:updated>2026-11-02T12:31:01.789Z</atom:updated>
            <content:encoded><![CDATA[<p>When the original Transformer architecture was introduced in the seminal 2017 paper <em>“Attention Is All You Need”</em>, it was designed as a dual-component system consisting of an <strong>Encoder</strong> and a <strong>Decoder</strong>. The encoder processed the input text in parallel, while the decoder generated the output text token-by-token, relying on a complex sub-layer called <em>Cross-Attention</em> to look back at the encoder’s data.</p><p>Modern Large Language Models (LLMs) like GPT, Llama, Mistral, and DeepSeek have stripped away this complexity. They utilize a streamlined, <strong>pure decoder-only architecture</strong>. By removing the encoder and the cross-attention mechanisms entirely, the model is optimized for a single, highly efficient task: predicting the next token in a sequence.</p><p>Today, when engineers talk about a “Transformer block” or a “GPT block,” they are referring to this modern decoder-only variant.</p><p>A decoder-only language model stacks dozens or hundreds of these blocks. Each individual block takes a sequence of vectors, gives tokens a chance to exchange information through self-attention, and then processes each token’s updated representation independently through a feed-forward network.</p><p>In a small teaching model, we choose an embedding width <strong>d_model = 128</strong>. This represents the number of dimensions in which a token has been passed through a trained embedding model to get its vector representation. In layman’s terms, this means we attempt to comprehend 128 different features of a given token (which is not always a word or a character). We then use four attention heads (<strong>heads = 4</strong>), which parallelizes learning across four threads. This allows each head to learn its own set of weights independently.</p><p>Once these features are learned, they are combined back together to pass through a feed-forward network, which first expands and then later contracts the representation to yield a final output of the exact same size. In a nutshell, we pass an embedding through these learning blocks to interpret its context relative to all other tokens in the sequence. In a production model, these architectural choices vary significantly; the numbers provided here are concrete examples rather than a universal recipe.</p><p>1. <strong>High School Concept: Matrix Foundations &amp; Projections</strong></p><p>Before exploring the core mechanics, let us review a foundational mathematical concept: <strong>a matrix represents data arranged in rows and columns</strong>.</p><p>In an LLM, the input data matrix <em>X</em> has dimensions of <strong>(Context Length × d_model)</strong>. For a production workload, this could mean a matrix of <strong>32,000 × 512</strong>, where 32,000 is the context length (number of tokens) and 512 is the input embedding size.</p><p>Matrix multiplication requires that the <strong>columns of the first matrix match the rows of the second matrix</strong>. Multiplying a matrix of size <em>(A × B)</em> by a matrix of size <em>(B × C)</em> yields a new matrix of size <em>(A × C)</em>.</p><p>For each embedding representation <em>X</em>, the layer applies learned linear projections to generate the Queries (<em>Q</em>), Keys (<em>K</em>), and Values (<em>V</em>):</p><blockquote><strong><em>Q = X · Wq</em></strong><em><br></em><strong><em>K = X · Wk</em></strong><em><br></em><strong><em>V = X · Wv</em></strong></blockquote><p>Here, <em>X</em> is our input embedding matrix, which is multiplied by the learned weight matrices <em>Wq</em>, <em>Wk</em>, and <em>Wv</em>. These weights are initialized randomly (using frameworks like PyTorch via torch.tensor(requires_grad=True)) and are optimized during training via backpropagation.</p><p><strong><em>Concrete Mathematical Example</em></strong></p><p>Let our sequence length be 4 tokens and <strong>d_model = 4</strong>. Let the input matrix <em>X</em> be:</p><pre> X= [ 1  0  1  2 ]<br>    [ 0  2  1  0 ]<br>    [ 1  1  0  1 ]<br>    [ 2  0  2  1 ]</pre><p>Assume a learned query projection matrix <em>Wq</em> of size (4 × 4) looks like this:</p><pre>     [ 1   0  1  0 ]<br>Wq = [ 0   1  0  1 ]<br>     [ 1  -1  0  0 ]<br>     [ 0   1  1 -1 ]</pre><p>To find the Query matrix <em>Q</em>, we compute <em>X × Wq</em>. For example:</p><p><strong>Row 1 Calculation (Token 1)</strong></p><blockquote><strong>Col 1:</strong> (1·1) + (0·0) + (1·1) + (2·0) = 1 + 0 + 1 + 0 = <strong>2</strong></blockquote><blockquote><strong>Col 2:</strong> (1·0) + (0·1) + (1·(-1)) + (2·1) = 0 + 0–1 + 2 = <strong>1</strong></blockquote><blockquote><strong>Col 3:</strong> (1·1) + (0·0) + (1·0) + (2·1) = 1 + 0 + 0 + 2 = <strong>3</strong></blockquote><blockquote><strong>Col 4:</strong> (1·0) + (0·1) + (1·0) + (2·(-1)) = 0 + 0 + 0–2 = <strong>-2</strong></blockquote><blockquote><strong>Result for Row 1:</strong> [ 2 1 3 -2 ]</blockquote><p><strong>Row 2 Calculation (Token 2)</strong></p><blockquote><strong>Col 1:</strong> (0·1) + (2·0) + (1·1) + (0·0) = 0 + 0 + 1 + 0 = <strong>1</strong></blockquote><blockquote><strong>Col 2:</strong> (0·0) + (2·1) + (1·(-1)) + (0·1) = 0 + 2–1 + 0 = <strong>1</strong></blockquote><blockquote><strong>Col 3:</strong> (0·1) + (2·0) + (1·0) + (0·1) = 0 + 0 + 0 + 0 = <strong>0</strong></blockquote><blockquote><strong>Col 4:</strong> (0·0) + (2·1) + (1·0) + (0·(-1)) = 0 + 2 + 0 + 0 = <strong>2</strong></blockquote><blockquote><strong>Result for Row 2:</strong> [ 1 1 0 2 ]</blockquote><p><strong>Row 3 Calculation (Token 3)</strong></p><blockquote><strong>Col 1:</strong> (1·1) + (1·0) + (0·1) + (1·0) = 1 + 0 + 0 + 0 = <strong>1</strong></blockquote><blockquote><strong>Col 2:</strong> (1·0) + (1·1) + (0·(-1)) + (1·1) = 0 + 1 + 0 + 1 = <strong>2</strong></blockquote><blockquote><strong>Col 3:</strong> (1·1) + (1·0) + (0·0) + (1·1) = 1 + 0 + 0 + 1 = <strong>2</strong></blockquote><blockquote><strong>Col 4:</strong> (1·0) + (1·1) + (0·0) + (1·(-1)) = 0 + 1 + 0–1 = <strong>0</strong></blockquote><blockquote><strong>Result for Row 3:</strong> [ 1 2 2 0 ]</blockquote><p><strong>Row 4 Calculation (Token 4)</strong></p><blockquote><strong>Col 1:</strong> (2·1) + (0·0) + (2·1) + (1·0) = 2 + 0 + 2 + 0 = <strong>4</strong></blockquote><blockquote><strong>Col 2:</strong> (2·0) + (0·1) + (2·(-1)) + (1·1) = 0 + 0–2 + 1 = <strong>-1</strong></blockquote><blockquote><strong>Col 3:</strong> (2·1) + (0·0) + (2·0) + (1·1) = 2 + 0 + 0 + 1 = <strong>3</strong></blockquote><blockquote><strong>Col 4:</strong> (2·0) + (0·1) + (2·0) + (1·(-1)) = 0 + 0 + 0–1 = <strong>-1</strong></blockquote><blockquote><strong>Result for Row 4:</strong> [ 4 -1 3 -1 ]</blockquote><p>Performing this calculation across all rows yields the final projected matrix:</p><pre>    [ 2  1  3 -2 ]<br>Q = [ 1  1  0  2 ]<br>    [ 1  2  2  1 ]<br>    [ 4  1  3 -1 ]</pre><p>Initially, all weight matrices are populated with small random values near zero. Over the course of training, they learn to capture distinct features. The three transformed vectors represent three different functional aspects of the same input:</p><ul><li><strong>Query (Q):</strong> Asks what this token position is looking for.</li><li><strong>Key (K):</strong> Describes what this token position offers to the context.</li><li><strong>Value (V):</strong> Carries the actual semantic information to be combined.</li></ul><p>Visualizing Projection Dimensions</p><pre>Input Matrix X             Weight Matrix W_q          Query Matrix Q<br>[Context x d_model]    ×    [d_model x d_out]    =    [Context x d_out]<br>┌─────────────────┐         ┌───────────────┐         ┌───────────────┐<br>│     32,000      │         │      512      │         │    32,000     │<br>│        ×        │         │       ×       │         │       ×       │<br>│       512       │         │     4,096     │         │     4,096     │<br>└─────────────────┘         └───────────────┘         └───────────────┘</pre><p>2. <strong>The Core Mechanics: Scaled Dot-Product Attention &amp; Causal Masking</strong></p><p>The attention mechanism allows token embeddings to calculate their mutual similarities across the provided context window. For a single head, the mathematical equation is defined as:</p><blockquote><strong><em>Attention(Q, K, V) = Softmax( (Q · Kᵀ / √d_head) + Causal Mask ) · V</em></strong></blockquote><p>Breaking Down the Math Step-by-Step</p><ol><li><strong>Similarity Matrix (Q · Kᵀ):</strong> If <em>Q</em> is of size (32,000 × 4,096), it is multiplied by the transpose of the Key matrix <em>Kᵀ</em>, which has dimensions of (4,096 × 32,000). This matrix multiplication results in an attention score matrix of size <strong>(32,000 × 32,000)</strong>, mapping the raw relationship score between <strong>every pair </strong>of positions.</li><li><strong>Scaling by √d_head:</strong> As the head dimension grows larger, the dot products grow larger in magnitude. This pushes the softmax function into regions with extremely small gradients. Dividing by <strong>√d_head</strong> scales these values down, stabilizing the gradient flow during backpropagation.</li><li><strong>Why Causal Masking is Required:</strong> In an autoregressive decoder, the model must not look into the future. When predicting the next word, it should only have access to current and past tokens. To enforce this condition, a causal mask is added to the scaled scores before applying the softmax function. This mask sets all upper-triangular positions (representing future tokens) to negative infinity (-∞). Since e^(-∞) = 0, the softmax operation assigns a probability of exactly 0 to those future tokens.</li></ol><pre>Raw Attention Matrix (QKᵀ)     After Causal Masking (-∞)      Post-Softmax Probabilities<br>     ┌───────────┐                   ┌───────────┐                   ┌───────────┐<br>     │ 12  4  8  │                   │ 12 -∞ -∞  │                   │1.0 0.0 0.0│<br>     │  3  9  2  │        ───&gt;       │  3  9 -∞  │        ───&gt;       │.002.998 0 │<br>     │  7  6 11  │                   │  7  6 11  │                   │.02 .01 .97│<br>     └───────────┘                   └───────────┘                   └───────────┘</pre><p><strong>Softmax Layer as a Probability Distribution</strong></p><p>Applying the softmax function guarantees that the weights across each row sum exactly to <strong>1.0</strong>, turning them into a valid probability distribution.</p><p>For instance, at token position 4, the model might assign the weights <strong>[0.05, 0.15, 0.70, 0.10]</strong> to positions 1 through 4. The final output vector for position 4 is computed as a weighted mixture of those historical value vectors:</p><blockquote><strong><em>Output_4 = 0.05·V₁ + 0.15·V₂ + 0.70·V₃ + 0.10·V₄</em></strong></blockquote><p><strong>3. PyTorch Code: Building the Attention Mechanisms From Scratch</strong></p><p>Below is the complete, self-contained implementation of the four major attention variants used across the industry.</p><p>python</p><pre>import torch<br>import torch.nn as nn<br>import torch.nn.functional as F<br>class MultiHeadAttention(nn.Module):<br>    def __init__(self, d_model, num_heads):<br>        super().__init__()<br>        assert d_model % num_heads == 0, &quot;d_model must be divisible by num_heads&quot;<br>        self.d_model = d_model<br>        self.num_heads = num_heads<br>        self.d_head = d_model // num_heads<br>        <br>        # MHA allocates distinct W_q, W_k, W_v for every single head<br>        self.W_q = nn.Linear(d_model, d_model, bias=False)<br>        self.W_k = nn.Linear(d_model, d_model, bias=False)<br>        self.W_v = nn.Linear(d_model, d_model, bias=False)<br>        self.out_proj = nn.Linear(d_model, d_model, bias=False)<br>        <br>    def forward(self, x):<br>        B, T, C = x.shape  # Batch size, Sequence length, Embedding dimension<br>        # 1. Project the input tensor x through the learned weight matrix (W_q)<br>        # 2. Reshape (view) the flat dimensions into separate parallel attention heads<br>        # 3. Swap (transpose) dimensions so the &#39;heads&#39; dimension comes before the &#39;sequence length&#39; (T)<br>        # This allows us to perform fast, parallel matrix multiplication across all batches and heads at once.<br>        q = self.W_q(x).view(B, T, self.num_heads, self.d_head).transpose(1, 2)<br>        k = self.W_k(x).view(B, T, self.num_heads, self.d_head).transpose(1, 2)<br>        v = self.W_v(x).view(B, T, self.num_heads, self.d_head).transpose(1, 2)<br>        <br>        # Multiplying q and K^T and dividing by sqrt(d_head)<br>        scores = (q @ k.transpose(-2, -1)) / (self.d_head ** 0.5)<br>        # Applying causal mask so that it does not learn from future tokens <br>        # 1. Create a square matrix of size (T, T) completely filled with negative infinity (-inf).<br>        # 2. Extract the upper triangular part (triu) of this matrix, starting 1 slot above the main diagonal.<br>        # 3. The zeros on the lower half allow past tokens to talk, while -inf blocks future tokens.<br>        mask = torch.triu(torch.full((T, T), float(&#39;-inf&#39;), device=x.device), diagonal=1)<br>        scores = scores + mask<br>        <br>        attn_weights = F.softmax(scores, dim=-1)<br>        <br>        # 1. Multiply attention weights by values (v) to get context vectors for each head<br>        # 2. Swap &#39;heads&#39; and &#39;sequence length&#39; (T) back to the original order<br>        # 3. Force PyTorch to arrange the data sequentially in memory (contiguous)<br>        # 4. Collapse the parallel head dimensions back into the single combined channel (C)<br>        context = (attn_weights @ v).transpose(1, 2).contiguous().view(B, T, C)<br>        return self.out_proj(context)<br><br># Another technique where Query heads together so they can share a single Key and Value head. <br>class GroupedQueryAttention(nn.Module):<br>    def __init__(self, d_model, num_query_heads, num_kv_groups):<br>        super().__init__()<br>        self.num_query_heads = num_query_heads<br>        # The number of distinct Key and Value heads. <br>        # Instead of matching the number of Query heads, we restrict this <br>        # to a smaller number to compress our memory footprint.<br>        self.num_kv_groups = num_kv_groups<br>        self.d_head = d_model // num_query_heads<br>        self.queries_per_group = num_query_heads // num_kv_groups<br>        <br>        self.W_q = nn.Linear(d_model, num_query_heads * self.d_head, bias=False)<br>        self.W_k = nn.Linear(d_model, num_kv_groups * self.d_head, bias=False)<br>        self.W_v = nn.Linear(d_model, num_kv_groups * self.d_head, bias=False)<br>        self.out_proj = nn.Linear(d_model, d_model, bias=False)<br>        <br>    def forward(self, x):<br>        B, T, _ = x.shape<br>        q = self.W_q(x).view(B, T, self.num_query_heads, self.d_head).transpose(1, 2)<br>        k = self.W_k(x).view(B, T, self.num_kv_groups, self.d_head).transpose(1, 2)<br>        v = self.W_v(x).view(B, T, self.num_kv_groups, self.d_head).transpose(1, 2)<br>        <br>        # Duplicates (clones) each Key head sequentially across the head dimension (dim=1).<br>        # This expands the shared Key matrix so it matches the exact shape of the Query matrix,<br>        # allowing us to perform fast, parallel batch matrix multiplication.<br>        k = k.repeat_interleave(self.queries_per_group, dim=1)<br>        v = v.repeat_interleave(self.queries_per_group, dim=1)<br>        <br>        scores = (q @ k.transpose(-2, -1)) / (self.d_head ** 0.5)<br>        mask = torch.triu(torch.full((T, T), float(&#39;-inf&#39;), device=x.device), diagonal=1)<br>        scores = scores + mask<br>        <br>        attn_weights = F.softmax(scores, dim=-1)<br>        context = (attn_weights @ v).transpose(1, 2).contiguous().view(B, T, -1)<br>        return self.out_proj(context)</pre><p><strong>DeepSeek’s Multi-head Latent Attention (MLA)</strong></p><p><strong>Multi-head Latent Attention (MLA)</strong> compresses the Key-Value cache during generation. Instead of caching large KV matrices, MLA compresses keys and values into a low-dimensional latent vector vector during inference. This reduces memory traffic overhead, speeding up generation times across long contexts.</p><p><strong>4. The Geometry of Space: Sinusoidal vs. Rotary Positional Embeddings (RoPE)</strong></p><p>Because self-attention processes tokens in parallel, it is inherently permutation-invariant. Without explicit positional context, the sequence &quot;Dog eats fish&quot; maps to the exact same attention presentation as &quot;Fish eats dog&quot;.</p><ul><li><strong>Sinusoidal Encodings:</strong> In the original 2017 Transformer paper, the authors faced a core dilemma: <strong>Attention treats tokens like a bag of words with no inherent order.</strong> To fix this, they introduced <strong>Sinusoidal Positional Encodings</strong> — a method that assigns a unique, mathematically calculated vector to each position in a sentence using sine and cosine waves.</li><li>Instead of forcing the model to <em>learn</em> positions during training, they hard-coded a fixed geometric blueprint.</li><li>The Intuition: The Structural Clock</li><li>Think of sinusoidal encoding like a high-precision binary clock or an odometer, where each slot moves at a different speed:</li><li>The first few dimensions fluctuate incredibly fast (like seconds).</li><li>The middle dimensions change moderately (like minutes).</li><li>The last dimensions change incredibly slowly (like hours).</li></ul><p>By looking at the combined values across all dimensions, the model can instantly tell exactly what “time” (position) it is, and exactly how far apart two tokens are.</p><p>The Mathematical Formula</p><p>For a token at position <strong>pos</strong> and a feature column index <strong>i</strong> inside the embedding :</p><blockquote><strong><em>If the index is Even (2i):</em></strong><em> PE(pos, 2i) = sin( pos / 10000^(2i / d_model) )<br></em><strong><em>If the index is Odd (2i+1):</em></strong><em> PE(pos, 2i+1) = cos( pos / 10000^(2i / d_model) )</em></blockquote><p>The reason <strong>i</strong> only goes up to 1 for an embedding size of 4 is because <strong>every single value of <em>i</em> populates two positions at the same time</strong> — one even position using Sine, and one odd position using Cosine.</p><p><strong>Step-by-Step Concrete Example</strong></p><p>Let’s calculate the positional embedding vector for the first two tokens in a sentence using a small teaching setup where <strong>d_model = 4</strong> (four dimensions).</p><p>Our frequency base denominator for the dimensions is calculated as:</p><ul><li>For the first pair of dimensions (i=0): 10000^(0/4) = 1</li><li>For the second pair of dimensions (i=1): 10000^(2/4) = √10000 = 100</li></ul><p><strong>1. Calculating Token at Position 0 (First Word)</strong></p><ul><li><strong>Dim 0 (Even, i=0):</strong> sin(0 / 1) = sin(0) = <strong>0.0</strong></li><li><strong>Dim 1 (Odd, i=0):</strong> cos(0 / 1) = cos(0) = <strong>1.0</strong></li><li><strong>Dim 2 (Even, i=1):</strong> sin(0 / 100) = sin(0) = <strong>0.0</strong></li><li><strong>Dim 3 (Odd, i=1):</strong> cos(0 / 100) = cos(0) = <strong>1.0</strong></li></ul><blockquote><strong><em>Position 0 Vector:</em></strong><em> </em><em>[ 0.0, 1.0, 0.0, 1.0 ]</em></blockquote><p><strong>2. Calculating Token at Position 1 (Second Word)</strong></p><ul><li><strong>Dim 0 (Even, i=0):</strong> sin(1 / 1) = sin(1 radian) ≈ <strong>0.841</strong></li><li><strong>Dim 1 (Odd, i=0):</strong> cos(1 / 1) = cos(1 radian) ≈ <strong>0.540</strong></li><li><strong>Dim 2 (Even, i=1):</strong> sin(1 / 100) = sin(0.01 radians) ≈ <strong>0.010</strong></li><li><strong>Dim 3 (Odd, i=1):</strong> cos(1 / 100) = cos(0.01 radians) ≈ <strong>0.999</strong></li></ul><blockquote><strong><em>Position 1 Vector:</em></strong><em> </em><em>[ 0.841, 0.540, 0.010, 0.999 ]</em></blockquote><p>How the Model Uses It</p><p>Once these fixed vectors are calculated, they are <strong>directly added</strong> to the token’s semantic word embeddings:</p><p>text</p><pre>[ Word Embedding ]  +  [ Positional Vector ]  =  [ Final Input Vector ]<br>(Carries Word Meaning)    (Carries Token Order)     (Carries Meaning + Order)</pre><p>Use code with caution.</p><p>Because sine and cosine follow strict linear rotation properties, the attention mechanism can easily calculate the dot product between two tokens and determine their distance. For example, the relationship between position <em>pos</em> and position <em>pos + k</em> can be completely broken down as a linear function of the offset <em>k</em>.</p><p><strong>Why Modern Models Moved Away From It</strong></p><p>While beautiful, sinusoidal encodings are <strong>absolute</strong> — they attach a specific coordinate to a specific slot. Modern architectures prefer <strong>Rotary Position Embeddings (RoPE)</strong> because RoPE injects this positioning directly <em>into the attention scores as relative angles</em>, making it vastly superior at handling dynamically scaled, ultra-long context windows.</p><ul><li><strong>Rotary Position Embeddings (RoPE):</strong> Instead of adding absolute vectors to the embeddings, RoPE treats pairs of coordinates in the Query and Key vectors as 2D complex numbers, rotating them in the complex plane by a position-dependent angle.</li></ul><p>RoPE Geometric Intuition Diagram</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*E0DPehkjbqEcOPwcvL0FnQ.png" /></figure><p>A 2D vector coordinate pair [q_0, q_1] is rotated by an angle dependent on its exact sequence position <em>m</em>:</p><p>Mathematically, the rotation matrix for a single 2D coordinate pair at position \(m\) is written as:</p><pre>$$<br>R_{\theta, m}^{2} = \begin{pmatrix} \cos(m\theta) &amp; -\sin(m\theta) \\ \sin(m\theta) &amp; \cos(m\theta) \end{pmatrix}</pre><p>When we calculate the dot product between a shifted Query vector at position \(m\) and a shifted Key vector at position \(n\), the absolute positions drop out. The remaining calculation depends entirely on their <strong>relative distance</strong> \((m — n)\):</p><p>\((R_{m}Q)\cdot (R_{n}K)=Q^{T}R_{m}^{T}R_{n}K=Q^{T}R_{n-m}K\)</p><p>This formulation allows the network to naturally generalize to long context lengths during inference.</p><p>When we calculate the dot product between a shifted Query vector at position <em>m</em> and a shifted Key vector at position <em>n</em>, the absolute positions drop out. The remaining calculation depends entirely on their <strong>relative distance (m — n)</strong>. This formulation allows the network to naturally generalize to long context lengths during inference.</p><p><strong>5. Stable Deep Stacking: LayerNorm vs. RMSNorm</strong></p><p>Normalization stabilizes internal activation scales across deep network architectures.</p><ul><li><strong>LayerNorm</strong> centers the input activations by subtracting their mean, and then scales them using their variance.</li><li><strong>RMSNorm</strong> operates under the premise that centering the data by subtracting the mean is computationally redundant. It simplifies the operation by scaling exclusively by the Root Mean Square, saving significant overhead on hardware.</li></ul><p>Numerical Walkthrough Comparison</p><p>Let our hidden activation vector be <strong>x = [1.0, 2.0, 3.0]</strong>.</p><p><strong>LayerNorm Calculations:</strong></p><ol><li>Mean = (1 + 2 + 3) / 3 = <strong>2.0</strong></li><li>Centered Vector (x — Mean) = <strong>[-1.0, 0.0, 1.0]</strong></li><li>Variance = ( (-1)² + ⁰² + ¹² ) / 3 = <strong>0.666</strong></li><li>Output (Divided by √Variance) = <strong>[-1.22, 0.0, 1.22]</strong></li></ol><p><strong>RMSNorm Calculations:</strong></p><ol><li>Mean Square = (¹² + ²² + ³²) / 3 = (1 + 4 + 9) / 3 = <strong>4.666</strong></li><li>Root Mean Square (RMS) = √4.666 = <strong>2.16</strong></li><li>Output (x / RMS) = [1/2.16, 2/2.16, 3/2.16] = <strong>[0.46, 0.93, 1.39]</strong></li></ol><p>6. Regularization and Nonlinearities: Dropout, SwiGLU, and Feed-Forward Networks</p><p>Once the attention layer completes its calculation, the token representations pass through a position-wise Feed-Forward Network (FFN).</p><p><strong>Why and How Dropout Operates</strong></p><p><strong>Dropout</strong> acts as a regularizer during training by randomly setting a fraction <em>p</em> of hidden activations to exactly 0.0. If <em>p</em> = 0.1, roughly 10% of the neurons are disabled on each forward pass. This prevents co-adaptation of features, ensuring the model does not overfit to specific training patterns. During inference and evaluation, dropout is completely disabled.</p><p><strong>Gated Nonlinearities: SwiGLU</strong></p><p>While classic architectures relied on standard activations like ReLU or GELU, modern frontier networks use <strong>SwiGLU (Swish Gated Linear Units)</strong>. SwiGLU uses a gated mechanism where one linear layer modulates the output of another:</p><blockquote><strong><em>SwiGLU(x) = Swish(x · W_gate) ⊙ (x · W_up) · W_down</em></strong></blockquote><p>Where <strong>⊙</strong> represents an element-wise multiplication. This gating mechanism allows the model to control information flow dynamically, determining which features to pass forward down the network stack.</p><p><strong>7. PyTorch Block Construction: Putting It All Together From Scratch</strong></p><p>Below is the complete implementation of a modern, production-grade Pre-LN Transformer decoder block using RMSNorm, Grouped-Query Attention, and SwiGLU nonlinearities.</p><p>python</p><pre>class RMSNorm(nn.Module):<br>    def __init__(self, dim, eps=1e-6):<br>        super().__init__()<br>        self.eps = eps<br>        self.gamma = nn.Parameter(torch.ones(dim))<br>    def forward(self, x):<br>        rms = torch.rsqrt(x.pow(2).mean(-1, keepdim=True) + self.eps)<br>        return x * rms * self.gamma<br>class SwiGLUFFN(nn.Module):<br>    def __init__(self, d_model, d_ffn):<br>        super().__init__()<br>        self.W_gate = nn.Linear(d_model, d_ffn, bias=False)<br>        self.W_up = nn.Linear(d_model, d_ffn, bias=False)<br>        self.W_down = nn.Linear(d_ffn, d_model, bias=False)<br>    def forward(self, x):<br>        # Element-wise multiplication of the Swish-gated branch and the up-projection branch<br>        return self.W_down(F.silu(self.W_gate(x)) * self.W_up(x))<br>class CompleteGPTBlock(nn.Module):<br>    def __init__(self, d_model, num_heads, num_kv_groups, d_ffn, dropout_p=0.1):<br>        super().__init__()<br>        self.attn_norm = RMSNorm(d_model)<br>        self.attn = GroupedQueryAttention(d_model, num_heads, num_kv_groups)<br>        self.attn_dropout = nn.Dropout(dropout_p)<br>        <br>        self.ffn_norm = RMSNorm(d_model)<br>        self.ffn = SwiGLUFFN(d_model, d_ffn)<br>        self.ffn_dropout = nn.Dropout(dropout_p)<br>    def forward(self, x):<br>        # Clean Pre-LN Residual Highway Layout<br>        x = x + self.attn_dropout(self.attn(self.attn_norm(x)))<br>        x = x + self.ffn_dropout(self.ffn(self.ffn_norm(x)))<br>        return x<br># Instantiate a single complete block matching our architectural layout<br>if __name__ == &quot;__main__&quot;:<br>    example_input = torch.randn(2, 64, 128)  # Batch=2, Sequence=64 tokens, d_model=128<br>    block = CompleteGPTBlock(d_model=128, num_heads=4, num_kv_groups=2, d_ffn=384)<br>    output = block(example_input)<br>    print(&quot;Execution successful! Output shape matches input tensor exactly:&quot;, output.shape)</pre><h3>When you read a model card</h3><p>Look for the specific choices it actually discloses: positional method, normalization, attention variant, feed-forward function, dense versus mixture-of-experts layers, and context length. Do not infer these from a brand name. Some commercial model vendors publish extensive model cards but not the full weight architecture.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=811db886581c" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/article-2-inside-a-gpt-block-attention-position-and-stable-training-811db886581c">Article 2 : Inside a GPT Block: Attention, Position, and Stable Training</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A Language Model, Built One Token at a Time]]></title>
            <link>https://pub.towardsai.net/a-language-model-built-one-token-at-a-time-b35095094725?source=rss-b087c43f503b------2</link>
            <guid isPermaLink="false">https://medium.com/p/b35095094725</guid>
            <dc:creator><![CDATA[Ravishanker Mnit]]></dc:creator>
            <pubDate>Wed, 28 Oct 2026 13:01:04 GMT</pubDate>
            <atom:updated>2026-10-31T03:25:48.602Z</atom:updated>
            <content:encoded><![CDATA[<h4>From Transformer to GPT: the next token becomes the training signal</h4><p>The first surprise about a large language model is how small its training instruction sounds: <strong>given the tokens so far, predict the next token</strong>.</p><p>Consider the sentence:</p><blockquote><em>The little robot opened the ___</em></blockquote><p>The model does not receive the answer “door” as a special label attached to the whole sentence. During training, it sees the prefix and learns a probability distribution for the next token. The desired continuation might be door, but other continuations can also be plausible. The model’s job is to put more probability on the training continuation over many examples.</p><p>This simple objective became unusually powerful when combined with the Transformer, large text collections, and enough compute.</p><h3>A short route from sequence models to GPT</h3><p>Before Transformers, a common pattern for translation and other sequence tasks used an encoder to read the source and a decoder to produce the answer. Recurrent networks processed tokens in order. That ordering was natural, but it limited parallel work across a long sentence: position 20 depended on the hidden state from position 19.</p><p>In 2017, <em>Attention Is All You Need</em> introduced the Transformer: attention layers and position-wise feed-forward layers, with residual connections and normalization. The original system was an encoder–decoder model for translation, not the decoder-only GPT architecture. Its key practical advantage was that the training computation for many positions could be organized in parallel. (<a href="https://arxiv.org/abs/1706.03762">Attention Is All You Need</a>)</p><p>The Transformer idea later appeared in three broad shapes. <strong>Encoder-only</strong> models such as BERT build contextual representations for understanding tasks. <strong>Encoder–decoder</strong> models such as T5 read an input sequence and generate a transformed output, which fits translation and summarization naturally. <strong>Decoder-only</strong> models such as GPT predict the next token and can continue any prompt. These are useful patterns, not rigid boundaries: modern systems can wrap a decoder in multimodal encoders, routers, tools, or other modules. (<a href="https://arxiv.org/abs/1810.04805">BERT</a>; <a href="https://arxiv.org/abs/1910.10683">T5</a>)</p><p>The GPT idea took the Transformer decoder and used it as a <strong>generative pre-training</strong> model. Train on a broad text corpus with a causal objective, then adapt the resulting representations to tasks. GPT-2 demonstrated how a larger text-trained model could perform several tasks in a zero-shot way; GPT-3 showed strong few-shot behavior when examples were included in the prompt. These are milestones in one research line, not a claim that every modern assistant has the same architecture or training recipe. (<a href="https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf">GPT-1 paper</a>; <a href="https://openai.com/index/better-language-models/">GPT-2 announcement</a>; <a href="https://arxiv.org/abs/2005.14165">GPT-3 paper</a>)</p><p>The progression is roughly:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ym5Q1yBIsnwoaOhQU9HrjQ.png" /></figure><h3>What does “causal” mean here?</h3><p>In ordinary self-attention, every position could read every other position. A language model that predicts the next token must not peek at the future answer. A <strong>causal mask</strong> blocks that information.</p><p>For four token positions, a simplified visibility mask is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*glHthfTawIQIUtYsvdieNQ.png" /></figure><p><strong>Figure 2.</strong> Each position can read its own token and preceding tokens; future tokens are masked.</p><p>The model can train all four predictions in one forward pass because the mask prevents leakage. Position 1 predicts token 2, position 2 predicts token 3, and so on. At generation time, it starts with a prompt and repeatedly samples or selects one next token.</p><h3>Text is converted to token IDs first</h3><p>Neural networks operate on numbers, so a tokenizer turns text into integer IDs. A token can be a whole word, a word fragment, a punctuation mark, or a byte-level unit. For example, a tokenizer might represent unbelievable as two or three pieces rather than one whole-word entry.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ByRGuTtf-9Vfzr_jJIy8Jw.png" /><figcaption><strong>Figure 3.</strong> Training compares each position’s logits with its next-token label.</figcaption></figure><p>The tiktoken package is an implementation of byte-pair-encoding tokenizers used with OpenAI model families. It can encode text and decode token IDs; it does <strong>not</strong> train a new tokenizer from your corpus. If you are truly creating a new model family, tokenizer design and vocabulary training are a separate step. For a compact experiment, reusing a public tokenizer is easier and lets us focus on the model. (<a href="https://github.com/openai/tiktoken">OpenAI tiktoken repository</a>)</p><pre>import tiktoken<br><br>enc = tiktoken.get_encoding(&quot;gpt2&quot;)<br>text = &quot;The little robot opened the door.&quot;<br>ids = enc.encode(text)<br>print(ids)<br>print(enc.decode(ids))</pre><p>The returned IDs are not word numbers with human meaning. They are indices into an embedding table. The model learns a vector for each ID. If the vocabulary has about 50,000 entries and the embedding width is 128, the token table alone has about 50,000 × 128 = 6.4 million learned values.</p><h3>The training target is shifted by one</h3><p>If token IDs are [A, B, C, D], training pairs are:</p><pre>input:  [A, B, C]<br>target: [B, C, D]</pre><p>t each position, the model produces scores (logits) over the entire vocabulary. A softmax converts scores into probabilities. Cross-entropy compares those probabilities with the correct next-token IDs. If the correct token has probability 0.30, its negative log probability is approximately -ln(0.30) = 1.20 nats. Training tries to lower this loss across many positions and batches.</p><p>The objective is not “write a perfect answer.” It is “predict the data well.” That difference explains why a base model can continue text fluently but still fail to follow a user’s request reliably. Assistant behavior is built in later stages.</p><h3>The corpus is part of the model</h3><p>Before tokenization, a training pipeline usually collects documents, removes malformed or duplicate material, applies quality and language filters, handles personal or sensitive information under the project’s data policy, and checks rights and provenance. It then tokenizes and packs examples into training sequences. Evaluation data should be held out by document or source before windows are sampled; randomly splitting adjacent token windows can put near-identical text on both sides of the test.</p><p>These choices affect what the model learns. More text is not automatically better if it contains duplicates, low-quality templates, contamination from evaluation answers, or material the project should not use.</p><h3>What to remember</h3><ul><li>GPT is a family name for generative pre-trained Transformer systems; it does not mean every current model is one plain decoder stack.</li><li>A causal mask lets a model learn next-token prediction without reading future tokens.</li><li>Tokenization is a learned or designed mapping from text fragments to IDs; token IDs then map to vectors.</li><li>Pre-training gives broad statistical capability. It does not, by itself, guarantee helpfulness, factuality, or safe tool use.</li></ul><p><strong>Next:</strong> inside one decoder block, including why position encoding, normalization, and residual paths matter.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b35095094725" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/a-language-model-built-one-token-at-a-time-b35095094725">A Language Model, Built One Token at a Time</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>