<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Hiru Dewmi on Medium]]></title>
        <description><![CDATA[Stories by Hiru Dewmi on Medium]]></description>
        <link>https://medium.com/@h.u.dewmi?source=rss-fe5e9e3bda19------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*OenopR0zA964uDk4d4GXJA.jpeg</url>
            <title>Stories by Hiru Dewmi on Medium</title>
            <link>https://medium.com/@h.u.dewmi?source=rss-fe5e9e3bda19------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 11:19:53 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@h.u.dewmi/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Implementing a Vision Transformer from Scratch with PyTorch]]></title>
            <link>https://medium.com/@h.u.dewmi/implementing-a-vision-transformer-from-scratch-with-pytorch-9791b9546585?source=rss-fe5e9e3bda19------2</link>
            <guid isPermaLink="false">https://medium.com/p/9791b9546585</guid>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[vision-transformer]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Hiru Dewmi]]></dc:creator>
            <pubDate>Mon, 14 Sep 2026 02:36:01 GMT</pubDate>
            <atom:updated>2026-09-14T02:36:01.424Z</atom:updated>
            <content:encoded><![CDATA[<h3>Building ViT Step by Step: Patch Embeddings, Multi-Head Attention, Transformer Blocks, and Image Classification</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FaJfiCpACF-G5wmXuQP7OA.png" /><figcaption>Understanding Vision Transformer architecture through a simple PyTorch code implementation</figcaption></figure><p>In my previous article, <a href="https://medium.com/@h.u.dewmi/vision-transformers-explained-a-comprehensive-technical-guide-a0273814de78?sharedUserId=h.u.dewmi"><strong>Vision Transformers Explained: A Comprehensive Technical Guide</strong></a>, I discussed how the Vision Transformer (ViT) changed computer vision by treating an image as a sequence of patches and processing those patches using a Transformer architecture.</p><p>Understanding the theory is one thing. Building the model yourself is another.</p><p>In this article, we’ll move from theory to practice and implement a <strong>Vision Transformer from scratch using PyTorch</strong>.</p><p>By the end, we will have built the complete pipeline:</p><pre>Input Image<br>      ↓<br>Patch Embedding<br>      ↓<br>Add Class Token<br>      ↓<br>Add Positional Embeddings<br>      ↓<br>Transformer Encoder × N<br>      ↓<br>Layer Normalization<br>      ↓<br>Classification Head<br>      ↓<br>Predicted Class</pre><p>We’ll implement the major components step by step:</p><ul><li>Patch Embedding</li><li>Multi-Head Self-Attention</li><li>MLP Block</li><li>Transformer Encoder Block</li><li>Class Token</li><li>Positional Embeddings</li><li>Complete Vision Transformer</li></ul><p>The goal is not to reproduce every optimization used in large production ViTs. Instead, we’ll build a clean implementation that makes the architecture easy to understand.</p><h3>Code Implementation</h3><p>The complete PyTorch implementation is available on GitHub:</p><p>👉 <a href="https://github.com/HiruDewmi/vision-transformer-from-scratch-pytorch.git"><strong><em>Vision Transformer from Scratch with PyTorch — GitHub</em></strong></a></p><p>The notebook includes the complete implementation, training on CIFAR-10, evaluation, and visualization of the results.</p><h3>1. Imports</h3><p>Let’s start with the required libraries.</p><pre>import torch<br>import torch.nn as nn</pre><p>We’ll use PyTorch nn.Module to define each component of the Vision Transformer.</p><h3>2. Understanding the Input</h3><p>A standard image tensor in PyTorch has the shape:</p><pre>(B, C, H, W)</pre><p>where:</p><ul><li><strong>B</strong> = batch size</li><li><strong>C</strong> = number of image channels</li><li><strong>H</strong> = image height</li><li><strong>W</strong> = image width</li></ul><p>For example, a batch of RGB images with resolution 224 × 224 might have the shape:</p><pre>(B, 3, 224, 224)</pre><p>The original Vision Transformer commonly uses:</p><pre>Image size: 224 × 224<br>Patch size: 16 × 16</pre><p>This divides the image into:</p><pre>14 × 14 = 196 patches</pre><p>Each patch will eventually become a token for the Transformer.</p><h3>3. Patch Embedding</h3><p>The first step is to convert an image into a sequence of patch embeddings.</p><p>Conceptually, the process looks like this:</p><pre>Image<br>224 × 224<br>      ↓<br>Split into 16 × 16 patches<br>      ↓<br>196 patches<br>      ↓<br>Linear Projection<br>      ↓<br>196 embedding vectors</pre><p>We can implement this efficiently using a convolution.</p><pre>class PatchEmbed(nn.Module):<br>  def __init__(<br>        self,<br>        img_size=224,<br>        patch_size=16,<br>        in_chans=3,<br>        embed_dim=768<br>    ):<br>        super().__init__()<br>        assert img_size % patch_size == 0, \<br>            &quot;Image size must be divisible by patch size&quot;<br>        self.n_patches = (img_size // patch_size) ** 2<br>        self.proj = nn.Conv2d(<br>            in_channels=in_chans,<br>            out_channels=embed_dim,<br>            kernel_size=patch_size,<br>            stride=patch_size<br>        )<br>    def forward(self, x):<br>        x = self.proj(x)<br>        # (B, D, H/P, W/P)<br>        x = x.flatten(2)<br>        # (B, D, N)<br>        x = x.transpose(1, 2)<br>        # (B, N, D)<br>        return x</pre><h3>Why Use Conv2d for Patch Embedding?</h3><p>This might initially seem strange.</p><p>We’re building a Transformer, so why are we using a convolution?</p><p>Using Conv2d with <em>kernel_size = patch_size</em> and <em>stride = patch_size</em> is functionally equivalent to extracting non-overlapping patches, flattening each patch, and applying the same learned linear projection to every patch.</p><p>Because:</p><pre>kernel_size = patch_size<br>stride = patch_size</pre><p>the convolution creates non-overlapping patches.</p><p>For an input:</p><pre>(B, 3, 224, 224)</pre><p>with:</p><pre>patch_size = 16<br>embed_dim = 768</pre><p>the shapes become:</p><pre>(B, 3, 224, 224)<br>↓<br>(B, 768, 14, 14)<br>        ↓ flatten<br>(B, 768, 196)<br>        ↓ transpose<br>(B, 196, 768)<br><br><br>Now we have a sequence of 196 tokens, each with an embedding dimension of 768.</pre><h3>4. Multi-Head Self-Attention</h3><p>Self-attention is the core mechanism behind Transformers.</p><p>Instead of using PyTorch’s built-in nn.MultiheadAttention, we&#39;ll implement the mechanism ourselves so that we can understand what happens internally.</p><h3>Queries, Keys, and Values</h3><pre>class MultiHeadAttention(nn.Module):<br>    def __init__(<br>        self,<br>        dim,<br>        heads=8,<br>        dropout=0.0<br>    ):<br>        super().__init__()<br>        assert dim % heads == 0, \<br>            &quot;Embedding dimension must be divisible by number of heads&quot;<br>        self.heads = heads<br>        self.head_dim = dim // heads<br>        self.scale = self.head_dim ** -0.5<br>        # Combined Query, Key, Value projection<br>        self.qkv = nn.Linear(<br>            dim,<br>            dim * 3<br>        )<br>        self.proj = nn.Linear(<br>            dim,<br>            dim<br>        )<br>        self.dropout = nn.Dropout(<br>            dropout<br>        )<br>    def forward(self, x):<br>        B, N, D = x.shape<br>        # Create Q, K and V<br>        qkv = self.qkv(x)<br>        # (B, N, 3 * D)<br>        qkv = qkv.reshape(<br>            B,<br>            N,<br>            3,<br>            self.heads,<br>            self.head_dim<br>        )<br>        # (B, N, 3, heads, head_dim)<br>        qkv = qkv.permute(<br>            2,<br>            0,<br>            3,<br>            1,<br>            4<br>        )<br>        # (3, B, heads, N, head_dim)<br>        q, k, v = qkv[0], qkv[1], qkv[2]<br>        # Compute attention scores<br>        attention = (<br>            q @ k.transpose(-2, -1)<br>        ) * self.scale<br>        # Normalize attention scores<br>        attention = attention.softmax(<br>            dim=-1<br>        )<br>        attention = self.dropout(<br>            attention<br>        )<br>        # Apply attention weights to values<br>        x = attention @ v<br>        # Combine attention heads<br>        x = x.transpose(1, 2)<br>        x = x.reshape(<br>            B,<br>            N,<br>            D<br>        )<br>        # Final projection<br>        x = self.proj(x)<br>        return x</pre><h3>5. Understanding the Attention Shapes</h3><p>This is often the most confusing part of implementing Transformers.</p><p>Suppose we have:</p><pre>Batch size = 32<br>Tokens = 197<br>Embedding dimension = 768<br>Heads = 12</pre><p>Each attention head receives:</p><pre>768 / 12 = 64 dimensions</pre><p>After splitting into heads:</p><pre>Q, K, V</pre><pre>(32, 12, 197, 64)</pre><p>The attention matrix becomes:</p><pre>Q × Kᵀ</pre><pre>(32, 12, 197, 197)</pre><p>This means that every token, including the class token and image patches, can compute relationships with every other token in the sequence.</p><p>For each image:</p><pre>Patch 1  → attends to all patches<br>Patch 2  → attends to all patches<br>Patch 3  → attends to all patches<br>...<br>Patch 196 → attends to all patches</pre><p>This global interaction is one of the major differences between Vision Transformers and CNNs.</p><h3>6. The MLP Block</h3><p>After self-attention, every Transformer block contains a feed-forward neural network.</p><p>In ViT, this is typically:</p><pre>Linear<br>   ↓<br>GELU<br>   ↓<br>Dropout<br>   ↓<br>Linear<br>   ↓<br>Dropout</pre><p>The hidden dimension is usually expanded by a factor of four.</p><p>For example:</p><pre>768 → 3072 → 768</pre><p>Let’s implement it as a separate module.</p><pre>class MLP(nn.Module):<br>  def __init__(<br>        self,<br>        dim,<br>        mlp_ratio=4.0,<br>        dropout=0.0<br>    ):<br>        super().__init__()<br>        hidden_dim = int(<br>            dim * mlp_ratio<br>        )<br>        self.net = nn.Sequential(<br>            nn.Linear(<br>                dim,<br>                hidden_dim<br>            ),<br>            nn.GELU(),<br>            nn.Dropout(<br>                dropout<br>            ),<br>            nn.Linear(<br>                hidden_dim,<br>                dim<br>            ),<br>            nn.Dropout(<br>                dropout<br>            )<br>        )<br>    def forward(self, x):<br>        return self.net(x)</pre><h3>7. Building the Transformer Encoder Block</h3><p>Now we can combine:</p><ul><li>Layer Normalization</li><li>Multi-Head Self-Attention</li><li>Residual Connections</li><li>MLP</li></ul><p>The architecture is:</p><pre>Input<br>   │<br>   ▼<br>LayerNorm<br>   │<br>   ▼<br>Multi-Head Attention<br>   │<br>   ▼<br>Residual Addition<br>   │<br>   ▼<br>LayerNorm<br>   │<br>   ▼<br>MLP<br>   │<br>   ▼<br>Residual Addition</pre><p>This is called a <strong>pre-normalization Transformer block</strong>.</p><pre>class EncoderBlock(nn.Module):<br>  def __init__(<br>        self,<br>        dim,<br>        heads,<br>        mlp_ratio=4.0,<br>        dropout=0.0<br>    ):<br>        super().__init__()<br>        self.norm1 = nn.LayerNorm(<br>            dim<br>        )<br>        self.attn = MultiHeadAttention(<br>            dim=dim,<br>            heads=heads,<br>            dropout=dropout<br>        )<br>        self.norm2 = nn.LayerNorm(<br>            dim<br>        )<br>        self.mlp = MLP(<br>            dim=dim,<br>            mlp_ratio=mlp_ratio,<br>            dropout=dropout<br>        )<br>    def forward(self, x):<br>        # Self-attention with residual connection<br>        x = x + self.attn(<br>            self.norm1(x)<br>        )<br>        # MLP with residual connection<br>        x = x + self.mlp(<br>            self.norm2(x)<br>        )<br>        return x</pre><h3>8. Adding the Class Token</h3><p>The Transformer needs a way to produce a single representation for the entire image.</p><p>ViT solves this by introducing a learnable <strong>class token</strong>.</p><p>Before the image patches are passed into the Transformer:</p><pre>Patch 1<br>Patch 2<br>Patch 3<br>...<br>Patch 196</pre><p>we prepend:</p><pre>[CLS]</pre><p>The sequence becomes:</p><pre>[CLS]<br>Patch 1<br>Patch 2<br>...<br>Patch 196</pre><p>After passing through the Transformer, the final representation of the class token is used for classification.</p><p>We define it as:</p><pre>self.cls_token = nn.Parameter(<br>    torch.zeros(1, 1, embed_dim)<br>)<br><br><br>nn.init.trunc_normal_(<br>    self.cls_token,<br>    std=0.02<br>)<br><br></pre><p>During the forward pass, it is repeated for every image in the batch.</p><h3>9. Positional Embeddings</h3><p>Transformers do not inherently understand spatial position.</p><p>Without positional information, these two sequences are indistinguishable:</p><pre>Patch A → Patch B → Patch C</pre><p>and:</p><pre>Patch C → Patch A → Patch B</pre><p>To preserve spatial information, ViT adds learnable positional embeddings.</p><pre>self.pos_embed = nn.Parameter(<br>    torch.zeros(<br>        1,<br>        n_patches + 1,<br>        embed_dim<br>    )<br>)<br><br>nn.init.trunc_normal_(<br>    self.pos_embed,<br>    std=0.02<br>)</pre><p>The +1 represents the class token.</p><p>The positional embeddings are simply added to the token embeddings:</p><pre>x = x + self.pos_embed</pre><h3>10. Building the Complete Vision Transformer</h3><p>Now let’s combine everything.</p><pre>class VisionTransformer(nn.Module):<br>  def __init__(<br>        self,<br>        img_size=224,<br>        patch_size=16,<br>        in_chans=3,<br>        num_classes=1000,<br>        embed_dim=768,<br>        depth=12,<br>        heads=12,<br>        mlp_ratio=4.0,<br>        dropout=0.0<br>    ):<br>        super().__init__()<br>        # Patch embedding<br>        self.patch_embed = PatchEmbed(<br>            img_size=img_size,<br>            patch_size=patch_size,<br>            in_chans=in_chans,<br>            embed_dim=embed_dim<br>        )<br>        n_patches = self.patch_embed.n_patches<br>        # Class token<br>        self.cls_token = nn.Parameter(<br>            torch.zeros(<br>                1,<br>                1,<br>                embed_dim<br>            )<br>        )<br>        # Positional embeddings<br>        self.pos_embed = nn.Parameter(<br>            torch.zeros(<br>                1,<br>                n_patches + 1,<br>                embed_dim<br>            )<br>        )<br>        self.pos_drop = nn.Dropout(<br>            dropout<br>        )<br>        # Transformer encoder blocks<br>        self.blocks = nn.ModuleList([<br>            EncoderBlock(<br>                dim=embed_dim,<br>                heads=heads,<br>                mlp_ratio=mlp_ratio,<br>                dropout=dropout<br>            )<br>            for _ in range(depth)<br>        ])<br>        # Final normalization<br>        self.norm = nn.LayerNorm(<br>            embed_dim<br>        )<br>        # Classification head<br>        self.head = nn.Linear(<br>            embed_dim,<br>            num_classes<br>        )<br>    def forward(self, x):<br>        B = x.shape[0]<br>        # Patch embeddings<br>        x = self.patch_embed(x)<br>        # (B, N, D)<br>        # Expand class token for batch<br>        cls_tokens = self.cls_token.expand(<br>            B,<br>            -1,<br>            -1<br>        )<br>        # Add class token<br>        x = torch.cat(<br>            (cls_tokens, x),<br>            dim=1<br>        )<br>        # Add positional embeddings<br>        x = x + self.pos_embed<br>        x = self.pos_drop(x)<br>        # Transformer encoder<br>        for block in self.blocks:<br>            x = block(x)<br>        # Final normalization<br>        x = self.norm(x)<br>        # Extract class token<br>        cls_token = x[:, 0]<br>        # Classification<br>        x = self.head(cls_token)<br>        return x</pre><p>And that’s it.</p><p>We now have a complete Vision Transformer.</p><h3>11. Creating a ViT-Base Style Model</h3><p>The original ViT-Base configuration uses approximately:</p><pre>Image Size:      224<br>Patch Size:      16<br>Embedding Dim:   768<br>Depth:           12<br>Attention Heads: 12<br>MLP Ratio:       4</pre><p>We can create a similar architecture using:</p><pre>model = VisionTransformer(<br>    img_size=224,<br>    patch_size=16,<br>    in_chans=3,<br>    num_classes=1000,<br>    embed_dim=768,<br>    depth=12,<br>    heads=12,<br>    mlp_ratio=4.0<br>)</pre><p>Let’s print the model.</p><pre>print(model)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*1YL0AkeD85VeYh1wo2IR3w.png" /><figcaption>The complete Vision Transformer architecture printed from the PyTorch model, showing its main components and layer structure.</figcaption></figure><h3>12. Testing the Model</h3><p>Let’s create a batch of random images.</p><pre>x = torch.randn(<br>    8,<br>    3,<br>    224,<br>    224<br>)</pre><p>Then perform a forward pass.</p><pre>output = model(x)</pre><pre>print(output.shape)</pre><p>The expected output is:</p><pre>torch.Size([8, 1000])</pre><p>This means:</p><pre>8 images<br>×<br>1000 class predictions</pre><p>Each row contains the classification logits for one image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zAzhsaUiqfCymDQxehokfA.png" /><figcaption>Prediction results on CIFAR-10 test dataset using a small Vit model</figcaption></figure><h3>13. Following the Tensor Through the Model</h3><p>Let’s summarize the tensor transformations.</p><p>Starting with:</p><pre>(B, 3, 224, 224)</pre><h4>Patch Embedding</h4><pre>(B, 196, 768)</pre><h4>Add Class Token</h4><pre>(B, 197, 768)</pre><h4>Add Positional Embeddings</h4><pre>(B, 197, 768)</pre><h4>Transformer Encoder</h4><pre>(B, 197, 768)</pre><p>The shape remains unchanged through every Transformer block.</p><h4>Extract Class Token</h4><pre>(B, 768)</pre><h4>Classification Head</h4><pre>(B, 1000)</pre><p>The complete pipeline looks like this:</p><pre>(B, 3, 224, 224)<br>        ↓ Patch Embedding<br>(B, 196, 768)<br>        ↓ Add CLS Token<br>(B, 197, 768)<br>        ↓ Add Position Embeddings<br>(B, 197, 768)<br>        ↓ Transformer Blocks × 12<br>(B, 197, 768)<br>        ↓ Extract CLS Token<br>(B, 768)<br>        ↓ Classification Head<br>(B, 1000)</pre><h3>14. Comparing This Implementation to the Original ViT</h3><p>Our implementation captures the core architecture of the Vision Transformer:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/576/1*_-WZ-XcJnP9UABUCy3HIng.png" /></figure><p>However, this is still a simplified educational implementation.</p><p>Production-grade implementations often include additional features such as:</p><ul><li>Stochastic Depth</li><li>Advanced weight initialization</li><li>Attention dropout</li><li>Layer-wise learning-rate decay</li><li>Mixed-precision training</li><li>Gradient checkpointing</li><li>Positional embedding interpolation</li><li>Efficient attention implementations</li></ul><p>Libraries such as timm also provide highly optimized Vision Transformer implementations.</p><p>The purpose here is to understand the architecture rather than reproduce every engineering optimization.</p><h3>15. A Small Practical Model for Experimentation</h3><p>Training ViT-Base from scratch requires significant computational resources and large amounts of data.</p><p>For experimentation, we can create a smaller Vision Transformer.</p><pre>small_vit = VisionTransformer(<br>    img_size=32,<br>    patch_size=4,<br>    in_chans=3,<br>    num_classes=10,<br>    embed_dim=192,<br>    depth=6,<br>    heads=3,<br>    mlp_ratio=4.0<br>)</pre><p>This configuration is more suitable for datasets such as:</p><ul><li>CIFAR-10</li><li>CIFAR-100</li><li>Small custom image datasets</li></ul><p>The input would be:</p><pre>x = torch.randn(<br>    32,<br>    3,<br>    32,<br>    32<br>)</pre><pre>output = small_vit(x)</pre><pre>print(output.shape)</pre><p>Expected output:</p><pre>torch.Size([32, 10])</pre><h3>16. Important Practical Considerations</h3><p>Implementing the architecture is only the first step.</p><p>Training a Vision Transformer successfully requires careful consideration.</p><h3>Data Size Matters</h3><p>Unlike CNNs, ViTs have relatively weak built-in assumptions about image structure.</p><p>Training a large ViT from scratch on a small dataset can easily lead to poor performance.</p><p>This is why techniques such as:</p><ul><li>Data augmentation</li><li>Transfer learning</li><li>Knowledge distillation</li><li>Self-supervised pretraining</li></ul><p>became important in Transformer-based vision research.</p><h3>Patch Size Matters</h3><p>Patch size directly affects the number of tokens.</p><p>For a 224 × 224 image:</p><h3>Patch Size = 32</h3><pre>7 × 7 = 49 patches</pre><h3>Patch Size = 16</h3><pre>14 × 14 = 196 patches</pre><h3>Patch Size = 8</h3><pre>28 × 28 = 784 patches</pre><p>Smaller patches preserve more spatial detail but dramatically increase attention computation.</p><p>Since self-attention compares tokens with each other, the computational cost grows approximately with [N²], where <strong>N</strong> is the number of patches.</p><h3>17. What Makes ViT Different from a CNN?</h3><p>A CNN processes information locally.</p><p>A small convolution sees only a small neighborhood.</p><p>Global information emerges gradually as layers are stacked.</p><p>A Vision Transformer works differently.</p><p>From the first attention layer:</p><pre>Patch A ↔ Patch B<br>Patch A ↔ Patch C<br>Patch A ↔ Patch D<br>...</pre><p>Every patch can directly interact with every other patch.</p><p>This ability to model global relationships is one of the defining characteristics of Vision Transformers.</p><h3>Final Thoughts</h3><p>Building a Vision Transformer from scratch makes the architecture much easier to understand.</p><p>What initially looks like a complicated model is actually built from a small number of components:</p><ol><li><strong>Convert the image into patches</strong></li><li><strong>Project patches into embeddings</strong></li><li><strong>Add positional information</strong></li><li><strong>Process the sequence using Transformer blocks</strong></li><li><strong>Use the class token for classification</strong></li></ol><p>The complete architecture can be summarized as:</p><pre>Image<br>  ↓<br>Patch Embedding<br>  ↓<br>Class Token + Position Embeddings<br>  ↓<br>Multi-Head Self-Attention<br>  +<br>MLP<br>  ↓<br>Repeat Transformer Blocks<br>  ↓<br>Final Normalization<br>  ↓<br>Classification Head</pre><p>The real power of Vision Transformers comes from the simplicity and scalability of this idea.</p><p>Once an image becomes a sequence of embeddings, the same fundamental Transformer architecture that revolutionized natural language processing can be applied to visual data.</p><p>And that idea has gone far beyond image classification.</p><p>Modern computer vision systems now build on this foundation for:</p><ul><li>Object Detection</li><li>Semantic Segmentation</li><li>Self-Supervised Learning</li><li>Vision-Language Models</li><li>Image Generation</li><li>Multimodal AI</li></ul><p>Understanding how to implement a basic ViT is therefore an excellent starting point for exploring the broader ecosystem of modern Transformer-based vision models.</p><h3>Companion Research Notebook</h3><p>To make this implementation easier to explore and reproduce, I have also included the accompanying research notebook.</p><p>The notebook contains the complete Vision Transformer implementation, including patch embedding, multi-head self-attention, Transformer encoder blocks, positional embeddings, and the final classification model.</p><p>It also provides the code needed to experiment with the model and better understand how the different components work together.</p><p>Feel free to explore, modify, and experiment with the implementation.</p><p>Check this out: <a href="https://github.com/HiruDewmi/vision-transformer-from-scratch-pytorch/blob/main/vision_transformer_from_scratch.ipynb"><strong><em>Implementing a Vision Transformer from Scratch with PyTorch , Notebook</em></strong></a></p><h3>Related Reading</h3><p>If you’d like to understand the theory behind the implementation, read my previous article:</p><p><a href="https://medium.com/@h.u.dewmi/vision-transformers-explained-a-comprehensive-technical-guide-a0273814de78?sharedUserId=h.u.dewmi"><strong>Vision Transformers Explained: A Comprehensive Technical Guide</strong></a></p><p>In that article, I explain the mathematical foundations of ViT, including patch embeddings, positional encodings, self-attention, training requirements, and how Vision Transformers compare with CNNs.</p><p><em>If you found this implementation useful, feel free to experiment with the architecture by changing the patch size, embedding dimension, number of attention heads, and Transformer depth. One of the best ways to understand Vision Transformers is to build them and then break them to see how each design decision affects the model.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9791b9546585" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vision Transformers Explained: A Comprehensive Technical Guide]]></title>
            <link>https://medium.com/@h.u.dewmi/vision-transformers-explained-a-comprehensive-technical-guide-a0273814de78?source=rss-fe5e9e3bda19------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0273814de78</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[vision-transformer]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Hiru Dewmi]]></dc:creator>
            <pubDate>Tue, 08 Sep 2026 04:39:59 GMT</pubDate>
            <atom:updated>2026-09-09T05:37:49.315Z</atom:updated>
            <content:encoded><![CDATA[<p>For nearly a decade, convolutional neural networks (CNNs) were the undisputed default architecture for computer vision. AlexNet (2012), VGG, ResNet, and EfficientNet: each generation improved on the last, but all shared the same core assumption: images should be processed with small, sliding convolutional filters that exploit locality and translation invariance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kYbB0PcdtkTOydmN7AKZaw.png" /></figure><p>In October 2020, a team at Google Research/Brain challenged that assumption with a paper titled <em>“An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”</em> (Dosovitskiy et al., ICLR 2021). The idea was almost provocatively simple: cut an image into patches, treat each patch like a “word” (token), and feed the sequence into a standard Transformer encoder — the same architecture behind BERT and GPT — with essentially no vision-specific modifications. The result, the <strong>Vision Transformer (ViT)</strong>, matched or beat state-of-the-art CNNs on image classification when pretrained on enough data, while using substantially less compute to reach that performance.</p><p>This article walks through ViT’s architecture end-to-end, the mathematics of self-attention as applied to images, the training dynamics that make it work (or fail), the major variants and descendants that followed (DeiT, Swin, BEiT, MAE, ViT-22B, CLIP), and its practical role in computer vision today. Facts, figures, and benchmark numbers below have been checked against the original papers and their abstracts/tables.</p><h3><strong>Why Transformers for Vision? The Core Motivation</strong></h3><p>Transformers were originally designed for sequences of discrete tokens (words). Images are continuous 2D grids of pixels. Two obstacles stood in the way of a naive transplant:</p><ol><li><strong>Quadratic cost of self-attention.</strong> Standard self-attention computes pairwise interactions between every token. If each <em>pixel</em> were a token, a 224×224 image would produce 50,176 tokens, and full self-attention would cost roughly the square of that, computationally intractable.</li><li><strong>No spatial inductive bias.</strong> CNNs bake in two assumptions: <em>locality</em> (nearby pixels are related) and <em>translation equivariance</em> (a feature detector that works in one part of the image works everywhere). Transformers have neither built in; they must learn spatial structure from data.</li></ol><p>ViT’s solution to problem (1) is patchification: instead of pixels, the atomic unit is a 16×16 (or 14×14, 32×32) patch. This shrinks a 224×224 image to just 196 tokens (14×14 patches), making standard self-attention computationally feasible.</p><p>Problem (2) is not “solved” so much as <em>accepted as a tradeoff</em>: ViT deliberately discards the convolutional inductive bias, betting that with enough training data, the model can learn locality and spatial relationships on its own — and learn representations that are less constrained by hand-designed priors than a CNN’s. As the original paper put it, this is a discouraging setup at moderate data scale but becomes an advantage at large scale, because “large scale training trumps inductive bias.”</p><h3>Architecture Walkthrough</h3><ol><li><strong>Patch Embedding</strong></li></ol><p>Given an input image x ∈ R^(H×W×C), ViT splits it into N non-overlapping square patches of size P×P, where N = (H×W)/P². Each patch is flattened into a vector of length P²·C (e.g., 16×16×3 = 768 for RGB images) and passed through a single trainable linear projection into the model&#39;s hidden dimension D:</p><pre>z0 = [x_class; x_p1·E; x_p2·E; ...; x_pN·E] + E_pos</pre><p>Where E is the learned patch-embedding matrix (equivalent to a convolution with kernel size and stride both equal to P — in practice, implementations often literally use a Conv2d(kernel=P, stride=P) for this step, which is mathematically identical to flattening then linear projecting).</p><p><strong>2</strong>. <strong>The Class Token</strong></p><p>Borrowing directly from BERT’s [CLS] token, ViT prepends a learnable embedding x_class to the sequence of patch embeddings. After passing through the Transformer encoder, the final-layer state corresponding to this token serves as the aggregate image representation, fed into a classification head (a small MLP) during pretraining, and typically replaced by a task-specific head during fine-tuning.</p><p><strong>3. Position Embeddings</strong></p><p>Since self-attention is permutation-invariant (it has no inherent notion of order or spatial position), ViT adds a learned 1D position embedding to each patch embedding. Notably, the original paper found that more sophisticated 2D-aware position embeddings gave no measurable benefit over simple learned 1D embeddings — the model apparently learns 2D spatial relationships on its own from the 1D embeddings during training (the paper visualizes this: nearby patches learn similar position embeddings, and row/column structure emerges automatically).</p><p><strong>4. The Transformer Encoder</strong></p><p>The sequence z0 (197 tokens for a 14×14-patch, 224×224 image: 196 patches + 1 class token) is passed through L identical Transformer encoder blocks. Each block consists of:</p><ul><li><strong>LayerNorm (LN)</strong></li><li><strong>Multi-Head Self-Attention (MSA)</strong></li><li><strong>Residual connection</strong></li><li><strong>LayerNorm (LN)</strong></li><li><strong>MLP block</strong> (two linear layers with a GELU nonlinearity in between, typically expanding to 4× the hidden dimension and back)</li><li><strong>Residual connection</strong></li></ul><p>Formally, for a block l:</p><pre>z&#39;_l = MSA(LN(z_(l-1))) + z_(l-1)<br>z_l  = MLP(LN(z&#39;_l)) + z&#39;_l</pre><p>This is the <strong>pre-norm</strong> formulation (LayerNorm before the sublayer, not after), a detail that matters for training stability at depth, and one ViT adopted from later Transformer practice rather than the original 2017 Transformer paper, which used post-norm.</p><p><strong>5. Multi-Head Self-Attention (the mathematical core)</strong></p><p>For each attention head, the input sequence z (shape N×D) is linearly projected into queries, keys, and values:</p><pre>Q = zW_Q,  K = zW_K,  V = zW_V</pre><p>Attention weights are computed as scaled dot-product similarity between queries and keys, normalized with softmax:</p><pre>Attention(Q,K,V) = softmax(QK^T / √d_k) · V</pre><p>The √d_k scaling factor (where d_k is the per-head key dimension) prevents dot products from growing large in magnitude as dimensionality increases, which would push softmax into regions with vanishing gradients.</p><p>Multiple heads run this operation in parallel with separate learned projections, and their outputs are concatenated and linearly projected back to dimension D. Intuitively, each head can specialize; some heads in trained ViTs attend very locally (similar to a small convolutional receptive field), while others attend globally across the entire image from the very first layer, a capability CNNs don&#39;t have without many stacked layers or explicit global-pooling tricks.</p><p><strong>6. Model Variants (Original Paper)</strong></p><p>The original paper defined three sizes, deliberately mirroring BERT’s Base/Large naming, plus a new larger “Huge” variant:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/544/1*zwc4MwWayjCgvrhbVKzNIQ.png" /><figcaption>Table 1. Original Vision Transformer configurations. As model size increases from ViT-Base to ViT-Huge, the number of layers, hidden dimensions, and parameters grow significantly, enabling higher representational capacity at the cost of increased computational requirements.</figcaption></figure><p>Naming convention: ViT-L/16 means the ViT-Large configuration with 16×16 patches; ViT-H/14 uses 14×14 patches (more patches → longer sequence → more compute, generally better accuracy).</p><h3>Training Regime: Why Data Scale Is Everything</h3><p>This is the single most important empirical finding of the ViT paper, and it’s worth stating precisely: <strong>when trained from scratch on ImageNet-1k alone (~1.3M images) without heavy regularization, ViT underperforms comparably-sized ResNets by a few percentage points.</strong> The lack of convolutional inductive bias hurts when data is limited.</p><p>The picture reverses at scale. The paper pretrained models on:</p><ul><li><strong>ImageNet-21k</strong>: ~14 million images, 21,000 classes</li><li><strong>JFT-300M</strong>: Google’s in-house dataset of ~300 million noisily labeled images</li></ul><p>When pretrained on JFT-300M and then fine-tuned on ImageNet, the best model (ViT-H/14) reached <strong>88.55% top-1 accuracy on ImageNet</strong>, 90.72% on the cleaned ImageNet-ReaL labels, 94.55% on CIFAR-100, and 77.63% average across the 19-task VTAB suite — at the time, matching or exceeding the best CNN-based results (BiT-L, based on large ResNets), while requiring roughly <strong>2–4× less pretraining compute</strong> on average across benchmarks.</p><p>Other practical training details from the paper and follow-up work:</p><ul><li>Pretraining used <strong>Adam</strong>, not SGD (unusual for vision at the time — CNNs were typically trained with SGD+momentum).</li><li>Fine-tuning at higher resolution than pretraining helps; when resolution changes, the pretrained position embeddings are 2D-interpolated to match the new (larger) patch grid.</li><li>Strong regularization and data augmentation (mixup, RandAugment, dropout, stochastic depth) are essential when training data is limited — this insight directly motivated DeiT.</li></ul><h3>Key Successors and Variants</h3><p>ViT sparked an entire subfield. A few of the most consequential follow-ups:</p><h4>1. DeiT — Data-efficient Image Transformers (Touvron et al., Facebook AI, 2021)</h4><p>DeiT showed that ViT-style models could be trained competitively <strong>on ImageNet-1k alone</strong>, without JFT-300M-scale pretraining, using a strong training recipe (aggressive augmentation, regularization) plus a <strong>distillation token, </strong>an extra token trained to mimic the output of a CNN teacher model via knowledge distillation. This made ViTs practical for researchers without access to Google-scale private datasets and compute.</p><h4>2. Swin Transformer (Liu et al., Microsoft Research, ICCV 2021 Best Paper)</h4><p>Swin reintroduces some CNN-like structure: it computes self-attention within local, non-overlapping <strong>windows</strong> rather than globally (Window Multi-head Self-Attention, W-MSA), then <strong>shifts</strong> the windows in alternating layers (SW-MSA) so information can flow across window boundaries. Combined with a <strong>patch-merging</strong> operation that progressively reduces token count and increases channel depth (mimicking a CNN’s downsampling pyramid), this gives Swin <strong>linear</strong> rather than quadratic computational complexity with respect to image size, and produces the kind of multi-scale hierarchical feature maps that dense prediction tasks (object detection, semantic segmentation) need. Swin became a dominant backbone for detection/segmentation benchmarks for several years, and was later extended by <strong>Swin V2</strong> to scale to much larger models and higher resolutions.</p><h4>3. BEiT (Bao et al., Microsoft, 2021)</h4><p>BEiT applies a BERT-style <strong>masked image modeling</strong> pretraining objective: patches are masked out, and the model must predict discrete visual “tokens” (produced by a separately trained discrete VAE tokenizer) for the masked regions, rather than raw pixels, bringing ViT pretraining conceptually in line with masked-language modeling in NLP.</p><h4>4. MAE — Masked Autoencoders Are Scalable Vision Learners (He et al., Meta AI, 2022)</h4><p>MAE simplified and scaled up masked pretraining considerably. It masks a very high proportion of image patches — typically <strong>75%</strong> — and trains an asymmetric encoder-decoder: a large ViT encoder only ever sees the small subset of <em>visible</em> patches (making pretraining 3–4× cheaper in compute), while a lightweight decoder reconstructs the missing patches’ raw pixel values using a simple mean-squared-error loss. Because the encoder never processes mask tokens, MAE pretraining is both faster and more memory-efficient than earlier masked-modeling approaches, and it became one of the most widely adopted self-supervised pretraining recipes for ViT backbones.</p><h4>5. ViT-22B — Scaling Vision Transformers to 22 Billion Parameters (Dehghani et al., Google, 2023)</h4><p>This work pushed ViT scale roughly two orders of magnitude past ViT-Huge, to 22 billion parameters, requiring several architectural modifications for training stability at that scale (e.g., parallel attention-and-MLP computation, query/key normalization). It demonstrated that ViT’s favorable scaling behavior — first observed at hundreds of millions of parameters — continues to hold at tens of billions, and that larger ViTs continue to improve on fairness/robustness metrics and downstream alignment with human perceptual judgments, not just raw accuracy.</p><h4>6. CLIP — Contrastive Language-Image Pretraining (Radford et al., OpenAI, 2021)</h4><p>CLIP is not a ViT variant per se, but it’s one of the most influential applications of the architecture: a ViT (or ResNet) image encoder and a Transformer text encoder are trained jointly on ~400M image-text pairs scraped from the internet, using a contrastive objective that pulls matching image/text embeddings together and pushes non-matching pairs apart. The resulting ViT image encoder produces representations useful for zero-shot classification and became a foundational building block for later multimodal and generative systems (including diffusion-model text conditioning).</p><h4>7. DINO / DINOv2 (Caron et al., Meta AI)</h4><p>DINO showed that self-distillation with no labels, applied to ViTs, produces attention maps that spontaneously segment objects — without any supervision for segmentation — and that ViT features trained this way transfer exceptionally well as general-purpose visual descriptors. DINOv2 extended this to a larger scale and became a widely used off-the-shelf visual feature extractor.</p><h3>Hybrid Architectures</h3><p>The original ViT paper also tested a <strong>hybrid</strong> variant: instead of raw image patches, the input sequence is formed from feature maps produced by a CNN backbone (e.g., a ResNet’s intermediate layers), with the same patch-embedding-and-Transformer pipeline applied on top. This slightly outperforms pure ViT at small compute budgets, since the CNN stem reintroduces some useful locality bias early on, though the gap closes and largely disappears as model and data scale grow.</p><h3>ViT vs. CNN: A Structured Comparison</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*whFFgFghfGyCEYDT14GyZw.png" /><figcaption>Table 2. Comparison of Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs). The two architectures differ fundamentally in their inductive biases, data requirements, computational characteristics, and scalability, making each suitable for different computer vision scenarios.</figcaption></figure><p>Neither architecture is strictly superior; the practical choice depends on data availability, target hardware, and whether the downstream task needs dense, multi-scale spatial features (favoring hierarchical/Swin-style models) versus a single global representation (favoring plain ViT, especially with strong self-supervised pretraining like MAE or DINO).</p><h3>Applications Beyond Classification</h3><ul><li><strong>Object detection</strong>: ViTDet and DETR-family models adapt ViT backbones (sometimes plain, non-hierarchical ViT with simple feature pyramid adaptations added only at the end) for detection.</li><li><strong>Semantic segmentation</strong>: SegFormer and Swin-based segmentation heads use hierarchical ViT variants for pixel-level dense prediction.</li><li><strong>Self-supervised representation learning</strong>: MAE, DINO/DINOv2, and BEiT-style masked modeling now typically outperform purely supervised ViT pretraining for transfer learning.</li><li><strong>Multimodal / vision-language</strong>: CLIP-style dual encoders and ViT as the vision tower inside vision-language models (e.g., feeding visual tokens into a language-model decoder) are now standard practice.</li><li><strong>Video</strong>: ViT has been extended to spatio-temporal patches (e.g., “tubelets”) for video classification and understanding.</li><li><strong>Generative modeling</strong>: ViT-style patch-based Transformers (as opposed to convolutional U-Nets) are used as the denoising backbone in some modern diffusion models (Diffusion Transformers / DiT), extending the “patches as tokens” idea from discriminative to generative vision models.</li></ul><h3>Limitations and Open Challenges</h3><ul><li><strong>Quadratic attention cost</strong> remains a real constraint for high-resolution images or long video sequences; windowed/hierarchical attention (Swin), linear-attention approximations, and token-merging/pruning methods are active mitigations.</li><li><strong>Data and compute hunger</strong> for training from scratch, though self-supervised pretraining (MAE, DINO) has substantially closed this gap versus supervised-only training.</li><li><strong>Positional generalization</strong>: models trained at one resolution/patch grid need interpolation tricks to generalize to different input sizes.</li><li><strong>Interpretability caveats</strong>: attention maps are visually compelling but are not a fully reliable explanation of model behavior; research on the faithfulness of attention-as-explanation is ongoing and contested, similar to debates in NLP.</li></ul><h3>Conclusion</h3><p>The Vision Transformer’s core insight was less about a clever new mechanism and more about a bet on scale: strip away vision-specific architectural priors, treat an image as a sequence of patches, and let a standard Transformer learn spatial structure from data provided there’s enough data to learn it from. That bet paid off, and the ensuing five years produced a rich ecosystem of variants that either lean into ViT’s strengths (scaling, as in ViT-22B; self-supervised pretraining, as in MAE and DINO) or patch its weaknesses (data efficiency via DeiT; multi-scale structure and compute efficiency via Swin). Today, ViT-based backbones, whether plain, hierarchical, or paired with language encoders as in CLIP, are a standard, often default, choice across image classification, detection, segmentation, self-supervised representation learning, and multimodal systems.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/680/1*k_1lMBq1js0G_kc5j3aEyw.png" /><figcaption>Table 3. Evolution of Vision Transformer architectures and related research from 2020 to 2023. The table highlights key milestones in the development of Transformer-based vision models, including improvements in data-efficient training, hierarchical attention, self-supervised learning, multimodal learning, and large-scale model scaling.</figcaption></figure><p><strong>Want to implement Vision Transformers from scratch?</strong> In my next article, I’ll walk through a practical implementation of a Vision Transformer using Python and PyTorch, including patch embeddings, multi-head self-attention, Transformer encoder blocks, and training.</p><p>👉 <a href="https://medium.com/@h.u.dewmi/practical-implementation-of-vision-transformer-e18edde85247?sharedUserId=h.u.dewmi"><strong>Read the Vision Transformer Implementation Guide</strong></a></p><p>👉 More comprehensive implementation guide: <a href="https://medium.com/@h.u.dewmi/9791b9546585?sharedUserId=h.u.dewmi"><strong><em>Implementing a Vision Transformer from Scratch with PyTorch</em></strong></a></p><h3>Key References</h3><ul><li>Dosovitskiy et al., <em>An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale</em>, ICLR 2021 (arXiv:2010.11929)</li><li>Touvron et al., <em>Training data-efficient image transformers &amp; distillation through attention</em> (DeiT), 2021</li><li>Liu et al., <em>Swin Transformer: Hierarchical Vision Transformer using Shifted Windows</em>, ICCV 2021</li><li>Bao et al., <em>BEiT: BERT Pre-Training of Image Transformers</em>, 2021</li><li>He et al., <em>Masked Autoencoders Are Scalable Vision Learners</em> (MAE), 2022</li><li>Radford et al., <em>Learning Transferable Visual Models From Natural Language Supervision</em> (CLIP), 2021</li><li>Caron et al., <em>Emerging Properties in Self-Supervised Vision Transformers</em> (DINO), 2021</li><li>Dehghani et al., <em>Scaling Vision Transformers to 22 Billion Parameters</em>, 2023</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0273814de78" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Practical Implementation of Vision Transformer]]></title>
            <link>https://medium.com/@h.u.dewmi/practical-implementation-of-vision-transformer-e18edde85247?source=rss-fe5e9e3bda19------2</link>
            <guid isPermaLink="false">https://medium.com/p/e18edde85247</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Hiru Dewmi]]></dc:creator>
            <pubDate>Fri, 04 Sep 2026 09:48:56 GMT</pubDate>
            <atom:updated>2026-09-08T10:26:45.813Z</atom:updated>
            <content:encoded><![CDATA[<p>If you’d like to understand the theory behind the implementation, read my previous article:</p><p><a href="https://medium.com/@h.u.dewmi/vision-transformers-explained-a-comprehensive-technical-guide-a0273814de78?sharedUserId=h.u.dewmi"><strong>Vision Transformers Explained: A Comprehensive Technical Guide</strong></a></p><p>Practical tips drawn from the literature:</p><ul><li><strong>Don’t train plain ViT from scratch on small datasets</strong> without a strong recipe (DeiT-style augmentation/distillation) or self-supervised pretraining (MAE/DINO) — you will likely underperform a ResNet baseline.</li><li><strong>Fine-tuning resolution</strong>: increasing input resolution at fine-tuning time versus pretraining time reliably improves accuracy; remember to bilinearly interpolate the position embeddings to the new patch grid size.</li><li><strong>Patch size tradeoff</strong>: smaller patches (e.g., 14×14 vs 32×32) increase sequence length, giving finer-grained spatial resolution and generally higher accuracy, at roughly quadratic extra attention cost.</li><li>For dense prediction tasks, prefer hierarchical backbones (Swin) or add explicit multi-scale adapters to plain ViT (as in ViTDet) rather than expecting a plain ViT’s single-resolution output to work well out of the box.</li></ul><h3>Implementing a Vision Transformer: Patch Embedding and Encoder Block in PyTorch</h3><p>This is a minimal, illustrative PyTorch implementation of the two core building blocks of a Vision Transformer: the <strong>patch embedding layer</strong> (which turns an image into a sequence of tokens) and a single <strong>Transformer encoder block</strong> (which processes that sequence via self-attention and an MLP). It is meant to make the architecture concrete and readable, not to serve as a drop-in, training-ready model; production implementations (e.g., in timm or torchvision) add things like class tokens, position embeddings, dropout scheduling, weight initialization schemes, and stacking of many blocks.</p><h3>1. Patch Embedding</h3><p>The job of this layer is to take an image of shape (B, C, H, W) and turn it into a sequence of N patch tokens of dimension D, where N = (H/P) × (W/P) for patch size P.</p><p>The cleanest way to do this is with a single strided convolution: a Conv2d with kernel_size = stride = patch_size is mathematically equivalent to slicing the image into non-overlapping P×P patches, flattening each one, and applying a shared linear projection, but it&#39;s faster and more memory-efficient than doing those steps explicitly.</p><pre>import torch<br>import torch.nn as nn<br><br>class PatchEmbed(nn.Module):<br>    &quot;&quot;&quot;Splits an image into non-overlapping patches and linearly projects<br>    each patch into a D-dimensional embedding.&quot;&quot;&quot;<br>    def __init__(self, img_size=224, patch_size=16, in_chans=3, embed_dim=768):<br>        super().__init__()<br>        self.n_patches = (img_size // patch_size) ** 2<br>        self.proj = nn.Conv2d(<br>            in_chans, embed_dim, kernel_size=patch_size, stride=patch_size<br>        )<br>    def forward(self, x):<br>        # x: (B, C, H, W)<br>        x = self.proj(x)                   # (B, D, H/P, W/P)<br>        x = x.flatten(2).transpose(1, 2)   # (B, N, D)<br>        return x</pre><p><strong>Shape trace</strong>, for a 224×224 RGB image with patch size 16 and embedding dimension 768:</p><ul><li>Input image shape (B, 3, 224, 224)</li><li>After proj (conv) shape (B, 768, 14, 14)</li><li>After flatten(2) shape (B, 768, 196)</li><li>After transpose(1, 2) shape (B, 196, 768)</li></ul><p>The final shape (B, N=196, D=768)is exactly the format a Transformer encoder expects: a batch of token sequences.</p><h3>2. Transformer Encoder Block</h3><p>Each encoder block applies, in order: <strong>LayerNorm → Multi-Head Self-Attention → residual add → LayerNorm → MLP → residual add</strong>. This is the <em>pre-norm</em> variant (normalization before each sublayer rather than after), which trains more stably at depth than the original 2017 Transformer’s post-norm design.</p><pre>class EncoderBlock(nn.Module):<br>    &quot;&quot;&quot;One pre-norm Transformer encoder block: MSA + MLP, each with a<br>    residual connection and preceding LayerNorm.&quot;&quot;&quot;<br><br>  def __init__(self, dim, heads, mlp_ratio=4.0, drop=0.0):<br>        super().__init__()<br>        self.norm1 = nn.LayerNorm(dim)<br>        self.attn = nn.MultiheadAttention(<br>            dim, heads, dropout=drop, batch_first=True<br>        )<br>        self.norm2 = nn.LayerNorm(dim)<br>        hidden = int(dim * mlp_ratio)<br>        self.mlp = nn.Sequential(<br>            nn.Linear(dim, hidden),<br>            nn.GELU(),<br>            nn.Dropout(drop),<br>            nn.Linear(hidden, dim),<br>            nn.Dropout(drop),<br>        )<br>    def forward(self, x):<br>        # --- Multi-Head Self-Attention sublayer ---<br>        y = self.norm1(x)<br>        attn_out, _ = self.attn(y, y, y)   # self-attention: Q=K=V=y<br>        x = x + attn_out                   # residual connection<br>        # --- MLP sublayer ---<br>        x = x + self.mlp(self.norm2(x))    # residual connection<br>        return x</pre><p><strong>Why each piece is there:</strong></p><ul><li>norm1 / norm2 (<strong>LayerNorm</strong>): normalizes activations across the feature dimension for each token independently, which stabilizes training in deep stacks of attention/MLP layers.</li><li>nn.MultiheadAttention(dim, heads, ...): implements scaled dot-product self-attention across heads parallel attention heads. Calling it with y, y, y as query, key, and value makes it <em>self</em>-attention — every token attends to every other token in the same sequence, including itself.</li><li>The <strong>residual connections</strong> (x + attn_out, x + self.mlp(...)) let gradients flow directly through the network, which is what makes it feasible to stack dozens of these blocks (ViT-Large uses 24, ViT-Huge uses 32) without vanishing gradients.</li><li>The <strong>MLP</strong> expands the hidden dimension by mlp_ratio (4× by default) with a GELU nonlinearity in between, giving the block its non-linear, per-token feature-mixing capacity; attention mixes information <em>across</em> tokens, the MLP transforms information <em>within</em> each token.</li></ul><h3>3. Putting Them Together</h3><p>A full (still simplified) ViT forward pass chains these two pieces, adding a learnable class token and position embeddings between them:</p><pre>class MiniViT(nn.Module):<br>    def __init__(self, img_size=224, patch_size=16, in_chans=3,<br>                 embed_dim=768, depth=12, heads=12, num_classes=1000):<br>        super().__init__()<br>        self.patch_embed = PatchEmbed(img_size, patch_size, in_chans, embed_dim)<br>        n_patches = self.patch_embed.n_patches<br>        <br>        self.cls_token = nn.Parameter(torch.zeros(1, 1, embed_dim))<br>        self.pos_embed = nn.Parameter(torch.zeros(1, n_patches + 1, embed_dim))<br>        <br>        self.blocks = nn.ModuleList(<br>            [EncoderBlock(embed_dim, heads) for _ in range(depth)]<br>        )<br>        self.norm = nn.LayerNorm(embed_dim)<br>        self.head = nn.Linear(embed_dim, num_classes)<br>    <br>    def forward(self, x):<br>        B = x.shape[0]<br>        x = self.patch_embed(x)                          # (B, N, D)<br>        cls_tokens = self.cls_token.expand(B, -1, -1)     # (B, 1, D)<br>        x = torch.cat((cls_tokens, x), dim=1)             # (B, N+1, D)<br>        x = x + self.pos_embed<br>        <br>        for block in self.blocks:<br>            x = block(x)<br>        <br>        x = self.norm(x)<br>        cls_out = x[:, 0]          # take the class token&#39;s final state<br>        return self.head(cls_out)</pre><h3>4. What’s Missing for a Production Implementation</h3><p>This sketch deliberately omits several details that real implementations (e.g., timm.models.vision_transformer) include:</p><ul><li><strong>Stochastic depth</strong> (per-block layer dropout) for regularizing very deep ViTs.</li><li><strong>Careful weight initialization</strong> (truncated normal for linear/conv layers, zero-init for the class token and position embeddings).</li><li><strong>Position embedding interpolation</strong> logic for fine-tuning at a different resolution than pretraining.</li><li><strong>Attention dropout</strong> tuned separately from MLP dropout.</li><li>Optional <strong>distillation token</strong> (DeiT-style) or <strong>register tokens</strong> (used in some newer variants to absorb high-norm “outlier” activations).</li></ul><p>For actual training or fine-tuning, use a maintained library (timm, torchvision.models.vision_transformer, or Hugging Face transformers&#39; ViTModel) rather than this illustrative version.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e18edde85247" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>