<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ankita Mungalpara on Medium]]></title>
        <description><![CDATA[Stories by Ankita Mungalpara on Medium]]></description>
        <link>https://medium.com/@mungalpara.ankita?source=rss-203731bd95fc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*wAzxhyevwoZ3wLr0</url>
            <title>Stories by Ankita Mungalpara on Medium</title>
            <link>https://medium.com/@mungalpara.ankita?source=rss-203731bd95fc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 15:55:27 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mungalpara.ankita/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building CLIP From Scratch: Unfolding Story Behind Pixels]]></title>
            <link>https://medium.com/@mungalpara.ankita/building-clip-from-scratch-unfolding-story-behind-pixels-38f1e9fd5e0d?source=rss-203731bd95fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/38f1e9fd5e0d</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mutilmodal-model]]></category>
            <category><![CDATA[clip]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Ankita Mungalpara]]></dc:creator>
            <pubDate>Fri, 21 Feb 2025 15:22:31 GMT</pubDate>
            <atom:updated>2025-02-21T15:22:31.374Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*5kIG5SdBUVsP5cveGi3D2A.jpeg" /></figure><p><strong>Contrastive Language-Image Pre-training</strong> (CLIP<strong>)</strong> was developed by <a href="https://openai.com/"><strong><em>OpenAI</em></strong></a> and first introduced in the paper <em>“</em><a href="https://arxiv.org/abs/2103.00020"><strong><em>Learning Transferable Visual Models From Natural Language Supervision</em></strong></a><em>”</em> in 2021. It was designed to improve <a href="https://huggingface.co/tasks/zero-shot-classification"><strong><em>zero-shot learning</em></strong></a> in computer vision by training on a vast number of image-text pairs from the internet.</p><p>The conventional approach to visual perception has long followed a standard formula:</p><ol><li>Pre-train a CNN on large labeled datasets (typically <a href="https://image-net.org/download.php"><strong><em>ImageNet</em></strong></a>).</li><li>Fine-tune on specific tasks with smaller datasets.</li></ol><p>While this approach has produced impressive results, it has significant limitations:</p><ul><li>Requires extensive manual annotation</li><li>Limited by the scope of pre-defined classes</li><li>Lacks flexibility for new tasks</li></ul><p>In 2016, Li et al. pioneered the use of natural language for visual prediction, achieving 11.4% zero-shot accuracy on ImageNet and Flickr comments. Mahajan et al. (2018) demonstrated that using Instagram hashtags for pre-training could significantly improve model performance, increasing ImageNet accuracy by over 5%.</p><p>OpenAI’s introduction of CLIP in 2021 revolutionized the field by training 400 million image-text pairs. Unlike its predecessors, CLIP combined a powerful vision encoder with a robust text encoder, trained through <a href="https://arxiv.org/abs/2010.05113"><strong><em>contrastive learning</em></strong></a>. This approach enabled zero-shot learning across various tasks, eliminating the need for task-specific training.</p><h3>Understanding CLIP’s Training Objective</h3><p>Early approaches to training vision-language models, such as Mahajan et al. (2018) and Xie et al. (2020), required extensive computational resources — ranging from 19 GPU years to 33 TPUv3 core years. Initially, the authors of CLIP attempted to train an image CNN and a transformer-based text model from scratch to predict image captions. However, this approach proved inefficient, as the language model struggled to generalize beyond memorizing hashtags and comments.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OVi8blLZw_wf2rrxdlfbdg.png" /><figcaption>Fig. 1: CLIP Architecture (Source: <a href="https://arxiv.org/pdf/2103.00020">CLIP Papaer</a>)</figcaption></figure><p>To overcome these challenges, CLIP adopts a contrastive learning objective, significantly improving efficiency. The model is trained on N image-text pairs, where both the image and text are encoded into a shared multimodal embedding space. The goal is to maximize the cosine similarity between correct image-text pairs while minimizing the similarity between incorrect pairs. This results in a more scalable and generalizable model that can recognize visual concepts without task-specific fine-tuning.</p><p>By utilizing contrastive learning, CLIP efficiently aligns visual and textual representations, enabling robust zero-shot classification and transfer learning across different domains.</p><h3>Model Architecture</h3><p>CLIP employs two different backbones for its image encoder — <a href="https://pytorch.org/vision/main/models/generated/torchvision.models.resnet50.html"><strong><em>ResNet-50</em></strong></a><strong><em> </em></strong>and <a href="https://arxiv.org/pdf/2010.11929"><strong><em>Vision Transformer (ViT)</em></strong></a> — while using a transformer-based text encoder. The largest ResNet variant, RN50x64, required 18 days of training on 592 V100 GPUs, whereas the largest Vision Transformer variant was trained in 12 days on 256 V100 GPUs.</p><p>The model is instantiated with attributes such as <em>embed_dim</em>, which defines the embedding space dimensions, and <em>width</em> and <em>layers</em>, which specify the architecture’s depth. If the <em>vision_layers</em> attribute is provided as a tuple or list, ResNet is used as the image encoder’s backbone; otherwise, the Visual Transformer is selected.</p><p>Let’s walk through how we can build and train CLIP from scratch. This workflow consists of several key components:</p><ol><li>Positional Embedding</li><li>Attention Head</li><li>Multi-Head Attention</li><li>Transformer Encoder</li><li>Image Encoder</li><li>Text Encoder</li><li>Dataset Preparation</li><li>Tokenization</li><li>CLIP Model</li><li>Training</li><li>Hyper-parameter Tuning</li><li>Model Evaluation and Inference with Gradio</li></ol><h4><strong>Import Required Dependencies</strong></h4><p>First, let’s set up all the necessary imports that are required for our implementation.</p><pre><br>import torch<br>import torch.nn as nn<br>from torch.utils.data import Dataset, DataLoader<br>from torch.optim import Adam<br><br>import torchvision.transforms as T<br>from PIL import Image<br><br>import numpy as np<br>import pandas as pd<br>import os<br>from tqdm import tqdm<br><br>import matplotlib.pyplot as plt<br>import gradio as gr<br>import plotly.graph_objects as go</pre><p>We are using PyTorch as the foundation of our implementation, providing efficient tensor operations.</p><pre>import torch<br>import torch.nn as nn</pre><p>We are working with DataSet and DataLoader modules for efficient data loading and batch processing.</p><pre>from torch.utils.data import Dataset, DataLoader</pre><p>Here, in our implementation, we are using an Adam optimizer, so let’s import it.</p><pre>from torch.optim import Adam</pre><p>Also, we are using torchvision.transforms module that provides transformations like data resizing, normalization, and augmentation.</p><pre>import torchvision.transforms as T</pre><p>For visualizing the CLIP model’s performance, we are using a couple of visualization packages: <a href="https://matplotlib.org/"><strong><em>Matplotlib</em></strong></a> for basic training plots, while <a href="https://www.gradio.app/"><strong><em>Gradio</em></strong></a> enables us to build a web user interface for model evaluation, and <a href="https://plotly.com/python/"><strong><em>Plotly</em></strong></a> provides interactive-dynamic visualizations.</p><pre>import matplotlib.pyplot as plt<br>import gradio as gr<br>import plotly.graph_objects as go</pre><h4>Positional Embedding</h4><p>We define a <em>PositionalEmbedding</em> class using PyTorch, which introduces positional information to input sequences. Since <a href="https://arxiv.org/pdf/2304.10557"><strong><em>transformers</em></strong></a> lack inherent order awareness, positional embeddings help sequence order, allowing the model to differentiate between token positions.</p><pre>class PositionalEmbedding(nn.Module):<br>    def __init__(self, width, max_seq_length):<br>        super().__init__()<br>        pe = torch.zeros(max_seq_length, width)<br>        for pos in range(max_seq_length):<br>            for i in range(width):<br>                if i % 2 == 0:<br>                    pe[pos][i] = np.sin(pos/(10000 ** (i/width)))<br>                else:<br>                    pe[pos][i] = np.cos(pos/(10000 ** ((i-1)/width)))<br>        self.register_buffer(&#39;pe&#39;, pe.unsqueeze(0))<br><br>    def forward(self, x):<br>        return x + self.pe</pre><p>At first, a positional embedding matrix, <em>pe,</em> is created and initialized to zeros.</p><pre>pe = torch.zeros(max_seq_length, width)</pre><p>Each position is assigned unique values using sinusoidal functions — sine for even indices and cosine for odd indices. It ensures smooth transitions between positions, which helps the model learn relative positional relationships.</p><pre>for pos in range(max_seq_length):<br>    for i in range(width):<br>        if i % 2 == 0:<br>            pe[pos][i] = np.sin(pos/(10000 ** (i/width)))<br>        else:<br>            pe[pos][i] = np.cos(pos/(10000 ** ((i-1)/width)))</pre><p>Also, we stored these embeddings as a buffer, ensuring it is not updated during training.</p><pre>self.register_buffer(&#39;pe&#39;, pe.unsqueeze(0))</pre><p>In the forward pass, the computed positional embeddings are added to the input, adding position-related information without changing the model’s learned weights.</p><pre>def forward(self, x):<br>    return x + self.pe</pre><h4>Attention Head</h4><p>Next up in our workflow is defining <em>the AttentionHead </em>class, which forms the core of the self-attention mechanism in Transformers.</p><p>Let’s understand attention first:</p><p>Attention is a mechanism that allows models to dynamically focus on different parts of an input sequence, enabling them to capture long-range dependencies effectively.</p><pre>class AttentionHead(nn.Module):<br>    def __init__(self, width, head_size):<br>        super().__init__()<br>        self.head_size = head_size<br>        self.query = nn.Linear(width, head_size)<br>        self.key = nn.Linear(width, head_size)<br>        self.value = nn.Linear(width, head_size)<br><br>    def forward(self, x, mask=None):<br>        B, T, C = x.shape  # batch, sequence length, channels<br>        Q = self.query(x)  # (B, T, head_size)<br>        K = self.key(x)    # (B, T, head_size)<br>        V = self.value(x)  # (B, T, head_size)<br>        <br>        # Compute attention scores<br>        attention = Q @ K.transpose(-2, -1)  # (B, T, T)<br>        attention = attention / (self.head_size ** 0.5)<br>        <br>        # Create attention mask<br>        if mask is not None:<br>            # Create a mask for attention scores<br>            # mask shape: (B, T) -&gt; attention_mask shape: (B, T, T)<br>            attention_mask = mask.unsqueeze(1).expand(-1, T, -1)<br>            attention = attention.masked_fill(attention_mask == 0, float(&#39;-inf&#39;))<br>        <br>        attention = torch.softmax(attention, dim=-1)<br>        return attention @ V</pre><p>In our implementation, we defined an <em>AttentionHead</em> class that captures token relationships through query (<em>Q</em>), key (<em>K</em>), and value (<em>V</em>) transformations.</p><p>At first, we declare three linear layers:</p><ul><li>Query (Q): Determines what the current token is looking for.</li><li>Key (K): Represents how relevant each token is to the query.</li><li>Value (V): Holds the actual token information to be weighted.</li></ul><p>Each transformation allows multiple heads to focus on different aspects of an input sequence.</p><pre>class AttentionHead(nn.Module):<br>    def __init__(self, width, head_size):<br>        super().__init__()<br>        self.head_size = head_size<br>        self.query = nn.Linear(width, head_size)<br>        self.key = nn.Linear(width, head_size)<br>        self.value = nn.Linear(width, head_size)</pre><p>In the forward pass, we extract the batch size, sequence length, and embedding dimension and compute query (Q), key (K), and value (V) projections.</p><pre>def forward(self, x, mask=None):<br>    B, T, C = x.shape  # batch, sequence length, channels<br>    Q = self.query(x)  # (B, T, head_size)<br>    K = self.key(x)    # (B, T, head_size)<br>    V = self.value(x)  # (B, T, head_size)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/374/1*rqazIJIwUXO-RsP5CqJaMA.png" /><figcaption>Fig. 2: Scaled Dot-Product Attention (Source: <a href="https://arxiv.org/pdf/1706.03762v7">Paper</a>)</figcaption></figure><p>After calculating the query, key, and value, we calculate attention scores using <a href="https://arxiv.org/pdf/1706.03762v7"><strong><em>scaled-dot product attention</em></strong></a>, which measures token similarity.</p><p>As discussed <a href="https://arxiv.org/pdf/1706.03762v7"><strong><em>here</em></strong></a>, the scaled-dot product attention equation is written as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/612/1*9kvtv7zdwomoo3vWFyuJxg.png" /></figure><p>Here <em>Q</em> (Query), <em>K </em>(Key), and <em>V</em> (Value) are linear projections of the input, and <em>dk</em> is the key dimension.</p><p>The dot product between queries and keys measure similarity, and the <em>softmax</em> normalizes these scores into a probability distribution, deciding how much focus should be given to each token.</p><pre># Compute attention scores<br>attention = Q @ K.transpose(-2, -1)  # (B, T, T)<br>attention = attention / (self.head_size ** 0.5)</pre><p>If the mask is provided (e.g., for casual attention in autoregressive models), it is used to prevent attention to certain positions, ensuring the model only attends to valid tokens.</p><pre># Create attention mask<br>if mask is not None:<br>    # Create a mask for attention scores<br>    # mask shape: (B, T) -&gt; attention_mask shape: (B, T, T)<br>    attention_mask = mask.unsqueeze(1).expand(-1, T, -1)<br>    attention = attention.masked_fill(attention_mask == 0, float(&#39;-inf&#39;))</pre><p>These scores are passed through <a href="https://pytorch.org/docs/stable/generated/torch.nn.Softmax.html"><strong><em>softmax</em></strong></a><strong><em> </em></strong>function, converting them into normalized attention weights. And we return the weighted sum of values (V) representing context-aware token embeddings.</p><pre>attention = torch.softmax(attention, dim=-1)<br>return attention @ V</pre><h4>Multi-Head Attention</h4><p>It is a key component in Transformer architecture that enhances self-attention using multiple independent attention heads. Each head computes attention separately, focusing on different relationships within the input, and the outputs are then concatenated and transformed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/352/1*HXnB8iSG548q-JJnbMZZlQ.png" /><figcaption>Fig. 3: Multi-Head Attention (Source: <a href="https://arxiv.org/pdf/1706.03762v7">Paper</a>)</figcaption></figure><pre>class MultiHeadAttention(nn.Module):<br>    def __init__(self, width, n_heads):<br>        super().__init__()<br>        self.head_size = width // n_heads<br>        self.heads = nn.ModuleList([AttentionHead(width, self.head_size) for _ in range(n_heads)])<br>        self.W_o = nn.Linear(width, width)<br><br>    def forward(self, x, mask=None):<br>        out = torch.cat([head(x, mask=mask) for head in self.heads], dim=-1)<br>        return self.W_o(out)</pre><p>While initializing, we divide the embedding dimension equally among the number of heads and create multiple attention heads. Next, we use a final transformation <em>W_o</em>, which projects concatenated outputs of all attention heads back to the original embedding dimension.</p><pre>def __init__(self, width, n_heads):<br>    super().__init__()<br>    self.head_size = width // n_heads<br>    self.heads = nn.ModuleList([AttentionHead(width, self.head_size) for _ in range(n_heads)])<br>    self.W_o = nn.Linear(width, width)</pre><p>In the forward method, we pass an input through each attention head separately and concatenate the outputs of all heads along the last dimension.</p><p>And we return a linear projection <em>W_o</em> that combines the information from different heads.</p><pre>def forward(self, x, mask=None):<br>    out = torch.cat([head(x, mask=mask) for head in self.heads], dim=-1)<br>    return self.W_o(out)</pre><h4>Transformer Encoder</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w9Cr5doYofKsKE6HfTSuqQ.png" /><figcaption>Fig. 4: Vision Transformer (ViT) Model Overview (Source: <a href="https://arxiv.org/pdf/2010.11929">Paper</a>)</figcaption></figure><p>With <a href="https://arxiv.org/pdf/1706.03762v7"><strong><em>multi-head attention</em></strong></a> in place, let’s move to the <strong>Transformer Encoder</strong>, which is a fundamental component of models like <a href="https://arxiv.org/pdf/1810.04805"><strong><em>BERT</em></strong></a> and Vision Transformers (ViTs).</p><p>It is responsible for processing input sequences while capturing contextual dependencies.</p><pre>class TransformerEncoder(nn.Module):<br>    def __init__(self, width, n_heads, r_mlp=4):<br>        super().__init__()<br>        self.width = width<br>        self.n_heads = n_heads<br>        self.ln1 = nn.LayerNorm(width)<br>        self.mha = MultiHeadAttention(width, n_heads)<br>        self.ln2 = nn.LayerNorm(width)<br>        self.mlp = nn.Sequential(<br>            nn.Linear(width, width*r_mlp),<br>            nn.GELU(),<br>            nn.Linear(width*r_mlp, width)<br>        )<br><br>    def forward(self, x, mask=None):<br>        x = x + self.mha(self.ln1(x), mask=mask)<br>        x = x + self.mlp(self.ln2(x))<br>        return x</pre><p>The class initializes with layer normalizations (<em>ln1</em> and <em>ln2</em>), which stabilizes activations and ensures smoother training. Next, it applies Multi-Head Attention (MHA), which allows the encoder to capture contextual dependencies across the input sequence by attending to different parts of the data simultaneously.</p><pre>def __init__(self, width, n_heads, r_mlp=4):<br>    super().__init__()<br>    self.width = width<br>    self.n_heads = n_heads<br>    self.ln1 = nn.LayerNorm(width)<br>    self.mha = MultiHeadAttention(width, n_heads)<br>    self.ln2 = nn.LayerNorm(width)<br>    self.mlp = nn.Sequential(<br>            nn.Linear(width, width*r_mlp),<br>            nn.GELU(),<br>            nn.Linear(width*r_mlp, width)<br>        )</pre><p>Additionally, a feedforward network (MLP) expands the feature dimension, applies the <a href="https://arxiv.org/pdf/1606.08415v5"><strong><em>GELU</em></strong></a> activation function, and projects it back to the original <em>width</em>.</p><p>The forward method processes input tensor <em>x</em> by first applying layer normalization (<em>ln1</em>), followed by multi-head attention (MHA) to capture contextual relationships. The result is added back to the input tensor <em>x </em>via a residual connection.</p><pre>def forward(self, x, mask=None):<br>    x = x + self.mha(self.ln1(x), mask=mask)<br>    x = x + self.mlp(self.ln2(x))<br>    return x</pre><p>Next, Layer Normalization<strong> (</strong><em>ln2</em><strong>)</strong> is applied, followed by an MLP block for further feature refinement.</p><h4><strong>Image Encoder</strong></h4><p>It is responsible for transforming input images into a high-dimensional representation that can be used for downstream tasks such as vision-language modeling.</p><pre>class ImageEncoder(nn.Module):<br>    def __init__(self, width, img_size, patch_size, n_channels, n_layers, n_heads, emb_dim):<br>        super().__init__()<br>        assert img_size[0] % patch_size[0] == 0 and img_size[1] % patch_size[1] == 0<br>        <br>        self.n_patches = (img_size[0] * img_size[1]) // (patch_size[0] * patch_size[1])<br>        self.max_seq_length = self.n_patches + 1<br>        <br>        self.linear_project = nn.Conv2d(n_channels, width, kernel_size=patch_size, stride=patch_size)<br>        self.cls_token = nn.Parameter(torch.randn(1, 1, width))<br>        self.positional_embedding = PositionalEmbedding(width, self.max_seq_length)<br>        self.encoder = nn.ModuleList([TransformerEncoder(width, n_heads) for _ in range(n_layers)])<br>        self.projection = nn.Parameter(torch.randn(width, emb_dim))<br><br>    def forward(self, x):<br>        x = self.linear_project(x)<br>        x = x.flatten(2).transpose(1, 2)<br>        x = torch.cat((self.cls_token.expand(x.size()[0], -1, -1), x), dim=1)<br>        x = self.positional_embedding(x)<br>        <br>        for encoder_layer in self.encoder:<br>            x = encoder_layer(x)<br>        <br>        x = x[:, 0, :]<br>        x = x @ self.projection<br>        return x / torch.norm(x, dim=-1, keepdim=True)</pre><p>The encoder initializes by ensuring that the image dimension is perfectly divisible by the patch size (<em>patch_size</em>). Next, it computes the total number of image patches and defines the maximum sequence length by adding a classification token.</p><pre>def __init__(self, width, img_size, patch_size, n_channels, n_layers, n_heads, emb_dim):<br>    super().__init__()<br>    assert img_size[0] % patch_size[0] == 0 and img_size[1] % patch_size[1] == 0<br>        <br>    self.n_patches = (img_size[0] * img_size[1]) // (patch_size[0] * patch_size[1])<br>    self.max_seq_length = self.n_patches + 1</pre><p>The patch embeddings I performed used a 2D convolution that maps input image patches to a lower-dimensional space with (<em>width</em>) channels. Also, a positional embedding is applied to retain spatial information.</p><pre>self.linear_project = nn.Conv2d(n_channels, width, kernel_size=patch_size, stride=patch_size)<br>self.cls_token = nn.Parameter(torch.randn(1, 1, width))<br>self.positional_embedding = PositionalEmbedding(width, self.max_seq_length)</pre><p>The encoder stack consists of multiple Transformer encoder layers, which refine patch representations through self-attention. Finally, a projection layer maps the encoded features into the final embedding.</p><pre>self.encoder = nn.ModuleList([TransformerEncoder(width, n_heads) for _ in range(n_layers)])<br>self.projection = nn.Parameter(torch.randn(width, emb_dim))</pre><p>In the forward pass, the image is first transformed into patch embeddings using a convolutional layer, then flattened and transposed to match the Transformer input format. A classification token <em>cls_token </em>is appended to serve as a global representation of the image. Next, positional embeddings are added to retain spatial structure.</p><pre>def forward(self, x):<br>    x = self.linear_project(x)<br>    x = x.flatten(2).transpose(1, 2)<br>    x = torch.cat((self.cls_token.expand(x.size()[0], -1, -1), x), dim=1)<br>    x = self.positional_embedding(x)</pre><p>The patch embeddings are then sequentially processed through multiple Transformer Encoder layers.</p><pre>for encoder_layer in self.encoder:<br>    x = encoder_layer(x)</pre><p>The final image representation is extracted from the CLS token’s output, followed by a projection step that maps it to the desired embedding dimension. The resulting vector is normalized, ensuring stable representations.</p><pre>x = x[:, 0, :]<br>x = x @ self.projection<br>return x / torch.norm(x, dim=-1, keepdim=True)</pre><h4>Text Encoder</h4><p>The Text Encoder converts input text into a dense, high-dimensional embedding, enabling effective representation learning for language understanding.</p><pre>class TextEncoder(nn.Module):<br>    def __init__(self, vocab_size, width, max_seq_length, n_heads, n_layers, emb_dim):<br>        super().__init__()<br>        self.max_seq_length = max_seq_length<br>        self.encoder_embedding = nn.Embedding(vocab_size, width)<br>        self.positional_embedding = PositionalEmbedding(width, max_seq_length)<br>        self.encoder = nn.ModuleList([TransformerEncoder(width, n_heads) for _ in range(n_layers)])<br>        self.projection = nn.Parameter(torch.randn(width, emb_dim))<br>        self.width = width<br><br>    def forward(self, text, mask=None):<br>        batch_size = text.shape[0]<br>        <br>        # [batch_size, seq_len, width]<br>        x = self.encoder_embedding(text)<br>        x = self.positional_embedding(x)<br>        <br>        # Apply transformer layers<br>        for encoder_layer in self.encoder:<br>            x = encoder_layer(x, mask=mask)<br>        <br>        # Extract features from the CLS token (first token)<br>        x = x[:, 0, :]  # Take the first token (CLS token) from each sequence<br>        <br>        # Project to embedding dimension and normalize<br>        x = x @ self.projection<br>        return x / torch.norm(x, dim=-1, keepdim=True)</pre><p>The model begins with an embedding layer that maps input tokens to a <em>width</em>-dimensional space. A positional embedding is added to retain token order. The text is then processed through a stack of Transformer Encoder layers, capturing long-range dependencies. Finally, a learnable projection maps the encoded features to the desired embedding dimension <em>emb_dim</em>.</p><pre>def __init__(self, vocab_size, width, max_seq_length, n_heads, n_layers, emb_dim):<br>    super().__init__()<br>    self.max_seq_length = max_seq_length<br>    self.encoder_embedding = nn.Embedding(vocab_size, width)<br>    self.positional_embedding = PositionalEmbedding(width, max_seq_length)<br>    self.encoder = nn.ModuleList([TransformerEncoder(width, n_heads) for _ in range(n_layers)])<br>    self.projection = nn.Parameter(torch.randn(width, emb_dim))<br>    self.width = width</pre><p>In the forward method, the input text is first embedded and enriched with positional information. It then passes through multiple Transformer Encoder layers, refining token representations. The CLS token’s output is extracted as the final sentence representation. A projection step follows, and the output is L2-normalized to ensure stable feature representations.</p><pre>def forward(self, text, mask=None):<br>    batch_size = text.shape[0]<br>        <br>    # [batch_size, seq_len, width]<br>    x = self.encoder_embedding(text)<br>    x = self.positional_embedding(x)<br>    <br>    # Apply transformer layers<br>    for encoder_layer in self.encoder:<br>        x = encoder_layer(x, mask=mask)<br>    <br>    # Extract features from the CLS token (first token)<br>    x = x[:, 0, :]  # Take the first token (CLS token) from each sequence<br>    <br>    # Project to embedding dimension and normalize<br>    x = x @ self.projection<br>    return x / torch.norm(x, dim=-1, keepdim=True)</pre><h4>Dataset Preparation</h4><p>For training, we will use the <a href="https://www.kaggle.com/datasets/adityajn105/flickr8k/data"><strong><em>Flickr8K</em></strong></a> dataset, which consists of 8,000 images, each accompanied by multiple textual captions. The dataset will be preprocessed and loaded using a custom PyTorch Dataset class.</p><pre>class FlickrDataset(Dataset):<br>    def __init__(self, df, image_path, transform=None):<br>        self.df = df<br>        self.image_path = image_path<br>        self.transform = transform or T.Compose([<br>            T.Resize((224, 224)),<br>            T.ToTensor(),<br>            T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])<br>        ])<br>        <br>        self.image_groups = df.groupby(&#39;image&#39;)<br>        self.unique_images = list(self.image_groups.groups.keys())<br><br>    def __len__(self):<br>        return len(self.unique_images)<br><br>    def __getitem__(self, idx):<br>        image_name = self.unique_images[idx]<br>        captions = self.image_groups.get_group(image_name)[&#39;caption&#39;].tolist()<br>        caption = np.random.choice(captions)<br>        image = Image.open(os.path.join(self.image_path, image_name)).convert(&#39;RGB&#39;)<br>        image = self.transform(image)<br>        tokens, mask = tokenizer(caption)<br>        return {&#39;image&#39;: image, &#39;caption&#39;: tokens, &#39;mask&#39;: mask, &#39;raw_caption&#39;: caption}</pre><p>The dataset is initialized with a DataFrame containing image-caption pairs and the respected image directory path. Now, transformations are applied, including resizing, normalization, and tensor conversion. Then images are grouped to allow considering multiple captions for each unique image.</p><pre>def __init__(self, df, image_path, transform=None):<br>    self.df = df<br>    self.image_path = image_path<br>    self.transform = transform or T.Compose([<br>        T.Resize((224, 224)),<br>        T.ToTensor(),<br>        T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])<br>    ])<br>    <br>    self.image_groups = df.groupby(&#39;image&#39;)<br>    self.unique_images = list(self.image_groups.groups.keys())</pre><p>The length of the dataset corresponds to the number of unique images, ensuring that each image is accessed individually rather than counting all caption duplicates.</p><pre>def __len__(self):<br>    return len(self.unique_images)</pre><p>Given an index, the <em>__getitem__</em> method retrieves an image filename and selects a corresponding caption. The image is loaded, converted to RGB, and transformed for model input. The caption is tokenized using a predefined tokenizer, returning the token sequence and an attention mask. The final output is a dictionary containing:</p><ul><li>The processed image</li><li>The tokenized caption</li><li>The attention mask, and</li><li>The raw caption text</li></ul><pre>def __getitem__(self, idx):<br>    image_name = self.unique_images[idx]<br>    captions = self.image_groups.get_group(image_name)[&#39;caption&#39;].tolist()<br>    caption = np.random.choice(captions)<br>    image = Image.open(os.path.join(self.image_path, image_name)).convert(&#39;RGB&#39;)<br>    image = self.transform(image)<br>    tokens, mask = tokenizer(caption)<br>    return {&#39;image&#39;: image, &#39;caption&#39;: tokens, &#39;mask&#39;: mask, &#39;raw_caption&#39;: caption}</pre><h4>Tokenization</h4><p>The tokenizer function is responsible for encoding text into a numerical format suitable for deep learning models and decoding tokenized sequences back into text. It ensures that input text adheres to a fixed sequence length while maintaining important structural elements like start and end tokens.</p><pre>def tokenizer(text, encode=True, mask=None, max_seq_length=77):  # Increased max_seq_length to match config<br>    if encode:<br>        # Add start and end tokens<br>        out = chr(2) + text + chr(3)<br>        <br>        # Truncate if longer than max_seq_length<br>        if len(out) &gt; max_seq_length:<br>            out = out[:max_seq_length-1] + chr(3) <br>        <br>        # Pad with zeros if shorter than max_seq_length<br>        out = out + &quot;&quot;.join([chr(0) for _ in range(max_seq_length - len(out))])<br>        <br>        # Convert to tensor<br>        out = torch.IntTensor(list(out.encode(&quot;utf-8&quot;)))<br>        <br>        # Create mask (1s for actual tokens, 0s for padding)<br>        n_actual_tokens = len(text) + 2  # +2 for start and end tokens<br>        n_actual_tokens = min(n_actual_tokens, max_seq_length)<br><br>        # Create a 1D mask tensor<br>        mask = torch.zeros(max_seq_length, dtype=torch.long)<br>        mask[:n_actual_tokens] = 1  # Set 1s for actual tokens<br>        <br>    else:<br>        # Decoding<br>        if mask is None:<br>            raise ValueError(&quot;Mask is required for decoding&quot;)<br>        <br>        # Convert back to text, removing start and end tokens<br>        out = [chr(x) for x in text[1:len(mask.nonzero())]]<br>        out = &quot;&quot;.join(out)<br>        mask = None<br><br>    return out, mask</pre><p>During encoding, the function first adds special tokens: a start token (<em>chr(2)</em>) at the beginning and an end token (<em>chr(3)</em>) at the end of the text.</p><p>If the text length exceeds the defined <em>max_seq_length</em>, it is truncated while ensuring the end token is retained. If the text is shorter than the required length, zero-padding is applied to maintain a uniform input size.</p><pre>if encode:<br>    # Add start and end tokens<br>    out = chr(2) + text + chr(3)<br>    <br>    # Truncate if longer than max_seq_length<br>    if len(out) &gt; max_seq_length:<br>        out = out[:max_seq_length-1] + chr(3) <br>    <br>    # Pad with zeros if shorter than max_seq_length<br>    out = out + &quot;&quot;.join([chr(0) for _ in range(max_seq_length - len(out))])</pre><p>The processed text is then converted into a tensor using UTF-8 encoding, allowing the model to interpret it numerically. Additionally, an attention mask is generated, marking actual tokens with <em>1s</em> and padded positions with <em>0s</em>, helping the model distinguish between meaningful input and padding.</p><pre># Convert to tensor<br>out = torch.IntTensor(list(out.encode(&quot;utf-8&quot;)))<br><br># Create mask (1s for actual tokens, 0s for padding)<br>n_actual_tokens = len(text) + 2  # +2 for start and end tokens<br>n_actual_tokens = min(n_actual_tokens, max_seq_length)<br><br># Create a 1D mask tensor<br>mask = torch.zeros(max_seq_length, dtype=torch.long)<br>mask[:n_actual_tokens] = 1  # Set 1s for actual tokens</pre><p>For decoding, the function reconstructs the text by converting tokenized numerical sequences back into readable format. The start and end tokens are removed, and the original text is retrieved. A mask is required during decoding to determine valid token positions and prevent unnecessary processing of padding elements.</p><pre># Decoding<br><br># Convert back to text, removing start and end tokens<br>out = [chr(x) for x in text[1:len(mask.nonzero())]]<br>out = &quot;&quot;.join(out)<br>mask = None</pre><p>By implementing this tokenizer, textual data is efficiently structured and aligned for our model architecture, ensuring consistency in sequence-based learning.</p><h4>CLIP Model</h4><p>The <em>CLIP</em> class is a multimodal learning model that processes both images and text, enabling them to be mapped into a shared embedding space. It integrates an <em>ImageEncoder</em> for processing visual data and an <em>TextEncoder</em> for handling textual input. The architecture allows these two different modalities to interact meaningfully, making it suitable for tasks like:</p><ul><li>Zero-Shot Image Classification</li><li>Semantic Image Retrieval</li><li>Image Ranking</li><li>Reverse Image Search</li><li>Image Capturing, etc.</li></ul><pre>class CLIP(nn.Module):<br>    def __init__(self, config):<br>        super().__init__()<br>        self.image_encoder = ImageEncoder(<br>            config[&#39;width&#39;],<br>            config[&#39;img_size&#39;],<br>            config[&#39;patch_size&#39;],<br>            config[&#39;n_channels&#39;],<br>            config[&#39;vit_layers&#39;],<br>            config[&#39;vit_heads&#39;],<br>            config[&#39;emb_dim&#39;]<br>        )<br>        self.text_encoder = TextEncoder(<br>            config[&#39;vocab_size&#39;],<br>            config[&#39;text_width&#39;],<br>            config[&#39;max_seq_length&#39;],<br>            config[&#39;text_heads&#39;],<br>            config[&#39;text_layers&#39;],<br>            config[&#39;emb_dim&#39;]<br>        )<br>        self.temperature = nn.Parameter(torch.ones([]) * np.log(1 / 0.07))<br>        self.device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br><br>    def forward(self, image, text, mask=None):<br>        # Move mask to the same device as the model<br>        if mask is not None:<br>            mask = mask.to(self.device)<br>            <br>        image_features = self.image_encoder(image)<br>        text_features = self.text_encoder(text, mask=mask)<br>        <br>        # Compute similarity<br>        logits = (image_features @ text_features.transpose(-2, -1)) * torch.exp(self.temperature)<br>        <br>        # Compute loss<br>        labels = torch.arange(logits.shape[0], device=self.device)<br>        loss_i = nn.functional.cross_entropy(logits.transpose(-2, -1), labels)<br>        loss_t = nn.functional.cross_entropy(logits, labels)<br>        <br>        return (loss_i + loss_t) / 2</pre><p>In the initialization method <em>__init__</em>, the image encoder extracts feature representations from images, breaking them down into patches and processing them through multiple transformer layers.</p><pre>self.image_encoder = ImageEncoder(<br>                          config[&#39;width&#39;],<br>                          config[&#39;img_size&#39;],<br>                          config[&#39;patch_size&#39;],<br>                          config[&#39;n_channels&#39;],<br>                          config[&#39;vit_layers&#39;],<br>                          config[&#39;vit_heads&#39;],<br>                          config[&#39;emb_dim&#39;]<br>                        )</pre><p>Similarly, the text encoder converts input text into meaningful embeddings.</p><pre>self.text_encoder = TextEncoder(<br>                        config[&#39;vocab_size&#39;],<br>                        config[&#39;text_width&#39;],<br>                        config[&#39;max_seq_length&#39;],<br>                        config[&#39;text_heads&#39;],<br>                        config[&#39;text_layers&#39;],<br>                        config[&#39;emb_dim&#39;]<br>                    )</pre><p>In the forward pass, images and text are encoded separately using their respective encoders.</p><pre>image_features = self.image_encoder(image)<br>text_features = self.text_encoder(text, mask=mask)</pre><p>The resulting feature vectors are then used to compute a similarity matrix between all image-text pairs using a dot product, scaled by the exponential of the temperature parameter. This similarity matrix serves as the foundation for contrastive learning.</p><pre># Compute similarity<br>logits = (image_features @ text_features.transpose(-2, -1)) * torch.exp(self.temperature)</pre><p>To train the model, <a href="https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html"><strong><em>cross-entropy loss</em></strong></a> is applied to ensure that correct image-text pairs are ranked higher than incorrect ones.</p><p>The final loss is an average of two objectives: aligning images with their corresponding texts and vice versa.</p><pre># Compute loss<br>labels = torch.arange(logits.shape[0], device=self.device)<br>loss_i = nn.functional.cross_entropy(logits.transpose(-2, -1), labels)<br>loss_t = nn.functional.cross_entropy(logits, labels)<br><br>return (loss_i + loss_t) / 2</pre><h4>Training</h4><p>The CLIP model will be trained using a contrastive loss function, which ensures that images and their corresponding text descriptions are mapped to similar representations in the embedding space.</p><pre>def train_clip(model, train_loader, optimizer, device, epochs=30):<br>    model.train()<br>    best_loss = float(&#39;inf&#39;)<br>    <br>    for epoch in range(epochs):<br>        total_loss = 0<br>        progress_bar = tqdm(train_loader, desc=f&#39;Epoch {epoch+1}/{epochs}&#39;)<br>        <br>        for batch in progress_bar:<br>            images = batch[&#39;image&#39;].to(device)<br>            captions = batch[&#39;caption&#39;].to(device)<br>            masks = batch[&#39;mask&#39;].to(device)<br>            <br>            loss = model(images, captions, masks)<br>            <br>            optimizer.zero_grad()<br>            loss.backward()<br>            optimizer.step()<br>            <br>            total_loss += loss.item()<br>            progress_bar.set_postfix({&#39;loss&#39;: loss.item()})<br>        <br>        avg_loss = total_loss / len(train_loader)<br>        print(f&#39;Epoch [{epoch+1}/{epochs}], Average Loss: {avg_loss:.4f}&#39;)</pre><p>In the training loop, the model is set to training <em>model.train()</em>, ensuring that parameters are updated during back-propagation.</p><p>The CLIP model’s forward pass computes the contrastive loss, which measures how well the model aligns image-text pairs. The optimizer is then used to update the model’s weights. And this function also keeps track of the total loss across batches.</p><pre>loss = model(images, captions, masks)<br>           <br>optimizer.zero_grad()<br>loss.backward()<br>optimizer.step()<br><br>total_loss += loss.item()</pre><p>After completing an epoch, it calculates and prints the average loss, helping monitor the model’s improvement over time.</p><pre>avg_loss = total_loss / len(train_loader)<br>print(f&#39;Epoch [{epoch+1}/{epochs}], Average Loss: {avg_loss:.4f}&#39;)</pre><h4>Hyperparameter Tuning</h4><p>The provided configuration defines the architectural and training parameters for the CLIP model.</p><pre><br>emb_dim: 256<br>width: 768<br>img_size: (224, 224)<br>patch_size: (16, 16)<br>n_channels: 3<br>vit_layers: 6<br>vit_heads: 8<br>vocab_size: 25000<br>text_width: 512<br>max_seq_length: 77<br>text_heads: 8<br>text_layers: 6<br>batch_size: 32<br>learning_rate: 1e-4<br>epochs: 10</pre><h4>Evaluation and Inference with Gradio</h4><p>Once trained, this model can be used to analyze images. The function <em>get_image_text_similarity</em> performs inference with the trained CLIP model, computing the similarity between an input image and text prompts.</p><pre>def get_image_text_similarity(model, image_tensor, text_tokens, text_masks):<br>    &quot;&quot;&quot;Calculate similarity between image and text prompts.&quot;&quot;&quot;<br>    with torch.no_grad():<br>        device = next(model.parameters()).device<br>        image_tensor = image_tensor.to(device)<br>        text_tokens = text_tokens.to(device)<br>        text_masks = text_masks.to(device)<br>        <br>        # Get embeddings<br>        image_features = model.image_encoder(image_tensor)<br>        text_features = model.text_encoder(text_tokens, mask=text_masks)<br>        <br>        # Compute similarities and probabilities<br>        similarities = (image_features @ text_features.transpose(-2, -1)).squeeze(0)<br>        similarities = similarities * torch.exp(model.temperature)<br>        probabilities = torch.nn.functional.softmax(similarities, dim=-1)<br>        <br>        return probabilities.cpu().numpy()</pre><p>The function extracts image and text embeddings using the respective encoders and computes their similarity via the dot product. This score is scaled using the learned temperature parameter to adjust distribution sharpness.</p><p>Finally, a <em>softmax</em> function normalizes the scores into probabilities, indicating how well the image matches different text prompts.</p><p>The Gradio-based UI allows to:</p><ol><li>Upload an image and input multiple text prompts.</li><li>Compute similarity scores between the image and text inputs.</li><li>Display results in a structured format with confidence scores.</li></ol><h4>CLIP Inference with Gradio</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RuCAooPTaWj6J-HqvMG-lQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Tje-FdO0m3aSyhnEDETLVA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Joxh9EplpauMZ9_J7gteuw.png" /></figure><h4>Conclusion</h4><p>In this blog, we explored the implementation of a CLIP-like model, leveraging transformer-based encoders for both images and text. We covered key components, including positional embeddings, multi-head attention, and transformer encoders, along with dataset handling and tokenization. The training pipeline was designed to optimize image-text alignment, while the inference function enabled practical applications like similarity computation and zero-shot classification.</p><p>By integrating these techniques, we built a robust multimodal system capable of understanding visual and textual inputs in a shared representation space. This framework can be further extended for applications such as image retrieval, caption generation, and multimodal reasoning, demonstrating the power of contrastive learning in bridging vision and language.</p><p>I have deployed this fine-tuned CLIP model using Gradio on <a href="https://huggingface.co/spaces/AnkitaMungalpara/CLIP_Inference_with_Gradio"><strong><em>Hugging Face Space</em></strong></a>, making it easy to test with custom images and text prompts. The complete code is also available in my <a href="https://github.com/AnkitaMungalpara/Building-CLIP-From-Scratch/tree/0555e70ef5fce1ec5930d7d74c14c898780f0092"><strong><em>GitHub</em></strong></a> repository, where you can explore, experiment, and build upon it for further development.</p><p>Thank you for reading! I hope this blog provided valuable insights into training a CLIP model from scratch. Happy coding!</p><h4>References:</h4><ol><li>Learning Transferable Visual Models From Natural Language Supervision. <a href="https://arxiv.org/pdf/2103.00020"><strong><em>Link</em></strong></a></li><li>An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. <a href="https://arxiv.org/pdf/2010.11929"><strong><em>Link</em></strong></a></li><li>An Introduction to Transformers. <a href="https://arxiv.org/pdf/2304.10557"><strong><em>Link</em></strong></a></li><li>Attention is All You Need. <a href="https://arxiv.org/pdf/1706.03762"><strong><em>Link</em></strong></a></li><li>BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. <a href="https://arxiv.org/pdf/1810.04805"><strong><em>Link</em></strong></a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=38f1e9fd5e0d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>