<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Matt Nguyen on Medium]]></title>
        <description><![CDATA[Stories by Matt Nguyen on Medium]]></description>
        <link>https://medium.com/@mbn312?source=rss-b66aa802e02f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dmbNkD5D-u45r44go_cf0g.png</url>
            <title>Stories by Matt Nguyen on Medium</title>
            <link>https://medium.com/@mbn312?source=rss-b66aa802e02f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 09:57:59 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mbn312/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[DALL-E 2 from Scratch]]></title>
            <link>https://medium.com/correll-lab/dall-e-2-from-scratch-c055bf881b9a?source=rss-b66aa802e02f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c055bf881b9a</guid>
            <category><![CDATA[text-to-image-generation]]></category>
            <category><![CDATA[clip]]></category>
            <category><![CDATA[dalle-2]]></category>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[diffusion-models]]></category>
            <dc:creator><![CDATA[Matt Nguyen]]></dc:creator>
            <pubDate>Mon, 12 Aug 2024 19:54:23 GMT</pubDate>
            <atom:updated>2024-08-27T16:28:31.070Z</atom:updated>
            <content:encoded><![CDATA[<h4>Text-Conditioned Image Generation on FashionMNIST using CLIP Latents</h4><p>Denoising diffusion probabilistic models (DDPM) are a popular type of generative AI model that were introduced by <a href="https://arxiv.org/pdf/2006.11239">Ho et al. in 2020</a> and improved upon by <a href="https://arxiv.org/pdf/2102.09672">Nichol et al. in 2021</a>. The basic idea behind these models is that noise is added to images in the forward diffusion process in order to train the model to predict the noise that should be removed at a certain timestep in the reverse diffusion process. When sampling images, you would start with an image containing pure noise and iteratively remove model’s predicted noise at each timestep until you get the final image.</p><p>In order to have a DDPM generate multiple types of images while still letting the user choose which type of image they want, the model needs to be conditioned on some input. <a href="https://cdn.openai.com/papers/dall-e-2.pdf">Ramesh et al.</a> introduced one such conditioning method called unCLIP, which is used in OpenAI’s DALL-E 2 model. In the method described by Ramesh et al., the input caption is first passed to a prior network which will use a trained CLIP model to get the CLIP text embeddings. These text embeddings are then used by a decoder-only transformer in order to generate possible CLIP image embeddings. The CLIP image embeddings generated by the prior network will be used by a decoder network, which consists of a UNet model, in order to condition the images that are created. In this article, we are going to be building a simple diffusion model using this process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/914/1*_rPBfkAmGcu_hHtz6UtMmw.png" /><figcaption>Figure 1: Model Overview. Image: own work.</figcaption></figure><p>Colab notebook with the code for this tutorial can be found <a href="https://colab.research.google.com/drive/1g4kWCBPaNEq1YBeBB7iwKeU3_3LRcEFf?usp=sharing">here</a>. Additionally, GitHub repo can be found <a href="https://github.com/mbn312/dalle2">here</a>.</p><p><strong>Libraries and Modules</strong></p><p>We are going to be building our models using PyTorch, so we will need to import the library plus others that we will be using in this tutorial.</p><pre>import torch<br>import torch.nn as nn</pre><p>We also need to import in torchvision.transforms in order to resize the input images and convert them to tensors. Resizing the input images is optional. You just need to make sure that the image size is divisible by the patch size.</p><pre>import torchvision.transforms as T</pre><p>We are going to be using Adam or AdamW for our optimizer depending on weight decay, so we need to import it in from torch.optim. We are also going to be using a learning rate scheduler, so we need need to import in lr_scheduler from torch.optim as well.</p><pre>from torch.optim import Adam, AdamW, lr_scheduler</pre><p>We are going to be importing in the FashionMNIST dataset from HuggingFace for this tutorial, so we need to import in datasets. We are going to be using Dataset and DataLoader from PyTorch to help load the data, so we need to import that in as well.</p><pre>from datasets import load_dataset<br>from torch.utils.data import Dataset, DataLoader</pre><p>We are going to import in pyplot from matplotlib in order to display the original images, noisy images, diffused images, etc.</p><pre>import matplotlib.pyplot as plt</pre><p>Calculating our timestep embeddings is going to require us to perform sin and cosine, so we need to import in numpy.</p><pre>import numpy as np</pre><p>Finally we are going to import in dataclass from dataclasses in order to create config classes that hold our parameters.</p><pre>from dataclasses import dataclass</pre><p>Putting this all together, we get:</p><pre>import torch<br>import torch.nn as nn<br>import torchvision.transforms as T<br>from torch.optim import Adam, AdamW, lr_scheduler<br>from datasets import load_dataset<br>from torch.utils.data import Dataset, DataLoader<br>import numpy as np<br>import matplotlib.pyplot as plt<br>from dataclasses import dataclass, field</pre><p><strong>CLIP</strong></p><p>In order to create diffusion images from text, we are going to be using the embeddings from the CLIP model. The text embeddings that we get from CLIP are used to condition the prior model to diffuse corresponding image embeddings. These image embeddings are then used to condition the decoder model to help guide it to the desired image.</p><p>For more information about the CLIP model and how to implement it from scratch, I recommend reading my other Medium article.</p><p><a href="https://medium.com/correll-lab/building-clip-from-scratch-68f6e42d35f4">Building CLIP From Scratch</a></p><p>My previous vision transformers article might be useful to read as well.</p><p><a href="https://medium.com/correll-lab/building-a-vision-transformer-model-from-scratch-a3054f707cc6">Building a Vision Transformer Model From Scratch</a></p><p>I will be leaving out some topics/code because I already went over them in my previous articles. Those topics include: multi-head attention, positional encodings, vision transformers, text transformers, CLIP, tokenization, and loading in the FashionMNIST dataset.</p><p><strong>Forward Diffusion</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wizOMcbdXSO2MVQEjT1cvw.png" /><figcaption>Figure 2: Forward diffusion using linear (top) and cosine (bottom) schedules (Source: <a href="https://arxiv.org/pdf/2102.09672">Nichol et al. 2021</a>)</figcaption></figure><p>Forward diffusion is a part of the diffusion process in which Gaussian noise is added to an image at each timestep over T steps. The amount of noise that should be given at each timestep is given by a variance schedule. The formula for forward diffusion is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YNat3pk0CZHvifmj4jrcsA.png" /></figure><p>That is, we keep applying Gaussian noise at every time step. While the formula above adds the noise iteratively, by using a reparameterization trick, we are able to directly sample a noisy image at any arbitrary timestep in closed form. We are going to be using this method for our forward diffusion process. The formula for this method is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/625/1*lXbkPtO4nc0vd9yMNKOQPw.png" /></figure><p>For the variance schedule, in the original DDPM paper by <a href="https://arxiv.org/pdf/2006.11239">Ho et al.</a>, a linear schedule was used. While this schedule worked well with high resolution images, when the images were smaller (64x64 or less), the forward process was too noisy at the end. In order to combat this, <a href="https://arxiv.org/pdf/2102.09672">Nichol et al.</a> suggested using a cosine schedule whose formula is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/986/1*s98nNK5p0KDftv7ICFwW3Q.png" /></figure><p>These values are clipped to be less than or equal to 0.999 in order to prevent singularities near t = T. Because we are using the FashionMNIST dataset which contains low resolution images, the cosine beta schedule is the one we are going to use.</p><p>Putting this all together, the code for the forward diffusion process is going to look something like this:</p><pre># Gets elements from indicies and makes sure output has a certain dimension<br>def extract_and_expand(x, idx, shape):<br>    return x[idx].reshape(idx.shape[0], *((1, ) * (len(shape) - 1)))<br><br># Returns beta schedule<br>def get_beta_schedule(schedule, max_time, s=0.008):<br>    if schedule == &quot;linear&quot;:<br>        scale = 1000 / max_time<br>        betas = torch.linspace(1e-4  * scale, 0.02  * scale, max_time)<br>    elif schedule == &quot;cosine&quot;:<br>        t = torch.linspace(0, max_time, max_time + 1)<br>        a_bars = torch.cos((((t / max_time) + s) / (1 + s)) * (np.pi / 2)) ** 2<br>        a_bars = a_bars / a_bars[0]<br>        betas = 1 - (a_bars[1:] / a_bars[:-1])<br>        betas = torch.clamp(betas, min=0, max=0.999)<br>    else:<br>        Exception(&quot;Beta schedule not implemented.&quot;)<br><br>    return betas<br><br>def get_schedule_values(config):<br>        schedule_values = {}<br>        schedule_values[&quot;betas&quot;] = get_beta_schedule(config.prior.schedule, config.prior.max_time).to(config.device)<br>        schedule_values[&quot;alphas&quot;] = 1.0 - schedule_values[&quot;betas&quot;]<br>        schedule_values[&quot;alpha_bars&quot;] = torch.cumprod(schedule_values[&quot;alphas&quot;], axis = 0)<br>        schedule_values[&quot;sqrt_recip_alphas&quot;] = torch.sqrt(1.0 / schedule_values[&quot;alphas&quot;])<br>        schedule_values[&quot;sqrt_alpha_bars&quot;] = torch.sqrt(schedule_values[&quot;alpha_bars&quot;])<br>        schedule_values[&quot;sqrt_one_minus_alpha_bars&quot;] = torch.sqrt(1.0 - schedule_values[&quot;alpha_bars&quot;])<br>        schedule_values[&quot;alpha_bars_prev&quot;] = torch.cat((torch.ones(1, device=config.device), schedule_values[&quot;alpha_bars&quot;][:-1]))<br>        schedule_values[&quot;sigma&quot;] = schedule_values[&quot;betas&quot;] * (1.0 - schedule_values[&quot;alpha_bars_prev&quot;]) / (1.0 - schedule_values[&quot;alpha_bars&quot;])<br>        return schedule_values<br><br># Gets noisy image at a certain timestep<br>def forward_diffusion(x_0, schedule_values, t):<br>    noise = torch.randn_like(x_0)<br>    sqrt_alpha_bars = extract_and_expand(schedule_values[&quot;sqrt_alpha_bars&quot;], t, x_0.shape)<br>    sqrt_one_minus_alpha_bars = extract_and_expand(schedule_values[&quot;sqrt_one_minus_alpha_bars&quot;], t, x_0.shape)<br>    x_noisy = (sqrt_alpha_bars * x_0) + (sqrt_one_minus_alpha_bars * noise)<br>    return x_noisy, noise</pre><p><strong>Timestep Embedding</strong></p><p>The timestep embedding is an important part of diffusion. This is because images at different timesteps have a different amount of noise. In order to use this information in our model, we are going to use the sinusoidal positional encodings. These positional encodings are the same that are commonly used for Transformers. The main difference is that our input timesteps are mostly likely not going to be in sequential order and contain all the possible timesteps, so we need to only get the positional encodings that correspond to the timesteps that are inputted.</p><pre>class SinusoidalPositionalEncodings(nn.Module):<br>    def __init__(self, <br>                 max_seq_length,    # Maximum sequence length<br>                 width              # Width of model<br>            ):<br>        super().__init__()<br><br>        # Create positional encodings<br>        pe = torch.zeros(max_seq_length, width)<br><br>        for pos in range(max_seq_length):<br>            for i in range(width):<br>                if i % 2 == 0:<br>                    pe[pos][i] = np.sin(pos / (10000 ** (i / width)))<br>                else:<br>                    pe[pos][i] = np.cos(pos / (10000 ** ((i - 1) / width)))<br><br>        self.register_buffer(&#39;pe&#39;, pe)<br>        <br>    def forward(self, x):<br>        # Get positional encodings corresponding to inputted timesteps<br>        x = self.pe[x]<br><br>        return x</pre><p>These time encodings are then passed through a MLP to further capture temporal information.</p><pre>self.time_mlp = nn.Sequential(<br>    SinusoidalPositionalEmbedding(config.decoder.max_time, config.decoder.model_channels),<br>    nn.Linear(config.decoder.model_channels, config.decoder.cond_channels),<br>    nn.SiLU(),<br>    nn.Linear(config.decoder.cond_channels, config.decoder.cond_channels)<br>)</pre><p>There are multiple ways to condition the input on this timestep embedding which are described in the UNet: Residual Blocks section of this article.</p><p><strong>Prior Model</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*J9XikjlQ-2XsqrYV8eRIrA.png" /><figcaption>Figure 3: Diffusion Prior Sampling Overview</figcaption></figure><p>The prior model is used to generate CLIP image embeddings from text captions. It is possible to forgo the prior model and condition on the CLIP text embeddings instead of the CLIP image embeddings generated by the prior, but it was that using the prior model performed the best.</p><p>There are two main model classes that can be used for the prior model, an autoregressive prior and a diffusion prior. For our model, we are going to be using the diffusion prior because Ramesh et al. 2022 found that it outperformed the autoregressive prior for comparable model size and reduced training compute. Classifier-free guidance is something that can be implemented in order to improve sample quality for both the autoregressive and diffusion prior, but we are not going to be using it for this article.</p><p>The diffusion prior starts by taking the text captions as an input and getting the CLIP text and image embeddings. When loading the CLIP model in, all of the layers should be frozen and the mode should be set to eval.</p><pre>def freeze_model(model, set_eval=True):<br>    if set_eval:<br>        model.eval()<br><br>    for param in model.parameters():<br>        param.requires_grad = False<br><br># Constructor<br>self.clip = CLIP(config).to(config.device)<br>self.clip.load_state_dict(torch.load(config.clip.model_location, map_location=config.device))<br>freeze_model(self.clip)<br><br># Forward<br>image_embeddings = self.clip.image_encoder(images) # (B, C, H, W) -&gt; (B, latent_dim)<br>text_embeddings = self.clip.text_encoder(captions, mask=masks) # (B, text_seq_length) -&gt; (B, latent_dim)</pre><p>It then gets random timesteps for each of the images in the batch and uses them to get the noisy CLIP image embeddings from forward diffusion.</p><pre># Forward<br>timesteps = torch.randint(0, self.config.prior.max_time, (images.shape[0],)) # (B, )<br><br>noisy_image_embedding, _ = forward_diffusion(image_embeddings, self.schedule_values, timesteps)</pre><p>The timesteps are then passed through a MLP in order to get the timestep embeddings.</p><pre># Constructor<br>self.time_mlp = nn.Sequential(<br>    SinusoidalPositionalEmbedding(config.prior.max_time, config.latent_dim),<br>    nn.Linear(config.latent_dim, config.latent_dim * config.prior.r_mlp, bias=config.prior.bias),<br>    nn.SiLU(),<br>    nn.Linear(config.latent_dim * config.prior.r_mlp, config.latent_dim, bias=config.prior.bias)<br>)<br><br># Forward<br>timestep_embeddings = self.time_mlp(timesteps) # (B, ) -&gt; (B, latent_dim)</pre><p>Another important part of the prior model are the learned embeddings. These embeddings are a torch parameter that are going to be used to predict the final output. The learned embedding from the constructor is going to need to be expanded in the forward method so that there is an embedding for each item in the batch.</p><pre># Constructor<br>self.learned_embedding = nn.Parameter(torch.randn(config.latent_dim))<br><br># Forward<br>learned_embeddings = self.learned_embedding.repeat(images.shape[0], 1) # (latent_dim) -&gt; (B, latent_dim)</pre><p>The text captions, CLIP text embeddings, timestep embeddings, noisy CLIP image embeddings, and learned embeddings are going to be concatenated into a sequence. All these items are going to have shape (B, latent_dim), but we are going to want to add an extra dimension in the middle to give them shape (B, 1, latent_dim). We are going to want to concatenate on this new dimension which should give the sequence shape (B, 5, latent_dim). If using a dataset with higher quality images, one possible way to improve the quality of the model could be to pass the text embeddings, image embeddings, and/or timestep embeddings through convolution layers to increase the sequence length.</p><pre>tokens = torch.cat((<br>    captions,               # Image Caption<br>    text_embeddings,        # CLIP Text Embedding<br>    timestep_embeddings,    # Timestep Embedding<br>    noisy_image_embedding,  # Noisy CLIP Image Embedding<br>    learned_embeddings      # Learned Embedding<br>), dim=1) # (B, 5, latent_dim)</pre><p>This sequence is then passed through a decoder-only Transformer with a causal attention mask. A causal attention mask is a lower triangular matrix of ones which makes it so that tokens can only attend to tokens that came before them.</p><pre># Constructor<br>self.decoder = nn.ModuleList(<br>    [TransformerBlock(<br>        config.latent_dim,<br>        cond_width=config.latent_dim,<br>        n_heads=config.prior.n_heads,<br>        dropout=config.prior.dropout,<br>        r_mlp=config.prior.r_mlp,<br>        bias=config.prior.bias<br>    ) for _ in range(config.prior.n_layers)]<br>)<br><br>self.register_buffer(&quot;causal_attention_mask&quot;, torch.tril(torch.ones(5, 5))[None, :])<br><br># Forward<br>for block in self.decoder:<br>    tokens = block(tokens, mask=self.causal_attention_mask)</pre><p>Finally, we take the learned embeddings from the output of the Transformer and pass it through a LayerNorm and Linear layer to get the predicted image embeddings.</p><pre><br># Constructor<br>self.output = nn.Sequential(<br>    nn.LayerNorm(config.latent_dim),<br>    nn.Linear(config.latent_dim, config.latent_dim, bias=config.decoder.bias)<br>)<br><br># Forward<br>pred_image_embeddings = self.output(tokens[:, -1, :])</pre><p>While normally diffusion models would predict the noise and generate the sample by removing the noise iteratively, it was mentioned in the unCLIP paper that it was better to just predict the CLIP image embeddings directly. The loss function for this model should be a mean-squared error loss between the predicted and actual CLIP image embeddings.</p><pre>loss = nn.functional.mse_loss(pred_image_embeddings, image_embeddings)</pre><p>During sampling, in order to improve quality, the model should generate two samples of the CLIP image embeddings and choose the one that has the higher dot product with the CLIP text embedding.</p><pre>def get_one_sample(self, text_embeddings, captions):<br>    # Get image embeddings that are pure noise<br>    noisy_image_embeddings = torch.randn(text_embeddings.shape, device=self.config.device)<br><br>    # timestep is max for all items because image embeddings are pure noise<br>    timesteps = torch.full((captions.shape[0],), self.config.prior.max_time - 1)<br><br>    # Get timestep embeddings<br>    timestep_embeddings = self.time_mlp(timesteps) # (B, ) -&gt; (B, latent_dim)<br>    timestep_embeddings = timestep_embeddings[:, None, :] # (B, latent_dim) -&gt; (B, 1, latent_dim)<br><br>    # Expand learned embedding so that there is one for each item in batch<br>    learned_embeddings = self.learned_embedding.repeat(captions.shape[0], 1) # (latent_dim) -&gt; (B, latent_dim)<br>    learned_embeddings = learned_embeddings[:, None, :] # (B, latent_dim) -&gt; (B, 1, latent_dim)<br><br>    tokens = torch.cat((<br>        captions,               # Image Caption<br>        text_embeddings,        # CLIP Text Embedding<br>        timestep_embeddings,    # Timestep Embedding<br>        noisy_image_embeddings,  # Noisy CLIP Image Embedding<br>        learned_embeddings      # Learned Embedding<br>    ), dim=1) # (B, 5, latent_dim)<br><br>    # Pass through transformer blocks with causal attention mask<br>    for block in self.decoder:<br>        tokens = block(tokens, mask=self.causal_attention_mask)<br><br>    # Get learned embeddings and pass through output projection to get CLIP image embeddings<br>    pred_image_embeddings = self.output(tokens[:, -1, :])<br><br>    return pred_image_embeddings<br><br>def sample(self, captions, masks=None):<br>    # Get CLIP text embeddings<br>    t_emb = self.clip.text_encoder(captions, mask=masks) # (B, text_seq_length) -&gt; (B, latent_dim)<br>    text_embeddings = t_emb[:, None, :] # (B, latent_dim) -&gt; (B, 1, latent_dim)<br><br>    # Make caption length equal to latent dimension<br>    if self.config.text_seq_length &gt;= self.config.latent_dim:<br>        captions = captions[:, :self.config.latent_dim]  # (B, max_seq_len) -&gt; (B, latent_dim)<br>    else:<br>        captions = nn.functional.pad(captions, (0, self.config.latent_dim - self.config.text_seq_length)) # (B, max_seq_len) -&gt; (B, latent_dim)<br><br>    captions = captions[:, None, :] # (B, latent_dim) -&gt; (B, 1, latent_dim)<br><br>    # Getting two samples<br>    sample_1 = self.get_one_sample(text_embeddings, captions)<br>    sample_2 = self.get_one_sample(text_embeddings, captions)<br><br>    gen_image_embeddings = torch.zeros(sample_1.shape)<br><br>    # Choosing the samples with the higher dot product with text embeddings<br>    for i in range(gen_image_embeddings.shape[0]):<br>        if sample_1[i] @ t_emb[i] &gt;= sample_2[i] @ t_emb[i]:<br>            gen_image_embeddings[i] = sample_1[i]<br>        else:<br>            gen_image_embeddings[i] = sample_2[i]<br><br>    return gen_image_embeddings</pre><p>Putting all of this together, the code for the diffusion prior model should look something like this:</p><pre>class DiffusionPrior(nn.Module):<br>    def __init__(self, config):<br>        super().__init__()<br>        # Loading CLIP Model<br>        self.clip = CLIP(config).to(config.device)<br>        self.clip.load_state_dict(torch.load(config.clip.model_location, map_location=config.device))<br>        freeze_model(self.clip)<br><br>        self.config = config<br><br>        self.time_mlp = nn.Sequential(<br>            SinusoidalPositionalEmbedding(config.prior.max_time, config.latent_dim),<br>            nn.Linear(config.latent_dim, config.latent_dim * config.prior.r_mlp, bias=config.prior.bias),<br>            nn.SiLU(),<br>            nn.Linear(config.latent_dim * config.prior.r_mlp, config.latent_dim, bias=config.prior.bias)<br>        )<br><br>        self.learned_embedding = nn.Parameter(torch.randn(config.latent_dim))<br><br>        self.schedule_values = get_schedule_values(config)<br><br>        # Transformer blocks<br>        self.decoder = nn.ModuleList(<br>            [TransformerBlock(<br>                config.latent_dim,<br>                cond_width=config.latent_dim,<br>                n_heads=config.prior.n_heads,<br>                dropout=config.prior.dropout,<br>                r_mlp=config.prior.r_mlp,<br>                bias=config.prior.bias<br>            ) for _ in range(config.prior.n_layers)]<br>        )<br><br>        # Output Projection<br>        self.output = nn.Sequential(<br>            nn.LayerNorm(config.latent_dim),<br>            nn.Linear(config.latent_dim, config.latent_dim, bias=config.decoder.bias)<br>        )<br><br>        self.register_buffer(&quot;causal_attention_mask&quot;, torch.tril(torch.ones(5, 5))[None, :])<br><br>    def get_one_sample(self, text_embeddings, captions):<br>        # Get image embeddings that are pure noise<br>        noisy_image_embeddings = torch.randn(text_embeddings.shape, device=self.config.device)<br><br>        # timestep is max for all items because image embeddings are pure noise<br>        timesteps = torch.full((captions.shape[0],), self.config.prior.max_time - 1)<br><br>        # Get timestep embeddings<br>        timestep_embeddings = self.time_mlp(timesteps) # (B, ) -&gt; (B, latent_dim)<br>        timestep_embeddings = timestep_embeddings[:, None, :] # (B, latent_dim) -&gt; (B, 1, latent_dim)<br><br>        # Expand learned embedding so that there is one for each item in batch<br>        learned_embeddings = self.learned_embedding.repeat(captions.shape[0], 1) # (latent_dim) -&gt; (B, latent_dim)<br>        learned_embeddings = learned_embeddings[:, None, :] # (B, latent_dim) -&gt; (B, 1, latent_dim)<br><br>        tokens = torch.cat((<br>            captions,               # Image Caption<br>            text_embeddings,        # CLIP Text Embedding<br>            timestep_embeddings,    # Timestep Embedding<br>            noisy_image_embeddings,  # Noisy CLIP Image Embedding<br>            learned_embeddings      # Learned Embedding<br>        ), dim=1) # (B, 5, latent_dim)<br><br>        # Pass through transformer blocks with causal attention mask<br>        for block in self.decoder:<br>            tokens = block(tokens, mask=self.causal_attention_mask)<br><br>        # Get learned embeddings and pass through output projection to get CLIP image embeddings<br>        pred_image_embeddings = self.output(tokens[:, -1, :])<br><br>        return pred_image_embeddings<br><br>    def sample(self, captions, masks=None):<br>        # Get CLIP text embeddings<br>        t_emb = self.clip.text_encoder(captions, mask=masks) # (B, text_seq_length) -&gt; (B, latent_dim)<br>        text_embeddings = t_emb[:, None, :] # (B, latent_dim) -&gt; (B, 1, latent_dim)<br><br>        # Make caption length equal to latent dimension<br>        if self.config.text_seq_length &gt;= self.config.latent_dim:<br>            captions = captions[:, :self.config.latent_dim]  # (B, max_seq_len) -&gt; (B, latent_dim)<br>        else:<br>            captions = nn.functional.pad(captions, (0, self.config.latent_dim - self.config.text_seq_length)) # (B, max_seq_len) -&gt; (B, latent_dim)<br><br>        captions = captions[:, None, :] # (B, latent_dim) -&gt; (B, 1, latent_dim)<br><br>        # Getting two samples<br>        sample_1 = self.get_one_sample(text_embeddings, captions)<br>        sample_2 = self.get_one_sample(text_embeddings, captions)<br><br>        gen_image_embeddings = torch.zeros(sample_1.shape)<br><br>        # Choosing the samples with the higher dot product with text embeddings<br>        for i in range(gen_image_embeddings.shape[0]):<br>            if sample_1[i] @ t_emb[i] &gt;= sample_2[i] @ t_emb[i]:<br>                gen_image_embeddings[i] = sample_1[i]<br>            else:<br>                gen_image_embeddings[i] = sample_2[i]<br><br>        return gen_image_embeddings<br><br>    def forward(self, images, captions, masks=None):<br>        # Get CLIP image embeddings<br>        image_embeddings = self.clip.image_encoder(images) # (B, C, H, W) -&gt; (B, latent_dim)<br><br>        # Get CLIP text embeddings<br>        text_embeddings = self.clip.text_encoder(captions, mask=masks) # (B, text_seq_length) -&gt; (B, latent_dim)<br>        text_embeddings = text_embeddings[:, None, :] # (B, latent_dim) -&gt; (B, 1, latent_dim)<br><br>        # Make caption length equal to latent dimension<br>        if self.config.text_seq_length &gt;= self.config.latent_dim:<br>            captions = captions[:, :self.config.latent_dim]  # (B, max_seq_len) -&gt; (B, latent_dim)<br>        else:<br>            captions = nn.functional.pad(captions, (0, self.config.latent_dim - self.config.text_seq_length)) # (B, max_seq_len) -&gt; (B, latent_dim)<br><br>        captions = captions[:, None, :] # (B, latent_dim) -&gt; (B, 1, latent_dim)<br><br>        # Get random timesteps for forward diffusion<br>        timesteps = torch.randint(0, self.config.prior.max_time, (images.shape[0],)) # (B, )<br><br>        # Get timestep embeddings<br>        timestep_embeddings = self.time_mlp(timesteps) # (B, ) -&gt; (B, latent_dim)<br>        timestep_embeddings = timestep_embeddings[:, None, :] # (B, latent_dim) -&gt; (B, 1, latent_dim)<br><br>        # Perform forward diffusion to get noisy CLIP image embeddings<br>        noisy_image_embedding, _ = forward_diffusion(image_embeddings, self.schedule_values, timesteps)<br>        noisy_image_embedding = noisy_image_embedding[:, None, :] # (B, latent_dim) -&gt; (B, 1, latent_dim)<br><br>        # Expand learned embedding so that there is one for each item in batch<br>        learned_embeddings = self.learned_embedding.repeat(images.shape[0], 1) # (latent_dim) -&gt; (B, latent_dim)<br>        learned_embeddings = learned_embeddings[:, None, :] # (B, latent_dim) -&gt; (B, 1, latent_dim)<br><br>        tokens = torch.cat((<br>            captions,               # Image Caption<br>            text_embeddings,        # CLIP Text Embedding<br>            timestep_embeddings,    # Timestep Embedding<br>            noisy_image_embedding,  # Noisy CLIP Image Embedding<br>            learned_embeddings      # Learned Embedding<br>        ), dim=1) # (B, 5, latent_dim)<br><br>        # Pass through transformer blocks with causal attention mask<br>        for block in self.decoder:<br>            tokens = block(tokens, mask=self.causal_attention_mask)<br><br>        # Get learned embeddings and pass through output projection to get CLIP image embeddings<br>        pred_image_embeddings = self.output(tokens[:, -1, :])<br><br>        loss = nn.functional.mse_loss(pred_image_embeddings, image_embeddings)<br><br>        return loss</pre><p><strong>Diffusion Decoder Overview</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1006/1*wJsOq133seopaPRyewZiDw.png" /><figcaption>Figure 4: Decoder UNet Overview. Image: own work.</figcaption></figure><p>The diffusion decoder is the part of the model where the image is created. It does this by predicting the noise that should be removed at each timestep and iteratively removing the predicted noise from a noisy image. In the unCLIP model used by <a href="https://cdn.openai.com/papers/dall-e-2.pdf">Ramesh et al.</a>, they used a model that was extremely similar to the <a href="https://arxiv.org/pdf/2112.10741">GLIDE model by Nichol et al.</a>, and we are also going to be following that pretty closely.</p><p>The two model architectures that are most often used for the backbone of of this are UNet and Transformers.</p><p>One of the main benefits of the Transformers method is its scalability, it scales well with large datasets and more complex models. Another benefit is that while UNet is used primarily for images, Transformers are more flexible and can be used for other data types with significant changes.</p><p>UNets are the main choice for many image related tasks. This is because it is good at getting the local information through its convolutional layers while maintaining high-resolution features through its skip connections. Another benefit of UNets is that the shape of the input and the output should be the same, which is useful in image diffusion.</p><p>UNet works by taking the input and passing it through the encoder layers which identify/capture relevant while decreasing the resolution. It is then passed through the decoder layers which try to locate the features while increasing the resolution back to its original shape. Because spatial information during the encoder layers, skip connections are added from the encoder to the decoder in order to help preserve them.</p><p><strong>Decoder: Residual Blocks</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*EbsthCp0nrs_8fTCoOH1PQ.png" /><figcaption>Figure 5: Residual Block Overview</figcaption></figure><p>Both encoder and decoder layers are made of residual blocks that contain the convolutions that are used to identify and locate features. These residual blocks usual are comprised of Normalization, Activation, Convolution, Normalization, Activation, Convolution; however, our residual blocks are going to be Normalization, Convolution, Normalization, Activation, Convolution, Normalization (Shown in figure). This architecture was shown to improve performance while still maintaining non-linearity in the paper by <a href="https://arxiv.org/pdf/1610.02915">Han et al</a>.</p><pre># Constructor<br>self.layers1 = nn.Sequential(<br>    nn.GroupNorm(n_groups, d_in),<br>    nn.Conv2d(d_in, d_out, kernel_size, padding=1)<br>)<br><br>self.layers2 = nn.Sequential(<br>    nn.GroupNorm(n_groups, d_out),<br>    nn.SiLU(),<br>    nn.Dropout(p=dropout),<br>    nn.Conv2d(d_out, d_out, kernel_size, padding=1),<br>    nn.GroupNorm(n_groups, d_out)<br>)<br><br># Forward<br>x = self.layers1(x_0)<br><br>x = self.layers2(x)</pre><p>For the Normalization portion of our residual blocks, we are going to be using GroupNorm. GroupNorm is used instead of BatchNorm because its performance is independent of batch size which makes it perform better on small or variable batch sizes. Using GroupNorm also tends to improve the stability during training.</p><p>The Activation portion of the residual blocks is used for non-linearity. While ReLU is commonly used for residual networks, we are going to be using SiLU for our activation function. In the paper <a href="https://arxiv.org/pdf/1710.05941">Ramachandran et al</a>., SiLU was shown to outperform ReLU and other activation functions despite models and hyperparameters being set specifically for ReLU. Because of SiLU’s simplicity and similarity to ReLU, it can be easily implemented by just using it in ReLU’s place.</p><p>For the Convolution, we are going to be using Conv2d with a 3x3 kernel size and SAME padding in order to preserves spatial dimensions. SAME padding works by adding zeros to the border of the input to insure that the output shape is the same as the input shape without padding. The first convolution will project the input from d_in channels to d_out channels while the second convolution will just keep d_out channels.</p><p>The residual block will also be conditioned on the inputted embedding information. One of the main ways that conditioning is performed is by just adding the embedding to the input. For our model, we are going to perform a linear projection in order to get a scale and a bias value. The input is then multiplied by the scale and the bias is added afterwards. <a href="https://arxiv.org/pdf/2105.05233">Nichol and Dhariwal</a> showed that using this method of conditioning improved the FID score compared to the addition method. One thing to note when coding this part is that the conditioning embedding is likely to have less dimensions than the input. Because of this, we need to add dimensions to the end of the embedding. For example, if the input had shape (B, C, L) and the embedding had shape (B, C), you would need to add a dimension to give it shape (B, C, 1).</p><pre># Constructor<br>self.use_scale_shift = use_scale_shift<br><br>self.cond_layers = nn.Sequential(<br>    nn.SiLU(),<br>    nn.Linear(model_channels, d_out * 2 if use_scale_shift else d_out)<br>)<br><br># Forward<br>emb = self.cond_layers(emb)<br><br>while len(emb.shape) &lt; len(x.shape):<br>    emb = emb[..., None]<br><br>if self.use_scale_shift:<br>    y_s, y_b = emb.chunk(2, dim=1)<br>    x = y_s * x + y_b<br>else:<br>    x += emb</pre><p>The conditioning is performed after the first convolution instead of at the beginning of the residual block because it allows the model to get the basic features before refining them with timestep information which results in better representations.</p><p>Finally skip connections are created by adding the original input of the ResNet block to the output. A 1x1 convolution is performed on the input if the number of channels needs to be changed in order to match the output. These skip connections are used to enhance feature propagation.</p><pre># Constructor<br>self.residual = nn.Conv2d(d_in, d_out, 1) if d_in != d_out else nn.Identity()<br><br># Forward<br>x += self.residual(x_0)</pre><p>Putting all of this together, the final code for the residual blocks should look something like this:</p><pre>class ResidualBlock(nn.Module):<br>    def __init__(self, d_in, d_out, cond_channels=128, n_groups=8, kernel_size=(3,3), dropout=0.0, use_scale_shift=True):<br>        super().__init__()<br><br>        self.use_scale_shift = use_scale_shift<br><br>        self.layers1 = nn.Sequential(<br>            nn.GroupNorm(n_groups, d_in),<br>            nn.SiLU(),<br>            nn.Conv2d(d_in, d_out, kernel_size, padding=1)<br>        )<br><br>        # Activation &amp; Linear Projection for Embedding<br>        self.cond_layers = nn.Sequential(<br>            nn.SiLU(),<br>            # d_out multiplied by 2 in order to split into scale &amp; shift if necessary<br>            nn.Linear(cond_channels, d_out * 2 if use_scale_shift else d_out)<br>        )<br><br>        self.layers2 = nn.Sequential(<br>            nn.GroupNorm(n_groups, d_out),<br>            nn.SiLU(),<br>            nn.Dropout(p=dropout),<br>            nn.Conv2d(d_out, d_out, kernel_size, padding=1),<br>            nn.GroupNorm(n_groups, d_out)<br>        )<br><br>        self.residual = nn.Conv2d(d_in, d_out, 1) if d_in != d_out else nn.Identity()<br><br>    def forward(self, x_0, emb):<br>        x = self.layers1(x_0)<br><br>        emb = self.cond_layers(emb)<br><br>        # Adding dimensions to embedding<br>        while len(emb.shape) &lt; len(x.shape):<br>            emb = emb[..., None]<br><br>        # Conditioning input with embedding<br>        if self.use_scale_shift:<br>            # Getting scale and shift<br>            y_s, y_b = emb.chunk(2, dim=1)<br>            # Performing scale and shift<br>            x = y_s * x + y_b<br>        else:<br>            # Adding embedding to input<br>            x += emb<br><br>        x = self.layers2(x)<br><br>        # Skip Connection<br>        x += self.residual(x_0)<br><br>        return x</pre><p><strong>Decoder: Attention Blocks</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*8FSreyjP0V37MMUCK4d4xQ.png" /><figcaption>Figure 6: Attention Block Overview</figcaption></figure><p>In our model, we are going to place attention blocks after the residual blocks of the inner layers of the encoder and decoder as well as the bottleneck (bottom layer connecting encoder and decoder). For example, if the encoder and decoder have four layers, there would be an attention blocks on layers two and three.</p><p>Using attention blocks has multiple benefits to our model. One of the benefits of attention blocks is that it helps capture information about the spatial relationships between different parts of the image. Another benefit is that it helps weight how important the model’s features are. Finally, attention blocks can be used to condition the image generation process.</p><p>There are multiple ways to implement attention-mechanism for the attention blocks. Some of the options include self-attention and cross-attention. The method that we are going to use is going to be sort of a mix of the two which was used in the GLIDE model. For this method, we are going to get the keys and values from both the input and the conditioning information and concatenate them together to get the final keys and values.</p><pre>Q, K, V = self.qkv(x).chunk(3, dim=-1)<br><br>Q = Q.view(B, L, self.n_heads, self.head_size).transpose(1, 2)<br><br>K = K.view(B, L, self.n_heads, self.head_size).transpose(1, 2)<br><br>V = V.view(B, L, self.n_heads, self.head_size).transpose(1, 2)<br><br># Concatenating keys and values of input and condition<br>if cond is not None:<br>    k_c, v_c = self.cond_kv(cond).chunk(2, dim=-1)<br>    k_c = k_c.view(B, cond.shape[1], self.n_heads, self.head_size).transpose(1, 2)<br>    v_c = v_c.view(B, cond.shape[1], self.n_heads, self.head_size).transpose(1, 2)<br>    K = torch.cat((K, k_c), dim=-2)<br>    V = torch.cat((V, v_c), dim=-2)</pre><p>One thing to note is that input are images that have shape (B, C, H, W), but for our attention mechanism we are going to want the shape to be (B, L, C). Because of this, we are going to need to combine the H and W dimensions and transpose it with the C dimension. After performing attention, we are going to need to convert the output back to its original shape.</p><pre>b, c, h, w = x_0.shape<br><br># Changing shape to perform attention<br>x = x.permute(0, 2, 3, 1).view(b, h * w, c) # (B, C, H, W) -&gt; (B, H * W, C)<br><br># Attention<br>x = self.attention(x, cond)<br><br># Changing back to original shape<br>x = x.view(b, h, w, c).permute(0, 3, 1, 2)</pre><p>The final code should look something like this:</p><pre>class AttentionBlock(nn.Module):<br>    def __init__(self, n_channels, cond_channels, n_groups=8, n_heads=1, dropout=0.0):<br>        super().__init__()<br><br>        assert n_channels % n_heads == 0, &quot;n_channels must be divisible by n_heads&quot;<br><br>        self.n_heads = n_heads<br>        self.head_size = n_channels // n_heads<br>        self.scale = self.head_size ** -0.5<br><br>        self.group_norm = nn.GroupNorm(n_groups, n_channels)<br><br>        self.qkv = nn.Linear(n_channels, n_channels * 3)<br><br>        self.cond_kv = nn.Linear(cond_channels, n_channels * 2)<br><br>        self.out_proj = nn.Linear(n_channels, n_channels)<br><br>        self.dropout = nn.Dropout(dropout)<br><br>    def attention(self, x, cond=None):<br>        B, L, _ = x.shape<br><br>        # Getting queries, keys, and values for input<br>        Q, K, V = self.qkv(x).chunk(3, dim=-1)<br><br>        Q = Q.view(B, L, self.n_heads, self.head_size).transpose(1, 2)<br><br>        K = K.view(B, L, self.n_heads, self.head_size).transpose(1, 2)<br><br>        V = V.view(B, L, self.n_heads, self.head_size).transpose(1, 2)<br><br>        # Concatenating keys and values of condition to keys and values of input<br>        if cond is not None:<br>            k_c, v_c = self.cond_kv(cond).chunk(2, dim=-1)<br>            k_c = k_c.view(B, cond.shape[1], self.n_heads, self.head_size).transpose(1, 2)<br>            v_c = v_c.view(B, cond.shape[1], self.n_heads, self.head_size).transpose(1, 2)<br>            K = torch.cat((K, k_c), dim=-2)<br>            V = torch.cat((V, v_c), dim=-2)<br><br>        # Get dot product between queries and keys<br>        attention = torch.matmul(Q, K.transpose(-2, -1))<br><br>        # Scale<br>        attention = attention * self.scale<br><br>        # Applying softmax<br>        attention = torch.softmax(attention, dim=-1)<br><br>        # Get dot product with values<br>        attention = torch.matmul(attention, V)<br><br>        # Combine heads<br>        attention = attention.transpose(1, 2)<br>        attention = attention.contiguous().view(x.shape)<br><br>        # Output projection<br>        attention = self.out_proj(attention)<br><br>        # Dropout<br>        attention = self.dropout(attention)<br><br>        return attention<br><br>    def forward(self, x_0, cond=None):<br>        b, c, h, w = x_0.shape<br><br>        # Group normalization<br>        x = self.group_norm(x_0)<br><br>        # Changing shape to perform attention<br>        x = x.permute(0, 2, 3, 1).view(b, h * w, c) # (B, C, H, W) -&gt; (B, H * W, C)<br><br>        # Attention<br>        x = self.attention(x, cond)<br><br>        # Changing back to original shape<br>        x = x.view(b, h, w, c).permute(0, 3, 1, 2)<br><br>        # Residual connection<br>        x = x + x_0<br><br>        return x</pre><p><strong>Decoder: Downsampling</strong></p><p>In between each of our encoder layers, we are going to need to reduce the resolution of the input. There are two main downsampling methods: pooling and using convolutional layers. Pooling is parameter-free which makes it more computationally efficient and can help with overfitting. The benefit of convolutional layers is that it does have parameters which makes it able to learn and preserve important features. We are going to use a stride of 2 for our model which will reduce the resolution of the inputs by a factor of 2.</p><p>We are going to code our model to be able to use either method, but when training our model, we are going to have it used the convolutional method.</p><pre>class Downsample(nn.Module):<br>    def __init__(self, n_channels, kernel_size=(3,3), stride=2, down_pool=False):<br>        super().__init__()<br><br>        if down_pool:<br>            self.down = nn.AvgPool2d(stride)<br>        else:<br>            self.down = nn.Conv2d(n_channels, n_channels, kernel_size, stride=stride, padding=1)<br><br>    def forward(self, x):<br>        x = self.down(x)<br>        return x</pre><p><strong>Decoder: Upsampling</strong></p><p>In between each of our decoder layers, we are going to need to increase the resolution of the inputs that are passed through. In order to do this, we are going to first perform interpolation, which will multiply the height and width of the input by a factor of 2. Afterwards, we are going to pass it through a convolutional layer, which will learn and preserve important features while also ensuring that the number of channels is correct.</p><pre>class Upsample(nn.Module):<br>    def __init__(self, d_in, d_out, kernel_size=(3,3)):<br>        super().__init__()<br><br>        self.conv = nn.Conv2d(d_in, d_out, kernel_size, padding=1)<br><br>    def forward(self, x):<br>        x = nn.functional.interpolate(x, scale_factor=2)<br>        x = self.conv(x)<br>        return x</pre><p><strong>Decoder: Final Model</strong></p><p>When training the decoder, the first thing that the model needs to do is set up the conditioning information. To do that, we first need to sample from the Prior model in order to get the CLIP image embeddings. Like with the CLIP model, when loading the Prior model in, the layers should be frozen and the mode should be set to eval.</p><pre># Constructor<br>self.prior = DiffusionPrior(config).to(config.device)<br>self.prior.load_state_dict(torch.load(config.prior.model_location, map_location=config.device))<br>freeze_model(self.prior)<br><br># Forward<br>img_embeddings = self.prior.sample(caption, mask).to(x.device)</pre><p>Afterwards, the inputted timesteps need to be used to get the timestep embeddings (more info in previous timestep embedding section). The CLIP image embeddings that were generated by the prior model are then projected and added to the timestep embeddings. These embeddings are the ones that are used to condition the model’s residual blocks.</p><pre># Constructor<br>self.img_projection = nn.Sequential(<br>    nn.Linear(config.latent_dim, config.decoder.cond_channels),<br>    nn.SiLU(),<br>    nn.Linear(config.decoder.cond_channels, config.decoder.cond_channels)<br>)<br><br>#Forward<br>c_emb = self.time_mlp(time) + self.img_projection(img_embeddings)<br><br>for module in self.encoder:<br>    if isinstance(module, ResidualBlock):<br>        x = module(x, c_emb)</pre><p>For the attention conditioning information, we first need to pass the text captions through a text Transformer encoder. In the unCLIP paper, these text encodings were used because Ramesh et al. thought that it would help learn aspects of natural language that CLIP couldn’t. During testing, they found that it did not have that much effect with this, so this part is optional.</p><pre># Constructor<br>self.text_embedding = nn.Embedding(config.vocab_size, config.latent_dim)<br>self.positional_encodings = nn.Parameter(torch.randn(config.text_seq_length,config.latent_dim) * (config.latent_dim ** -0.5))<br><br>self.text_encoder = nn.ModuleList(<br>    [TransformerBlock(<br>        config.latent_dim, <br>        cond_width=config.latent_dim, <br>        n_heads=config.decoder.n_heads, <br>        dropout=config.decoder.dropout, <br>        r_mlp=config.decoder.r_mlp, <br>        bias=config.decoder.bias<br>     ) for _ in range(config.decoder.text_layers)]<br>)<br><br>self.final_ln = nn.LayerNorm(config.latent_dim)<br><br># Function<br>def encode_text(self, text, mask=None):<br>    x = self.text_embedding(text)<br><br>    x = x + self.positional_encodings<br><br>    for block in self.text_encoder:<br>        x = block(x, mask=mask)<br><br>    x = self.final_ln(x)<br><br>    return x<br><br># Forward<br>text_encodings = self.encode_text(text, mask)</pre><p>After encoding the text, the CLIP image embeddings are projected into four extra tokens and concatenated onto the end of the encoded text.</p><pre># Constructor<br>self.get_img_tokens = nn.Linear(1, config.decoder.n_img_tokens)<br><br># Forward<br>img_tokens = self.get_img_tokens(img_embeddings[..., None]).permute(0, 2, 1)<br><br>c_attn = torch.cat([text_encodings, img_tokens], dim=1)</pre><p>After setting up the conditioning information, the noisy images are passed through an initial convolutional layer in order to get the number of channels to the initial model channels.</p><pre># Constructor<br>ch = config.decoder.model_channels<br><br>self.in_conv = nn.Conv2d(config.img_channels, ch, config.decoder.kernel_size, padding=1)<br><br># Forward<br>x = self.in_conv(x)</pre><p>Now that the noisy images have the desired amount of channels, they can be passed through the UNet layer along with the conditioning information. The UNet is going to have four encoder and decoder layers with a bottleneck layer in between. For the number of channels at each layer, we are going to be using [1, 2, 4, 8] as the ratios of layer channels to model channels. The encoder, decoder, and bottleneck layers of the UNet all are going to have two residual blocks. In between the bottleneck residual blocks and after each residual block in the inner layers of the encoder and decoder, attention blocks are placed (Figure 4). Skip connections are placed between the residual blocks of the encoder and decoder.</p><pre># Config<br>model_channels:int = 32<br>channel_ratios:list[int] = field(default_factory=lambda: [1, 2, 4, 8])<br>n_layer_blocks:int = 2<br><br># Constructor<br># UNet Encoder Layers<br>self.encoder = nn.ModuleList([])<br>for r in config.decoder.channel_ratios:<br>    for _ in range(config.decoder.n_layer_blocks):<br>        self.encoder.append(ResidualBlock(ch, config.decoder.model_channels * r, config.decoder.cond_channels, config.decoder.n_groups, config.decoder.kernel_size, config.decoder.dropout, config.decoder.use_scale_shift))<br><br>        ch = config.decoder.model_channels * r<br>        if r != config.decoder.channel_ratios[0] and r != config.decoder.channel_ratios[-1]:<br>            self.encoder.append(AttentionBlock(ch, config.latent_dim, config.decoder.n_groups, config.decoder.n_heads, config.decoder.dropout))<br><br>    if r != config.decoder.channel_ratios[-1]:<br>        self.encoder.append(Downsample(ch, config.decoder.kernel_size, config.decoder.stride, config.decoder.down_pool))<br><br># UNet Bottleneck Layers<br>self.bottleneck = nn.ModuleList([])<br>for block in range(config.decoder.n_layer_blocks):<br>    self.bottleneck.append(ResidualBlock(ch, ch, config.decoder.cond_channels, config.decoder.n_groups, config.decoder.kernel_size, config.decoder.dropout, config.decoder.use_scale_shift))<br><br>    if block != config.decoder.n_layer_blocks - 1:<br>        self.bottleneck.append(AttentionBlock(ch, config.latent_dim, config.decoder.n_groups, config.decoder.n_heads, config.decoder.dropout))<br><br># UNet Decoder Layers<br>self.decoder = nn.ModuleList([])<br>for r in range(len(config.decoder.channel_ratios))[::-1]:<br>    for _ in range(config.decoder.n_layer_blocks):<br>        self.decoder.append(ResidualBlock(ch * 2, ch, config.decoder.cond_channels, config.decoder.n_groups, config.decoder.kernel_size, config.decoder.dropout, config.decoder.use_scale_shift))<br><br>        if r != 0 and r!= len(config.decoder.channel_ratios) - 1:<br>            self.decoder.append(AttentionBlock(ch, config.latent_dim, config.decoder.n_groups, config.decoder.n_heads, config.decoder.dropout))<br><br>    if r != 0:<br>        ch = config.decoder.model_channels * config.decoder.channel_ratios[r-1]<br>        self.decoder.append(Upsample(config.decoder.model_channels * config.decoder.channel_ratios[r], ch, config.decoder.kernel_size))<br><br># Forward<br>for module in self.encoder:<br>    if isinstance(module, ResidualBlock):<br>        x = module(x, c_emb)<br>        self.connections.append(x)<br>    elif isinstance(module, AttentionBlock):<br>        x = module(x, cond=c_attn)<br>    else:<br>        x = module(x)<br><br>for module in self.bottleneck:<br>    if isinstance(module, ResidualBlock):<br>        x = module(x, c_emb)<br>    elif isinstance(module, AttentionBlock):<br>        x = module(x, cond=c_attn)<br>    else:<br>        x = module(x)<br><br>for module in self.decoder:<br>    if isinstance(module, ResidualBlock):<br>        x = torch.cat([x, self.connections.pop()], dim=1)<br>        x = module(x, c_emb)<br>    elif isinstance(module, AttentionBlock):<br>        x = module(x, cond=c_attn)<br>    else:<br>        x = module(x)</pre><p>The output of the UNet decoder layers are then passed through a GroupNorm and SiLU activation layer before using Conv2d in order to get the output back to the original number of channels.</p><pre># Constructor<br>self.output = nn.Sequential(<br>    nn.GroupNorm(config.decoder.n_groups, config.decoder.model_channels),<br>    nn.SiLU(),<br>    nn.Conv2d(config.decoder.model_channels, config.img_channels, config.decoder.kernel_size, padding=1)<br>)<br><br># Forward<br>x = self.output(x)</pre><p>Putting everything together, the final code for the Decoder model will look something like this:</p><pre>class Decoder(nn.Module):<br>    def __init__(self, config):<br>        super().__init__()<br>        # Loading Prior Model<br>        self.prior = DiffusionPrior(config).to(config.device)<br>        self.prior.load_state_dict(torch.load(config.prior.model_location, map_location=config.device))<br>        freeze_model(self.prior)<br><br>        # MLP to get time embeddings<br>        self.time_mlp = nn.Sequential(<br>            SinusoidalPositionalEmbedding(config.decoder.max_time, config.decoder.model_channels),<br>            nn.Linear(config.decoder.model_channels, config.decoder.cond_channels),<br>            nn.SiLU(),<br>            nn.Linear(config.decoder.cond_channels, config.decoder.cond_channels)<br>        )<br><br>        # MLP to project CLIP image embeddings<br>        self.img_projection = nn.Sequential(<br>            nn.Linear(config.latent_dim, config.decoder.cond_channels),<br>            nn.SiLU(),<br>            nn.Linear(config.decoder.cond_channels, config.decoder.cond_channels)<br>        )<br><br>        # Projection to get image tokens<br>        self.get_img_tokens = nn.Linear(1, config.decoder.n_img_tokens)<br><br>        # Embedding layer for text captions<br>        self.text_embedding = nn.Embedding(config.vocab_size, config.latent_dim)<br><br>        # Learned positional encodings for text captions<br>        self.positional_encodings = nn.Parameter(torch.randn(config.text_seq_length,config.latent_dim) * (config.latent_dim ** -0.5))<br><br>        # Transformer encoder blocks to encoder text captions<br>        self.text_encoder = nn.ModuleList(<br>            [TransformerBlock(<br>                config.latent_dim,<br>                cond_width=config.latent_dim,<br>                n_heads=config.decoder.n_heads,<br>                dropout=config.decoder.dropout,<br>                r_mlp=config.decoder.r_mlp,<br>                bias=config.decoder.bias<br>            ) for _ in range(config.decoder.text_layers)]<br>        )<br><br>        self.final_ln = nn.LayerNorm(config.latent_dim)<br><br>        ch = config.decoder.model_channels<br><br>        # Initial convolution <br>        self.in_conv = nn.Conv2d(config.img_channels, ch, config.decoder.kernel_size, padding=1)<br><br>        # UNet Encoder Layers<br>        self.encoder = nn.ModuleList([])<br>        for r in config.decoder.channel_ratios:<br>            for _ in range(config.decoder.n_layer_blocks):<br>                self.encoder.append(ResidualBlock(ch, config.decoder.model_channels * r, config.decoder.cond_channels, config.decoder.n_groups, config.decoder.kernel_size, config.decoder.dropout, config.decoder.use_scale_shift))<br><br>                ch = config.decoder.model_channels * r<br>                if r != config.decoder.channel_ratios[0] and r != config.decoder.channel_ratios[-1]:<br>                    self.encoder.append(AttentionBlock(ch, config.latent_dim, config.decoder.n_groups, config.decoder.n_heads, config.decoder.dropout))<br><br>            if r != config.decoder.channel_ratios[-1]:<br>                self.encoder.append(Downsample(ch, config.decoder.kernel_size, config.decoder.stride, config.decoder.down_pool))<br><br>        # UNet Bottleneck Layers<br>        self.bottleneck = nn.ModuleList([])<br>        for block in range(config.decoder.n_layer_blocks):<br>            self.bottleneck.append(ResidualBlock(ch, ch, config.decoder.cond_channels, config.decoder.n_groups, config.decoder.kernel_size, config.decoder.dropout, config.decoder.use_scale_shift))<br><br>            if block != config.decoder.n_layer_blocks - 1:<br>                self.bottleneck.append(AttentionBlock(ch, config.latent_dim, config.decoder.n_groups, config.decoder.n_heads, config.decoder.dropout))<br><br>        # UNet Decoder Layers<br>        self.decoder = nn.ModuleList([])<br>        for r in range(len(config.decoder.channel_ratios))[::-1]:<br>            for _ in range(config.decoder.n_layer_blocks):<br>                self.decoder.append(ResidualBlock(ch * 2, ch, config.decoder.cond_channels, config.decoder.n_groups, config.decoder.kernel_size, config.decoder.dropout, config.decoder.use_scale_shift))<br><br>                if r != 0 and r!= len(config.decoder.channel_ratios) - 1:<br>                    self.decoder.append(AttentionBlock(ch, config.latent_dim, config.decoder.n_groups, config.decoder.n_heads, config.decoder.dropout))<br><br>            if r != 0:<br>                ch = config.decoder.model_channels * config.decoder.channel_ratios[r-1]<br>                self.decoder.append(Upsample(config.decoder.model_channels * config.decoder.channel_ratios[r], ch, config.decoder.kernel_size))<br><br>        # Output projection<br>        self.output = nn.Sequential(<br>            nn.GroupNorm(config.decoder.n_groups, config.decoder.model_channels),<br>            nn.SiLU(),<br>            nn.Conv2d(config.decoder.model_channels, config.img_channels, config.decoder.kernel_size, padding=1)<br>        )<br><br>        # Skip connections<br>        self.connections = []<br><br>    def encode_text(self, text, mask=None):<br>        x = self.text_embedding(text)<br><br>        x = x + self.positional_encodings<br><br>        for block in self.text_encoder:<br>            x = block(x, mask=mask)<br><br>        x = self.final_ln(x)<br><br>        return x<br><br>    def forward(self, x, time, caption=None, mask=None):<br>        # Sample prior model to get CLIP image embeddings<br>        img_embeddings = self.prior.sample(caption, mask).to(x.device)<br><br>        # Get conditioning information for residual blocks<br>        c_emb = self.time_mlp(time) + self.img_projection(img_embeddings)<br><br>        # Get conditioning information for attention blocks<br>        c_attn = self.get_img_tokens(img_embeddings[..., None]).permute(0, 2, 1)<br>        if caption is not None:<br>            c_attn = torch.cat([self.encode_text(caption, mask), c_attn], dim=1)<br><br>        # Initial convolution<br>        x = self.in_conv(x)<br><br>        # UNet encoder layers<br>        for module in self.encoder:<br>            if isinstance(module, ResidualBlock):<br>                x = module(x, c_emb)<br>                self.connections.append(x)<br>            elif isinstance(module, AttentionBlock):<br>                x = module(x, cond=c_attn)<br>            else:<br>                x = module(x)<br><br>        # UNet bottleneck layers<br>        for module in self.bottleneck:<br>            if isinstance(module, ResidualBlock):<br>                x = module(x, c_emb)<br>            elif isinstance(module, AttentionBlock):<br>                x = module(x, cond=c_attn)<br>            else:<br>                x = module(x)<br><br>        # UNet decoder layers<br>        for module in self.decoder:<br>            if isinstance(module, ResidualBlock):<br>                x = torch.cat([x, self.connections.pop()], dim=1)<br>                x = module(x, c_emb)<br>            elif isinstance(module, AttentionBlock):<br>                x = module(x, cond=c_attn)<br>            else:<br>                x = module(x)<br><br>        # Output projection<br>        x = self.output(x)<br><br>        return x</pre><p><strong>Decoder: Loss</strong></p><p>An important part of training the decoder model is the loss function. The first thing that were are going to do in our loss function is sample random timesteps for each image in the batch.</p><pre>timesteps = torch.randint(0, config.decoder.max_time, (image.shape[0],), device=config.device, dtype=torch.long)</pre><p>These timesteps are then used, along with the schedule values, to get the noisy images and noise added through forward diffusion.</p><pre>schedule_values = get_schedule_values(config)<br>noisy_image, noise = forward_diffusion(image, schedule_values, timesteps)</pre><p>The noisy images, timesteps, text captions, and text masks are then passed through the model to get the predicted noises.</p><pre>pred_noise = decoder(noisy_image, timesteps, caption, mask)</pre><p>Finally, we can use the predicted noise from the model and the actual noise from the forward diffusion function to calculate the loss. The loss that we are going to be using for the decoder is the mean squared error loss between the predicted noise and actual noise.</p><pre>loss = nn.functional.mse_loss(pred_noise, noise)</pre><p>Putting this all together we get:</p><pre># Calculating Loss<br>get_schedule_values(config)<br>timesteps = torch.randint(0, config.decoder.max_time, (image.shape[0],), device=config.device, dtype=torch.long)<br>noisy_image, noise = forward_diffusion(image, schedule_values, timesteps)<br>pred_noise = decoder(noisy_image, timesteps, caption, mask)<br>loss = nn.functional.mse_loss(pred_noise, noise)</pre><p><strong>Config</strong></p><pre>@dataclass<br>class CLIPConfig:<br>    # Vision Transformer<br>    patch_size:tuple[int,int] = (4,4)<br>    vit_width:int = 256<br>    vit_layers:int = 6<br>    vit_heads:int = 8<br>    # Text Transformer<br>    text_width:int = 256<br>    text_layers:int = 6<br>    text_heads:int = 8<br>    # Attention<br>    dropout:float = 0.2<br>    r_mlp:int = 4<br>    bias:bool = False<br>    # Training<br>    augment_data:bool = True<br>    num_workers:int = 0<br>    batch_size:int = 128<br>    lr:float = 5e-4<br>    lr_min:float = 1e-5<br>    weight_decay:float = 1e-4<br>    epochs:int = 200<br>    warmup_epochs:int = 5<br>    grad_max_norm:float = 1.0<br>    get_val_accuracy:bool = False<br>    model_location:str = &quot;../clip_fmnist.pt&quot;<br><br>@dataclass<br>class PriorConfig:<br>    # Diffusion<br>    max_time:int = 1000<br>    schedule:str = &quot;cosine&quot;<br>    schedule_offset:float = 0.008<br>    # Transformer Decoder<br>    width:int = 256<br>    n_layers:int = 6<br>    n_heads:int = 8<br>    # Attention<br>    dropout:float = 0.2<br>    r_mlp:int = 4<br>    bias:bool = False<br>    # Training<br>    augment_data:bool = False<br>    num_workers:int = 0<br>    batch_size:int = 128<br>    lr:float = 5e-4<br>    lr_min:float = 1e-5<br>    weight_decay:float = 1e-4<br>    epochs:int = 150<br>    warmup_epochs:int = 5<br>    grad_max_norm:float = 1.0<br>    model_location:str = &quot;../prior_fmnist.pt&quot;<br><br>@dataclass<br>class DecoderConfig:<br>    # Diffusion<br>    max_time:int = 1000<br>    schedule:str = &quot;cosine&quot;<br>    # UNet<br>    n_groups:int = 8<br>    kernel_size:tuple[int, int] = (3,3)<br>    model_channels:int = 32<br>    cond_channels:int = 128<br>    channel_ratios:list[int] = field(default_factory=lambda: [1, 2, 4, 8])<br>    n_layer_blocks:int = 2<br>    dropout:float = 0.1<br>    use_scale_shift:bool = True<br>    n_heads:int = 8<br>    stride:int = 2<br>    down_pool:bool = False<br>    r_mlp:int = 4<br>    bias:bool = False<br>    text_layers:int = 4<br>    n_img_tokens:int = 4<br>    # Training<br>    augment_data:bool = False<br>    num_workers:int = 0<br>    batch_size:int = 32<br>    lr:float = 5e-4<br>    lr_min:float = 1e-5<br>    weight_decay:float = 1e-4<br>    epochs:int = 100<br>    warmup_epochs:int = 5<br>    grad_max_norm:float = 1.0<br>    sample_after_epoch:bool = True<br>    model_location:str = &quot;../decoder_fmnist.pt&quot;<br><br>@dataclass<br>class FMNISTConfig:<br>    latent_dim:int = 256<br>    # Dataset Info<br>    dataset:str = &quot;fashion_mnist&quot;<br>    data_location:str = &quot;./../datasets&quot;<br>    img_size:tuple[int,int] = (32,32)<br>    img_channels:int = 1<br>    vocab_size:int = 256<br>    text_seq_length:int = 64<br>    # Data Augmentation / Normalization<br>    prob_hflip:float = 0.5<br>    crop_padding:int = 4<br>    train_mean:list[float] = field(default_factory=lambda: [0.2855552])<br>    train_std:list[float] = field(default_factory=lambda: [0.33848408])<br>    # Training<br>    train_val_split:tuple[int,int] = (50000, 10000)<br>    device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br>    # Model Configs<br>    clip = CLIPConfig()<br>    prior = PriorConfig()<br>    decoder = DecoderConfig()</pre><p><strong>Training</strong></p><p>For this text-to-image diffusion model, we are actually going to have to train 3 separate models: CLIP, diffusion prior, and the decoder. Including all the code of the three training scripts would be a lot, so if you want to see them, you can find them in the GitHub repo that I linked in the introduction.</p><p>For the dataset, all of the images were resized from (28,28) to (32,32). The images were also normalized with the mean and standard deviation set to the mean and standard deviation of the training split.</p><pre>transform = T.Compose([<br>    T.Resize(config.img_size),<br>    T.ToTensor(),<br>    T.Normalize(config.train_mean, config.train_std)<br>])</pre><p>For the training set of the CLIP model, I also implemented data augmentation for the training split by randomly flipping horizontally and also randomly cropping.</p><pre>transform = T.Compose([<br>    T.Resize(config.img_size),<br>    T.RandomHorizontalFlip(p=config.prob_hflip)<br>    T.RandomCrop(config.img_size[0], padding=config.crop_padding)<br>    T.ToTensor(),<br>    T.Normalize(config.train_mean, config.train_std)<br>])</pre><p>When training all three of the models, I used AdamW for the optimizer when using weight decay and Adam otherwise.</p><pre>if config.clip.weight_decay == 0:<br>    optimizer = Adam(clip.parameters(), lr=config.clip.lr)<br>else:<br>    optimizer = AdamW(clip.parameters(), lr=config.clip.lr, weight_decay=config.clip.weight_decay)</pre><p>For all of the models, I also used a cosine annealing learning rate scheduler with a linear warmup. The schedule was updated at the end of every epoch.</p><pre>if config.clip.warmup_epochs &gt; 0:<br>    warmup = lr_scheduler.LinearLR(optimizer=optimizer, start_factor=(1 / config.clip.warmup_epochs), end_factor=1.0, total_iters=(config.clip.warmup_epochs - 1), last_epoch=-1)<br><br>scheduler = lr_scheduler.CosineAnnealingLR(optimizer, T_max=(config.clip.epochs - config.clip.warmup_epochs), eta_min=config.clip.lr_min)</pre><p>Gradient clipping with a max norm of 1.0 was also used for all of the models to help prevent exploding gradients.</p><pre> torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=config.[model].grad_max_norm)</pre><p>When training the CLIP model, the training data was randomly split into a training split and a validation split. The model weights were saved when the validation loss for an epoch was lower than or equal to the previous lowest validation loss.</p><p><strong>Sampling</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mFuVpd-Vkt0Us8jKo6o5mA.png" /><figcaption>Figure 7: Sampling Algorithm (Source: <a href="https://arxiv.org/pdf/2006.11239">Ho et al. 2020</a>)</figcaption></figure><p>The first step in the sampling process is to randomly generate an image that contains only noise.</p><pre>B = prompts.shape[0]<br># Get completely noisy image<br>img = torch.randn((B, config.img_channels, config.img_size[0], config.img_size[1]), device=config.device)</pre><p>After getting the noisy image, we are going to calculate some of the schedule values that we are going to need to calculate x_{t-1}.</p><pre>def get_schedule_values(config):<br>        schedule_values = {}<br>        schedule_values[&quot;betas&quot;] = get_beta_schedule(config.decoder.schedule, config.decoder.max_time).to(config.device)<br>        schedule_values[&quot;alphas&quot;] = 1.0 - schedule_values[&quot;betas&quot;]<br>        schedule_values[&quot;alpha_bars&quot;] = torch.cumprod(schedule_values[&quot;alphas&quot;], axis = 0)<br>        schedule_values[&quot;sqrt_recip_alphas&quot;] = torch.sqrt(1.0 / schedule_values[&quot;alphas&quot;])<br>        schedule_values[&quot;sqrt_alpha_bars&quot;] = torch.sqrt(schedule_values[&quot;alpha_bars&quot;])<br>        schedule_values[&quot;sqrt_one_minus_alpha_bars&quot;] = torch.sqrt(1.0 - schedule_values[&quot;alpha_bars&quot;])<br>        schedule_values[&quot;alpha_bars_prev&quot;] = torch.cat((torch.ones(1, device=config.device), schedule_values[&quot;alpha_bars&quot;][:-1]))<br>        schedule_values[&quot;sigma&quot;] = schedule_values[&quot;betas&quot;] * (1.0 - schedule_values[&quot;alpha_bars_prev&quot;]) / (1.0 - schedule_values[&quot;alpha_bars&quot;])<br>        return schedule_values<br><br>schedule_values = get_schedule_values(config)</pre><p>We then need to iteratively go through all of the timesteps, from 0 to max_time-1, in reverse order. We also need to expand the timestep so that there is a timestep for every item in the batch.</p><pre>for t in range(0, config.max_time)[::-1]:<br>    timesteps = torch.full((B,), t, device=config.device, dtype=torch.long)</pre><p>For sampling at each timestep, the first step is to get the schedule values for only that timestep. After getting the timestep’s schedule values, they will need to be expanded so that the number of dimensions is equal to that of the images.</p><pre># Getting schedule values for timestep<br>sqrt_recip_alphas_t = extract_and_expand(schedule_values[&quot;sqrt_recip_alphas&quot;], timesteps, img.shape)<br>betas_t = extract_and_expand(schedule_values[&quot;betas&quot;], timesteps, img.shape)<br>sqrt_one_minus_alpha_bars_t = extract_and_expand(schedule_values[&quot;sqrt_one_minus_alpha_bars&quot;], timesteps, img.shape)<br>sigma_t = extract_and_expand(schedule_values[&quot;sigma&quot;], timesteps, img.shape)</pre><p>Afterwards, we use the decoder model to predict the noise at timestep t.</p><pre># Predicting noise at timestep t with decoder<br>pred_noise = decoder(img, timesteps, caption=prompt, mask=mask)</pre><p>If it is not the final timestep, we also need to generate random noise (z). We need this and sigma_t because of the fact that our model predicts the noise from x_t to x_0 directly instead of x_t to x_{t-1}. Because of this, the model predicts the noise from x_t to x_0 and adds the noise from x_{t-1} to x_0 back in. This makes it so that only the noise from x_t to x_{t-1} is removed. The sigma_t and z values are not need at the final step because the predicted noise from x_t to x_0 is the same as x_t to x_{t-1}.</p><pre># Generating random noise<br>z = torch.randn_like(img) if t &gt; 0 else 0</pre><p>With the generated noise, schedule values, and predicted noise, we can calculate the image at timestep t-1.</p><pre># Calculating image at timestep t-1<br>img = sqrt_recip_alphas_t * (img - (betas_t / sqrt_one_minus_alpha_bars_t) * pred_noise) + (sigma_t * z)<br>img = torch.clamp(img, -1.0, 1.0)</pre><p>Putting this together, the code for sampling and image will look something like this:</p><pre>@torch.no_grad()<br>def sample_image(config, prompt, mask, schedule_values=None):<br>    # Load decoder model<br>    decoder = Decoder(config).to(config.device)<br>    decoder.load_state_dict(torch.load(config.decoder.model_location, map_location=config.device))<br>    decoder.eval()<br><br>    B = prompt.shape[0]<br>    # Get completely noisy image<br>    img = torch.randn((B, config.img_channels, config.img_size[0], config.img_size[1]), device=config.device)<br><br>    # Calculate schedule values<br>    if schedule_values is None:<br>        schedule_values = get_schedule_values(config)<br><br>    for t in range(0, config.decoder.max_time)[::-1]:<br>        # Setting the timesteps for all the items in the batch<br>        timesteps = torch.full((B,), t, device=config.device, dtype=torch.long)<br><br>        # Getting schedule values for timestep<br>        sqrt_recip_alphas_t = extract_and_expand(schedule_values[&quot;sqrt_recip_alphas&quot;], timesteps, img.shape)<br>        betas_t = extract_and_expand(schedule_values[&quot;betas&quot;], timesteps, img.shape)<br>        sqrt_one_minus_alpha_bars_t = extract_and_expand(schedule_values[&quot;sqrt_one_minus_alpha_bars&quot;], timesteps, img.shape)<br>        sigma_t = extract_and_expand(schedule_values[&quot;sigma&quot;], timesteps, img.shape)<br><br>        # Predicting noise at timestep t with decoder<br>        pred_noise = decoder(img, timesteps, caption=prompt, mask=mask)<br><br>        # Generating random noise<br>        z = torch.randn_like(img) if t &gt; 0 else 0<br><br>        # Calculating image at timestep t-1<br>        img = sqrt_recip_alphas_t * (img - (betas_t / sqrt_one_minus_alpha_bars_t) * pred_noise) + (sigma_t * z)<br><br>        img = torch.clamp(img, -1.0, 1.0)<br><br>    return img</pre><p><strong>Results</strong></p><p>To view the results of our model, I am going to display the reverse diffusion process for each of the captions. In order to view this, we created a modified version of the sample_image function to plot the images at ten timesteps during the reverse diffusion process.</p><pre># Displaying Results<br>config = FMNISTConfig()<br>captions = {<br>    0: &quot;An image of a t-shirt/top&quot;,<br>    1: &quot;An image of trousers&quot;,<br>    2: &quot;An image of a pullover&quot;,<br>    3: &quot;An image of a dress&quot;,<br>    4: &quot;An image of a coat&quot;,<br>    5: &quot;An image of a sandal&quot;,<br>    6: &quot;An image of a shirt&quot;,<br>    7: &quot;An image of a sneaker&quot;,<br>    8: &quot;An image of a bag&quot;,<br>    9: &quot;An image of an ankle boot&quot;<br>}<br>sample_captions = torch.stack([tokenizer(x, text_seq_length=config.text_seq_length)[0] for x in captions.values()]).to(config.device)<br>sample_masks = torch.stack([tokenizer(x, text_seq_length=config.text_seq_length)[1] for x in captions.values()]).to(config.device)<br>for i in range(len(sample_captions)):<br>    caption = sample_captions[None, (i % len(sample_captions))]<br>    mask = sample_masks[None, (i % len(sample_masks))]<br>    test = sample_plot_image(config, caption, mask)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*UOK5ET1JulWC-pogtN4UVA.png" /></figure><p><strong>Possible Ways to Improve Models</strong></p><ul><li>Implementing classifier free guidance</li><li>Learn sigma and use hybrid loss objective in order to improve log-likelihoods.</li><li>Increasing the number of tokens for the sequence in the prior model that is passed through the decoder-only Transformer.</li><li>Learned padding for text captions</li><li>K-Fold cross validation to help find optimal parameters (training time would significantly increase).</li></ul><p><strong>Relevant Papers</strong></p><ul><li>Ho et al. 2020, DDPM Paper: <a href="https://arxiv.org/pdf/2006.11239">https://arxiv.org/pdf/2006.11239</a></li><li>Nichol et al. 2021, Improved DDPM: <a href="https://arxiv.org/pdf/2102.09672">https://arxiv.org/pdf/2102.09672</a></li><li>Nichol et al. 2022, GLIDE: <a href="https://arxiv.org/pdf/2112.10741">https://arxiv.org/pdf/2112.10741</a></li><li>Han et al. 2017, Pyramidal Residual Networks: <a href="https://arxiv.org/pdf/1610.02915">https://arxiv.org/pdf/1610.02915</a></li><li>Ramachandran et al. 2017, Activation Functions: <a href="https://arxiv.org/pdf/1710.05941">https://arxiv.org/pdf/1710.05941</a></li><li>Dhariwal et al. 2021, Diffusion Beats GANs: <a href="https://arxiv.org/pdf/2105.05233">https://arxiv.org/pdf/2105.05233</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c055bf881b9a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/correll-lab/dall-e-2-from-scratch-c055bf881b9a">DALL-E 2 from Scratch</a> was originally published in <a href="https://medium.com/correll-lab">Toward Humanoids</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building CLIP From Scratch]]></title>
            <link>https://medium.com/correll-lab/building-clip-from-scratch-68f6e42d35f4?source=rss-b66aa802e02f------2</link>
            <guid isPermaLink="false">https://medium.com/p/68f6e42d35f4</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[object-recognition]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Matt Nguyen]]></dc:creator>
            <pubDate>Thu, 16 May 2024 08:34:10 GMT</pubDate>
            <atom:updated>2024-09-19T04:46:52.339Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Open World Object Recognition on the FashionMNIST Dataset</strong></h4><p>Computer vision systems were historically limited to a fixed set of classes, CLIP has been a revolution allowing open world object recognition by “predicting which image and text pairings go together&quot;. CLIP is able to predict this by learning the cosine similarity between image and text feature for batches of training data. This is shown in the contrastive pre-training portion of Figure 1 where the dot product between the image features {I_1 … I_N} and the text features {T_1 … T_N} is taken.</p><p>In this tutorial, we are going to build CLIP from scratch and test it on the fashion MNIST dataset. Some of the sections in this article are taken from my <a href="https://medium.com/correll-lab/building-a-vision-transformer-model-from-scratch-a3054f707cc6">vision transformers article</a>, which will help you to understand how transformer models can be applied to images.</p><p><a href="https://medium.com/correll-lab/building-a-vision-transformer-model-from-scratch-a3054f707cc6">Building a Vision Transformer Model From Scratch</a></p><p>Notebook with the code from this tutorial can be found <a href="https://colab.research.google.com/drive/1E4sEg7RM8HBv4PkIhjWZuwCXXbB_MinS?usp=sharing">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5V3OGdcahofT2J8JkoHiAg.png" /><figcaption>Figure 1: CLIP Model Overview. Image: <a href="https://arxiv.org/pdf/2103.00020">CLIP Paper</a>.</figcaption></figure><p><strong>Import Libraries and Modules</strong></p><pre>import torch<br>import torch.nn as nn<br>import torch.optim as optim<br>import torchvision.transforms as T<br>from torch.utils.data import Dataset, DataLoader<br>from datasets import load_dataset<br>import matplotlib.pyplot as plt<br>import numpy as np</pre><p>We are going to be building CLIP using PyTorch, so we will need to import the library plus others that we will be using in this tutorial.</p><pre>import torch<br>import torch.nn as nn</pre><p>We also need to import in torchvision.transforms in order to resize the input images and convert them to tensors. Resizing the input images is optional. You just need to make sure that the image size is divisible by the patch size.</p><pre>import torchvision.transforms as T</pre><p>We are going to be using Adam for our optimizer, so we need to import it in from torch.optim.</p><pre>from torch.optim import Adam</pre><p>We are going to be importing in the fashion MNIST dataset from HuggingFace for this tutorial, so we need to import in datasets. We are going to be using the DataLoader from PyTorch to help load the data, so we need to import that in as well.</p><pre>from datasets import load_dataset<br>from torch.utils.data import DataLoader</pre><p>We are going to import in matplotlib in order to display images when doing zero-shot classification.</p><pre>import matplotlib.pyplot as plt</pre><p>Finally, we need to import in numpy which we will use to perform sin and cosine when creating the positional encodings.</p><pre>import numpy as np</pre><h3>Image and Text Encoders</h3><p>We will first build the image and text encoders. Both embed images and text, respectively, into a single token, which can then be used in the contrastive loss computation. If you already know how image and text encoding work, you can skip right to the Section “CLIP Model” further down.</p><p><strong>Positional Embedding</strong></p><pre>class PositionalEmbedding(nn.Module):<br>  def __init__(self, width, max_seq_length):<br>    super().__init__()<br><br>    pe = torch.zeros(max_seq_length, width)<br><br>    for pos in range(max_seq_length):<br>      for i in range(width):<br>        if i % 2 == 0:<br>          pe[pos][i] = np.sin(pos/(10000 ** (i/width)))<br>        else:<br>          pe[pos][i] = np.cos(pos/(10000 ** ((i-1)/width)))<br><br>    self.register_buffer(&#39;pe&#39;, pe.unsqueeze(0))<br><br>  def forward(self, x):<br><br>    x = x + self.pe<br><br>    return x</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*8hq5drJzWOxFhbGEfceISQ.png" /><figcaption>Figure 2: Changing patch order can change an O into an X. Image: <a href="https://medium.com/correll-lab/building-a-vision-transformer-model-from-scratch-a3054f707cc6">ViT From Scratch Article</a>.</figcaption></figure><p>Unlike models like LSTMs which take embeddings in sequentially, transformers take embeddings in parallel. While this increases the speed, transformers are not aware of what order sequences are supposed to be in. This is a problem because changing the order of the sequence would most likely alter its meaning. An example of this is Figure 2 which shows that changing an image’s patch order can change the image from an O to something that more closely resembles an X. In order to fix this problem, positional encodings need to be added to the embeddings. Each positional encoding is unique to the position that it represents which allows model to identify which position each embedding is supposed to go. In order for the positional encodings to be added to the embeddings, they have to have the same dimension, d_model. We get the positional encodings by using the equations in Figure 3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*G7lAIXZyRVn0QzeN" /><figcaption>Figure 3: Positional encoding equations. Image: <a href="https://arxiv.org/pdf/1706.03762">Transformer Paper.</a></figcaption></figure><pre>pe = torch.zeros(max_seq_length, width)<br><br>for pos in range(max_seq_length):<br>  for i in range(width):<br>    if i % 2 == 0:<br>      pe[pos][i] = np.sin(pos/(10000 ** (i/width)))<br>    else:<br>      pe[pos][i] = np.cos(pos/(10000 ** ((i-1)/width)))<br><br>self.register_buffer(&#39;pe&#39;, pe.unsqueeze(0))</pre><p>Notice that <em>pe</em> is only a local variable and only gets added to the class using the <a href="https://pytorch.org/docs/stable/generated/torch.nn.Module.html"><em>register_buffer</em></a> method. This way, the positional encoding become a non-trainable part of the model.</p><p>In the forward method, the positional encodings that we calculated above are added to the input.</p><pre>x = x + self.pe<br><br>return x</pre><p><strong>Attention Head</strong></p><pre>class AttentionHead(nn.Module):<br>  def __init__(self, width, head_size):<br>    super().__init__()<br>    self.head_size = head_size<br><br>    self.query = nn.Linear(width, head_size)<br>    self.key = nn.Linear(width, head_size)<br>    self.value = nn.Linear(width, head_size)<br><br>  def forward(self, x, mask=None):<br>    # Obtaining Queries, Keys, and Values<br>    Q = self.query(x)<br>    K = self.key(x)<br>    V = self.value(x)<br><br>    # Dot Product of Queries and Keys<br>    attention = Q @ K.transpose(-2,-1)<br><br>    # Scaling<br>    attention = attention / (self.head_size ** 0.5)<br><br>    # Applying Attention Mask<br>    if mask is not None:<br>        attention = attention.masked_fill(mask == 0, float(&quot;-inf&quot;))<br><br>    attention = torch.softmax(attention, dim=-1)<br><br>    attention = attention @ V<br><br>    return attention</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/0*N99XOqa8iH4WK_u9" /><figcaption>Figure 4: Scaled Dot-Product Attention and Multi-Head Attention diagrams. Image: <a href="https://arxiv.org/pdf/1706.03762">Transformer Paper</a>.</figcaption></figure><p>Transformers use attention which is a communication mechanism that allows the model to focus on important parts of an image. Attention scores can be calculated using the equation in Figure 5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*nc_6eJDIlzGTqJAAihLPGA.png" /><figcaption>Figure 5: Attention equation. Image: <a href="https://arxiv.org/pdf/1706.03762">Transformer Paper</a>.</figcaption></figure><p>The first step in calculating attention is obtaining the queries, keys, and values of the tokens. The query of a token is what the token is looking for, the key is what the token contains, and the value is what is communicated between the tokens. The queries, keys, and values can be calculated by passing tokens through linear modules.</p><pre>def forward(self, x):<br>   # Obtaining Queries, Keys, and Values<br>   Q = self.query(x)<br>   K = self.key(x)<br>   V = self.value(x)</pre><p>We are able to get the relationship between the tokens in a sequence by getting the dot product of the queries and keys.</p><pre># Dot Product of Queries and Keys<br>attention = Q @ K.transpose(-2,-1)</pre><p>We need to scale these values to control variance at initialization so that tokens are able to aggregate information from more than one other token. Scaling is applied by dividing the dot product by the square root of the size of the attention head.</p><pre># Scaling<br>attention = attention / (self.head_size ** 0.5)</pre><p>The main difference between transformer encoders and decoders is that decoders apply an attention mask while the encoders do not. While CLIP is an encoder only model, a mask still needs to be applied with the text encoder due to the padding that is applied to the input text during tokenization. Note that the mask is optional, so this Attention head can be used in both the text and vision encoder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DiztJc24Nsl_Hql1vIby0A.png" /><figcaption>Figure 6: Example of applying mask to attention scores. Image: Own work.</figcaption></figure><pre># Applying Attention Mask<br>if mask is not None:<br>    attention = attention.masked_fill(mask == 0, float(&quot;-inf&quot;))</pre><p>We then need to apply a SoftMax operation on the scaled dot product. Here, values with negative infinity will simply be ignored.</p><pre>attention = torch.softmax(attention, dim=-1)</pre><p>Finally, we need to get the dot product between the SoftMax and the values matrix. This is essentially communicating the information between the corresponding tokens.</p><pre>attention = attention @ V<br><br>return attention</pre><p><strong>Multi-Head Attention</strong></p><pre>class MultiHeadAttention(nn.Module):<br>  def __init__(self, width, n_heads):<br>    super().__init__()<br>    self.head_size = width // n_heads<br><br>    self.W_o = nn.Linear(width, width)<br><br>    self.heads = nn.ModuleList([AttentionHead(width, self.head_size) for _ in range(n_heads)])<br><br>  def forward(self, x, mask=None):<br>    # Combine attention heads<br>    out = torch.cat([head(x, mask=mask) for head in self.heads], dim=-1)<br><br>    out = self.W_o(out)<br><br>    return out</pre><p>Multi-head attention is just running multiple heads of self-attention in parallel and combining them. We can do this by adding the attention heads into a module list,</p><pre>self.heads = nn.ModuleList([AttentionHead(width, self.head_size) for _ in range(n_heads)])</pre><p>passing through the input and concatenating the results.</p><pre>def forward(self, x, mask=None):<br>    # Combine attention heads<br>    out = torch.cat([head(x, mask=mask) for head in self.heads], dim=-1)</pre><p>We then need to pass the output through another linear module.</p><pre>out = self.W_o(out)<br><br>return out</pre><p><strong>Transformer Encoder</strong></p><pre>class TransformerEncoder(nn.Module):<br>    def __init__(self, width, n_heads, r_mlp=4):<br>        super().__init__()<br>        self.width = width<br>        self.n_heads = n_heads<br><br>        # Sub-Layer 1 Normalization<br>        self.ln1 = nn.LayerNorm(width)<br><br>        # Multi-Head Attention<br>        self.mha = MultiHeadAttention(width, n_heads)<br><br>        # Sub-Layer 2 Normalization<br>        self.ln2 = nn.LayerNorm(width)<br><br>        # Multilayer Perception<br>        self.mlp = nn.Sequential(<br>            nn.Linear(self.width, self.width*r_mlp),<br>            nn.GELU(),<br>            nn.Linear(self.width*r_mlp, self.width)<br>        )<br><br><br>    def forward(self, x, mask=None):<br>        x = x + self.mha(self.ln1(x), mask=mask)<br><br>        x = x + self.mlp(self.ln2(x))<br><br>        return x</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RgiuEOa1imuUMHERAdazTw.png" /><figcaption>Figure 7: Transformer Encoder Diagram. Image: Own work.</figcaption></figure><p>The transformer encoder is made up of two sub-layers: the first sub-layer performs multi-head attention and the second sub-layer contains a multi-layer perceptron. The multi-head attention sub-layer performs communication between tokens while the multi-layer perceptron sub-layers allows the tokens to individually “think” on what was communicated to them.</p><p>Layer normalization is an optimization technique that normalizes each input in the batch independently across its features. For our model, we will pass our inputs through a layer norm module at the beginning of each sub-layer.</p><pre># Sub-Layer 1 Normalization<br>self.ln1 = nn.LayerNorm(width)<br><br># Sub-Layer 2 Normalization<br>self.ln2 = nn.LayerNorm(width)</pre><p>The MLP will consist of two linear layers with a GELU layer in between. GELU is used instead of RELU because it doesn’t have RELU’s limitation of being non-differentiable at zero.</p><pre># Multilayer Perception<br>self.mlp = nn.Sequential(<br>    nn.Linear(width, width*r_mlp),<br>    nn.GELU(),<br>    nn.Linear(width*r_mlp, width)<br>)</pre><p>In the forward method for the encoder, the input is passed through the first layer normalization module before performing multi-head attention. The original input is added to the output from performing multi-head attention to create a residual connection.</p><p>This is then passed through another layer normalization module before being inputted into the MLP. Another residual connection is created by adding the output from the MLP to the out from the first residual connection.</p><p>The residual connections are used to help prevent the vanishing gradient problem by creating a path for the gradient to be back-propagated unimpeded back to the original input.</p><pre>def forward(self, x):<br>    # Residual Connection After Sub-Layer 1<br>    out = x + self.mha(self.ln1(x))<br><br>    # Residual Connection After Sub-Layer 2<br>    out = out + self.mlp(self.ln2(out))<br><br>    return out</pre><p><strong>Tokenization</strong></p><pre>def tokenizer(text, encode=True, mask=None, max_seq_length=32):<br>    if encode:<br>        out = chr(2) + text + chr(3) # Adding SOT and EOT tokens<br>        out = out + &quot;&quot;.join([chr(0) for _ in range(max_seq_length-len(out))]) # Adding Padding<br>        out = torch.IntTensor(list(out.encode(&quot;utf-8&quot;))) # Encoding Text<br>        mask = torch.ones(len(out.nonzero()))<br>        mask = torch.cat((mask,torch.zeros(max_seq_length-len(mask)))).type(torch.IntTensor)<br>    else:<br>        out = [chr(x) for x in text[1:len(mask.nonzero())-1]]<br>        out = &quot;&quot;.join(out)<br>        mask = None<br><br>    return out, mask</pre><p>Transformers are unable to process raw text, so the first thing that we need to do is tokenize the input strings before passing them through the text encoder.</p><p>In this tutorial, we are going to be doing a simple version of tokenization where we just use the UTF-8 encoding. We are able to naively use the UTF-8 encoding for tokenization because we are only going to be using simple text for our examples. For more complex examples, you may want to use a BPE tokenizer. This is because with UTF encoding, you have a max vocab size of 256, which means that with more complex examples you may have longer input sequences which would be inefficient when doing attention due to limited context length.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qTFmmRJD1awtQowxl94now.png" /><figcaption>Figure 8: Tokenization Process with Max Sequence Length of 10. Image: Own work.</figcaption></figure><p>The first step for our tokenizer is to add the start of text and end of text tokens to the input string.</p><pre>text = chr(2) + text + chr(3)</pre><p>After adding the start of text and end of text tokens, we need to pad the length of the sequence to the maximum sequence length.</p><pre>text = text + &quot;&quot;.join([chr(0) for _ in range(10-len(text))])</pre><p>We complete the tokenization by encoding the text sequence to UTF-8 and converting the output to an IntTensor.</p><pre>text = torch.IntTensor(list(text.encode(&quot;utf-8&quot;)))</pre><p>After tokenizing the text, we need to create a mask for the text. While the mask that is normally used in transformers is used to ensure that tokens do not communicate with future tokens, the mask that we are applying here just makes it so that padding tokens are ignored. Because of this, the mask is just going to be a tensor of size equal to the max sequence length where the elements is 0 where there is padding and 1 otherwise.</p><pre>mask = torch.ones(len(text.nonzero()))<br>mask = torch.cat((mask,torch.zeros(10-len(mask)))).type(torch.IntTensor)</pre><p><strong>Text Encoder</strong></p><pre>class TextEncoder(nn.Module):<br>    def __init__(self, vocab_size, width, max_seq_length, n_heads, n_layers, emb_dim):<br>        super().__init__()<br><br>        self.max_seq_length = max_seq_length<br><br>        self.encoder_embedding = nn.Embedding(vocab_size, width)<br><br>        self.positional_embedding = PositionalEmbedding(width, max_seq_length)<br><br>        self.encoder = nn.ModuleList([TransformerEncoder(width,n_heads) for _ in range(n_layers)])<br><br>        # learned proj of image to embed<br>        self.projection = nn.Parameter(torch.randn(width, emb_dim))<br><br>    def forward(self, text, mask=None):<br>        # Text Embedding<br>        x = self.encoder_embedding(text)<br><br>        # Positional Embedding<br>        x = self.positional_embedding(x)<br><br>        # Transformer Encoder<br>        for encoder_layer in self.encoder:<br>            x = encoder_layer(x, mask=mask)<br><br>        # Takes features from the EOT Embedding<br>        x = x[torch.arange(text.shape[0]), torch.sub(torch.sum(mask[:,0],dim=1),1)]<br><br>        # joint multimodal embedding<br>        if self.projection is not None:<br>            x = x @ self.projection<br><br>        x = x / torch.norm(x, dim=-1, keepdim=True)<br><br>        return x</pre><p>For the text encoder, we are going to be using a regular transformer model. The first step in creating the text encoder is creating an embedding table of size (vocab_size, width). This embedding table contains a vector representation with a size equal to the width of the transformer model for each token in the vocabulary.</p><pre>self.encoder_embedding = nn.Embedding(vocab_size, width)</pre><p>Before outputting the results for the transformer, we are going to need to embed the features in the joint embedding space. We are going to do this by getting the dot product of the text features and a learned projection that we create by using a nn.Parameter.</p><pre># learned proj of image to embed<br>self.projection = nn.Parameter(torch.randn(width, emb_dim))</pre><p>In the forward method, the first thing that we are going to do is pass the text tokens through the embedding table.</p><pre># Text Embedding<br>x = self.encoder_embedding(text)</pre><p>We then need to add the positional encodings to the output of the embedding table.</p><pre># Positional Embedding<br>x = self.positional_embedding(x)</pre><p>With the positional encodings added, we can now pass it through the encoder layers along with the masks.</p><pre># Transformer Encoder<br>for encoder_layer in self.encoder:<br>    x = encoder_layer(x, mask=mask)</pre><p>The output of the encoder layers is the text features. We are going to be using the features from the EOT embedding. If you are using something like the BERT model for the text encoder, you would want to use the class tokens here instead.</p><pre># Takes features from the EOT Embedding<br>x = x[torch.arange(text.shape[0]),torch.sub(torch.sum(mask[:,0],dim=1),1)]</pre><p>Finally, we embed the text features in the joint embedding space by getting the dot product between the features and the learned projection and normalize it by dividing by the normalized dot product.</p><pre># joint multimodal embedding<br>if self.projection is not None:<br>    x = x @ self.projection<br><br>x = x / torch.norm(x, dim=-1, keepdim=True)<br><br>return x</pre><p><strong>Image Encoder</strong></p><pre>class ImageEncoder(nn.Module):<br>    def __init__(self, width, img_size, patch_size, n_channels, n_layers, n_heads, emb_dim):<br>        super().__init__()<br><br>        assert img_size[0] % patch_size[0] == 0 and img_size[1] % patch_size[1] == 0, &quot;img_size dimensions must be divisible by patch_size dimensions&quot;<br>        assert width % n_heads == 0, &quot;width must be divisible by n_heads&quot;<br><br>        self.n_patches = (img_size[0] * img_size[1]) // (patch_size[0] * patch_size[1])<br><br>        self.max_seq_length = self.n_patches + 1<br><br>        self.linear_project = nn.Conv2d(n_channels, width, kernel_size=patch_size, stride=patch_size)<br><br>        self.cls_token = nn.Parameter(torch.randn(1, 1, width))<br><br>        self.positional_embedding = PositionalEmbedding(width,self.max_seq_length)<br><br>        self.encoder = nn.ModuleList([TransformerEncoder(width,n_heads) for _ in range(n_layers)])<br><br><br>        # learned proj of image to embed<br>        self.projection = nn.Parameter(torch.randn(width, emb_dim))<br><br><br>    def forward(self,x):<br>        # Patch Embedding<br>        x = self.linear_project(x)<br>        x = x.flatten(2).transpose(1, 2)<br><br>        # Positional Embedding<br>        x = torch.cat((self.cls_token.expand(x.size()[0], -1, -1),x), dim=1)<br>        x = self.positional_embedding(x)<br><br>        # Transformer Encoder<br>        for encoder_layer in self.encoder:<br>            x = encoder_layer(x)<br><br>        # Getting Class Tokens<br>        x = x[:, 0, :]<br><br>        # joint multimodal embedding<br>        if self.projection is not None:<br>            x = x @ self.projection<br><br>        x = x / torch.norm(x, dim=-1, keepdim=True)<br><br>        return x</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/980/1*_5RE7btB01N77KHlgUXbIg.png" /><figcaption>Figure 9: Vision Transformer Model Diagram. Image:<a href="https://arxiv.org/pdf/2010.11929"> ViT Paper</a>.</figcaption></figure><p>For the image encoder, we are going to be using a vision transformer. When creating our image encoder, we first need to make sure that the input images can be split evenly into patches of size patch_size and that the dimensionality of the model is divisible by the number of attention heads.</p><pre>assert img_size[0] % patch_size[0] == 0 and img_size[1] % patch_size[1] == 0, &quot;img_size dimensions must be divisible by patch_size dimensions&quot;<br>assert width % n_heads == 0, &quot;width must be divisible by n_heads&quot;</pre><p>We also need to calculate the maximum sequence length for the positional encoding, which will be equal to the number of patches plus one. The number of patches can be found by dividing the product of the height and width of the input image by the product of the height and width of the patch size.</p><pre>self.n_patches = (self.img_size[0] * self.img_size[1]) // (self.patch_size[0] * self.patch_size[1])<br>self.max_seq_length = self.n_patches + 1</pre><p>The vision transformer will also need to be able to have multiple encoder modules. This can be achieved by putting a list of encoder layers inside of a ModuleList.</p><pre>self.encoder = nn.ModuleList([TransformerEncoder(width,n_heads) for _ in range(n_layers)])</pre><p>Before we pass are inputs through the encoder layers, we first need to split the input image into patches and create a sequence of linear embeddings of these patches. We are able to achieve this by using PyTorch’s Conv2d method</p><p>The Conv2d method takes the input images, splits them into patches and provides a linear projection of a size equal to the width of the model. By setting kernel_size and stride to patch size, we ensure that the patches are the correct size and there is no overlap.</p><pre>self.linear_project = nn.Conv2d(n_channels, width, kernel_size=patch_size, stride=patch_size)</pre><p>In the forward method we pass through the input that has shape (B, C, H, W) through the linear_project/Conv2D method and receives an output of shape (B, d_model, P_col, P_row).</p><pre>def forward(self, x):<br>    x = self.linear_project(x) # (B, C, H, W) -&gt; (B, width, P_col, P_row)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YAiaEmajmui1tLnkRP5jLw.jpeg" /><figcaption>Figure 10: Conv2D applied on a single image. Each color represents which patch an element belongs to. Image: <a href="https://medium.com/correll-lab/building-a-vision-transformer-model-from-scratch-a3054f707cc6">ViT From Scratch Article</a>.</figcaption></figure><p>We use the flatten method to combine the patch column and patch row dimensions into a single patch dimension giving us a shape of (B, d_model, P)</p><pre>x = x.flatten(2) # (B, width, P_col, P_row) -&gt; (B, width, P)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hUMTVjVHizYQVfTz5dC5Pg.jpeg" /><figcaption>Figure 11: Flatten applied to Conv2d output. Image: <a href="https://medium.com/correll-lab/building-a-vision-transformer-model-from-scratch-a3054f707cc6">ViT From Scratch Article</a>.</figcaption></figure><p>Finally we use the transpose method to switch the d_model and patch dimensions to get a shape of (B, P, d_model).</p><pre>x = x.transpose(-2, -1) # (B, width, P) -&gt; (B, P, width)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V-7s4y66aEnLQQKF-IK6RA.jpeg" /><figcaption>Figure 12: Transpose applied to flatten output. Image: <a href="https://medium.com/correll-lab/building-a-vision-transformer-model-from-scratch-a3054f707cc6">ViT From Scratch Article</a>.</figcaption></figure><p>Vision transformers uses the standard approach of adding a learnable classification token to the patch embeddings in order to perform classification.</p><pre>self.cls_token = nn.Parameter(torch.randn(1, 1, width))</pre><p>Each image in the batch needs to have class token, so we are going to use the expand function in order to use self.cls_token to create class tokens for every image in the batch.</p><pre>x = torch.cat((self.cls_token.expand(x.size()[0], -1, -1),x), dim=1)</pre><p>After adding the class tokens, we need to add the positional encodings to the embeddings.</p><pre>x = self.positional_embedding(x)</pre><p>With the positional encodings added, we can now pass the embeddings through the encoder layers.</p><pre># Transformer Encoder<br>for encoder_layer in self.encoder:<br>    x = encoder_layer(x)</pre><p>From the output of the encoder layers, we only need the information from the learned class tokens.</p><pre># Getting Class Tokens<br>x = x[:, 0, :]</pre><p>Finally, we embed the image features in the joint embedding space by getting the dot product between the features and the learned projection and normalize it by dividing by the normalized dot product.</p><pre>if self.projection is not None:<br>    x = x @ self.projection<br><br>x = x / torch.norm(x, dim=-1, keepdim=True)<br><br>return x</pre><h3><strong>CLIP Model</strong></h3><pre>class CLIP(nn.Module):<br>    def __init__(self, emb_dim, vit_width, img_size, patch_size, n_channels, vit_layers, vit_heads, vocab_size, text_width, max_seq_length, text_heads, text_layers):<br>        super().__init__()<br><br>        self.image_encoder = ImageEncoder(vit_width, img_size, patch_size, n_channels, vit_layers, vit_heads, emb_dim)<br><br>        self.text_encoder = TextEncoder(vocab_size, text_width, max_seq_length, text_heads, text_layers, emb_dim)<br><br>        self.temperature = nn.Parameter(torch.ones([]) * np.log(1 / 0.07))<br><br>        self.device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br><br><br>    def forward(self,image,text, mask=None):<br>        I_e = self.image_encoder(image)<br>        T_e = self.text_encoder(text, mask=mask)<br><br>        # scaled pairwise cosine similarities [n, n]<br>        logits = (I_e @ T_e.transpose(-2,-1)) * torch.exp(self.temperature)<br><br>        # symmetric loss function<br>        labels = torch.arange(logits.shape[0]).to(self.device)<br><br>        loss_i = nn.functional.cross_entropy(logits.transpose(-2,-1), labels)<br>        loss_t = nn.functional.cross_entropy(logits, labels)<br><br>        loss = (loss_i + loss_t) / 2<br><br>        return loss</pre><p>When given a batch of images and captions, CLIP is supposed to tell you which captions goes with which images. It does this by training the text and image encoder together to maximize the pairwise cosine similarity scores of the pairs that are supposed to go together and minimizing the pairs that are not supposed to go together.</p><p>To do this, we first need to get the embedded features from the image and text encoders.</p><pre>def forward(self,image,text, mask=None):<br>        I_e = self.image_encoder(image)<br>        T_e = self.text_encoder(text, mask=mask)</pre><p>Using the embedded features, we can calculate the scaled pairwise cosine similarities by using a dot product between the embedded image features and a transposed version of the embedded text features. The cosine similarities should be maximized along the diagonal in the figure where the correct image and text are paired together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/461/1*bnHvQEaCg5kVHRBM6YfrQA.png" /><figcaption>Figure 13: Calculating cosine similarity. Image: <a href="https://arxiv.org/pdf/2103.00020">CLIP Paper</a>.</figcaption></figure><pre>logits = (I_e @ T_e.transpose(-2,-1)) * torch.exp(self.temperature)</pre><p>This works as I_e and T_e are each containing N batches, resulting in the matrix shown in Figure 13. In order to maximize the cosine similarity between related images, CLIP uses symmetric/contrastive loss. We can calculate this loss by first creating labels that correspond to the items in the batch.</p><pre># symmetric loss function<br>labels = torch.arange(logits.shape[0]).to(self.device)</pre><p>We then calculate the cross entropy loss along the rows of the logits to get the loss for the images.</p><pre>loss_i = nn.functional.cross_entropy(logits.transpose(-2,-1), labels)</pre><p>The loss for the text is calculated by calculating the cross entropy loss along the columns.</p><pre>loss_t = nn.functional.cross_entropy(logits, labels)</pre><p>We get the final loss by calculating the average between the loss for the images and the loss for the text.</p><pre>loss = (loss_i + loss_t) / 2<br><br>return loss</pre><p><strong>Dataset</strong></p><pre>class FashionMNIST(Dataset):<br>    def __init__(self, train=True):<br>        self.dataset = load_dataset(&quot;fashion_mnist&quot;)<br><br>        self.transform = T.ToTensor()<br><br>        if train:<br>            self.split = &quot;train&quot;<br>        else:<br>            self.split = &quot;test&quot;<br><br><br>        self.captions = {0: &quot;An image of a t-shirt/top&quot;,<br>                        1: &quot;An image of trousers&quot;,<br>                        2: &quot;An image of a pullover&quot;,<br>                        3: &quot;An image of a dress&quot;,<br>                        4: &quot;An image of a coat&quot;,<br>                        5: &quot;An image of a sandal&quot;,<br>                        6: &quot;An image of a shirt&quot;,<br>                        7: &quot;An image of a sneaker&quot;,<br>                        8: &quot;An image of a bag&quot;,<br>                        9: &quot;An image of an ankle boot&quot;}<br><br><br>    def __len__(self):<br>        return self.dataset.num_rows[self.split]<br><br>    def __getitem__(self,i):<br>        img = self.dataset[self.split][i][&quot;image&quot;]<br>        img = self.transform(img)<br><br>        cap, mask = tokenizer(self.captions[self.dataset[self.split][i][&quot;label&quot;]])<br><br>        mask = mask.repeat(len(mask),1)<br><br>        return {&quot;image&quot;: img, &quot;caption&quot;: cap, &quot;mask&quot;: mask}</pre><p>For this tutorial, we are going to be using the Fashion MNIST dataset from HuggingFace. We have chosen this dataset because it is rather small and keeps training time reasonable.</p><pre>self.dataset = load_dataset(&quot;fashion_mnist&quot;)</pre><p>For each entry in the dataset, we are going to need three things: the image, caption, and text mask.</p><p>For the image, the only change we need to make is to transform the image to a tensor.</p><pre>img = self.dataset[self.split][i][&quot;image&quot;]<br>img = self.transform(img)</pre><p>For the caption, we need to pass it through the tokenizer that we created to get the token representation along with the mask for the tokens.</p><pre>cap, mask = tokenizer(self.captions[self.dataset[self.split][i][&quot;label&quot;]])</pre><p>The mask that we get from the tokenizer has is a 1D tensor of size max_seq_length. In the text encoder, the mask is going to be applied to the attention scores which has a shape of (max_seq_length, max_seq_length). Because of this, we need to expand the mask so that it is applied to each row of the attention scores.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7pw-xy1zQcF3g8Tx7g10tg.png" /><figcaption>Figure 14: Mask before and after expanding. Image: Own work.</figcaption></figure><pre>mask = mask.repeat(len(mask),1)</pre><p>The image, caption, and mask are held in the dataset as a dictionary.</p><pre>return {&quot;image&quot;: img, &quot;caption&quot;: cap, &quot;mask&quot;: mask}</pre><p><strong>Training Parameters</strong></p><pre>emb_dim = 32<br>vit_width = 9<br>img_size = (28,28)<br>patch_size = (14,14)<br>n_channels = 1<br>vit_layers = 3<br>vit_heads = 3<br>vocab_size = 256<br>text_width = 32<br>max_seq_length = 32<br>text_heads = 8<br>text_layers = 4<br>lr = 1e-3<br>epochs = 10<br>batch_size = 128</pre><p><strong>Loading Dataset</strong></p><pre>train_set = FashionMNIST(train = True)<br>test_set = FashionMNIST(train = False)<br><br>train_loader = DataLoader(train_set, shuffle=True, batch_size=batch_size)<br>test_loader = DataLoader(test_set, shuffle=False, batch_size=batch_size)</pre><p><strong>Training</strong></p><pre>device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br>print(&quot;Using device: &quot;, device, f&quot;({torch.cuda.get_device_name(device)})&quot; if torch.cuda.is_available() else &quot;&quot;)<br><br>model = CLIP(emb_dim, vit_width, img_size, patch_size, n_channels, vit_layers, vit_heads, vocab_size, text_width, max_seq_length, text_heads, text_layers).to(device)<br><br>optimizer = optim.Adam(model.parameters(), lr=lr)<br><br>best_loss = np.inf<br>for epoch in range(epochs):<br>    for i, data in enumerate(train_loader, 0):<br>        img, cap, mask = data[&quot;image&quot;].to(device), data[&quot;caption&quot;].to(device), data[&quot;mask&quot;].to(device)<br>        loss = model(img,cap,mask)<br>        optimizer.zero_grad()<br>        loss.backward()<br>        optimizer.step()<br><br>    print(f&quot;Epoch [{epoch+1}/{epochs}], Batch Loss: {loss.item():.3f}&quot;)<br><br>    # Saves model if it performed better than the previous best<br>    if loss.item() &lt;= best_loss:<br>        best_loss = loss.item()<br>        torch.save(model.state_dict(), &quot;/content/drive/MyDrive/clip.pt&quot;)<br>        print(&quot;Model Saved.&quot;)</pre><p><strong>Testing</strong></p><pre># Loading Best Model<br>model = CLIP(emb_dim, vit_width, img_size, patch_size, n_channels, vit_layers, vit_heads, vocab_size, text_width, max_seq_length, text_heads, text_layers).to(device)<br>model.load_state_dict(torch.load(&quot;/content/drive/MyDrive/clip.pt&quot;, map_location=device))<br><br># Getting dataset captions to compare images to<br>text = torch.stack([tokenizer(x)[0] for x in test_set.captions.values()]).to(device)<br>mask = torch.stack([tokenizer(x)[1] for x in test_set.captions.values()])<br>mask = mask.repeat(1,len(mask[0])).reshape(len(mask),len(mask[0]),len(mask[0])).to(device)<br><br>correct, total = 0,0<br>with torch.no_grad():<br>    for data in test_loader:<br>        images, labels = data[&quot;image&quot;].to(device), data[&quot;caption&quot;].to(device)<br>        image_features = model.image_encoder(images)<br>        text_features = model.text_encoder(text, mask=mask)<br><br>        image_features /= image_features.norm(dim=-1, keepdim=True)<br>        text_features /= text_features.norm(dim=-1, keepdim=True)<br>        similarity = (100.0 * image_features @ text_features.T).softmax(dim=-1)<br>        _, indices = torch.max(similarity,1)<br>        pred = torch.stack([tokenizer(test_set.captions[int(i)])[0] for i in indices]).to(device)<br>        correct += int(sum(torch.sum((pred==labels),dim=1)//len(pred[0])))<br>        total += len(labels)<br><br>print(f&#39;\nModel Accuracy: {100 * correct // total} %&#39;)</pre><p>We tested the model by getting the captions that the model was trained on and comparing it to the actual captions. When training, we used the same caption template (“An image of a(n) {class}”), so this testing stage is pretty much the same as any other image classifier. We achieved a model accuracy of around 85%.</p><p><strong>Zero-Shot Classification</strong></p><pre># Loading Best Model<br>model = CLIP(emb_dim, vit_width, img_size, patch_size, n_channels, vit_layers, vit_heads, vocab_size, text_width, max_seq_length, text_heads, text_layers).to(device)<br>model.load_state_dict(torch.load(&quot;/content/drive/MyDrive/clip.pt&quot;, map_location=device))<br><br><br># Captions to compare images to<br>class_names =[&quot;t-shirt/top&quot;,<br>                        &quot;trousers&quot;,<br>                        &quot;pullover&quot;,<br>                        &quot;dress&quot;,<br>                        &quot;coat&quot;,<br>                        &quot;sandal&quot;,<br>                        &quot;shirt&quot;,<br>                        &quot;sneaker&quot;,<br>                        &quot;bag&quot;,<br>                        &quot;ankle boot&quot;]<br><br>text = torch.stack([tokenizer(x)[0] for x in class_names]).to(device)<br>mask = torch.stack([tokenizer(x)[1] for x in class_names])<br>mask = mask.repeat(1,len(mask[0])).reshape(len(mask),len(mask[0]),len(mask[0])).to(device)<br><br>idx = 1000<br><br>img = test_set[idx][&quot;image&quot;][None,:]<br>plt.imshow(img[0].permute(1, 2, 0)  ,cmap=&quot;gray&quot;)<br>plt.title(tokenizer(test_set[idx][&quot;caption&quot;], encode=False, mask=test_set[idx][&quot;mask&quot;][0])[0])<br>plt.show()<br>img = img.to(device)<br>with torch.no_grad():<br>  image_features = model.image_encoder(img)<br>  text_features = model.text_encoder(text, mask=mask)<br><br><br>image_features /= image_features.norm(dim=-1, keepdim=True)<br>text_features /= text_features.norm(dim=-1, keepdim=True)<br>similarity = (100.0 * image_features @ text_features.T).softmax(dim=-1)<br>values, indices = similarity[0].topk(5)<br><br># Print the result<br>print(&quot;\nTop predictions:\n&quot;)<br>for value, index in zip(values, indices):<br>    print(f&quot;{class_names[int(index)]:&gt;16s}: {100 * value.item():.2f}%&quot;)</pre><p>For zero-shot classification, we are comparing the image to just the class names. We input in the labels to compare against the image and it will return the top 5 predictions with the predicted likelihoods. This isn’t the greatest example of CLIP performing zero-shot classification. Using the fashion MNIST dataset makes the model easy to train, but captions are not very rich. To truly appreciate the zero-shot capabilities of CLIP, a training set with multiple nouns would be more appropriate. True zero-shot detection would then allow to detect previously unseen permutations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/428/1*6V_feGzgy1goQTU1DpJ3Hw.png" /><figcaption>Figure 15: Zero-shot classification output. Image: Own work.</figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=68f6e42d35f4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/correll-lab/building-clip-from-scratch-68f6e42d35f4">Building CLIP From Scratch</a> was originally published in <a href="https://medium.com/correll-lab">Toward Humanoids</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Vision Transformer Model From Scratch]]></title>
            <link>https://medium.com/correll-lab/building-a-vision-transformer-model-from-scratch-a3054f707cc6?source=rss-b66aa802e02f------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3054f707cc6</guid>
            <category><![CDATA[vision-transformer]]></category>
            <category><![CDATA[object-classification]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Matt Nguyen]]></dc:creator>
            <pubDate>Thu, 04 Apr 2024 22:45:33 GMT</pubDate>
            <atom:updated>2025-02-04T03:02:39.021Z</atom:updated>
            <content:encoded><![CDATA[<p>The self-attention-based transformer model was first introduced by Vaswani et al. in their paper <a href="https://arxiv.org/pdf/1706.03762.pdf"><em>Attention Is All You Need</em></a><em> </em>in 2017 and has been widely used in natural language processing. A transformer model is what is used by OpenAI to create ChatGPT. Transformers not only work on text, but also on images, and essentially any sequential data. In 2021, Dosovitsky et al. introduced the idea of using the transformers for computer vision tasks such as image classification in their paper <a href="https://arxiv.org/pdf/2010.11929.pdf"><em>An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale</em></a>. In their paper, they were able to achieve excellent results with their vision transformer model compared to convolutional networks and required a lot less resources to train. Check out this article to understand how a transformer works when its input are simply characters:</p><p><a href="https://medium.com/towards-data-science/explaining-the-attention-mechanism-29a0e7b448a9">Explaining the Attention Mechanism</a></p><p>In this tutorial, we are going to build a vision transformer model from scratch and test is on the MNIST dataset, a collection of handwritten digits that have become a standard benchmark in machine learning. Notebook with the code from tutorial can be found <a href="https://colab.research.google.com/drive/1rabTm93y39FNbu-21tDhlvYh2gp8edVR?usp=sharing">here</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/0*dq9DdNL4BQrdEuuR" /><figcaption>Figure 1: Vision Transformer Model Overview. Image: <a href="https://arxiv.org/pdf/2010.11929.pdf">ViT Paper.</a></figcaption></figure><p><strong>Import Libraries and Modules</strong></p><pre>import torch<br>import torch.nn as nn<br>import torchvision.transforms as T<br>from torch.optim import Adam<br>from torchvision.datasets.mnist import MNIST<br>from torch.utils.data import DataLoader<br>import numpy as np</pre><p>We are going to be building our vision transformer using PyTorch, so we will need to import the library plus others that we will be using in this tutorial.</p><pre>import torch<br>import torch.nn as nn</pre><p>We also need to import in torchvision.transforms in order to resize the input images and convert them to tensors. Resizing the input images is optional. You just need to make sure that the image size is divisible by the patch size.</p><pre>import torchvision.transforms as T</pre><p>We are going to be using Adam for our optimizer, so we need to import it in from torch.optim.</p><pre>from torch.optim import Adam</pre><p>We are going to import the MNIST dataset that we are using for this tutorial from torchvision. We are going to be using the DataLoader from PyTorch to help load the data, so we need to import that in as well.</p><pre>from torchvision.datasets.mnist import MNIST<br>from torch.utils.data import DataLoader</pre><p>Finally, we need to import in numpy which we will use to perform sin and cosine when creating the positional encodings.</p><pre>import numpy as np</pre><p><strong>Patch Embeddings</strong></p><pre>class PatchEmbedding(nn.Module):<br>  def __init__(self, d_model, img_size, patch_size, n_channels):<br>    super().__init__()<br><br>    self.d_model = d_model # Dimensionality of Model<br>    self.img_size = img_size # Image Size<br>    self.patch_size = patch_size # Patch Size<br>    self.n_channels = n_channels # Number of Channels<br><br>    self.linear_project = nn.Conv2d(self.n_channels, self.d_model, kernel_size=self.patch_size, stride=self.patch_size)<br><br>  # B: Batch Size<br>  # C: Image Channels<br>  # H: Image Height<br>  # W: Image Width<br>  # P_col: Patch Column<br>  # P_row: Patch Row<br>  def forward(self, x):<br>    x = self.linear_project(x) # (B, C, H, W) -&gt; (B, d_model, P_col, P_row)<br><br>    x = x.flatten(2) # (B, d_model, P_col, P_row) -&gt; (B, d_model, P)<br><br>    x = x.transpose(1, 2) # (B, d_model, P) -&gt; (B, P, d_model)<br>    <br>    return x</pre><p>The first step in creating a vision transformer is splitting the input image into patches and creating a sequence of linear embeddings of these patches. We are able to achieve this by using PyTorch’s Conv2d method.</p><p>The Conv2d method takes the input images, splits them into patches and provides a linear projection of a size equal to the width of the model. By setting kernel_size and stride to patch size, we ensure that the patches are the correct size and there is no overlap.</p><pre>self.linear_project = nn.Conv2d(self.n_channels, self.d_model, kernel_size=self.patch_size, stride=self.patch_size)</pre><p>In the forward method we pass through the input that has shape (B, C, H, W) through the linear_project/Conv2D method and receives an output of shape (B, d_model, P_col, P_row).</p><pre>  def forward(self, x):<br>    x = self.linear_project(x) # (B, C, H, W) -&gt; (B, d_model, P_col, P_row)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YAiaEmajmui1tLnkRP5jLw.jpeg" /><figcaption>Figure 2: Conv2D applied on a single image. Each color represents which patch an element belongs to. Image: own.</figcaption></figure><p>We use the flatten method to combine the patch column and patch row dimensions into a single patch dimension giving us a shape of (B, d_model, P)</p><pre>x = x.flatten(2) # (B, d_model, P_col, P_row) -&gt; (B, d_model, P)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hUMTVjVHizYQVfTz5dC5Pg.jpeg" /><figcaption>Figure 3: Flatten applied to Conv2d output. Image: own.</figcaption></figure><p>Finally we use the transpose method to switch the d_model and patch dimensions to get a shape of (B, P, d_model).</p><pre>x = x.transpose(-2, -1) # (B, d_model, P) -&gt; (B, P, d_model)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V-7s4y66aEnLQQKF-IK6RA.jpeg" /><figcaption>Figure 4: Transpose applied to flatten output. Image: own.</figcaption></figure><p>This is actually quite a lot to unpack. Check out this article, which develops this step from simply unfolding (cutting the image up) and using a linear transformation on the patches.</p><p><a href="https://medium.com/correll-lab/understanding-image-patch-embeddings-3d66c14fe7ed">Understanding Image Patch Embeddings</a></p><p>Both approaches have the same outcome, with the convolution operator providing a more compact representation and a more efficient computational implementation on a graphic card.</p><p><strong>Class Token and Positional Encoding</strong></p><pre>class PositionalEncoding(nn.Module):<br>  def __init__(self, d_model, max_seq_length):<br>    super().__init__()<br><br>    self.cls_token = nn.Parameter(torch.randn(1, 1, d_model)) # Classification Token<br><br>    # Creating positional encoding<br>    pe = torch.zeros(max_seq_length, d_model)<br><br>    for pos in range(max_seq_length):<br>      for i in range(d_model):<br>        if i % 2 == 0:<br>          pe[pos][i] = np.sin(pos/(10000 ** (i/d_model)))<br>        else:<br>          pe[pos][i] = np.cos(pos/(10000 ** ((i-1)/d_model)))<br><br>    self.register_buffer(&#39;pe&#39;, pe.unsqueeze(0))<br><br>  def forward(self, x):<br>    # Expand to have class token for every image in batch<br>    tokens_batch = self.cls_token.expand(x.size()[0], -1, -1)<br><br>    # Adding class tokens to the beginning of each embedding<br>    x = torch.cat((tokens_batch,x), dim=1)<br><br>    # Add positional encoding to embeddings<br>    x = x + self.pe<br><br>    return x</pre><p>The vision transformer model uses the standard approach of adding a learnable classification token to the patch embeddings in order to perform classification.</p><pre>self.cls_token = nn.Parameter(torch.randn(1, 1, d_model))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/779/1*8hq5drJzWOxFhbGEfceISQ.png" /><figcaption>Figure 5: Changing patch order can change an O into an X</figcaption></figure><p>Unlike models like LSTMs which take embeddings in sequentially, transformers take embeddings in parallel. While this increases the speed, transformers are not aware of what order sequences are supposed to be in. This is a problem because changing the order of the patches for an image would most likely alter the content of the image and what it is supposed to represent. An example of this is Figure 5 which shows that changing an image’s patch order can change the image from an O to something that more closely resembles an X. In order to fix this problem, positional encodings need to be added to the patch embeddings. Each positional encoding is unique to the position that it represents which allows model to identify which position each embedding is supposed to go. In order for the positional encodings to be added to the embeddings, they have to have the same dimension, d_model. We get the positional encodings by using the equations in Figure 6.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*G7lAIXZyRVn0QzeN" /><figcaption>Figure 6: Positional encoding equations</figcaption></figure><pre>pe = torch.zeros(max_seq_length, d_model)<br><br>for pos in range(max_seq_length):<br>  for i in range(d_model):<br>    if i % 2 == 0:<br>      pe[pos][i] = np.sin(pos/(10000 ** (i/d_model)))<br>    else:<br>      pe[pos][i] = np.cos(pos/(10000 ** ((i-1)/d_model)))<br><br>self.register_buffer(&#39;pe&#39;, pe.unsqueeze(0))</pre><p>In the forward method, the input is a batch of patch embeddings for multiple images. For example, a 32x32 image can be broken down into 16 patches of 8x8 size. In this <em>max_seq_length</em> would need to be 16+1=17 to create sufficient position embeddings, one for each patch, and one for the class token.</p><p>Because of this, we need to use the expand function in order to use self.cls_token to create class tokens for every image in the batch. As you, the attention model will encode information about the entire sequence into every token in the sequence. As each token is biased by their own information, the class token creates an independent summary of all of the tokens in a sequence.</p><pre>def forward(self, x):<br>    tokens_batch = self.cls_token.expand(x.size()[0], -1, -1)</pre><p>These classification tokens are then added to the beginning of each of the patch embeddings by using the torch.cat method.</p><pre>x = torch.cat((tokens_batch,x), dim=1)</pre><p>The positional encodings are added before being outputted.</p><pre>x = x + self.pe<br><br>return x</pre><p>This is how the data looks like before and after adding the class token:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0kOFYPz031d50CiGqI3vxQ.png" /><figcaption>Left: A 32x32 MNIST image broken into 16 8x8 patches using the Conv2D operation. Right: The 16 image patches after adding positional encoding and the class token, initialized with random data.</figcaption></figure><p>Note that we have initialized the Conv2D operation with 64 kernels, which each only take one pixel per patch, in order not to distort the image. The article that explains Conv2D in depth is linked further up.</p><p><strong>Attention Head</strong></p><pre>class AttentionHead(nn.Module):<br>  def __init__(self, d_model, head_size):<br>    super().__init__()<br>    self.head_size = head_size<br><br>    self.query = nn.Linear(d_model, head_size)<br>    self.key = nn.Linear(d_model, head_size)<br>    self.value = nn.Linear(d_model, head_size)<br><br>  def forward(self, x):<br>    # Obtaining Queries, Keys, and Values<br>    Q = self.query(x)<br>    K = self.key(x)<br>    V = self.value(x)<br><br>    # Dot Product of Queries and Keys<br>    attention = Q @ K.transpose(-2,-1)<br><br>    # Scaling<br>    attention = attention / (self.head_size ** 0.5)<br><br>    attention = torch.softmax(attention, dim=-1)<br><br>    attention = attention @ V<br><br>    return attention</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/0*N99XOqa8iH4WK_u9" /><figcaption>Figure 7: Scaled Dot-Product Attention and Multi-Head Attention diagrams</figcaption></figure><p>Vision transformers use attention which is a communication mechanism that allows the model to focus on important parts of an image. Attention scores can be calculated using the equation in Figure 8.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*nc_6eJDIlzGTqJAAihLPGA.png" /><figcaption>Figure 8: Attention equation</figcaption></figure><p>The first step in calculating attention is obtaining the queries, keys, and values of the tokens. The query of a token is what the token is looking for, the key is what the token contains, and the value is what is communicated between the tokens. The queries, keys, and values can be calculated by passing tokens through linear modules.</p><pre> def forward(self, x):<br>   # Obtaining Queries, Keys, and Values<br>   Q = self.query(x)<br>   K = self.key(x)<br>   V = self.value(x)</pre><p>We are able to get the relationship between the tokens in a sequence by getting the dot product of the queries and keys.</p><pre># Dot Product of Queries and Keys<br>attention = Q @ K.transpose(-2,-1)</pre><p>We need to scale these values to control variance at initialization so that tokens are able to aggregate information from more than one other token. Scaling is applied by dividing the dot product by the square root of the size of the attention head.</p><pre>attention = attention / (self.head_size ** 0.5)</pre><p>We then need to apply a soft max on the scaled dot product.</p><pre>attention = torch.softmax(attention, dim=-1)</pre><p>Finally, we need to get the dot product between the soft max and the values matrix. This is essentially communicating the information between the corresponding tokens.</p><pre>attention = attention @ V<br><br>return attention</pre><p><strong>Multi-Head Attention</strong></p><pre>class MultiHeadAttention(nn.Module):<br>  def __init__(self, d_model, n_heads):<br>    super().__init__()<br>    self.head_size = d_model // n_heads<br><br>    self.W_o = nn.Linear(d_model, d_model)<br><br>    self.heads = nn.ModuleList([AttentionHead(d_model, self.head_size) for _ in range(n_heads)])<br><br>  def forward(self, x):<br>    # Combine attention heads<br>    out = torch.cat([head(x) for head in self.heads], dim=-1)<br><br>    out = self.W_o(out)<br><br>    return out</pre><p>Multi-head attention is just running multiple heads of self-attention in parallel and combining them. We can do this by adding the attention heads into a module list,</p><pre>self.heads = nn.ModuleList([AttentionHead(d_model, self.head_size) for _ in range(n_heads)])</pre><p>passing through the input and concatenating the results.</p><pre>def forward(self, x):<br>  # Combine attention heads<br>  out = torch.cat([head(x) for head in self.heads], dim=-1)</pre><p>We then need to pass the output through another linear module.</p><pre>out = self.W_o(out)<br>return out</pre><p><strong>Transformer Encoder</strong></p><pre>class TransformerEncoder(nn.Module):<br>  def __init__(self, d_model, n_heads, r_mlp=4):<br>    super().__init__()<br>    self.d_model = d_model<br>    self.n_heads = n_heads<br><br>    # Sub-Layer 1 Normalization<br>    self.ln1 = nn.LayerNorm(d_model)<br><br>    # Multi-Head Attention<br>    self.mha = MultiHeadAttention(d_model, n_heads)<br><br>    # Sub-Layer 2 Normalization<br>    self.ln2 = nn.LayerNorm(d_model)<br><br>    # Multilayer Perception<br>    self.mlp = nn.Sequential(<br>        nn.Linear(d_model, d_model*r_mlp),<br>        nn.GELU(),<br>        nn.Linear(d_model*r_mlp, d_model)<br>    )<br><br>  def forward(self, x):<br>    # Residual Connection After Sub-Layer 1<br>    out = x + self.mha(self.ln1(x))<br><br>    # Residual Connection After Sub-Layer 2<br>    out = out + self.mlp(self.ln2(out))<br><br>    return out</pre><p>The transformer encoder is made up of two sub-layers: the first sub-layer performs multi-head attention and the second sub-layer contains a multi-layer perceptron. The multi-head attention sub-layer performs communication between tokens while the multi-layer perceptron sub-layers allows the tokens to individually “think” on what was communicated to them.</p><p>Layer normalization is an optimization technique that normalizes each input in the batch independently across its features. For our model, we will pass our inputs through a layer norm module at the beginning of each sub-layer.</p><pre># Sub-Layer 1 Normalization<br>self.ln1 = nn.LayerNorm(d_model)<br><br># Sub-Layer 2 Normalization<br>self.ln2 = nn.LayerNorm(d_model)</pre><p>The MLP will consist of two linear layers with a GELU layer in between. GELU is used instead of RELU because it doesn’t have RELU’s limitation of being non-differentiable at zero.</p><pre>    # Encoder Multilayer Perception<br>    self.mlp = nn.Sequential(<br>        nn.Linear(width, width*r_mlp),<br>        nn.GELU(),<br>        nn.Linear(width*r_mlp, width)<br>    )</pre><p>In the forward method for the encoder, the input is passed through the first layer normalization module before performing multi-head attention. The original input is added to the output from performing multi-head attention to create a residual connection.</p><p>This is then passed through another layer normalization module before being inputted into the MLP. Another residual connection is created by adding the output from the MLP to the out from the first residual connection.</p><p>The residual connections are used to help prevent the vanishing gradient problem by creating a path for the gradient to be back-propagated unimpeded back to the original input.</p><pre>def forward(self, x):<br>    # Residual Connection After Sub-Layer 1<br>    out = x + self.mha(self.ln1(x))<br><br>    # Residual Connection After Sub-Layer 2<br>    out = out + self.mlp(self.ln2(out))<br><br>    return out</pre><p><strong>Vision Transformer</strong></p><pre>class VisionTransformer(nn.Module):<br>  def __init__(self, d_model, n_classes, img_size, patch_size, n_channels, n_heads, n_layers):<br>    super().__init__()<br><br>    assert img_size[0] % patch_size[0] == 0 and img_size[1] % patch_size[1] == 0, &quot;img_size dimensions must be divisible by patch_size dimensions&quot;<br>    assert d_model % n_heads == 0, &quot;d_model must be divisible by n_heads&quot;<br><br>    self.d_model = d_model # Dimensionality of model<br>    self.n_classes = n_classes # Number of classes<br>    self.img_size = img_size # Image size<br>    self.patch_size = patch_size # Patch size<br>    self.n_channels = n_channels # Number of channels<br>    self.n_heads = n_heads # Number of attention heads<br><br>    self.n_patches = (self.img_size[0] * self.img_size[1]) // (self.patch_size[0] * self.patch_size[1])<br>    self.max_seq_length = self.n_patches + 1<br><br>    self.patch_embedding = PatchEmbedding(self.d_model, self.img_size, self.patch_size, self.n_channels)<br>    self.positional_encoding = PositionalEncoding( self.d_model, self.max_seq_length)<br>    self.transformer_encoder = nn.Sequential(*[TransformerEncoder( self.d_model, self.n_heads) for _ in range(n_layers)])<br><br>    # Classification MLP<br>    self.classifier = nn.Sequential(<br>        nn.Linear(self.d_model, self.n_classes),<br>        nn.Softmax(dim=-1)<br>    )<br><br>  def forward(self, images):<br>    x = self.patch_embedding(images)<br><br>    x = self.positional_encoding(x)<br><br>    x = self.transformer_encoder(x)<br>    <br>    x = self.classifier(x[:,0])<br><br>    return x</pre><p>When creating our vision transformer class, we first need to make sure that the input images can be split evenly into patches of size patch_size and that the dimensionality of the model is divisible by the number of attention heads.</p><pre>assert img_size[0] % patch_size[0] == 0 and img_size[1] % patch_size[1] == 0, &quot;img_size dimensions must be divisible by patch_size dimensions&quot;<br>assert d_model % n_heads == 0, &quot;d_model must be divisible by n_heads&quot;</pre><p>We also need to calculate the maximum sequence length for the positional encoding, which will be equal to the number of patches plus one. The number of patches can be found by dividing the product of the height and width of the input image by the product of the height and width of the patch size.</p><pre>self.n_patches = (self.img_size[0] * self.img_size[1]) // (self.patch_size[0] * self.patch_size[1])<br>self.max_seq_length = self.n_patches + 1</pre><p>The vision transformer will also need to be able to have multiple encoder modules. This can be achieved by putting a list of encoder layers inside of a sequential wrapper.</p><pre>self.encoder = nn.Sequential(*[TransformerEncoder(self.d_model, self.n_heads) for _ in range(n_layers)])</pre><p>The last part of the vision transformer model is the MLP classification head. This is consists of a linear layer followed by a soft-max layer.</p><pre>self.classifier = nn.Sequential(<br>        nn.Linear(self.d_model, self.n_classes),<br>        nn.Softmax(dim=-1)<br>)</pre><p>In the forward method, the input images are first passed through the patch embeddings layer to split the image into patches and get the sequence of linear embeddings for those patches. They are then passed through the positional encoding layer to add the classification token and positional encoding before being passed through the encoder modules. The classification tokens are then passed through the classification MLP to determine the classes of the images.</p><pre>def forward(self, images):<br>    x = self.patch_embedding(images)<br><br>    x = self.position_embedding(x)<br><br>    x = self.encoder(x)<br><br>    x = self.classifier(x[:,0])<br><br>    return x</pre><p>We are done building the model. Now we need to train and test it.</p><p><strong>Training Parameters</strong></p><pre>d_model = 9<br>n_classes = 10<br>img_size = (32,32)<br>patch_size = (16,16)<br>n_channels = 1<br>n_heads = 3<br>n_layers = 3<br>batch_size = 128<br>epochs = 5<br>alpha = 0.005</pre><p><strong>Loading MNIST Dataset</strong></p><pre>transform = T.Compose([<br>  T.Resize(img_size),<br>  T.ToTensor()<br>])<br><br>train_set = MNIST(<br>  root=&quot;./../datasets&quot;, train=True, download=True, transform=transform<br>)<br>test_set = MNIST(<br>  root=&quot;./../datasets&quot;, train=False, download=True, transform=transform<br>)<br><br>train_loader = DataLoader(train_set, shuffle=True, batch_size=batch_size)<br>test_loader = DataLoader(test_set, shuffle=False, batch_size=batch_size)</pre><p><strong>Training</strong></p><p>The code below trains our VisionTransformer class using the MNIST training set, showing the average loss over an entire epoch. It takes less than a minute to train using an entire epoch on a Tesla T4 GPU on colab. To better understand PyTorch optimizers and what happens in the code below check out the following optimizer:</p><p><a href="https://medium.com/@nikolaus.correll/a-primer-on-using-pytorch-optimizers-7a97e0999095">A primer on using PyTorch Optimizers</a></p><pre>device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br>print(&quot;Using device: &quot;, device, f&quot;({torch.cuda.get_device_name(device)})&quot; if torch.cuda.is_available() else &quot;&quot;)<br><br>transformer = VisionTransformer(d_model, n_classes, img_size, patch_size, n_channels, n_heads, n_layers).to(device)<br><br>optimizer = Adam(transformer.parameters(), lr=alpha)<br>criterion = nn.CrossEntropyLoss()<br><br>for epoch in range(epochs):<br><br>  training_loss = 0.0<br>  for i, data in enumerate(train_loader, 0):<br>    inputs, labels = data<br>    inputs, labels = inputs.to(device), labels.to(device)<br><br>    optimizer.zero_grad()<br><br>    outputs = transformer(inputs)<br>    loss = criterion(outputs, labels)<br>    loss.backward()<br>    optimizer.step()<br><br>    training_loss += loss.item()<br><br>  print(f&#39;Epoch {epoch + 1}/{epochs} loss: {training_loss  / len(train_loader) :.3f}&#39;)</pre><p><strong>Testing</strong></p><pre>correct = 0<br>total = 0<br><br>with torch.no_grad():<br>  for data in test_loader:<br>    images, labels = data<br>    images, labels = images.to(device), labels.to(device)<br><br>    outputs = transformer(images)<br><br>    _, predicted = torch.max(outputs.data, 1)<br>    total += labels.size(0)<br>    correct += (predicted == labels).sum().item()<br>  print(f&#39;\nModel Accuracy: {100 * correct // total} %&#39;)</pre><p><strong>Results</strong></p><p>Using this model, we were able to achieve and accuracy of ~ 92% on the MNIST dataset training over only 5 epochs. This example demonstrates that self-attention can be used as a stand-in for a deep convolutional network. Read on to learn how to combine vision transformers with text.</p><p><a href="https://medium.com/correll-lab/building-clip-from-scratch-68f6e42d35f4">Building CLIP From Scratch</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3054f707cc6" width="1" height="1" alt=""><hr><p><a href="https://medium.com/correll-lab/building-a-vision-transformer-model-from-scratch-a3054f707cc6">Building a Vision Transformer Model From Scratch</a> was originally published in <a href="https://medium.com/correll-lab">Toward Humanoids</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>