<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Subho Ghosh on Medium]]></title>
        <description><![CDATA[Stories by Subho Ghosh on Medium]]></description>
        <link>https://medium.com/@ighoshsubho?source=rss-a1b5dc939fb7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*pfnAhDZCR5lKkYuZ</url>
            <title>Stories by Subho Ghosh on Medium</title>
            <link>https://medium.com/@ighoshsubho?source=rss-a1b5dc939fb7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:37:12 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ighoshsubho/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[DiTs explained from scratch]]></title>
            <link>https://medium.com/@ighoshsubho/dits-explained-from-scratch-4fdb9192b1cf?source=rss-a1b5dc939fb7------2</link>
            <guid isPermaLink="false">https://medium.com/p/4fdb9192b1cf</guid>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[vision-transformer]]></category>
            <category><![CDATA[flux]]></category>
            <category><![CDATA[diffusion-models]]></category>
            <category><![CDATA[attention-mechanism]]></category>
            <dc:creator><![CDATA[Subho Ghosh]]></dc:creator>
            <pubDate>Mon, 14 Oct 2024 16:26:46 GMT</pubDate>
            <atom:updated>2024-10-14T18:30:56.956Z</atom:updated>
            <content:encoded><![CDATA[<p>Diffusion transformers are new paradigm of image generation, they power both models like SD3 and Flux as Multi-Modal Diffusion Transformer backbone. But there seems to be a dearth of quality resources to refer to, so I hope its helpful not only to you but me as well for future reference.</p><p>Since DiTs root lies from Vision Transformers its better have quick but clear understanding of ViTs first.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RptwU8iskyQhE8D28PnBvQ.png" /><figcaption>Fig1: Illustration of the Vision Transformer encoder</figcaption></figure><p>The number of research works in vision increased since the paper <em>Attention is all you need</em> was on air. ViTs apply a basic approach,</p><ol><li>First they convert images to number of patches, lets consider 4 patches for now, and since images have 3 channels, we have input size - Number of patches * (Channels * Patch Height * Patch Width). And this gets projected to D dimensional vector space. The positional embedding is added to store the information related to position of pixels in the image. <em>A CLS token is added which would be later used to get the classification scores.</em></li><li>Then comes our smart and intelligent friend called Attention, which helps to understand how pixels relates among one another, this is done by Multi Head Attention which in itself a great topic to dive deep. We will refer this in later sections.</li><li>The MLP is the last layer for single ViT block and in between all three, comes layer norm, which helps get smoother gradient flows and even benefits Multi Head Attention for stability during training. And hence we connect N layers of ViT block and there lies a residual connection after each block and layer norm.</li></ol><p>Well this is not a ViT class, its DiT! Why the heck its here! That’s coz if you understand ViT, you know 70% of how DiT works.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pb6WU8aQ_OQ9u_O4wKsBeQ.png" /><figcaption>Fig2: The Diffusion Transformer (DiT) architecture.In left are the latent DiT models which gets trained while in right are the details of DiT blocks which will be explained later in this post. :)</figcaption></figure><p>Just like regular diffusion models DiTs require VAEs to encode and decode images from pixels to latent space and vice versa.</p><p>Lets consider for now the entire DiT is a vision transformer with VAE, and the last layer outputs embeddings of size D dimension and transforms it to size <em>(Channels * Patch Height * Patch Width)</em> latent patches.</p><p>The latents which gets out is then decoded to pixel space by pre trained VAE decoder such that we can get a high resolution image in pixel space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SbZcETP8p5PCGaGUnJMcFg.png" /><figcaption>Fig3: Time step and class representations</figcaption></figure><p>Diffusion on simple terms is a denoising process and it needs some time step information as well. This is done with an embedding layer and 2 MLP layers to project it into D dimensional space. The class representation is trainable and it gets projected to that dimension too.</p><p>Now lets deep dive into Fig2 of DiT. And lets disect it a little bit. There are 2 variants which has been proposed by authors, which are as follows,</p><ol><li><strong>In context conditioning </strong>— Simply the time step and class embedding is added as two additional tokens and before the un-patchify block we shall get rid of these tokens and work with remaining tokens.</li><li><strong>Cross attention </strong>— In this variant, a cross attention layer is added between multi head attention and MLP, such that it can attend to class embedding and time steps representation with this layer.</li><li><strong>Adaptive layer norm </strong>— Here time step embedding gets added with class embedding, which is passed through an MLP layer which projects it to 4 D dimensional affine parameters. These acts as the β, γ for magnitude and shift of weights activated during forward pass.</li><li><strong>Adaptive layer norm zero</strong> — This adds additional 2 D dimensional parameters which ensures the scale we need to multiply before the residual connections α.</li></ol><p>The scale is initially set to zero such that its a identity block (outputs same as input) in the beginning to ensure better training and gradients.</p><p>Based on DiT scale, we all know that as we throw more compute and parameter count, the quality gets better and better. Same applies here. The official paper had these model configurations with layers, dimensions and heads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/666/1*tCV21jIvz0r1fodZ4A4-4w.png" /><figcaption>Fig4: Details of DiTs scale, model configurations for the Small (S), Base (B) and Large (L) and XLarge (XL) variants</figcaption></figure><p>Not just that, taking model parameters as constant, its found out that lesser the patch size (or more the patch count for an image), the image quality gets better and better.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/684/1*ypNYzbmgpto0L3yJuxljMg.png" /><figcaption>Fig5: FID-50k comparison for different conditioning strategies — lower is better</figcaption></figure><p>As the flops/compute increases, we get improved performance with DiT. The largest model trained in this paper is DiT XL of patch size 2, which generates images of really high quality and gets FID score lower than even latent diffusion models.</p><p>Wooh! A lot to digest, take your time and relax a little bit. Now lets implement DiT from scratch,</p><p>Lets implement the extract patches and reconstruct images function first, and since I’m GPU poor, please don’t mind me training on patch size 8 and also the implementation would be adaptive layer norm.</p><pre>def extract_patches(image_tensor, patch_size=8):<br>    &quot;&quot;&quot;<br>    Extracts patches from an image tensor.<br><br>    Args:<br>        image_tensor (torch.Tensor): Input image tensor with shape (bs, c, h, w).<br>        patch_size (int, optional): Size of the patches to extract. Defaults to 8.<br><br>    Returns:<br>        torch.Tensor: Extracted patches with shape (bs, L, c * patch_size * patch_size),<br>                      where L is the number of patches.<br>    &quot;&quot;&quot;<br>    bs, c, h, w = image_tensor.size()<br><br>    unfold = torch.nn.Unfold(kernel_size=patch_size, stride=patch_size)<br>    unfolded = unfold(image_tensor)<br><br>    unfolded = unfolded.transpose(1, 2).reshape(bs, -1, c * patch_size * patch_size)<br>    return unfolded<br><br>def reconstruct_image(patch_sequence, image_shape, patch_size=8):<br>    &quot;&quot;&quot;<br>    Reconstructs the original image tensor from a sequence of patches.<br><br>    Args:<br>        patch_sequence (torch.Tensor): Sequence of patches with shape<br>                                       (bs, L, c * patch_size * patch_size).<br>        image_shape (tuple): Shape of the original image tensor (bs, c, h, w).<br>        patch_size (int, optional): Size of the patches used in extraction. Defaults to 8.<br><br>    Returns:<br>        torch.Tensor: Reconstructed image tensor with shape (bs, c, h, w).<br>    &quot;&quot;&quot;<br>    bs, c, h, w = image_shape<br>    num_patches_h = h // patch_size<br>    num_patches_w = w // patch_size<br>    <br>    unfolded_shape = (bs, num_patches_h, num_patches_w, patch_size, patch_size, c)<br>    patch_sequence = patch_sequence.view(*unfolded_shape)<br>    <br>    patch_sequence = patch_sequence.permute(0, 5, 1, 3, 2, 4).contiguous()<br>    <br>    reconstructed = patch_sequence.view(bs, c, h, w)<br>    <br>    return reconstructed</pre><p>Now its time to make the conditional layer norm which scales the layer norm with the affine parameters from time and class embeddings. (In this implementation only time embeddings)</p><pre>import torch.nn as nn<br><br>class ConditionalNorm2d(nn.Module):<br>    &quot;&quot;&quot;<br>    Conditional Layer Normalization module for 2D inputs.<br><br>    This module applies layer normalization and then scales and shifts the normalized<br>    input based on input features.<br><br>    Args:<br>        hidden_size (int): The size of the hidden dimension to normalize.<br>        num_features (int): The number of input features for condition.<br><br>    Attributes:<br>        norm (nn.LayerNorm): Layer normalization module.<br>        fcw (nn.Linear): Linear layer for generating the scaling factor.<br>        fcb (nn.Linear): Linear layer for generating the shift factor.<br>    &quot;&quot;&quot;<br><br>    def __init__(self, hidden_size, num_features):<br>        super(ConditionalNorm2d, self).__init__()<br>        self.norm = nn.LayerNorm(hidden_size, elementwise_affine=False)<br>        self.fcw = nn.Linear(num_features, hidden_size)<br>        self.fcb = nn.Linear(num_features, hidden_size)<br><br>    def forward(self, x, features):<br>        &quot;&quot;&quot;<br>        Forward pass of the ConditionalNorm2d module.<br><br>        Args:<br>            x (torch.Tensor): Input tensor of shape (batch_size, sequence_length, hidden_size).<br>            features (torch.Tensor): Conditioning features of shape (batch_size, num_features).<br><br>        Returns:<br>            torch.Tensor: Normalized and conditioned output tensor of the same shape as input x.<br>        &quot;&quot;&quot;<br>        bs, s, l = x.shape<br>        <br>        out = self.norm(x)<br>        w = self.fcw(features).reshape(bs, 1, -1)<br>        b = self.fcb(features).reshape(bs, 1, -1)<br><br>        return w * out + b</pre><p>Now lets implement the Transformer block and we shall incorporate all the above classes in the main DiT block.</p><pre>import torch<br>import torch.nn as nn<br>from sinusoidal_pos_emb import SinusoidalPosEmb<br>from patch_utils import extract_patches, reconstruct_image<br>from conditional_norm2d import ConditionalNorm2d<br><br>class TransformerBlock(nn.Module):<br>    &quot;&quot;&quot;<br>    Transformer block with self-attention and conditional normalization.<br><br>    Args:<br>        hidden_size (int): Size of the hidden dimension. Default is 128.<br>        num_heads (int): Number of attention heads. Default is 4.<br>        num_features (int): Number of features for conditional normalization. Default is 128.<br><br>    Attributes:<br>        norm (nn.LayerNorm): Layer normalization for input.<br>        multihead_attn (nn.MultiheadAttention): Multi-head attention mechanism.<br>        con_norm (ConditionalNorm2d): Conditional normalization layer.<br>        mlp (nn.Sequential): Multi-layer perceptron for feature processing.<br>    &quot;&quot;&quot;<br><br>    def __init__(self, hidden_size=128, num_heads=4, num_features=128):<br>        super(TransformerBlock, self).__init__()<br>        <br>        self.norm = nn.LayerNorm(hidden_size)<br>        self.multihead_attn = nn.MultiheadAttention(hidden_size, num_heads=num_heads, <br>                                                    batch_first=True, dropout=0.0)<br>        self.con_norm = ConditionalNorm2d(hidden_size, num_features)<br>        self.mlp = nn.Sequential(<br>            nn.Linear(hidden_size, hidden_size * 4),<br>            nn.LayerNorm(hidden_size * 4),<br>            nn.ELU(),<br>            nn.Linear(hidden_size * 4, hidden_size)<br>        )<br>                <br>    def forward(self, x, features):<br>        &quot;&quot;&quot;<br>        Forward pass of the TransformerBlock.<br><br>        Args:<br>            x (torch.Tensor): Input tensor.<br>            features (torch.Tensor): Conditional features for normalization.<br><br>        Returns:<br>            torch.Tensor: Processed tensor after attention and MLP layers.<br>        &quot;&quot;&quot;<br>        norm_x = self.norm(x)<br>        x = self.multihead_attn(norm_x, norm_x, norm_x)[0] + x<br>        norm_x = self.con_norm(x, features)<br>        x = self.mlp(norm_x) + x<br>        return x<br><br>class DiT(nn.Module):<br>    &quot;&quot;&quot;<br>    Diffusion Transformer (DiT) module for vision encoding.<br><br>    Args:<br>        image_size (int): Size of the input image (assuming square images).<br>        channels_in (int): Number of input channels.<br>        patch_size (int): Size of image patches. Default is 16.<br>        hidden_size (int): Size of the hidden dimension. Default is 128.<br>        num_features (int): Number of features for time embedding. Default is 128.<br>        num_layers (int): Number of transformer layers. Default is 3.<br>        num_heads (int): Number of attention heads in each transformer block. Default is 4.<br><br>    Attributes:<br>        time_mlp (nn.Sequential): MLP for time step embedding.<br>        patch_size (int): Size of image patches.<br>        fc_in (nn.Linear): Linear layer for patch embedding.<br>        pos_embedding (nn.Parameter): Learnable positional embeddings.<br>        blocks (nn.ModuleList): List of TransformerBlock modules.<br>        fc_out (nn.Linear): Linear layer for output projection.<br>    &quot;&quot;&quot;<br><br>    def __init__(self, image_size, channels_in, patch_size=16, <br>                 hidden_size=128, num_features=128, <br>                 num_layers=3, num_heads=4):<br>        super(DiT, self).__init__()<br>        <br>        self.time_mlp = nn.Sequential(<br>            SinusoidalPosEmb(num_features),<br>            nn.Linear(num_features, 2 * num_features),<br>            nn.GELU(),<br>            nn.Linear(2 * num_features, num_features),<br>            nn.GELU()<br>        )<br>        <br>        self.patch_size = patch_size<br>        self.fc_in = nn.Linear(channels_in * patch_size * patch_size, hidden_size)<br>        <br>        seq_length = (image_size // patch_size) ** 2<br>        self.pos_embedding = nn.Parameter(torch.empty(1, seq_length, hidden_size).normal_(std=0.02))<br>        <br>        self.blocks = nn.ModuleList([<br>            TransformerBlock(hidden_size, num_heads) for _ in range(num_layers)<br>        ])<br>        <br>        self.fc_out = nn.Linear(hidden_size, channels_in * patch_size * patch_size)<br>                <br>    def forward(self, image_in, index):<br>        &quot;&quot;&quot;<br>        Forward pass of the DiT module.<br><br>        Args:<br>            image_in (torch.Tensor): Input image tensor.<br>            index (torch.Tensor): Time step index tensor.<br><br>        Returns:<br>            torch.Tensor: Processed image tensor.<br>        &quot;&quot;&quot;<br>        index_features = self.time_mlp(index)<br><br>        patch_seq = extract_patches(image_in, patch_size=self.patch_size)<br>        patch_emb = self.fc_in(patch_seq)<br><br>        embs = patch_emb + self.pos_embedding<br>        <br>        for block in self.blocks:<br>            embs = block(embs, index_features)<br>        <br>        image_out = self.fc_out(embs)<br>        <br>        return reconstruct_image(image_out, image_in.shape, patch_size=self.patch_size)</pre><p>Now comes our very old friend from DDIM paper, the term which helped us to add noise from very first step in diffusion process in a very deterministic approach,</p><p>α cum prod is the product of all α which indicates the amount of original image remaining in the image, while β is the amount of gaussian noise you add to the image as time step progresses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/648/1*JdyhfcNqhLMPzXAe7FwiAg.png" /><figcaption>Fig6: Conditional probability for a pixel at time t given its noise at time 0</figcaption></figure><p>Here we are gonna implementing this,</p><pre>import torch<br>from tqdm import trange<br>import copy<br><br>def noise_from_x0(curr_img, img_pred, alpha):<br>    &quot;&quot;&quot;<br>    Calculate the noise from the current image and its prediction.<br><br>    Args:<br>        curr_img (torch.Tensor): The current image.<br>        img_pred (torch.Tensor): The predicted image.<br>        alpha (float): The alpha value for the diffusion process.<br><br>    Returns:<br>        torch.Tensor: The calculated noise.<br>    &quot;&quot;&quot;<br>    return (curr_img - alpha.sqrt() * img_pred) / ((1 - alpha).sqrt() + 1e-4)<br><br>def cold_diffuse(diffusion_model, sample_in, total_steps, start_step=0):<br>    &quot;&quot;&quot;<br>    Perform cold diffusion on the input sample.<br><br>    Args:<br>        diffusion_model (torch.nn.Module): The diffusion model to use.<br>        sample_in (torch.Tensor): The input sample to diffuse.<br>        total_steps (int): The total number of diffusion steps.<br>        start_step (int, optional): The step to start diffusion from. Defaults to 0.<br><br>    Returns:<br>        torch.Tensor: The diffused output image.<br>    &quot;&quot;&quot;<br>    diffusion_model.eval()<br>    bs = sample_in.shape[0]<br>    device = sample_in.device<br>    alphas = torch.flip(cosine_alphas_bar(total_steps), (0,)).to(device)<br>    random_sample = copy.deepcopy(sample_in)<br><br>    with torch.no_grad():<br>        for i in trange(start_step, total_steps - 1):<br>            index = (i * torch.ones(bs, device=device)).long()<br><br>            img_output = diffusion_model(random_sample, index)<br><br>            noise = noise_from_x0(random_sample, img_output, alphas[i])<br>            x0 = img_output<br><br>            rep1 = alphas[i].sqrt() * x0 + (1 - alphas[i]).sqrt() * noise<br>            rep2 = alphas[i + 1].sqrt() * x0 + (1 - alphas[i + 1]).sqrt() * noise<br><br>            random_sample += rep2 - rep1<br><br>        index = ((total_steps - 1) * torch.ones(bs, device=device)).long()<br>        img_output = diffusion_model(random_sample, index)<br><br>    return img_output</pre><p>Voilà! You made it to the end, wait a little so that I could show you the possibilities. Though you did already become a DiT professional, why not get to an expert level.</p><p>Time to train our model, we use Adam as our optimizer, use L1 loss for noise prediction and initialize model for the training,</p><pre>dit = DiT(latent_size, channels_in=latents.shape[1], patch_size=patch_size, <br>            hidden_size=768, num_layers=10, num_heads=8).to(device)<br><br>optimizer = optim.Adam(dit.parameters(), lr=lr)<br><br>alphas = torch.flip(cosine_alphas_bar(timesteps), (0,)).to(device)<br><br>dit.train()<br>for epoch in pbar:<br>    pbar.set_postfix_str(&#39;Loss: %.4f&#39; % (mean_loss/len(train_loader)))<br>    mean_loss = 0<br><br>    for num_iter, (latents) in enumerate(tqdm(train_loader, leave=False)):<br>        latents = latents.to(device)<br>        <br>        #the size of the current minibatch<br>        bs = latents.shape[0]<br><br>        rand_index = torch.randint(timesteps, (bs, ), device=device)<br>        random_sample = torch.randn_like(latents)<br>        alpha_batch = alphas[rand_index].reshape(bs, 1, 1, 1)<br>        <br>        noise_input = alpha_batch.sqrt() * latents +\<br>        (1 - alpha_batch).sqrt() * random_sample<br>        <br>        with torch.cuda.amp.autocast():<br>            latent_pred = dit(noise_input, rand_index)<br>            loss = F.l1_loss(latent_pred, latents)<br>        <br>        # Backpropagation<br>        optimizer.zero_grad()<br>        scaler.scale(loss).backward()<br>        scaler.step(optimizer)<br>        scaler.update()<br>        <br>        #log the generator training loss<br>        loss_log.append(loss.item())<br>        mean_loss += loss.item()<br><br>    # Quick save of the model every epoch<br>    torch.save({&#39;epoch&#39;: epoch + 1,<br>                &#39;train_data_logger&#39;: loss_log,<br>                &#39;model_state_dict&#39;: dit.state_dict(),<br>                &#39;optimizer_state_dict&#39;: optimizer.state_dict(),<br>                 }, &quot;latent_dit.pt&quot;)<br><br>plt.plot(loss_log[1000:])</pre><p>when we plot the loss after 1000 iterations we get and this was trained for a million times and could take a couple of days but you don’t need to wait that long, directly pull in weights from HF,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yEBl-kNaKKFuUSz1Y7YSSQ.png" /><figcaption>Fig7: Log Loss after 1000 iterations</figcaption></figure><p>Lets see how our custom trained diffusion transformer performed, lets make 8 random noise samples and let the diffusion transformer sample from these images, I’m CURIOUS to find out the results,</p><pre>latent_noise = 0.95 * torch.randn(8, 4, latent_size, latent_size, device=device)<br>with torch.no_grad():<br>    with torch.cuda.amp.autocast():<br>        fake_latents = cold_diffuse(dit, latent_noise, total_steps=timesteps)<br>        fake_sample = vae.decode(fake_latents / 0.18215).sample<br><br>plt.figure(figsize = (20, 10))<br>out = vutils.make_grid(fake_sample.detach().float().cpu(), nrow=4, normalize=True)<br>_ = plt.imshow(out.numpy().transpose((1, 2, 0)))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XFL0MJrm8Q75cTjKSDtTJA.png" /><figcaption>Fig8: These AI figures might take over our society</figcaption></figure><p>Damn! I can’t believe these were generated by training a simple Diffusion Transformer. What are the possibilities? ENDLESS!</p><p>Image to Image is an interesting workflow to explore, here instead of starting with pure noise, we can start with an image that has been noised up via the forward process and use this as a starting point for generation! The more noise we add, the further the generated image will be from the source image!</p><pre>with torch.no_grad():<br>    with torch.cuda.amp.autocast():<br>        latents = vae.encode(test_tensor).latent_dist.sample().mul_(0.18215)<br>        latents = latents.expand(mini_batch_size, 4, latent_size, latent_size)<br>        latent_noise = 0.95 * torch.randn_like(latents)<br><br>        alpha_batch = alphas[index].expand(mini_batch_size).reshape(mini_batch_size, <br>                                                                    1, 1, 1)<br>        noise_input = alpha_batch.sqrt() * latents +\<br>        (1 - alpha_batch).sqrt() * latent_noise<br>        <br>        fake_latents = cold_diffuse(dit, noise_input, <br>                                    total_steps=timesteps, <br>                                    start_step=index)<br>        <br>        fake_sample = vae.decode(fake_latents / 0.18215).sample</pre><p>RESULTs? Here it is!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/878/1*GOmiZOKrJ-bnyCrGel-BQA.png" /><figcaption>Fig9: Obama is gonna love this generated images</figcaption></figure><p>Another great implementation could be inpainting of an image, that is, painting (generating) over specific sections of a source image.</p><p>To do this we need to update our diffusion generation loop to include the target image (or latent) and the “mask”. This “mask” in it’s simplest form is a binary (1s and 0s) that define what section of the target image we want to keep and what section we want to remove!</p><pre><br>def cold_diffuse_inpaint(diffusion_model, sample_in, target, mask, total_steps, start_step=0):<br>    &quot;&quot;&quot;<br>    Perform cold diffusion inpainting on the input sample.<br><br>    This function applies the cold diffusion process while incorporating a target image<br>    and a mask for inpainting purposes.<br><br>    Args:<br>        diffusion_model (torch.nn.Module): The diffusion model to use.<br>        sample_in (torch.Tensor): The input sample to diffuse.<br>        target (torch.Tensor): The target image for inpainting.<br>        mask (torch.Tensor): The mask indicating which parts of the image to inpaint.<br>        total_steps (int): The total number of diffusion steps.<br>        start_step (int, optional): The step to start diffusion from. Defaults to 0.<br><br>    Returns:<br>        torch.Tensor: The inpainted output image.<br>    &quot;&quot;&quot;<br>    diffusion_model.eval()<br>    bs = sample_in.shape[0]<br>    device = sample_in.device<br>    alphas = torch.flip(cosine_alphas_bar(total_steps), (0,)).to(device)<br>    random_sample = copy.deepcopy(sample_in)<br>    <br>    with torch.no_grad():<br>        for i in trange(start_step, total_steps - 1):<br>            index = (i * torch.ones(bs, device=device)).long()<br><br>            noisy_target = alphas[i].sqrt() * target + \<br>                           (1 - alphas[i]).sqrt() * torch.randn_like(target)<br>                <br>            random_sample = mask * random_sample + (1 - mask) * noisy_target<br>            <br>            img_output = diffusion_model(random_sample, index)<br><br>            noise = noise_from_x0(random_sample, img_output, alphas[i])<br>            x0 = img_output<br><br>            rep1 = alphas[i].sqrt() * x0 + (1 - alphas[i]).sqrt() * noise<br>            rep2 = alphas[i + 1].sqrt() * x0 + (1 - alphas[i + 1]).sqrt() * noise<br>            <br>            random_sample += rep2 - rep1<br>            <br>        index = ((total_steps - 1) * torch.ones(bs, device=device)).long()<br>        img_output = diffusion_model(random_sample, index)<br><br>    return img_output<br><br><br>with torch.no_grad():<br>    with torch.cuda.amp.autocast():<br>        latents = vae.encode(test_tensor).latent_dist.sample().mul_(0.18215)<br>        latents = latents.expand(mini_batch_size, 4, latent_size, latent_size)<br>        noise_input = 0.9 * torch.randn_like(latents)<br><br>        fake_latents = cold_diffuse_inpaint(dit, <br>                                            noise_input, <br>                                            total_steps=timesteps,<br>                                            target=latents,<br>                                            mask=mask)<br>        <br>        fake_sample = vae.decode(fake_latents / 0.18215).sample</pre><p>The images are not that promising given we are not generating any smooth mask and the patch size and the model we did choose. We can also try by inverting the mask,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/918/1*M_cYHbKdBMMkS-9cSBDkHA.png" /><figcaption>Fig10: Obama in parallel universe (kiddin just different looks)</figcaption></figure><p>Lets revert the mask and see how much diversity we can get in the background for these images. Now this could be pretty hilarious,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/968/1*cW6tNLKSU5exVvMP03Dg7g.png" /><figcaption>Fig11: Obama in different roles</figcaption></figure><p>Wait its not over yet, you can incorporate text conditioning with cross attention layers which might diversify your image generation more.</p><p>Here is how PixArt α implemented their first diffusion transformer with text conditioning with additional multi-head cross attention layers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/872/1*Zq3YVKndzOChnYctfv-dWQ.png" /><figcaption>Fig12: Model architecture of PIXART-α. A cross-attention module is integrated into each block to inject textual conditions.</figcaption></figure><p>Not All of the code shared are shared in this post, I have gone through the below references and took an inspiration from them mostly the theory behind diffusion. For detailed implementation you can refer <a href="https://github.com/explainingai-code/DiT-PyTorch/tree/main">this</a>.</p><p>So you did become a DiT expert huh! Don’t forget to share this blog, took an entire week to build this whole implementation by going through research only for you.</p><p>Kindly support by sharing this with your friends! See you in next deep dive posts :)</p><p>References:</p><ol><li><a href="https://github.com/chuanyangjin/fast-DiT">https://github.com/chuanyangjin/fast-DiT</a></li><li><a href="https://github.com/facebookresearch/DiT">https://github.com/facebookresearch/DiT</a></li><li><a href="https://arxiv.org/pdf/2212.09748">https://arxiv.org/pdf/2212.09748</a></li><li><a href="https://www.youtube.com/watch?v=aSLDXdc2hkk&amp;t=1322s">https://www.youtube.com/watch?v=aSLDXdc2hkk&amp;t=1322s</a></li><li><a href="https://www.youtube.com/watch?v=tU_ix9UU-g0&amp;t=16s">https://www.youtube.com/watch?v=tU_ix9UU-g0&amp;t=16s</a></li><li><a href="https://arxiv.org/pdf/2310.00426">https://arxiv.org/pdf/2310.00426</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4fdb9192b1cf" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>