<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dor Zvulun on Medium]]></title>
        <description><![CDATA[Stories by Dor Zvulun on Medium]]></description>
        <link>https://medium.com/@dzdata?source=rss-aaf4831d9b34------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*kAIs1Tz6M-YW1sOKnHJwcw.png</url>
            <title>Stories by Dor Zvulun on Medium</title>
            <link>https://medium.com/@dzdata?source=rss-aaf4831d9b34------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 04:00:23 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dzdata/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Introduction to Stable Diffusion with Hugging Face]]></title>
            <link>https://dzdata.medium.com/introduction-to-stable-diffusion-with-hugging-face-2d9b061cc18f?source=rss-aaf4831d9b34------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d9b061cc18f</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[stable-diffusion]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Dor Zvulun]]></dc:creator>
            <pubDate>Mon, 05 Oct 2026 16:01:02 GMT</pubDate>
            <atom:updated>2026-10-05T16:01:02.182Z</atom:updated>
            <content:encoded><![CDATA[<p>In a previous <a href="https://medium.com/@dzdata/intro-to-diffusion-model-part-1-29fe7724c043">series of posts</a>, we introduced the basic diffusion model and the ideas behind it, such as iterative refinement. The model we wrote let us generate beautiful images of butterflies, but we had no control over the final image (we got a random butterfly), and it was time-consuming. In this post, we will leverage the ideas behind a model called Stable Diffusion to condition the generated image using text descriptions and also improve the generation process. We are also going to use the Hugging Face library since it already implements the necessary components of Stable Diffusion</p><h3>Latent Diffusion Models</h3><p>Before we talk about conditioning image generation, let’s start with another problem we have. One of the main challenges in image generation is to generate a high-quality, high-resolution image. The problem is that as the image size increases, so does the computational power required to generate the image. If our model uses <a href="https://medium.com/@dzdata/from-attention-to-self-attention-and-transformers-a-mathematical-introduction-45ed053a1ed3">self-attention</a>, the computational power grows quadratically! For example, suppose we want to increase the image resolution from a 64x64 image to a 128x128 image. The new image has four times as many pixels as the first one, which means it requires 16 times the memory and computational cost in a self-attention layer. This is a high price to pay for just double the image size.</p><p>Here comes Latent Diffusion. The idea is to stop working directly in the image&#39;s pixel space and instead move to a lower-dimensional representation in what we call a latent space. We can achieve this by using a <a href="https://medium.com/python-in-plain-english/variational-autoencoder-1eb543f5f055">Variational AutoEncoder</a> (VAE) before the actual diffusion model. The main reason to use it is that the VAE can compress images to a smaller spatial dimension while it removes a lot of redundant information the image may contain. A good VAE reconstructs an image from the latent space back into a high-fidelity image in pixel space.</p><p>In Stable Diffusion, they use a VAE that takes as input an image of 3 channels and produces a four-channel latent representation with a spatial reduction factor of 8 for each dimension. That means that if we use a 512x512 input image, the VAE compresses it to a 4x64x64 latent, i.e., from 3x512x512=786,432 values to 16,384 values, a reduction factor of 48.</p><p>There are many benefits to using smaller latent representations in the diffusion process, such as lower memory usage, fewer layers needed in the network, and faster generation. And the important thing is that we can still enjoy high-resolution generated images.</p><p>Now that we understand the core idea behind latent diffusion models, let’s break Stable Diffusion down into its components and explain each one of them. We will use the Hugging Face <em>diffusers</em> module to also see how to implement each component in code.</p><p>We are going to handle the code by creating a class that centralizes all the components.</p><pre>from PIL import Image<br>import numpy as np<br>import torch<br>from diffusers import StableDiffusionPipeline<br>from tqdm import tqdm<br><br><br>class LDM:<br>    def __init__(self, device, seed=42):<br>        torch.manual_seed(seed)<br>        self.device = device<br>        self.pipe = self.create_pipe(device)<br><br>    def create_pipe(self, device):<br>        return StableDiffusionPipeline.from_pretrained(<br>            &quot;stable-diffusion-v1-5/stable-diffusion-v1-5&quot;,<br>            torch_dtype=torch.float16,<br>            variant=&quot;fp16&quot;,<br>        ).to(device)</pre><p>The main component Hugging Face gives us to handle a diffusion model, and specifically Stable Diffusion in our case, is the <em>StableDiffusionPipeline</em>. This object contains all the necessary components and information (such as scaling factor, normalization, etc.). In fact, we don’t need anything else to generate images. We can use the following:</p><pre>pipe = StableDiffusionPipeline.from_pretrained(<br>            &quot;stable-diffusion-v1-5/stable-diffusion-v1-5&quot;,<br>            torch_dtype=torch.float16,<br>            variant=&quot;fp16&quot;,<br>        ).to(device)<br>pipe(&quot;A paint of a green oak tree&quot;).images[0]</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*yKawgW-hP7GJgOW0QJ-nVg.png" /></figure><h4>Text Encoder</h4><p>To control the generated image, we must add additional information to the diffusion process. In the <a href="https://medium.com/@dzdata/intro-to-diffusion-model-part-1-29fe7724c043">post</a> about classic diffusion, the process takes random noise and generates an image from the distribution it was trained on. To make Stable Diffusion generate an image according to a description, we add a text caption as additional information to the process. Instead of relying on the noisy image alone, the model conditions its prediction on the text caption, making the noise estimation significantly more accurate.</p><p>Since we can’t just use raw text in the network, the original Stable Diffusion uses <a href="https://medium.com/@dzdata/clip-contrastive-language-image-pre-training-d16f1b082c5a">CLIP</a> to get a text embedding of the caption by encoding the tokens that represent the caption text. The UNet gets all embedding vectors, not just the final combined representation vector, so it can use each token&#39;s information separately rather than only the text&#39;s overall meaning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OrIGJ9acMPd4sHpVQN77XA.png" /><figcaption>Overview of the inputs that condition the UNet’s prediction</figcaption></figure><p>The first method, therefore, would be to create tokens from text.</p><pre>    def make_tokens(self, prompt: str):<br>        # Tokenize the input<br>        inputs = self.pipe.tokenizer(<br>            prompt, padding=&quot;max_length&quot;, truncation=True, return_tensors=&quot;pt&quot;<br>        )<br><br>        # Tokenize an empty string as the unconditional input<br>        unconditional_inputs = self.pipe.tokenizer(<br>            &quot;&quot;, padding=&quot;max_length&quot;, truncation=True, return_tensors=&quot;pt&quot;<br>        )<br><br>        return inputs, unconditional_inputs</pre><p>The method takes a text prompt and converts it into tokens. You can see that we also get tokens for an empty prompt in <em>unconditional_input, </em>which will be explained later.</p><p>After we have the token representation, the next step is to create the embedding.</p><pre>def make_embedding(self, inputs, unconditional_inputs):<br>        with torch.inference_mode():<br>            conditional_output = self.pipe.text_encoder(<br>                input_ids=inputs.input_ids.to(self.device),<br>                return_dict=True,<br>            )<br><br>            conditional_embeddings = conditional_output.last_hidden_state<br><br>            unconditional_output = self.pipe.text_encoder(<br>                input_ids=unconditional_inputs.input_ids.to(self.device),<br>                return_dict=True,<br>            )<br><br>            unconditional_embeddings = unconditional_output.last_hidden_state<br><br>        text_embedding = torch.cat(<br>            [unconditional_embeddings, conditional_embeddings], dim=0<br>        )<br><br>        return text_embedding</pre><p>We use the text encoder (CLIP in this case) to get the embedding. Since the text embedding is the internal representation of the CLIP model, it’s referred to as the encoder hidden state, or, as in the code, the <em>last_hidden_state</em>.</p><h4>The Latent Space</h4><p>As explained earlier, in a latent diffusion model, we don’t work in pixel space, but in a latent space we get from a VAE. When we train the model, we use the VAE to convert the image from a pixel representation with shape (3, 512, 512) into a latent representation with shape (4, 64, 64). When we want to generate a new image, we start from Gaussian noise with shape (4, 64, 64), and after the denoising process finishes, we use the VAE to decode the clean latent representation back into an image in the pixel space.</p><p>So, to start the process, we have to get some random noise in the latent space.</p><pre>    def make_latents(self, height, width):<br>        latents = torch.randn(<br>            (1, self.pipe.unet.config.in_channels, height // 8, width // 8),<br>            dtype=torch.float16,<br>        ).to(self.device)<br><br>        latents = latents * self.pipe.scheduler.init_noise_sigma<br><br>        return latents</pre><p>The <em>init_noise_sigma</em> scales standard Gaussian noise to match the initial noise level required by the active scheduler (in the classic schedulers, it’s just 1).</p><h4>The Denoising Process</h4><p>Now that we have the latent noise, we can use the denoising process to get a real image latent representation.</p><pre>def predict(self, latents, text_embedding, guidance_scale):<br>        for t in tqdm(<br>            self.pipe.scheduler.timesteps, desc=&quot;Generating image&quot;, unit=&quot;steps&quot;<br>        ):<br>            latent_input = torch.cat([latents] * 2, dim=0)<br>            latent_input = self.pipe.scheduler.scale_model_input(latent_input, t)<br><br>            with torch.inference_mode():<br>                noise_prediction = self.pipe.unet(<br>                    latent_input, t, encoder_hidden_states=text_embedding<br>                ).sample<br><br>            noise_prediction_unconditional, noise_prediction_text = (<br>                noise_prediction.chunk(2, dim=0)<br>            )<br><br>            # Perform guidance<br>            noise_prediction = noise_prediction_unconditional + guidance_scale * (<br>                noise_prediction_text - noise_prediction_unconditional<br>            )<br><br>            latents = self.pipe.scheduler.step(noise_prediction, t, latents).prev_sample<br><br>        return latents</pre><p>We loop over the timesteps of the scheduler, where in each iteration we</p><ol><li>Take the latent representation and duplicate it on the first dimension</li><li>Scale the model input. scale_model_input adjusts the latents at every timestep to match the exact input scale the UNet expects to see, and it depends on the scheduler (in classic schedulers, it does nothing).</li><li>Run the latent input (with the current time step and the text embedding) through the UNet to predict the noise.</li><li>Recall that the text embedding contains two representations: one is the prompt that describes the image, and one is an empty prompt we call the <em>unconditional_embeddings</em>. We also duplicated our latent. That means our network made two predictions, one for the unconditional embedding and the other for the prompt embedding on the same latent representation. So we split the result into <em>noise_prediction_unconditional</em> and <em>noise_prediction_text</em>.</li><li>The next step is performing Classifier-Free Guidance, which we will explain immediately.</li><li>Finally, we use the predicted noise to get a little bit clearer latent representation of the final image.</li><li>And we continue with the new latent representation to the next iteration until we go through all the time steps and get the clean latent representation of a real image</li></ol><h4>Classifier-Free Guidance</h4><p>Recall that one of the main goals with this model is to condition the result on a prompt. However, it appears that the model tends to rely mostly on the noisy image rather than the prompt for making its predictions. One reason we can think of why that may happen is that, during training, the image contains much more information than a very general caption that describes it.</p><p>This can be a problem, since we want the result to follow the prompt. To direct the process better, we can use <em>guidance. </em>Guidance is a method to get more control of the process. For example, we can use CLIP on the result image to get its embedding, compare it to the original prompt we used, and measure similarity.</p><p>A better method is <em>classifier-free guidance</em> (CFG). The main idea is to combine the prompt-conditioned process with an unconditioned process (empty prompt). During training, we sometimes use an empty prompt to force the network to also learn to generate images without additional information. At inference time, as you already saw in the code above, we make two predictions: one uses the prompt we gave the network, and another one uses an empty prompt.</p><p>You can think about the noise prediction with a prompt as a combination of the noise prediction according to the prompt only and the noise prediction according to the image only, without any prompt. Since we also have the noise prediction according to the image only, we can, by subtracting, get the noise prediction according to the prompt: noise_prediction_text — noise_prediction_unconditional. Now we can add it back to the noise prediction of the image only, but with some scaling to emphasize the part of the text:</p><pre>noise_prediction = noise_prediction_unconditional + guidance_scale * (<br>                noise_prediction_text - noise_prediction_unconditional<br>            )</pre><p>The stronger the scale, the stronger the impact of the prompt on the result, and the final image follows the prompt better. However, applying a guidance scale that is too high pushes the latents outside the model’s learned distribution, resulting in color clipping, blown-out contrast, and unnatural visual artifacts.</p><h4>Decoding with VAE</h4><p>At the end of the diffusion process, we get the clean image. But in a latent diffusion, it’s not really an image, but some representation of it in the latent space of the VAE. So to get the actual image, we have to use the VAE to decode the result.</p><pre>    def decode(self, prediction):<br>        # Use VAE to decode the latents into an image<br>        prediction_scaled = 1 / self.pipe.vae.config.scaling_factor * prediction<br><br>        with torch.inference_mode():<br>            image = self.pipe.vae.decode(prediction_scaled).sample<br><br>        return image</pre><p>We first scale the prediction with a scaling factor used in the training process to get it back to the correct latent representation of the VAE. The reason is that when a standard VAE encodes an image into a latent space, the resulting latent vectors do not naturally have a variance of 1. For example, in Stable Diffusion v1.5, the raw latents produced by the VAE encoder have a standard deviation around 1/0.18215 = 5.49.</p><p>Most diffusion models operate under the strict assumption that their input starts as standard Gaussian noise with a mean of 0 and a variance of 1. That’s why we use the scaling factor to bring the VAE encoding to variance 1 in the training process, and use it at inference to move the result back to the original statistics.</p><p>After the scaling, we decode the result with the VAE to get the image.</p><h4>Helper Function for Visualization</h4><p>The image we get from the VAE is in the range [-1, 1]. For visualization, we want to move it to the range of [0, 1]:</p><pre>def normalize_image(self, image):<br>      return (image / 2 + 0.5).clamp(0, 1)</pre><p>and we also want to convert Pytorch tensor to numpy and numpy to PIL</p><pre>@staticmethod<br>def convert_numpy_to_pil(image: np.ndarray) -&gt; Image:<br>    return Image.fromarray((image * 255).astype(&quot;uint8&quot;))<br><br>@staticmethod<br>def convert_tensor_to_numpy(tensor: torch.Tensor) -&gt; np.ndarray:<br>    tensor = torch.squeeze(tensor)<br>    image = tensor.detach().cpu().float().numpy()<br>    image = image.transpose(1, 2, 0)<br>    return image</pre><h3>Gradio</h3><p>It is much nicer to generate images and control the parameters with a web interface rather than in code. Gradio is a library that can help us to build a convenient UI with minimal code.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LOvtJb0DkYMEAZ3LN1pBWA.png" /><figcaption>Gradio interface example</figcaption></figure><p>The code for the Gradio app is very simple.</p><pre>import gradio as gr<br><br>prompt_input = gr.Textbox(label=&quot;Prompt&quot;, lines=2)<br><br>height_input = gr.Slider(<br>    minimum=256,<br>    maximum=1024,<br>    step=64,<br>    value=512,<br>    label=&quot;Height (pixels)&quot;,<br>    info=&quot;Image height in pixels&quot;,<br>)<br><br>width_input = gr.Slider(<br>    minimum=256,<br>    maximum=1024,<br>    step=64,<br>    value=512,<br>    label=&quot;Width (pixels)&quot;,<br>    info=&quot;Image width in pixels&quot;,<br>)<br><br>cfg_input = gr.Slider(<br>    minimum=1.0,<br>    maximum=20.0,<br>    step=0.5,<br>    value=7.5,<br>    label=&quot;CFG Scale&quot;,<br>    info=&quot;Classifier-Free Guidance scale&quot;,<br>)<br><br>steps_input = gr.Slider(<br>    minimum=1,<br>    maximum=150,<br>    step=1,<br>    value=50,<br>    label=&quot;Inference steps&quot;,<br>    info=&quot;Number of denoising steps&quot;,<br>)<br><br>output_image = gr.Image(label=&quot;Output Image&quot;, type=&quot;pil&quot;, format=&quot;png&quot;)<br><br><br>def make_gradio_interface(function: callable):<br>    return gr.Interface(<br>        fn=function,<br>        inputs=[prompt_input, height_input, width_input, steps_input, cfg_input],<br>        outputs=[output_image],<br>        title=&quot;Image Generation Interface&quot;,<br>        description=&quot;Generate images based on the given prompt and parameters.&quot;,<br>    )</pre><p>We chose the components we want to use, like Textbox, Slider, and Image, set some properties for each of them, and used the <em>Interface</em> function to generate the interface. The <em>Interface </em>function takes as input a function that uses the inputs and returns something that the output element expects to get (a PIL image in our case). See how simple it is!</p><h3>Run the App</h3><p>We have all the components we need and we can make our main script to put it all together.</p><pre>import torch<br><br>from app import make_gradio_interface<br>from models import LDM<br><br>device = torch.device(<br>    &quot;mps&quot;<br>    if torch.backends.mps.is_available()<br>    else (&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br>)<br><br>ldm = LDM(device)<br><br><br>def make_image(prompt, height, width, num_inference_steps, guidance_scale):<br>    tensor_image = ldm(prompt, height, width, num_inference_steps, guidance_scale)<br>    numpy_image = LDM.convert_tensor_to_numpy(tensor_image)<br>    pil_image = LDM.convert_numpy_to_pil(numpy_image)<br>    return pil_image<br><br><br>app = make_gradio_interface(make_image)<br><br>if __name__ == &quot;__main__&quot;:<br>    print(f&quot;Using device: {device}&quot;)<br>    app.launch()</pre><p>We start by selecting the device we are going to use (MPS for Apple Silicon, CUDA for Nvidia GPU, or CPU if we don’t have an accelerator). We create the LDM object and define a function for the Gradio interface that runs the latent diffusion process and returns a tensor that is converted to a PIL image. This PIL image will show as the result in the interface. app.launch() will start the Gradio app, usually at the address <a href="http://127.0.0.1:7860/">http://127.0.0.1:7860</a></p><h3>Summary</h3><p>In this post, we take the principles of diffusion models a step further by diving into Stable Diffusion and Latent Diffusion Models (LDMs). We examine how shifting the denoising process from high-dimensional pixel space into a lower-dimensional latent space using a VAE drastically reduces computational cost while preserving high-resolution outputs. We then break down each essential component — including CLIP text embeddings, UNet noise estimation, Classifier-Free Guidance (CFG), and VAE decoding — implementing them step-by-step using Hugging Face diffusers before wrapping everything into an interactive Gradio web app.</p><h3>Code Example</h3><p>The full code example can be found at this <a href="https://github.com/dorzv/ComputerVision/tree/master/latent_diffusion">link</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d9b061cc18f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CLIP — Contrastive Language-Image Pre-training]]></title>
            <link>https://dzdata.medium.com/clip-contrastive-language-image-pre-training-d16f1b082c5a?source=rss-aaf4831d9b34------2</link>
            <guid isPermaLink="false">https://medium.com/p/d16f1b082c5a</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Dor Zvulun]]></dc:creator>
            <pubDate>Tue, 04 Aug 2026 16:01:03 GMT</pubDate>
            <atom:updated>2026-08-04T16:01:03.140Z</atom:updated>
            <content:encoded><![CDATA[<h4>Breaking the Single-Modality Barrier</h4><p>In the world of modern AI, computer vision has long been treated as a solitary field. For years, the standard approach to understanding images was to analyze them in a vacuum — either by classifying pixels into rigid, predefined categories or trying to reconstruct them using architectures like Autoencoders and VAEs. But the human experience isn’t unimodal; we don’t just look at the world, we can also describe it.</p><p>This is where <strong>Contrastive Language-Image Pre-training (CLIP)</strong> changes the game. By bridging the gap between vision and language, CLIP changes the traditional analysis of images and explores a powerful approach: multimodal learning. Instead of training a model to analyze images in isolation, CLIP learns to connect the dots between two entirely different worlds — images and the natural language captions that describe them. By training on massive amounts of paired data, its mission is to take any arbitrary, never-before-seen image-text pair and accurately measure how well they align. The secret to unlocking this flexible, cross-modal understanding? It all comes down to a brilliantly designed loss function.</p><h3>The Core Intuition: What is Contrastive Learning?</h3><p>Contrastive learning is a technique where a model learns to distinguish between similar and dissimilar data points. The core idea is to map similar samples (positives) closer together in the feature space while pushing dissimilar samples (negatives) farther apart. There are many cases where contrastive learning and contrastive loss can be used. For example, it is used extensively across self-supervised learning, with prominent examples including methods like <a href="https://python.plainenglish.io/self-supervised-learning-for-visual-representation-b242bde77241">SimCLR</a> or <a href="https://dzdata.medium.com/paper-review-bootstrap-your-own-latent-byol-545c6f7acee3">BYOL</a>.</p><p>In CLIP, the main goal is to map an image directly to its descriptive text. For each batch of data, the model aims to produce similar representations for an image and its associated caption, while keeping that representation distant from the captions of all other images in the batch.</p><p>Through this approach, the model learns to connect visual information with textual information. By operating across two distinct modalities at once, CLIP achieved a major milestone in representation learning.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AMsXU-6wGOvNniFEGFTFNg.png" /><figcaption>Similar images and captions should be close together, while different concepts should be kept apart</figcaption></figure><h3>The Architecture: Two Encoders, One Shared Space</h3><p>Images and text belong to two distinct modalities — meaning they are fundamentally different information structures. If our goal is to relate an image to a text caption, we must first bring them into a shared space where they can be directly compared and measured for similarity.</p><p>To accomplish this, CLIP uses two dedicated encoders to extract feature representations from each domain: a ResNet or Vision Transformer (ViT) for images, and a standard Transformer for text. Ultimately, these two encoders map both distinct data types into fixed-size feature vectors within the exact same embedding dimension, enabling direct mathematical comparison.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OYHlXbP0QtA_9BaNjmm8_w.png" /><figcaption>Images and text pairs are processed by two types of encoders to produce a single vector representation that can be compared. Source:<a href="https://arxiv.org/pdf/2103.00020"> Learning Transferable Visual Models From Natural Language Supervision</a></figcaption></figure><h3>Multi-Modal Contrastive Loss</h3><p>Once we have these vector representations, the next step is determining how to compare them and how to use that comparison for training. The central intuition is straightforward: for a true <em>(image, text)</em> pair, the two feature vectors should be highly similar, whereas vectors from unmatched pairs should be distinct.</p><p>A standard way to measure similarity between two vectors is <strong>cosine similarity</strong>, which is the normalized dot product of the vectors.</p><p>Given a batch of <em>N (image, text)</em> pairs, the encoders produce <em>N </em>image embedding vectors and <em>N</em> text embedding vectors. Pairwise evaluation yields an <em>N×N</em> similarity matrix containing <em>N² </em>similarity scores. The <em>N</em> diagonal entries correspond to the true matching <em>(image, text)</em> pairs and should have high similarity scores, while the remaining <em>N² — N</em> off-diagonal entries represent unmatched pairs and should have low scores.</p><p>To maximize diagonal scores while minimizing off-diagonal scores, CLIP uses a symmetric cross-entropy loss. This involves computing standard cross-entropy loss on the similarity matrix in both directions and averaging the two terms:</p><p><strong>Image-to-Text Loss (L_I):</strong> For each image, predict which of the <em>N</em> text captions matches it (Given Image <em>i</em>, how well can the model pick out Text <em>i</em> from all <em>N</em> texts?). For each row <em>i</em>, a <strong>Softmax </strong>function is applied across the columns to turn raw similarity scores into a probability distribution over the text captions. Categorical cross-entropy is then calculated against the ground-truth diagonal index:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*huImt_Aa1qaHdrYssDR-2w.png" /><figcaption>The cross-entropy loss over the similarity between each image (i) and all text captions (j)</figcaption></figure><p><strong>Text-to-Image Loss (L_T):</strong> For each text caption, predict which of the <em>N</em> images matches it. (Given Text <em>j</em>, how well can the model pick out Image <em>j</em> from all <em>N</em> texts?). For each column <em>j</em>, a <strong>Softmax</strong> function is applied across the rows to form a probability distribution over the images, followed by categorical cross-entropy against the ground truth:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nxDH2JI3Jqpl9Icfv4bRAA.png" /><figcaption>The cross-entropy loss over the similarity between each Text (<em>j</em>) and all Images (<em>i</em>)</figcaption></figure><p>The total loss is simply the average of the two directional losses:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pMwJDSe8W04Xt7cAzezDUg.png" /></figure><p>Why use a symmetric cross-entropy loss? If the model optimized only row-wise (L_I), it would learn to map images to text effectively, but the text encoder wouldn’t be explicitly penalized if multiple texts collapsed onto the same image region in representation space (and vice versa).</p><p>By enforcing symmetry:</p><ul><li>The <strong>image encoder</strong> is forced to produce distinct, separable features for different images.</li><li>The <strong>text encoder</strong> is simultaneously forced to produce distinct, separable features for different captions.</li><li>Neither modality is allowed to dominate the shared embedding space.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jdQhUU6GMsns8iyaThZOjw.png" /><figcaption>Similarly, matrix. Values on the main diagonal should be maximized, while the off-diagonal ones should be minimized</figcaption></figure><p>It is worth noting that the actual similarity score Sᵢⱼ is not just the cosine similarity, but it is also scaled by a learnable temperature parameter τ:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a6oebi1ZyKqCcKLKgUsbqA.png" /></figure><h3>Zero-Shot Classification</h3><p>Classification is one of the foundational tasks in machine learning, particularly in computer vision. In standard supervised classification, given a data point and a set of predefined classes, the model outputs a probability distribution across those classes. However, those classes must be defined before training, restricting the model to categorizing data strictly within those labels.</p><p>A model capable of zero-shot classification, however, can categorize data into new classes without ever being explicitly trained on them. Because CLIP is trained to match images with descriptive text, we can construct a set of text prompts (e.g., “a photo of a dog” vs. “a photo of a cat”) and evaluate the similarity between an input image and each prompt. This turns classification into a dynamic search across arbitrary text labels. Even without dedicated training on a specific dataset or class list, CLIP can classify images flexibly on the fly.</p><p>In <a href="https://openai.com/index/clip/">OpenAI’s original blog post</a>, they show that CLIP can achive performance similar to models that were trained for ImageNet, without using ImageNet for training at all. Furthermore, you can see in the figure below that CLIP keeps very high performance even when tested on other datasets, while the network trained for ImageNet shows degradation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qYpVqxyxbGyGFT_GaOJO_w.png" /><figcaption>source: <a href="https://openai.com/index/clip/">OpenAI</a></figcaption></figure><h3>More Uses</h3><p>Beyond zero-shot classification, CLIP serves as a powerful foundational block for several downstream applications:</p><ul><li><strong>Semantic search: </strong>Because CLIP maps text and images into the same vector space, an image archive can be indexed by pre-computing CLIP embeddings for every photo. Users can then search the archive using natural language queries — CLIP converts the query text into an embedding vector and retrieves the closest matching photo vectors.</li><li><strong>Rich Visual-Textual Embeddings:</strong> Trained on hundreds of millions of image-caption pairs, CLIP learns rich, generalized representations. These embeddings are widely used to guide downstream tasks, such as conditioning text-to-image generative models (e.g., Stable Diffusion and DALL-E) on natural language prompts.</li></ul><h3>Conclusion</h3><p>CLIP marked a fundamental paradigm shift in representation learning. Instead of forcing models to analyze images in a vacuum or memorize rigid category labels, it proved that connecting vision directly to language makes models far more flexible and adaptable. This shift allows AI to tackle brand-new tasks on the fly without any extra training. Today, teaching models to understand text and images together isn’t just a clever alternative — it’s quickly becoming the standard way we build AI that actually understands the world around it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d16f1b082c5a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrating from Pandas to Polars: A Comprehensive Guide]]></title>
            <link>https://dzdata.medium.com/migrating-from-pandas-to-polars-a-comprehensive-guide-97e91dfa42b0?source=rss-aaf4831d9b34------2</link>
            <guid isPermaLink="false">https://medium.com/p/97e91dfa42b0</guid>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Dor Zvulun]]></dc:creator>
            <pubDate>Sat, 29 Mar 2025 09:12:37 GMT</pubDate>
            <atom:updated>2025-03-29T09:12:37.508Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5jv684HngXYePUs7ENlHng.png" /></figure><h3>Introduction</h3><p>Pandas has been the go-to library for data manipulation in Python for over a decade. It is powerful, flexible, and widely used in data science, machine learning, and analytics. However, as datasets grow in size, Pandas often struggles with performance due to its reliance on single-threaded execution and memory-intensive operations.</p><p>Polars, a relatively new DataFrame library, aims to address these limitations. Built using Rust, Polars is designed for speed and efficiency. It offers multi-threaded execution, a powerful query engine, and memory-efficient operations, making it an excellent alternative to Pandas for large-scale data processing.</p><p>In this post, we will explore the key advantages of Polars over Pandas and demonstrate how to migrate common Pandas operations to Polars.</p><h3>Why Polars?</h3><h4>1. Performance</h4><p>Polars leverages Rust and Arrow memory format to enable lightning-fast computations. It is optimized for multi-threading, allowing it to outperform Pandas significantly on large datasets.</p><h4>2. Memory Efficiency</h4><p>Unlike Pandas, which often requires large amounts of memory, Polars employs zero-copy operations and efficient memory layouts, reducing RAM usage considerably.</p><h4>3. Lazy Execution</h4><p>Polars supports lazy execution, which allows it to optimize query plans before execution, making it better for complex operations on big data.</p><h4>4. Better API Design</h4><p>Polars enforces stricter data types and provides a more functional API, reducing common issues like unintended type conversions and silent data corruption.</p><p>Now, let’s dive into how to migrate common Pandas operations to Polars.</p><h3>Migrating from Pandas to Polars</h3><h4>1. Installation</h4><p>Before we start, install Polars using:</p><pre>pip install polars</pre><p>Then, import the libraries:</p><pre>import pandas as pd<br>import polars as pl</pre><h4>2. Creating DataFrames</h4><p><strong>Pandas</strong></p><pre>df_pd = pd.DataFrame({<br>    &quot;name&quot;: [&quot;Alice&quot;, &quot;Bob&quot;, &quot;Charlie&quot;],<br>    &quot;age&quot;: [25, 30, 35],<br>    &quot;salary&quot;: [50000, 60000, 70000]<br>})</pre><p><strong>Polars</strong></p><pre>df_pl = pl.DataFrame({<br>    &quot;name&quot;: [&quot;Alice&quot;, &quot;Bob&quot;, &quot;Charlie&quot;],<br>    &quot;age&quot;: [25, 30, 35],<br>    &quot;salary&quot;: [50000, 60000, 70000]<br>})</pre><h4>3. Data Inspection</h4><p><strong>Pandas</strong></p><pre>df_pd.info()<br>df_pd.describe()</pre><p><strong>Polars</strong></p><pre>df_pl.schema  # Shows data types<br>df_pl.describe()</pre><p>Polars’ .schema replaces .info() and .describe() provides statistical summaries.</p><h4>4. Selecting Columns</h4><p><strong>Pandas</strong></p><pre>df_pd[&quot;name&quot;]<br>df_pd[[&quot;name&quot;, &quot;salary&quot;]]</pre><p><strong>Polars</strong></p><pre>df_pl[&quot;name&quot;]  # Series output<br>df_pl.select([&quot;name&quot;, &quot;salary&quot;])  # DataFrame output</pre><p>In Polars, .select() is preferred for multi-column selection.</p><h4>5. Filtering Rows</h4><p><strong>Pandas</strong></p><pre>df_pd[df_pd[&quot;age&quot;] &gt; 28]</pre><p><strong>Polars</strong></p><pre>df_pl.filter(pl.col(&quot;age&quot;) &gt; 28)</pre><p>Polars uses expressions like pl.col(&quot;age&quot;) for filtering.</p><h4>6. Adding New Columns</h4><p><strong>Pandas</strong></p><pre>df_pd[&quot;bonus&quot;] = df_pd[&quot;salary&quot;] * 0.1</pre><p><strong>Polars</strong></p><pre>df_pl = df_pl.with_columns((pl.col(&quot;salary&quot;) * 0.1).alias(&quot;bonus&quot;))</pre><p>Polars uses .with_columns() to add new columns.</p><h4>7. Applying Functions</h4><p>Applying the following function on the salary column.</p><pre>def categorize_salary(salary):<br>    if salary &gt; 60000:<br>        return &quot;High&quot;<br>    elif salary &gt; 50000:<br>        return &quot;Medium&quot;<br>    else:<br>        return &quot;Low&quot;</pre><p><strong>Pandas</strong></p><pre><br>df_pd[&quot;salary_category&quot;] = df_pd[&quot;salary&quot;].apply(categorize_salary)</pre><p><strong>Polars</strong></p><pre>df_pl = df_pl.with_columns(<br>    pl.when(pl.col(&quot;salary&quot;) &gt; 60000)<br>    .then(pl.lit(&quot;High&quot;))<br>    .when(pl.col(&quot;salary&quot;) &gt; 50000)<br>    .then(pl.lit(&quot;Medium&quot;))<br>    .otherwise(pl.lit(&quot;Low&quot;))<br>    .alias(&quot;salary_category&quot;)<br>)</pre><p>Polars avoids .apply(), whenever possible, replacing it with efficient vectorized expressions like when().then().otherwise().</p><h4>8. Grouping and Aggregations</h4><p><strong>Pandas</strong></p><pre>df_pd.groupby(&quot;age&quot;)[&quot;salary&quot;].mean()</pre><p><strong>Polars</strong></p><pre>df_pl.group_by(&quot;age&quot;).agg(pl.col(&quot;salary&quot;).mean())</pre><p>Polars’ .agg() method explicitly defines aggregations.</p><h4>9. Sorting</h4><p><strong>Pandas</strong></p><pre>df_pd.sort_values(by=&quot;salary&quot;, ascending=False)</pre><p><strong>Polars</strong></p><pre>df_pl.sort(&quot;salary&quot;, descending=True)</pre><h4>10. Merging DataFrames</h4><p><strong>Pandas</strong></p><pre>df2_pd = pd.DataFrame({&quot;name&quot;: [&quot;Alice&quot;, &quot;Bob&quot;], &quot;department&quot;: [&quot;HR&quot;, &quot;Engineering&quot;]})<br>df_merged = df_pd.merge(df2_pd, on=&quot;name&quot;, how=&quot;left&quot;)</pre><p><strong>Polars</strong></p><pre>df2_pl = pl.DataFrame({&quot;name&quot;: [&quot;Alice&quot;, &quot;Bob&quot;], &quot;department&quot;: [&quot;HR&quot;, &quot;Engineering&quot;]})<br>df_merged = df_pl.join(df2_pl, on=&quot;name&quot;, how=&quot;left&quot;)</pre><p>Polars replaces .merge() with .join().</p><h4>11. Handling Missing Values</h4><p><strong>Pandas</strong></p><pre>df_pd.fillna(0)<br>df_pd.dropna()</pre><p><strong>Polars</strong></p><pre>df_pl.fill_null(0)<br>df_pl.drop_nulls()</pre><p>Polars uses .fill_null() instead of .fillna() and drop_nulls() instead of dropna() .</p><h4>12. Lazy Execution for Big Data</h4><p>One of Polars’ biggest advantages is <strong>lazy execution</strong>, which optimizes query plans before execution.</p><pre>lazy_df = df_pl.lazy()<br>lazy_df = lazy_df.filter(pl.col(&quot;salary&quot;) &gt; 55000)<br>lazy_df.collect()  # Executes the query</pre><p>Lazy execution helps process large datasets efficiently. Lazy operations in Polars <strong>are not in place</strong>,<strong> </strong>so you must assign the result to a new variable or overwrite the original one because .collect() returns a <strong>new DataFrame</strong> and does not modify the original LazyFrame.</p><h3>Conclusion</h3><p>Migrating from Pandas to Polars provides significant performance and efficiency gains, especially for large datasets. With multi-threading, better memory management, and an optimized query engine, Polars is a strong alternative for data manipulation.</p><h4>Key Takeaways</h4><ul><li><strong>Polars is much faster</strong> due to multi-threading and Rust optimizations.</li><li><strong>Memory usage is lower</strong>, making it suitable for big data.</li><li><strong>Stronger type enforcement</strong> reduces unexpected bugs.</li><li><strong>Lazy execution</strong> optimizes queries before execution.</li></ul><p>While Pandas remains a powerful tool for data analysis, Polars is a compelling choice for those seeking better performance. If you’re working with large-scale data, Polars might be the better solution.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=97e91dfa42b0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DETR: End-to-End Object Detection with Transformers]]></title>
            <link>https://dzdata.medium.com/detr-end-to-end-object-detection-with-transformers-f40ce77bfe44?source=rss-aaf4831d9b34------2</link>
            <guid isPermaLink="false">https://medium.com/p/f40ce77bfe44</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Dor Zvulun]]></dc:creator>
            <pubDate>Fri, 28 Feb 2025 17:07:00 GMT</pubDate>
            <atom:updated>2025-02-28T17:07:00.681Z</atom:updated>
            <content:encoded><![CDATA[<h4>The Future of Object Detection with Transformers</h4><h3>Object Detection</h3><p>The task of object detection is composed of two main elements: classification of the object into a category (for example, a dog or a cat), and localization of the object in the image. The localization part is the heart of the object detection task, which differentiates it from the regular classification task.</p><p>The outcome of object detection is usually a bounding box that marks the object with an assigned class. In the following image, you can see an example of an image with two objects, a dog and a cat, each with its bounding box. The object’s bounding box is represented with a vector containing the class and the properties of the box (for example, the <em>x</em> and <em>y </em>coordinates of the top left corner and the height and width of the box).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZHLpGZhXVsOLiOcgd5EGAA.png" /></figure><h3>How to Find the Box?</h3><p>There are several methods to find the bounding box.</p><h4>Sliding Windows</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/236/0*fzGErC2A2TpHz400.gif" /><figcaption><a href="https://medium.com/@rajshekhar_k/object-detection-series-3-0-basic-of-object-detection-196d91550671">source</a></figcaption></figure><p>In this approach, we use a window that captures a portion of the image and then perform a classification on this portion to see which (if any) object is in there. Basically, we reduce the problem to many classification tasks. However, this method has many disadvantages. First, it is very hard to pick the size and stride of the window, and the result may include a huge amount of candidates. If we want to check more candidates we will need more calculations. And even then there is a good chance we dismiss objects or get a loose bounding box on the object.</p><h4>Region Proposals</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6HoDS2eV6q3PKcBaUvW5LA.png" /><figcaption><a href="https://arxiv.org/pdf/1311.2524">source</a></figcaption></figure><p>The region proposals method is used by R-CNN. Instead of using predefined windows that slide over the image, we first use an algorithm to extract region proposals and we run a classification network only on these regions. The number of candidates in this method should be significantly lower than the number in the sliding window approach. However, it is still a slow method since we need to run the classification network many times, and it depends on the quality of the region proposals algorithm.</p><h4>Anchors</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*elidg650ExjkBn8ZZbDR2A.png" /><figcaption><a href="https://arxiv.org/pdf/1512.02325">source</a></figcaption></figure><p>The anchor approach is used by some of the famous object detection architectures like Faster R-CNN, YOLO, and SSD. In this method, we pre-define boxes to capture objects. We divide the feature maps by a grid, and each grid element gets a set of boxes with different sizes and aspect ratios. If this element thinks the object is mainly inside it, it should assign it to the most suitable box, and make some small corrections to the output bounding box relative to the anchor box.</p><p>The anchor boxes method lets the process be in one shot and makes the end-to-end prediction much faster. However, there are still some problems. First, the performance is dependent on the design of the anchor sets. Second, many grid elements may think the object belongs to them, and we need a post-process step to collapse near-duplicate predictions (like non-maximum suppression). We may also need heuristics that assign target boxes to anchors.</p><p>In general, we can say that in all previous methods, the detection task is performed indirectly by defining a surrogate regression and classification problems. In the next section, we will discuss DETR, and see how it can handle the problem directly and overcome the issues that arise from the previous methods.</p><h3>DEtection TRansformer (DETR)</h3><p>DETR presents a CNN features extraction + encoder-decoder transformer architecture. It predicts all objects at once and drops multiple hand-designed components that encode prior knowledge, like spatial anchors or non-maximal suppression.</p><h4>The Architecture</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QrLL4d-uVtZ3ohOwX1TqIg.png" /></figure><p>In the first stage, the image is processed by a Convolution Neural Network that extracts features from it. The output is a set of feature vectors. Next, we use a transformer-based encoder and decoder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*N17YsYqkBdGNcM-H6W8I-Q.png" /></figure><p>The vectors are processed in a self-attention-based encoder, after adding to them positional encoding since the self-attention is invariant to permutations and we want to preserve the location information of the different parts in the image. If you are not familiar with these terms you can read my <a href="https://medium.com/@dzdata/from-attention-to-self-attention-and-transformers-a-mathematical-introduction-45ed053a1ed3">post about self-attention</a>. We stack <em>N</em> encoder blocks together and save the result vectors for the next step.</p><p>The encoding process looks at the relations between the different feature vectors and helps to disentangle objects.</p><p>In the next image, you can see the attention maps of the last encoder layer of a trained model. It seems the encoder indeed separates instances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MvNW7uVd6Vzl36ch_HOjvw.png" /></figure><p>In the next step, we take the output of the encoder and use cross-attention between them and <em>object queries</em>. The object queries are learnable vectors used as the query input for the decoder. Each object query is responsible for detecting an object in the image, and the number of object queries corresponds to the maximum number of objects the model can detect. The decoder gets all the queries together, meaning the model predicts all objects in a single shot.</p><p>In the next figure, you can see the position of objects associated with specific object queries. As you can see the different queries focus on different areas in the image. For example, the first query mainly looks for objects in the bottom left corner of the image, while the last one mainly focuses on objects in the top left corner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*itqIYjKEifnSZlgxPRjN1g.png" /></figure><p>For each object query, we get an output. Each of the outputs processes then in a Feed Forward Network which predicts if there is an object related to this output and if so where it is.</p><h4>The Loss</h4><p>DETR is a direct set predictions detector, meaning it predicts a set of <em>N</em> boxes each time. Since we don’t want to dismiss objects, we set <em>N</em> to be a much larger number than the typical number of objects in an image. Such detectors need a loss that forces unique matching between predicted and ground truth boxes. Because of that, DETR’s loss first produces an optimal bipartite matching between predicted and ground truth objects and then optimizes object-specific (bounding box) losses.</p><p>Let us denote y as the ground truth set of objects and ŷ as the set of <em>N</em> predictions. Since N ≫ GT (Ground Truth), we pad y with <em>no object</em> (∅) so y is also a set of size <em>N. </em>We want to find a bipartite matching between the two sets by searching for a permutation of <em>N</em> elements, σ, with the lowest cost.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JH1yvmr736YRQnrsjtezLQ.png" /></figure><p>The permutation σ is rearranging the order of the prediction so each prediction will be matched to its closest grout truth. This optimal assignment is computed efficiently with the Hungarian algorithm which is beyond the scope of this post.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bxTLpxUbTW8iTV7ttBir6w.png" /></figure><p>The loss function is a pair-wise matching cost between ground truth yᵢ=(cᵢ,bᵢ) (class and bounding box) and a prediction with index σ(i).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mBFLfwFdYXXwqzX_DZ-SHA.png" /></figure><p>where p̂(cᵢ) is the probability of the prediction to be the class cᵢ, and the other loss is a loss between the bounding boxes.</p><p>After finding the optimal matching between the two sets we can calculate the loss itself.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SIct3cFNhFDoC01jdsm5IQ.png" /></figure><p>where σ̂ is the optimal assignment we found in the previous step. We left to talk about the box loss. The box prediction is not with respect to some initial guesses but directly predicted, so we calculate the loss relative to the ground truth box. The loss itself is composed of generalized IoU (intersection over union) loss and L₁ loss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0mi682b1ddzcQuSaxwQfVg.png" /></figure><p>where the generalized IoU is given by</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p6A1p9YcofRth-r-llK7qA.png" /></figure><p>where B(bᵢ,b̂_σ(i)) is the box containing bᵢ,b̂_σ(i). As opposed to the regular IoU, the generalized IoU also considers how far the true box and the prediction box are from each other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mxcXaTZhQm1UW31SyLcgTQ.png" /></figure><h3>Bonus: DETR for Panoptics segmentation</h3><p>DETR is created for object detection tasks, but the author of the paper showed that they can add a segmentation head after the original architecture and get a panoptic segmentation model. Panoptics means it can set for each pixel in the image a class (as in semantic segmentation task), but also can separate different objects of the same class (as in instance segmentation).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1_8zYhtgmsjd7v_wZlqCbQ.png" /><figcaption>The additional head for panoptic segmentation</figcaption></figure><p>Here are some results of the panoptic segmentation of DETR.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xH0LKG2jf_pyIelyvz2oag.png" /></figure><h3>Conclusion</h3><p>DETR brings a new approach to object detection by leveraging a transformer-based architecture that enables end-to-end training, eliminating the need for pre-designed anchors and traditional post-processing steps like Non-Maximum Suppression (NMS). Its ability to perform parallel decoding allows for more efficient and simultaneous predictions, streamlining the detection process. Additionally, its flexibility makes it a powerful model that can be extended to other tasks, such as panoptic segmentation, showcasing its versatility. With these innovations, DETR paves the way for more streamlined, scalable, and efficient approaches in computer vision.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f40ce77bfe44" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Different Types of Normalizations in Deep Learning]]></title>
            <link>https://dzdata.medium.com/the-different-types-of-normalizations-in-deep-learning-03eece7fa789?source=rss-aaf4831d9b34------2</link>
            <guid isPermaLink="false">https://medium.com/p/03eece7fa789</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dor Zvulun]]></dc:creator>
            <pubDate>Fri, 31 Jan 2025 10:52:51 GMT</pubDate>
            <atom:updated>2025-01-31T10:52:51.565Z</atom:updated>
            <content:encoded><![CDATA[<h4>Exploring the Types of Normalization in Deep Learning and How They Work</h4><p>Normalization is a key part of deep learning that helps models train more effectively and perform better. It works by keeping data or calculations in a consistent range, making it easier for the model to learn patterns. There are several types of normalization, like batch normalization and layer normalization, each with its own purpose. In this blog, we’ll look at these methods, how they work, and why they are useful in deep learning.</p><h3>Why Use Normalization</h3><p>Normalization helps improve the efficiency, stability, and accuracy of training models. It is much easier for the model to find a good solution when its feature space is in scale. It’s best to understand this concept with visualization.</p><p>Let’s assume we have a simple linear regression task with two features. One feature, x1, comes from a normal distribution with a mean of 5 and a standard division of 5, while the other one, x2, comes from another normal distribution with parameters 0.5 and 0.5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/674/1*zOtbZR9u0QyoljJC_SCdqA.png" /><figcaption>Normal distribuation with mean and std of 5 (blue) and 0.5 (orange)</figcaption></figure><p>You can see the two features have very different scales. Now we assume these features relate to some target y, according to the formula:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/183/0*lZFhWyeLbC_SDr8x.png" /></figure><p>with A and B some coefficients.</p><p>We want to find the parameters <em>a</em> and <em>b </em>that give us the best estimation for <em>y</em>. We can use linear regression to do it by training a model to predict y, and update <em>a</em> and <em>b </em>parameters to minimize a loss function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/285/0*ckVfxeJPTkVfjeFr.png" /></figure><p>Now we can look at the loss function <em>J</em> as a function of <em>a</em> and <em>b.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*V2V3P2c2FO1ilANjZihOWQ.png" /><figcaption>A contour plot of the loss as function of a and b.</figcaption></figure><p>We can see the loss function is very skewed. The gradients in the direction of <em>a </em>are much steeper than those in the direction of <em>b</em>.</p><p>This can make the training slower and unstable. Let’s see what will happen if we normalize (or more precisely, standardize) the features before using them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/172/0*bzJ8SumTIlUgOMW-.png" /></figure><p>where μ is the mean and σ is the standard deviation. This transformation changes the distribution of the features to have a zero mean and standard deviation of one.</p><p>Now the loss is much uniformer in both directions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/711/1*-6CUdovf4b-o_5dGnExJNA.png" /></figure><p>which means the gradient directions are more uniform and the training will be faster and stabilizer.</p><h3>Normalization in Deep Learning</h3><p>The previous example was a classical machine-learning problem where we have input, one formula with parameters, and output. It was sufficient to normalize the input features. In deep learning, however, we have many layers, where each layer&#39;s output is the input to the next layer. In such a case we would like to normalize the features after (almost) each layer.</p><p>In the machine learning case, we have all the input features in advance, and we can normalize them before the training. In the deep learning case, we don’t know the feature distribution of each layer before the training due to the weights update. This means the next layer may see a different distribution in each step. So we need some new methods to handle the normalization.</p><p>In the following sections, we explore popular normalization techniques and see code implementation for them.</p><h3>Batch Normalization</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*BeRdzOsXQba-bFKb.png" /><figcaption>Feature map tensor, with N as the batch axis, C as the channel axis, and (H, W) as the spatial axes. The pixels in blue are normalized by the same mean and variance, computed by aggregating the values of these pixels. <a href="https://arxiv.org/pdf/1803.08494">Source</a></figcaption></figure><p>Batch Normalization is a normalization that normalizes the inputs to each layer within a mini-batch. For each mini-batch of data, Batch Normalization computes the mean (μ) and variance (σ²) of the inputs. These values are used to normalize the inputs by subtracting the mean and dividing by the standard deviation in a similar way to what we saw previously:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/223/1*PhA75ezRf1BjK9OuMNUzCA.png" /></figure><p>where ϵ is a small constant added for numerical stability. After normalization, the inputs are scaled and shifted using learnable parameters γ (scale) and β (shift):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/172/1*3pGOthZ-osAzQev3AueWBg.png" /></figure><p>We can code it as:</p><pre>def BatchNorm(x, gamma, beta, eps=1e-5):<br>    # x shape [N, C, H, W]<br><br>    mean = torch.mean(x, dim=[0,2,3], keepdim=True)  # [1, C, 1, 1]<br>    var = torch.var(x, dim=[0,2,3], keepdim=True)    # [1, C, 1, 1]<br><br>    x_hat = (x - mean) / torch.sqrt(var + eps)<br>    <br>    return gamma * x_hat + beta</pre><p>In addition to stabilizing the training and accelerating the convergence, Batch Normalization is also used as regularization. Since its statistics depend on the specific samples in the mini-batch, they introduce slight randomness (noise) into the normalized output. This noise acts similarly to dropout or other regularization methods by preventing the model from relying too heavily on specific patterns in the training data.</p><p>Because Batch normalization relies on computing mean and variance within each mini-batch, if the mini-batch size is too small the estimates of mean and variance become noisy, reducing its effectiveness. This may make the training unstable or lead to poor performance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*wo2NRhU6A0e3xD7O.png" /><figcaption><a href="https://arxiv.org/pdf/1803.08494">Source</a></figcaption></figure><h3>Layer Normalization</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*a3RaZXVst9e2YAUy.png" /><figcaption>Feature map tensor, with N as the batch axis, C as the channel axis, and (H, W) as the spatial axes. The pixels in blue are normalized by the same mean and variance, computed by aggregating the values of these pixels. <a href="https://arxiv.org/pdf/1803.08494">Source</a></figcaption></figure><p>Layer normalization (LN) is a normalization technique designed to address some of the limitations of batch normalization, particularly in scenarios with small batch sizes or sequential data. Instead of normalizing across the batch dimension, it normalizes across the features of each individual sample in a layer.</p><p>Unlike batch normalization, LN does not depend on the mini-batch size. It works effectively for small batch sizes and for single-sample training (e.g., in online learning). Moreover, because it normalizes each sample independently it is particularly effective in sequential processing models (like RNN) where the sequence length and structure may vary.</p><pre>def LayerNorm(x, gamma, beta, eps=1e-5):<br>    # x shape [N, C, H, W]<br>    <br>    mean = torch.mean(input=x, dim=[1,2,3], keepdim=True) # [N, 1, 1, 1]<br>    var = torch.var(input=x, dim=[1,2,3], keepdim=True)   # [N, 1, 1, 1]<br>    <br>    x_hat = (x - mean) / torch.sqrt(var + eps)<br>    <br>    return gamma * x_hat + beta</pre><h3>Instance Normalization</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*GyE0P4Dj4xZCek3g.png" /><figcaption>Feature map tensor, with N as the batch axis, C as the channel axis, and (H, W) as the spatial axes. The pixels in blue are normalized by the same mean and variance, computed by aggregating the values of these pixels. <a href="https://arxiv.org/pdf/1803.08494">Source</a></figcaption></figure><p>Instance Normalization (IN) is a normalization technique primarily used in tasks where preserving style or content is important, such as style transfer or generative models. It operates at the level of individual samples, normalizing each feature map independently for each instance (or image) in a batch. By normalizing within each feature map, Instance Normalization preserves spatial structure, which is critical in image generation tasks.</p><p>Similar to Layer Normalization, Instance Normalization does not depend on batch size and works well even for a batch size of one.</p><pre>def InstanceNorm(x, gamma, beta, eps=1e-5):<br>    # x shape [N, C, H, W]<br><br>    mean = torch.mean(input=x, dim=[2,3], keepdim=True)  # [N, C, 1, 1]<br>    var = torch.var(input=x, dim=[2,3], keepdim=True)    # [N, C, 1, 1]<br><br>    x_hat = (x - mean) / torch.sqrt(var + eps)<br>    <br>    return gamma * x_hat + beta</pre><h3>Group Normalization</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/875/0*cdmtYtwUktFkahfX.png" /><figcaption>Feature map tensor, with N as the batch axis, C as the channel axis, and (H, W) as the spatial axes. The pixels in blue are normalized by the same mean and variance, computed by aggregating the values of these pixels. <a href="https://arxiv.org/pdf/1803.08494">Source</a></figcaption></figure><p>Group Normalization (GN) is a normalization technique designed to address the limitations of Batch Normalization, particularly for tasks where the batch size is small or variable. It is a combination of Layer and Instance Normalization, divides the features in each layer into groups, and normalizes them within each group independently.</p><p>As in Layer Normalization and Instance Normalization, it is independent of the batch size. GN often outperforms Layer Normalization and Instance Normalization for convolutional networks, especially in tasks with limited data or memory.</p><pre><br>def GroupNorm(x, gamma, beta, group_num, eps=1e-5):<br>    # x shape [N, C, H, W]<br><br>    x = torch.reshape(x, shape=[N, G, C // G, H, W])<br>    mean = torch.mean(x, dim=[2,3,4], keepdim=True)  # [N, G, 1, 1, 1]<br>    var = torch.var(x, dim=[2,3,4], keepdim=True)    # [N, G, 1, 1, 1]<br><br>    x_hat = (x - mean) / torch.sqrt(var + eps)<br><br>    x_hat = torch.reshape(x, shape=[N, C, H, W])<br>    <br>    return gamma * x_hat + beta</pre><h3>Conclusion</h3><p>In this post, we saw the main ideas behind normalization and explored four of the most popular normalization techniques in deep learning. We can summarize their properties in a table.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rzWVvVhCFaPQ-5bdBgLURg.png" /></figure><p>Normalization techniques are important components for stabilizing and accelerating the training of deep learning models. From the widely-used Batch Normalization, which shines in large-batch scenarios, to the more specialized approaches like Layer Normalization, Instance Normalization, and Group Normalization, each method has its strengths and use cases. While Batch Normalization is the default choice for many tasks, alternative techniques address its limitations in scenarios such as small batch sizes, sequential data, or style-transfer applications. Understanding these differences allows you to select the most suitable normalization method, improving both model performance and training efficiency.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=03eece7fa789" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Knowledge Distillation — How Networks Can Teach]]></title>
            <link>https://python.plainenglish.io/knowledge-distillation-how-networks-can-teach-a3e287d28eea?source=rss-aaf4831d9b34------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3e287d28eea</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Dor Zvulun]]></dc:creator>
            <pubDate>Sat, 28 Dec 2024 11:38:11 GMT</pubDate>
            <atom:updated>2025-01-06T04:27:19.783Z</atom:updated>
            <content:encoded><![CDATA[<h3>Knowledge Distillation — How Networks Can Teach</h3><h4>An introduction to teacher-student models</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BDczIdUQroxbhLuepoOSOQ.png" /><figcaption><a href="https://arxiv.org/pdf/2006.05525">Source</a></figcaption></figure><p>In the rapidly advancing field of deep learning, deploying high-performance models in real-world applications often comes with a key challenge: the balance between accuracy and computational efficiency. Many state-of-the-art models are large and resource-intensive, making them impractical for deployment on devices with limited memory or processing power, such as mobile phones, embedded systems, or autonomous vehicles. This is where<strong> </strong>knowledge distillation becomes a powerful technique.</p><p>The core idea of knowledge distillation is to transfer knowledge from a large, complex model to a smaller, more efficient one without significant loss of accuracy.</p><p>This post will explore the principles behind knowledge distillation, its various techniques, and how it can be applied to deep learning models in computer vision. Let’s dive into how it works and what makes it so effective.</p><h3>Knowledge Distillation</h3><p>Knowledge distillation is a technique for transferring insights from a large, complex model or group of models to a single, more compact model that is feasible for deployment in real-world scenarios with practical limitations. In essence, it serves as a method of model compression.</p><p>This technique is predominantly applied to neural network models that feature complex architectures with multiple layers and numerous parameters. While deep learning models become more successful and bigger than ever, the deployment of extensive deep neural networks poses significant challenges, particularly for edge devices that have restricted memory and computational resources. This is why knowledge distillation methods have become increasingly important for real-world implementations.</p><p>In knowledge distillation, we take advantage of the complex network and use it as a “teacher”, to help a smaller and resource-friendly “student” model to learn from it, aiming to replicate the teacher&#39;s performance. The following section will explore the structure and mechanisms of the knowledge distillation technique in greater detail.</p><h3>How to Transfer the Knowledge</h3><p>As mentioned previously, our goal is to use an intricate network with high performance (the “teacher”), to teach a small and simpler network the knowledge it needs to perform well.</p><p>When we refer to knowledge in this context, we mainly talk about the weights and biases the network learns during training. These weights and biases are responsible for the results of the networks. However, since the student network has significantly fewer parameters than the teacher, we can’t just copy the values of the teacher parameters to the student. We need a way to transfer the knowledge to the student.</p><p>There are three techniques to transfer the knowledge.</p><h4>Response-based</h4><p>This is the most straightforward and commonly used method in knowledge distillation. It focuses on training the student model to replicate the final output predictions of the teacher model. This approach uses the prediction (or “response”) of the teacher as the primary source of knowledge to transfer to the student.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/938/1*9lypa_66RnjkGXKpi_Le9w.png" /><figcaption>The generic response-based knowledge distillation.<a href="https://arxiv.org/pdf/2006.05525"> Source.</a></figcaption></figure><h4>Feature-based</h4><p>This type of knowledge distillation extends the knowledge transfer process beyond just the final outputs of the teacher model to include the internal feature representations learned by the teacher. This approach aims to make the student model learn the intermediate representations (the feature maps) that the teacher extracts from the input data, providing a deeper level of guidance.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/965/1*kypyYCrFhm6-8oMEdeSgow.png" /><figcaption>The generic feature-based knowledge distillation. <a href="https://arxiv.org/pdf/2006.05525">Source.</a></figcaption></figure><h4>Relation-based</h4><p>This type of distillation method focuses on transferring knowledge about the relations among data samples, layers, or feature representations from the teacher model to the student model. Instead of directly mimicking the teacher’s outputs or internal features, relation-based distillation trains the student to capture the relationships between multiple feature representations or data points learned by the teacher, such as correlations, similarities, embedding, or probabilistic distributions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/939/1*KCye_xuiD_WBUEV3devJGA.png" /><figcaption>Generic relation-based knowledge distillation. <a href="https://arxiv.org/pdf/2006.05525">Source.</a></figcaption></figure><h3>Distillation Schemes</h3><p>There are several techniques to train the teacher and student models to perform knowledge distillation. The three main categories of knowledge distillation are <strong>offline distillation, online distillation,</strong> and <strong>self-distillation</strong>.</p><p>Offline distillation is the most popular. We first train the large teacher model, and after that, we use the pre-trained teacher to train the small student model. In the online distillation case, we train the teacher and the student simultaneously, so it is not a two-step process like the offline one, but an end-to-end training. Self-distillation is a special case of online distillation where the same network is used for both the teacher and student. An example of such a technique is where knowledge from deeper sections of the network is distilled into shallow sections of it.</p><p>There is a lot more to say about knowledge distillation techniques, and you can learn more from this <a href="https://arxiv.org/pdf/2006.05525">paper</a>.</p><h3>Offline Response-base Knowledge Distillation</h3><p>From now on we will focus on the most popular and straightforward technique of knowledge distillation, which is an offline learning of a response-based scheme.</p><p>The core idea behind it is to train a student model to approximate the results of a pre-trained teacher model. This involves several key steps:</p><ol><li><strong>Soft Targets</strong>: Instead of simply training the student to match the hard labels (e.g., a class label like “cat” or “dog”), like in the regular training scheme, knowledge distillation leverages the “soft targets” produced by the teacher. These soft targets are probability distributions over all possible classes (usually achieved by soft-max), and they contain information about the teacher’s confidence in its predictions.</li><li><strong>Temperature Scaling</strong>: During distillation, the teacher’s output is often scaled by a temperature parameter. The teacher’s predictions become softer by setting a higher temperature, which helps the student learn from subtle relationships between classes. For example, if an image has a high probability of both “dog” and “wolf,” this hint from the teacher helps the student understand that these classes are visually similar, thus improving its generalization.</li><li><strong>Loss Function</strong>: Knowledge distillation involves a special loss function that combines two terms: one that aligns the student’s predictions with the hard ground truth labels (the one true label of the image, like “dog”) as in regular training, and another that aligns the student’s predictions with the teacher’s soft targets (which give each class a probability). By optimizing this combined loss, the student learns from the ground truth and the teacher’s nuanced output, allowing it to approximate the teacher’s performance more closely.</li></ol><h3>Let’s Code It</h3><p>In this part, we will implement a simple knowledge-distillation scheme using offline response-based learning. We will use a large network as the teacher, and a smaller version as a student.</p><p>The full notebook can be found <a href="https://github.com/dorzv/ComputerVision/blob/master/knowledge_distillation/knowledge_distillation.ipynb">here</a>.</p><p>We start by importing all the necessary modules and setting a device to use.</p><pre>from copy import deepcopy<br>import torch<br>import torch.nn as nn<br>import torch.optim as optim<br>from torch.utils.data import DataLoader<br>import torchvision.transforms.v2 as transforms<br>import torchvision.datasets as datasets<br><br>device = &quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;<br>torch.manual_seed(42)</pre><p>We will use the CIFAR-10 dataset, which contains RGB images of size 32x32 for ten classes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/838/1*HXWzprR1_LEeSElrkH6yhg.png" /><figcaption>A sample of images from the CIFAR-10 dataset. Source: <a href="https://www.cs.toronto.edu/~kriz/cifar.html">https://www.cs.toronto.edu/~kriz/cifar.html</a></figcaption></figure><p>We will use torchvision to download and use the dataset and to define a transformation that converts the images to Pytorch tensors.</p><pre>transform = transforms.Compose([<br>    transforms.ToTensor()<br>])<br><br>train_dataset = datasets.CIFAR10(root=&#39;data&#39;, train=True, download=True, transform=transform)<br>test_dataset = datasets.CIFAR10(root=&#39;data&#39;, train=False, download=True, transform=transform)</pre><p>We use the datasets to create DataLoader objects.</p><pre>train_loader = DataLoader(train_dataset, batch_size=128, shuffle=True, num_workers=2)<br>test_loader = DataLoader(test_dataset, batch_size=128, shuffle=False, num_workers=2)</pre><p>After having the data, we can define the models. Let’s start with the teacher model, which will be a CNN model with a classifier.</p><pre>class TeacherModel(nn.Module):<br>    def __init__(self, num_classes=10):<br>        super().__init__()<br>        self.features_extractor = nn.Sequential(<br>            nn.Conv2d(3, 128, kernel_size=3, padding=1),<br>            nn.BatchNorm2d(128),<br>            nn.ReLU(),<br>            nn.Conv2d(128, 64, kernel_size=3, padding=1),<br>            nn.BatchNorm2d(64),<br>            nn.ReLU(),<br>            nn.MaxPool2d(kernel_size=2, stride=2),<br>            nn.Conv2d(64, 64, kernel_size=3, padding=1),<br>            nn.BatchNorm2d(64),<br>            nn.ReLU(),<br>            nn.Conv2d(64, 32, kernel_size=3, padding=1),<br>            nn.BatchNorm2d(32),<br>            nn.ReLU(),<br>            nn.MaxPool2d(kernel_size=2, stride=2),<br>        )<br><br>        self.classifier = nn.Sequential(<br>            nn.Linear(2048, 512),<br>            nn.ReLU(),<br>            nn.Dropout(0.1),<br>            nn.Linear(512, num_classes)<br>        )<br><br>    def forward(self, x):<br>        x = self.features_extractor(x)<br>        x = torch.flatten(x, 1)<br>        x = self.classifier(x)<br>        return x</pre><p>And the student model.</p><pre>class StudentModel(nn.Module):<br>    def __init__(self, num_classes=10):<br>        super().__init__()<br>        self.features_extractor = nn.Sequential(<br>            nn.Conv2d(3, 16, kernel_size=3, padding=1),<br>            nn.BatchNorm2d(16),<br>            nn.ReLU(),<br>            nn.MaxPool2d(kernel_size=2, stride=2),<br>            nn.Conv2d(16, 16, kernel_size=3, padding=1),<br>            nn.BatchNorm2d(16),<br>            nn.ReLU(),<br>            nn.MaxPool2d(kernel_size=2, stride=2),<br>        )<br><br>        self.classifier = nn.Sequential(<br>            nn.Linear(1024, 256),<br>            nn.ReLU(),<br>            nn.Dropout(0.1),<br>            nn.Linear(256, num_classes)<br>        )<br><br>    def forward(self, x):<br>        x = self.features_extractor(x)<br>        x = torch.flatten(x, 1)<br>        x = self.classifier(x)<br>        return x</pre><p>As you can see, the student model is shallower and narrower than the teacher model but has a similar architecture.</p><p>We want a training function and also a test function to measure accuracy.</p><pre>def train(model, dataloader, epochs, learning_rate, device):<br>    criterion = nn.CrossEntropyLoss()<br>    optimizer = optim.Adam(model.parameters(), lr=learning_rate)<br><br>    model.to(device)<br>    model.train()<br><br>    for epoch in range(epochs):<br>        running_loss = 0.0<br>        for images, labels in dataloader:<br>            images = images.to(device)<br>            labels = labels.to(device)<br><br>            optimizer.zero_grad()<br>            outputs = model(images)<br><br>            loss = criterion(outputs, labels)<br>            loss.backward()<br>            optimizer.step()<br><br>            running_loss += loss.item()<br><br>        print(f&quot;Epoch {epoch+1}/{epochs}, Mean Loss: {running_loss / len(train_loader)}&quot;)</pre><pre>def test(model, data_loader, device):<br>    model.to(device)<br>    model.eval()<br><br>    correct = 0<br>    total = 0<br><br>    with torch.no_grad():<br>        for images, labels in data_loader:<br>            images = images.to(device)<br>            labels = labels.to(device)<br><br>            outputs = model(images)<br>            predicted_class = torch.argmax(outputs, dim=1)<br><br>            total += labels.size(0)<br>            correct += (predicted_class == labels).sum().item()<br><br>    accuracy = 100 * correct / total<br>    print(f&quot;Test Accuracy: {accuracy:.2f}%&quot;)<br>    return accuracy</pre><p>Now we create instances of the teacher models, and of the student model, and train them regularly. We also clone the student model to use later with knowledge distillation.</p><pre>print(&#39;Teacher&#39;)<br>teacher = TeacherModel(num_classes=10)<br>train(teacher, train_loader, epochs=30, learning_rate=0.001, device=device)<br>test_accuracy_teacher = test(teacher, test_loader, device)<br><br>print(&#39;\nStudent&#39;)<br>student1 = StudentModel(num_classes=10)<br>student2 = deepcopy(student1)<br>train(student1, train_loader, epochs=30, learning_rate=0.001, device=device)<br>test_accuracy_student1 = test(student1, test_loader, device)</pre><pre>Teacher<br>Epoch 1/30, Mean Loss: 1.227013551975455<br>Epoch 2/30, Mean Loss: 0.8006297277520075<br>.<br>.<br>.<br>Epoch 30/30, Mean Loss: 0.042777400205383447<br>Test Accuracy: 80.35%<br><br>Student<br>Epoch 1/30, Mean Loss: 1.3870741598441472<br>Epoch 2/30, Mean Loss: 1.0735777561621898<br>.<br>.<br>.<br>Epoch 30/30, Mean Loss: 0.19093702329546594<br>Test Accuracy: 68.70%</pre><p>As we can see, the teacher model performs better than the student model. Now, we want to use knowledge distillation to see if it improves the student’s performance. First, we define a new training function that uses the teacher&#39;s result for the training of the student.</p><pre>def train_knowledge_distillation(teacher, student, data_loader, epochs, learning_rate, temperature, alpha, device):<br>    criterion = nn.CrossEntropyLoss()<br>    criterion_distill = nn.KLDivLoss(reduction=&#39;batchmean&#39;)<br>    optimizer = optim.Adam(student.parameters(), lr=learning_rate)<br>    teacher.to(device)<br>    student.to(device)<br>    teacher.eval()<br>    student.train()<br><br>    for epoch in range(epochs):<br>        running_loss = 0.0<br>        for images, labels in data_loader:<br>            images = images.to(device)<br>            labels = labels.to(device)<br><br>            optimizer.zero_grad()<br><br>            # Forward pass with the teacher model<br>            with torch.no_grad():<br>                teacher_logits = teacher(images)<br><br>            # Forward pass with the student model<br>            student_logits = student(images)<br><br>            # Calculate the soft label loss<br>            soft_targets = torch.softmax(teacher_logits / temperature, dim=1)<br>            soft_probs = torch.log_softmax(student_logits / temperature, dim=1)<br><br>            distillation_loss = criterion_distill(soft_probs, soft_targets)<br><br>            # Calculate the true label loss<br>            label_loss = criterion(student_logits, labels)<br><br>            # Weighted sum of the two losses<br>            loss = alpha * label_loss + (1.0 - alpha) * temperature**2 * distillation_loss<br>            loss.backward()<br>            optimizer.step()<br><br>            running_loss += loss.item()<br><br>        print(f&quot;Epoch {epoch+1}/{epochs}, Loss: {running_loss / len(train_loader)}&quot;)</pre><p>Most of the function is identical to the previous function, but we add a distillation loss term that uses soft labels based on the teacher results. This loss uses KL divergence (Kullback–Leibler divergence), which measures how much a model distribution is different from a true distribution. In our case, we assume the probability for each class that the teacher produces is the target probability, and we want to measure the differences between the probabilities the student gives and the teacher&#39;s result. Pay attention that Pytorch KLDivLoss assumes the input is in a log scale (but the target doesn’t have to), so we use log_softmax on the student’s logists. You can also see that we use a <em>temperature </em>term to smooth the probabilities that the softmax produces.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/0*2Rf6FePn9ZqkArqg.png" /><figcaption>A probability calculated using sot-max with temperature</figcaption></figure><p>In addition, we also calculate the regular cross-entropy loss with the true labels. The total loss can be written as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/345/0*Sh9SJQu-KoWfJiKX.png" /></figure><p>where the hard loss is a cross-entropy with the real labels, the soft loss is the KL divergence on the smoothed probabilities, T is the temperature we have used for the smoothed probabilities, and α is a weight parameter.</p><p>Now we can train the clone of the student with the new procedure. It’s important to notice that the clone is identical to the original student model before the training.</p><pre>train_knowledge_distillation(teacher, student2, train_loader, epochs=30, learning_rate=0.001, temperature=2, alpha=0.8, device=device)<br>test_accuracy_student_with_distillation = test(student2, test_loader, device)</pre><pre>Epoch 1/30, Loss: 2.1964448251382773<br>Epoch 2/30, Loss: 1.6716872932356033<br>.<br>.<br>.<br>Epoch 30/30, Loss: 0.3442998861946413<br>Test Accuracy: 71.08%</pre><p>You can see we achieve a nice improvement by only adding information from the teacher model.</p><h3>Conclusion</h3><p>Knowledge distillation offers a practical and effective way to bridge the gap between state-of-the-art performance and real-world constraints. By transferring the knowledge from a large, complex model to a smaller, more efficient one, developers can deploy their systems on resource-limited devices without sacrificing accuracy. We saw various methods to use the knowledge distillation idea, and in the example we explored, we demonstrated how a teacher-student framework can be implemented, showcasing the simplicity and effectiveness of applying this method in computer vision tasks. As AI continues to evolve, techniques like knowledge distillation will remain essential for enabling advanced models to perform efficiently in resource-constrained environments.</p><h3>Thank you for being a part of the community</h3><p><em>Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a></li><li><a href="https://cofeed.app/"><strong>Check out CoFeed, the smart way to stay up-to-date with the latest in tech</strong></a> <strong>🧪</strong></li><li><a href="https://differ.blog/"><strong>Start your own free AI-powered blog on Differ</strong></a> 🚀</li><li><a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Join our content creators community on Discord</strong></a> 🧑🏻‍💻</li><li>For more content, visit <a href="https://plainenglish.io/"><strong>plainenglish.io</strong></a> + <a href="https://stackademic.com/"><strong>stackademic.com</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3e287d28eea" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/knowledge-distillation-how-networks-can-teach-a3e287d28eea">Knowledge Distillation — How Networks Can Teach</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Paper Review: Bootstrap Your Own Latent (BYOL)]]></title>
            <link>https://dzdata.medium.com/paper-review-bootstrap-your-own-latent-byol-545c6f7acee3?source=rss-aaf4831d9b34------2</link>
            <guid isPermaLink="false">https://medium.com/p/545c6f7acee3</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Dor Zvulun]]></dc:creator>
            <pubDate>Fri, 29 Nov 2024 08:53:05 GMT</pubDate>
            <atom:updated>2024-11-29T08:53:05.445Z</atom:updated>
            <content:encoded><![CDATA[<h4>A New Approach to Self-Supervised Learning</h4><p>In a <a href="https://python.plainenglish.io/self-supervised-learning-for-visual-representation-b242bde77241">previous post</a>, we talked about the importance of self-supervised learning and why we would like to use it. Let’s repeat the main points. As we know, deep learning models need an extensive labeled dataset to learn from. While in a lot of cases, collecting the data itself is not a too hard task, the labeling step could be very time and money consuming, but this step is crucial for the classic supervised learning methods.</p><p>Self-supervised techniques can solve this problem by letting models learn from unlabeled data. It does so by learning meaningful representations directly from the data, without any need for labels.</p><p>In the<a href="https://python.plainenglish.io/self-supervised-learning-for-visual-representation-b242bde77241"> previous post</a>, we focused on the <a href="https://arxiv.org/pdf/2002.05709">SimCLR</a> architecture based on the contrastive learning method, which uses positive and negative sample pairs to teach models distinguishing features, by mapping similar samples (positives) closer together in the feature space while pushing dissimilar samples (negatives) farther apart. However, this method comes with significant computational costs and practical challenges. “Bootstrap Your Own Latent” (BYOL) brings a new approach to the field by proposing a way to learn without the need for negative pairs, making it both efficient and highly scalable.</p><p>In this post, we’ll explore how BYOL works, the problem it solves, its architectural design, and why it’s considered a breakthrough in self-supervised learning.</p><h3>The Problem: Dependency on Negative Samples</h3><p>As mentioned above, contrastive learning tries to find representations that keep pairs of positive examples (which are two augmented views of the same image) close, while negative samples (views from different images) representations are pushed farther apart. In this way, the model can learn distinct representations.</p><p>However, this method has limitations:</p><ul><li><strong>Risk of representation collapse</strong>: Without enough negative examples, contrastive learning methods can sometimes yield trivial representations that fail to capture useful features.</li><li><strong>High computational costs</strong>: Contrastive learning often requires large batch sizes to provide diverse negative samples, leading to increased memory and computational requirements.</li><li><strong>Batch size dependency</strong>: The method’s success often hinges on having a substantial number of negative samples, which becomes a problem when the batch size is small.</li></ul><p>As the last two points are a result of the first one, let’s expand a bit on the first point. Because we ask our model to find representations such as two views of the same image have similar representations, the model may come up with a constant representation for all images. Such a representation fulfills this goal, but it is not much of a help. By adding negative samples (i.e., other images), we add to the original task another task of discrimination between the representations of different images. This addition is the key to preventing collapsed representations.</p><p>It is found empirically that the number of negative samples and their quality have a strong influence on the result of this technique. This is the reason why it needs a large batch size and has high computational costs.</p><p>BYOL proposes a novel approach that eliminates the need for negative samples. Instead, it leverages two networks — a target network and an online network — that interact with different augmented views of the same image to refine each other’s representations iteratively. BYOL avoids collapse and complex negative sampling strategies and presents an efficient and stable learning solution that we will talk about next.</p><h3>BYOL’s Core Idea</h3><p>A simple solution to prevent collapse is to use a fixed randomly initialized network to produce the target representation of the predictions our main network produces. The authors found empirically that although it does avoid collapse, the result representations are not very good, but still much better than the initial fixed representation.</p><p>This experimental finding is the main idea behind BYOL. We start with a target representation, and we train a new and potentially enhanced representation, referred to as online, by predicting the target representation. Later we can use the updated online representation as the new target and iteratively repeat the procedure. In practice, the authors suggest using a slowly moving exponential average of the online network as the target network, instead of fixed checkpoints</p><h3>BYOL’s Architecture: How It Works</h3><p>BYOL’s approach is to create two augmented views of an image — two different transformations of the same input — and pass them through separate networks, “online” and “target”, where the online network tries to predict the target output.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rMKYA1unE35dLk6xL1OJWA.png" /><figcaption>BYOL’s architecture. BYOL minimizes a similarity loss between q_θ(z_θ) and sg(z’_ξ ), where θ are the trained weights, ξ are an exponential moving average of θ and sg means stop-gradient. At the end of training, everything but f_θ is discarded, and y_θ is used as the image representation. Source: <a href="https://arxiv.org/pdf/2006.07733">Bootstrap Your Own Latent A New Approach to Self-Supervised Learning</a></figcaption></figure><p>Let’s break down the architecture:</p><p><strong>Online Network</strong><br>The online network processes one view of the image and consists of three key stages:</p><ul><li><strong>Encoder f_θ</strong>: This is the primary network, often a convolutional neural network (CNN) like ResNet, that processes the image and extracts features. The encoder produces high-dimensional embeddings that capture the main features.</li><li><strong>Projector g_θ</strong>: Following the encoder there is a multi-layer perceptron (MLP) projector. This projector transforms the high-dimensional representation into a lower-dimensional latent space.</li><li><strong>Predictor q_θ</strong>: Exclusive to the online network, the predictor maps the projected feature representation to align with the target network’s output. This component helps the online network to closely match the target’s output.</li></ul><p><strong>Target Network</strong><br>The target network mirrors the online network’s structure, containing an encoder and a projector. However, it lacks a predictor. The target network serves as a stable reference point for the online network to align with, and its weights are updated gradually using an exponential moving average (EMA) of the online network’s weights. This EMA process is crucial for maintaining stability, preventing the model from collapsing to trivial solutions.</p><p><strong>The Loss Function</strong><br>BYOL employs a simple mean squared error between the normalized prediction and target projections, which can be shown to be a sort of cosine similarity loss.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/0*D4D9J_FuHdUu_toK.png" /></figure><p>The goal is to minimize the difference between these two representations, encouraging the online network to match the target’s stable reference. Unlike contrastive methods that require both positive and negative samples, BYOL’s loss is based solely on positive alignment, eliminating the need for direct contrast.</p><p>At each step, this loss is computed twice, wherein the second time we switch between the two augmented views so each of them will be passed to the other network. The final loss is the addition of the two losses.</p><p><strong>Exponential Moving Average (EMA) Update</strong><br>A pivotal part of BYOL is the slow, smoothing update of the target network’s weights using EMA. In each training step, the target network’s weights become a mix of its current weights and those of the online network. This creates a progressively refined target, which helps prevent representation collapse and allows the network to focus on meaningful features over time.</p><p>More formally, at each step, we compute the two losses, add them, make the optimization step to update the online network weights, and finally update the target weights as follows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/401/0*e6h81PKHRxpv7ahq.png" /></figure><p>where τ is the decay rate of the EMA and η is the learning rate.</p><h3>Main Code Implementation</h3><p>In this section, we will write the main components of BYOL architecture. Although this is not a full implementation, it still should give you an understanding of how to implement it.</p><p>Let’s import some necessary modules.</p><pre>from copy import deepcopy<br><br>import torch<br>from torch import nn<br>from torchvision import models</pre><p>We start with the components of the online network. For the encoder, we can use ResNet50, for example. ResNet50 can be downloaded directly from torchvision.</p><pre>backbone = models.resnet50(weights=None)<br>backbone_without_classifier = nn.Sequential(*list(backbone.children())[:-1])<br>online_encoder = backbone_without_classifier</pre><p>In the code, we got the ResNet50 architecture with random initialization. The original ResNet is built to predict ImageNet classes, so we remove the classifier component from it since we only want the features extracted from it.</p><p>We continue to the online projection which is a simple MLP.</p><pre>online_projection = nn.Sequential(<br>            nn.Flatten(),<br>            nn.Linear(backbone.fc.in_features, 4096),<br>            nn.BatchNorm1d(4096),<br>            nn.ReLU(inplace=True),<br>            nn.Linear(4096, 256)<br>        )</pre><p>And the predictor is the same.</p><pre>online_prediction = nn.Sequential(<br>            nn.Linear(256, 4096),<br>            nn.BatchNorm1d(4096),<br>            nn.ReLU(inplace=True),<br>            nn.Linear(4096, 256)<br>        )</pre><p>For the target network, we clone the online network’s encoder and projector.</p><pre>target_encoder = deepcopy(online_encoder)<br>target_projection = deepcopy(online_projection)</pre><p>and since we don’t need gradients for the target network, we can set all its parameters to not track gradients.</p><pre>for param in self.target_encoder.parameters():<br>    param.requires_grad = False<br><br>for param in self.target_projection.parameters():<br>    param.requires_grad = False</pre><p>We define the loss as a module (you can also define it as a regular function)</p><pre>class BYOLLoss(nn.Module):<br>    &quot;&quot;&quot;<br>    Loss function for self supervised learning based on<br>    the paper &quot;Bootstrap Your Own Latent A New Approach<br>    to Self-Supervised Learning&quot; equation (2).<br>    &quot;&quot;&quot;<br>    @staticmethod<br>    def forward(prediction, projection):<br>        &quot;&quot;&quot;<br><br>        Args:<br>            prediction (torch.Tensor): the instance representation prediction<br>                of the online network.<br>            projection (torch.Tensor): the instance representation of the<br>                target network.<br><br>        Returns:<br><br>        &quot;&quot;&quot;<br>        loss = 2 - 2 * F.cosine_similarity(prediction, projection)<br>        return loss.mean()</pre><p>and finally, a function to update the target network’s weights with an exponential moving average.</p><pre>def update_target_models(online, target, moving_average_decay):<br>    &quot;&quot;&quot;<br>    Update the weights of the target models (encoder and projection)<br>    using exponential moving average with the online models. For more <br>    information about the technique, refer to the paper &quot;Bootstrap Your <br>    Own Latent A New Approach to Self-Supervised Learning&quot;<br><br>    Args:<br>      online (nn.Module): a component of the online network<br>      target (nn.Module): a component of the traget network, <br>        the same as the online one<br>      moving_average_decay (float): the decay rate of the exponential <br>        moving average. Number between 0 and 1.<br>    &quot;&quot;&quot;<br>    for online_params, target_params in zip(online.parameters(), target.parameters()):<br>        target_params.data = (target_params.data * moving_average_decay<br>                              + (1 - self.moving_average_decay) * online_params.data)</pre><p>These are the main components we need to implement BYOL. For the training itself, you need to create a dataset that returns two augmented views of the image (you can read about the specific augmentations in the paper). Then, first pass the first view to the online network, and the second to the target network and compute the loss. After that, pass the second view to the online network and the first one to the target network, and compute the loss. Use the combined loss to update the online network’s weights, and then update the target network’s weight using the exponential moving average.</p><h3>Conclusion</h3><p>BYOL brings several significant benefits to self-supervised learning, thanks to its architecture and loss design. BYOL’s approach gives us several advantages:</p><ol><li><strong>Avoidance of Negative Samples</strong><br>By eliminating the need for negative pairs, BYOL bypasses the high resource demands of contrastive learning. It’s a considerable advantage in applications with limited computational resources, as it reduces the dependence on large batch sizes and complicates negative sample strategies. BYOL’s innovative design shows that useful features can be learned solely through positive reinforcement, making it a more efficient solution.</li><li><strong>Reduced Risk of Representation Collapse</strong><br>Representation collapse — where a network produces the same outputs regardless of the input — can be a common problem in self-supervised models. BYOL’s use of a predictor in the online network, combined with EMA-updated target weights, allows it to avoid collapse, making it more stable and robust.</li><li><strong>Simplicity and Scalability</strong><br>BYOL’s architecture is simple and highly scalable, requiring fewer hyperparameters than contrastive methods. This simplicity enables it to be easily adapted to various architectures.</li></ol><p>BYOL has a great impact on the field of self-supervised learning by showing that meaningful representations can be learned without negative samples. The success of BYOL shows the potential of innovative self-supervised learning methods and lays the foundations for future advancements in resource-efficient models.</p><p>By demonstrating that robust features can be learned without complex contrastive methods, BYOL is used as a base for new research directions. If you work on a task with a big dataset, but limited labels, BYOL offers you a highly effective solution to increase your system performance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=545c6f7acee3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Introduction to TensorRT]]></title>
            <link>https://python.plainenglish.io/introduction-to-tensorrt-b3edf88c746c?source=rss-aaf4831d9b34------2</link>
            <guid isPermaLink="false">https://medium.com/p/b3edf88c746c</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Dor Zvulun]]></dc:creator>
            <pubDate>Mon, 28 Oct 2024 20:02:07 GMT</pubDate>
            <atom:updated>2024-10-29T03:39:31.869Z</atom:updated>
            <content:encoded><![CDATA[<p>Deep Learning is a great tool that is incredibly successful in many tasks including vision and natural language tasks. While the model’s training could be very long, the inference is relatively quick. However, we sometimes need to speed up the inference, usually for real-time tasks (for example, for autonomous vehicles).</p><p>To address this issue, several methods were developed, such as layer fusion, model quantization, pruning, etc. Although there are many ways to use such methods, in this post we will focus on TensorRT by Nvidia.</p><p>TensorRT is an optimized deep-learning inference library developed by Nvidia for accelerating the performance of models <strong>on Nvidia GPUs</strong>. It includes features that enable developers to maximize model performance, reduce memory usage, and lower latency. To name some of them:</p><ul><li>Layer and Kernel Fusion — TensorRT fuses compatible layers to reduce the number of operations, enabling faster computation by decreasing the number of kernel calls and memory accesses.</li><li>Quantization — TensorRT supports various precisions, and by reducing precision (like using FP16 or INT8) it can speed up inference.</li><li>Optimized Kernels — TensorRT includes highly optimized kernels for different operations (like convolutions, pooling, etc.).</li><li>Graph Optimization — TensorRT provides graph-level optimizations, such as eliminating redundant operations, which reduce computation and improve runtime performance.</li><li>Tensor Memory Management — TensorRT optimizes memory usage by efficiently managing tensor storage.</li></ul><p>and more.</p><p>It’s important to note that TensorRT is an optimization library specifically designed for Nvidia hardware. This means that TensorRT only works with Nvidia GPUs, and the optimizations it applies are tailored to the specific Nvidia GPU architecture it will run on. As a result, models optimized with TensorRT are highly efficient on the specific hardware it runs on, but they may not run or would require different optimization, on non-NVIDIA hardware or even other Nvidia hardware.</p><p>The following image illustrates the typical deep learning development cycle using TensorRT.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/0*u8N8KglCOBFRnAtX.png" /><figcaption>Source: Nvidia</figcaption></figure><p>There are several options to convert a model into an optimized version by using TensorRT: using an ONNX file, using PyTorch with TensorRT, or using the TensorRT API in Python or C++. We can also deploy the optimized model in several ways, including using Pytorch, TensorRT API in Python or C++, or by using Nvidia Triton Inference.</p><p>To my opinion, it is easiest to go with <a href="https://github.com/pytorch/TensorRT">Torch TensorRT</a> with Pytorch, so let’s focus on it in this post.</p><h3>Torch-TensorRT</h3><p>We start by installing Torch TensorRT.</p><pre>! pip install torch-tensorrt -q</pre><p>First, let’s import all the necessary modules.</p><pre>import time<br><br>import torch<br>import torch.nn.functional as F<br>import torch_tensorrt<br>import torchvision.models as models<br>import torchvision.transforms.v2 as transforms<br><br>from PIL import Image<br>import requests<br>from io import BytesIO<br>import numpy as np</pre><p>For our example, we will use ResNet50 as our model to be optimized. We can use TorchVision to download the network with its weights pre-trained on ImageNet.</p><pre>model = models.resnet50(weights=&#39;DEFAULT&#39;).eval().to(&quot;cuda&quot;)</pre><p>We set the state of the model to <em>eval</em> since we don’t train it, and move it to the GPU.</p><p>Before we optimize the network, let’s make sure it works as expected. We will get an image from the internet and ask the network to predict what’s in the image.</p><p>We can download the image and load it to our code, but it will be nicer to get it from the web directly.</p><pre>url = &quot;https://images.pexels.com/photos/1108099/pexels-photo-1108099.jpeg?cs=srgb&amp;dl=pexels-chevanon-1108099.jpg&amp;fm=jpg&amp;w=640&amp;h=480&quot;<br>response = requests.get(url)<br>image = Image.open(BytesIO(response.content)).convert(&quot;RGB&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*trFM64dtSOwGTA0U" /><figcaption><a href="https://www.pexels.com/photo/two-yellow-labrador-retriever-puppies-1108099/">Photo by Chevanon Photography</a></figcaption></figure><p>Before we can feed the image to the network, we need to apply some transformations to make the image an appropriate input to the network.</p><pre>transform = transforms.Compose([<br>    transforms.Resize(256),<br>    transforms.CenterCrop(224),<br>    transforms.ToTensor(),<br>    transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),<br>])</pre><p>We apply the transformations to the image and move it to the GPU.</p><pre>input_image = transform(image).unsqueeze(0).to(&quot;cuda&quot;)</pre><p>The network will output logits, but it will be more convenient if we get as output the name of the predicted class together with the probability. For the class names, we first need to get all the names. There are a lot of GitHubs where we can find the list of classes for ImageNet. I just picked one and read the data from it.</p><pre>url = &#39;https://raw.githubusercontent.com/pytorch/hub/master/imagenet_classes.txt&#39;<br>response = requests.get(url)<br>labels = {index: label for index, label in enumerate(response.text.split(&#39;\n&#39;))}</pre><p>Now we have a dictionary with all the classes, and we can make a function to predict the class of an image.</p><pre>def predict(model, image):<br>    with torch.no_grad():<br>        output = model(image)<br>        probabilities = F.softmax(output, dim=1)<br>    probability , class_index = torch.max(probabilities, dim=1)<br>    return labels[class_index.item()], probability.item()<br><br>predict(model, input_image)<br># (&#39;golden retriever&#39;, 0.44795194268226624)</pre><p>The network predicted the right class!</p><p>Now, we want to measure the time it takes for the network to predict a batch. For that, we will write a function to measure the average prediction time.</p><pre>def benchmark(model, input_shape=(32, 3, 224, 224), warmup_iterations=50, run_iterations=100):<br>    input_data = torch.randn(input_shape)<br>    input_data = input_data.to(&quot;cuda&quot;)<br>        <br>    print(&quot;Warm up ...&quot;)<br>    with torch.no_grad():<br>        for _ in range(warmup_iterations):<br>            outputs = model(input_data)<br>    <br>    torch.cuda.synchronize()<br>    print(&quot;Start timing ...&quot;)<br>    timings = []<br>    with torch.no_grad():<br>        for i in range(run_iterations):<br>            start_time = time.time()<br>            outputs = model(input_data)<br>            torch.cuda.synchronize()<br>            end_time = time.time()<br>            timings.append(end_time - start_time)<br><br>    print(f&#39;Average batch time: {np.mean(timings)*1000:.2f} ms&#39;)</pre><p>The function gets a model, input shape, and the number of warmups and benchmark iterations.</p><p>At first, the function creates a random tensor with <em>input_shape</em> (batch size, channels, height, width) and moves it to the GPU. Then the model processes the input <em>warmup_iterations </em>times without measuring the time. The purpose of this stage is to warm up the GPU for more stable performance measurements. During this phase, any initialization that PyTorch might perform internally on the GPU occurs before the actual timed runs.</p><p>Now we move to the actual benchmarking. We use torch.cuda.synchronize() to ensure that all GPU operations are complete before timing starts. This is important to get accurate measurements because CUDA operations are asynchronous. We define a list for keeping on the batch processing time, and then process the input in a loop for <em>run_iterations </em>times. Inside the loop, we record the start time before we process the input, process it in the network, and then call again to torch.cuda.synchronize() to ensure that all GPU operations related to model inference are complete before recording the end time. Only then, we can record the end time and append the process time to the list. At the end, we will print the average batch time it took to the network to infer the batch.</p><pre>benchmark(model)<br># Warm up ...<br># Start timing ...<br># Average batch time: 89.47 ms</pre><p>Now it’s time to optimize the model with TensorRT. Basic optimization using Torch TensorRT is very straightforward. We choose the input size to optimize for and then compile the model.</p><pre>inputs = [torch.randn((1, 3, 224, 224)).cuda()]<br>optimized_model = torch_tensorrt.compile(model,&#39;default&#39;, inputs)</pre><p>That’s it. We have three main arguments in the compile function: the model we want to optimize, the strategy to compile (default means let Torch TensorRT decide), and the inputs we want to optimize for (in this case we optimize the model for input with shape (1, 3, 224, 224)).</p><p>Let’s see if we still get the same result as before.</p><pre>predict(optimized_model, input_image)<br># (&#39;golden retriever&#39;, 0.4479517638683319)</pre><p>Indeed, we get pretty same results.</p><p>Before benchmarking the optimized model, let’s optimize it again with a batch size of 32.</p><pre>inputs = [torch.randn((32, 3, 224, 224)).cuda()]<br>optimized_model = torch_tensorrt.compile(model,&#39;default&#39;, inputs)</pre><p>and run the benchmark.</p><pre>benchmark(optimized_model)<br># Warm up ...<br># Start timing ...<br># Average batch time: 66.77 ms</pre><p>We get a nice improvement with very minimal work. We can use more settings to optimize even more. For example, let’s use half-precision (float 16) for the optimization.</p><pre>inputs = [torch.randn((32, 3, 224, 224)).cuda()]<br>optimized_model = torch_tensorrt.compile(model,&#39;default&#39;, inputs, enabled_precisions={torch.float16})</pre><p>and run the benchmark again.</p><pre>benchmark(optimized_model)<br># Warm up ...<br># Start timing ...<br># Average batch time: 15.26 ms</pre><p>This is a very big improvement! But is it still accurate?</p><pre>inputs = [torch.randn((1, 3, 224, 224)).cuda()]<br>optimized_model = torch_tensorrt.compile(model,&#39;default&#39;, inputs, enabled_precisions={torch.float16})<br><br>predict(optimized_model, input_image)<br># (&#39;golden retriever&#39;, 0.44798213243484497)<br></pre><p>Yes, we still get the right result. You have probably noticed we had to compile the model again because we have a batch of size one in the prediction. It would be more convenient if we could use more input shapes in our optimized model.</p><pre>inputs = torch_tensorrt.Input(<br>    min_shape=(1, 3, 224, 224),<br>    opt_shape=(32, 3, 224, 224),<br>    max_shape=(64, 3, 224, 224),<br>    dtype=torch.float32)<br>optimized_model = torch_tensorrt.compile(model,&#39;default&#39;, inputs, enabled_precisions={torch.float16})</pre><p>In the previous code, we tell TensorRT that the minimum shape of inputs is for a batch of size one, and the maximum is a batch size of 64. We also tell it that we want the main optimization to be for a size of 32, and the format of the input will be float32.</p><p>We optimize the model again and check the benchmark and prediction.</p><pre>benchmark(optimized_model)<br># Warm up ...<br># Start timing ...<br># Average batch time: 15.73 ms<br><br>predict(optimized_model, input_image)<br># (&#39;golden retriever&#39;, 0.4470345973968506)</pre><p>We got similar results as in the previous experiment, but we didn’t need to create different optimized models to test the benchmark and the prediction.</p><h3>Conclusion</h3><p>TensorRT is a very powerful optimization tool for neural networks that run on Nvidia hardware. It lets us gain significant speed up for the model inference, with a little work from our side, using a wide variety of optimization techniques. In this post, we saw some basic examples of how we can use Torch-TensorRT to leverage the power of TensorRT directly into our Pytorch models with very minimal effort, but there is a lot more to do with it, and if you like, you can read more about it in <a href="https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html">here</a>.</p><h3>In Plain English 🚀</h3><p><em>Thank you for being a part of the </em><a href="https://plainenglish.io/"><strong><em>In Plain English</em></strong></a><em> community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://x.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a> | <a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a> | <a href="https://open.spotify.com/show/7qxylRWKhvZwMz2WuEoua0"><strong>Podcast</strong></a></li><li><a href="https://differ.blog/"><strong>Create a free AI-powered blog on Differ.</strong></a></li><li>More content at <a href="https://plainenglish.io/"><strong>PlainEnglish.io</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b3edf88c746c" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/introduction-to-tensorrt-b3edf88c746c">Introduction to TensorRT</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Self-Supervised Learning for Visual Representation]]></title>
            <link>https://python.plainenglish.io/self-supervised-learning-for-visual-representation-b242bde77241?source=rss-aaf4831d9b34------2</link>
            <guid isPermaLink="false">https://medium.com/p/b242bde77241</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Dor Zvulun]]></dc:creator>
            <pubDate>Sat, 28 Sep 2024 09:00:28 GMT</pubDate>
            <atom:updated>2024-09-30T07:54:30.524Z</atom:updated>
            <content:encoded><![CDATA[<h4>How we can improve our models by using unlabeled data.</h4><p>Deep learning is a great tool that outperforms many tasks such as image recognition, detection, and segmentation. However, one of the main challenges we encounter when we want to use deep learning is collecting enough data for the task. Most of the time, the data itself is insufficient, and we also need to label it to supervise the learning process.</p><p>The need for labels makes a lot of data unusable for learning tasks and prevents us from improving our models. Moreover, if you think about it, we humans don’t need so much labeled data. By seeing just a few labeled images, we can very quickly understand the <em>representation </em>of the object and can classify non-previously seen images just by understanding this representation. For example, you don’t need to see all types of dogs to classify an image of an unseen dog, because you understand what represents a dog in an image (four legs, tail, face structure, etc.).</p><p>To solve the problem of unlabeled data we can use self-supervised learning instead of regular supervised learning. Self-supervised learning enables models to learn meaningful features from vast amounts of unlabeled data. Self-supervised methods leverage the inherent structure of data and allow models to learn rich and transferable representations. In this post, we will understand the basic ideas behind self-supervised learning and use the <a href="https://arxiv.org/pdf/2002.05709">SimCLR </a>architecture to see how to train a self-supervised model. Finally, we will implement and train an image recognition model based on SimCLR using Pytorch and Pytorch Lightning.</p><h3>Contrastive Learning</h3><p>Contrastive learning is a self-supervised learning technique where a model learns to distinguish between similar and dissimilar data points. The core idea is to map similar samples (positives) closer together in the feature space while pushing dissimilar samples (negatives) farther apart.</p><p>But how we can know which images are similar and which are dissimilar if we don&#39;t have labels? One way to do that is when given an image, generate different “views” of it (such as different crops, rotations, or any other augmentations). The model then tries to make the representations of these views closer to each other and farther from representations of different images. This makes sense since essentially an image of a dog represents the same thing even if we flip it or rotate it, for example, and we expect the model to represent the augmented image in a similar way to the original image.</p><p>This process helps the model learn meaningful features and representations that can be used for various downstream tasks, like classification or object detection.</p><p>Next, we will talk about the SimCLR architecture which is based on contrastive learning.</p><h3>SimCLR</h3><p>SimCLR, or a Simple framework for Contrastive Learning of visual representations, is a contrastive learning architecture for self-supervised learning proposed in the paper “<a href="https://arxiv.org/pdf/2002.05709"><em>A Simple Framework for Contrastive Learning of Visual Representations</em></a>”. In the paper, they show that using SimCLR to learn representations and then fine-tune on ImageNet with just 10% of the labels achieves great results compared to learning without self-supervised and on 100% of the labels.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yQ6Mg0WzEd6pWOBlY7sszw.png" /><figcaption>Source: “<a href="https://arxiv.org/pdf/2002.05709">A Simple Framework for Contrastive Learning of Visual Representations</a>”</figcaption></figure><h4>The Architecture</h4><p>As we already mentioned, in contrastive learning we want similar images to have a close representation, while different images should be farther away from each other.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dj9tmigOsNeHc_Ec_iAHlA.png" /><figcaption>A simple framework for contrastive learning of visual representations. Two separate data augmentation operators are sampled from the same family of augmentations (tᵢ ~T and tⱼ ~T ) and applied to each data example to obtain two correlated views. A base encoder network f(·) and a projection head g(·) are trained to maximize agreement using a contrastive loss. After training is completed, we throw away the projection head g(·) and use encoder f(·) and representation h for downstream tasks.</figcaption></figure><p>As you can see in the figure above, we take an image and sample two augmentations to apply to it. Then, we use an encoder network to get a representation of the image. Finally, we use an MLP to project the representation and try to maximize the similarity between the two projections. Pay attention that after the training, we will remove the MLP and use only the representation for training downstream tasks like recognition. For different images, we will do a similar process, but we want to minimize the similarity between them.</p><p>As always, the best way to understand the whole process is to write it by ourselves, so let’s code it.</p><h3>Let’s Code It</h3><p>The full code can be found <a href="https://github.com/dorzv/ComputerVision/blob/master/self_supervised_learning/SimCLR.ipynb">here</a>.</p><p>In this section, we are going to implement SimCLR architecture using Pytorch. We will also use Pytorch Lightning to save some work and boilerplates. <a href="https://lightning.ai/docs/pytorch/stable/">PyTorch Lightning</a> is a lightweight wrapper for PyTorch, designed to simplify the research process while maintaining the flexibility and power of native PyTorch. It abstracts away much of the boilerplate code and provides a high-level interface for model training, validation, and testing. We will also install <a href="https://medium.com/python-in-plain-english/introduction-to-torch-metrics-5aff01f4731f">TorchMetrics </a>to track performance metrics later. To install it we use</p><pre>!pip install lightning torchmetrics -q</pre><p>We also want to import all the modules we want to use.</p><pre>import os<br>import matplotlib.pyplot as plt<br><br>import torch<br>import torch.nn as nn<br>import torch.nn.functional as F<br>import torch.optim as optim<br>from torch.utils.data import DataLoader, Subset<br>from torch.optim.lr_scheduler import OneCycleLR<br><br>import lightning as L<br>from lightning.pytorch.callbacks import ModelCheckpoint<br><br>import torchvision<br>import torchvision.transforms.v2 as transforms<br>import torchvision.models as models<br>from torchvision.datasets import STL10<br><br>from torchmetrics import Accuracy<br><br>from sklearn.model_selection import train_test_split<br><br>import numpy as np</pre><p>Let’s also set a seed for reproducibility.</p><pre>L.seed_everything(0)</pre><p>Finally, we set a device (GPU or CPU) and the number of workers we will use for the dataloader.</p><pre>device = &#39;cuda&#39; if torch.cuda.is_available() else &#39;cpu&#39;<br>num_worker = os.cpu_count()<br>print(f&#39;{device=}&#39;)<br>print(f&#39;{num_worker=}&#39;)</pre><h4>The Dataset</h4><p>We will use the <a href="https://cs.stanford.edu/%7Eacoates/stl10/">STL-10</a> dataset. This dataset is an image recognition dataset for self-supervised training. It consists of a small amount of labeled images, and a lot of unlabeled images (which are not necessarily of the same classes as the labeled images, but from a similar distribution, for example, other animals). The dataset is described as follows according to the STL-10 website:</p><ul><li>10 classes: airplane, bird, car, cat, deer, dog, horse, monkey, ship, truck.</li><li>Images are 96x96 pixels, in color.</li><li>500 training images (10 pre-defined folds), 800 test images per class.</li><li>100,000 unlabeled images for unsupervised learning. These examples are extracted from a similar but broader distribution of images. For instance, it contains other types of animals (bears, rabbits, etc.) and vehicles (trains, buses, etc.) in addition to the ones in the labeled set.</li><li>Images were acquired from labeled examples on <a href="http://www.image-net.org/">ImageNet</a>.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/681/0*l7HlLy1LtgvG0KUF.png" /><figcaption>Source: <a href="https://cs.stanford.edu/%7Eacoates/stl10/">https://cs.stanford.edu/%7Eacoates/stl10/</a></figcaption></figure><p>First, we define the folder where we will save the data</p><pre>DATASET_PATH = &#39;STL10&#39;</pre><p>Then, we define a class that creates multiple augmentation views of an image (as we want to give the network the same image with different augmentations as a positive sample).</p><pre>class AugmentationViews:<br>  &quot;&quot;&quot;<br>  Create multiple augmentations for an image<br>  &quot;&quot;&quot;<br>  def __init__(self, transform, num_views):<br>    self.transform = transform<br>    self.num_views = num_views<br><br>  def __call__(self, image):<br>    return [self.transform(image) for _ in range(self.num_views)]</pre><p>And finally, we define a basic transformation using torchvision, to apply to our images (randomly).</p><pre>base_transform = transforms.Compose([<br>    transforms.RandomHorizontalFlip(),<br>    transforms.RandomResizedCrop(size=96),<br>    transforms.RandomApply([transforms.ColorJitter(brightness=0.8, contrast=0.8, saturation=0.8, hue=0.1)], p=0.8),<br>    transforms.RandomGrayscale(p=0.2),<br>    transforms.ToTensor(),<br>    transforms.Normalize((0.5,), (0.5,))<br>])</pre><p>Now it’s time to get the data itself. We will do it using torchvision dataset module that supports STL10.</p><pre>unlabeled_data = STL10(<br>    root=DATASET_PATH,<br>    split=&quot;unlabeled&quot;,<br>    download=True,<br>    transform=AugmentationViews(base_transform, num_views=2))<br><br>train_data = STL10(<br>    root=DATASET_PATH,<br>    split=&quot;train&quot;,<br>    download=True,<br>    transform=AugmentationViews(base_transform, num_views=2)<br>)</pre><p>The module will download the files of the STL10 dataset to <em>DATASET_PATH, </em>create Dataset objects for the unlabeled and the labeled training data, and also apply the augmentation we set earlier to the data.</p><p>It’s always a good idea to look at the images in the dataset, so we will write code to show us a sample of images including the augmentation views.</p><pre>NUM_IMAGES = 20<br>images = torch.stack([image for ii in range(NUM_IMAGES)<br>                    for image in unlabeled_data[ii][0]],<br>                   dim=0)<br>images_grid = torchvision.utils.make_grid(images, nrow=10, normalize=True)<br>images_grid = images_grid.permute(1, 2, 0)<br><br>plt.figure(figsize=(20, 10))<br>plt.imshow(images_grid)<br>plt.axis(&quot;off&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U8VY8pfs3uDLQ1Nm2AlyWQ.png" /><figcaption>A sample and images with augmentations. Pay attention that for each image there are two versions with different augmentation.</figcaption></figure><h4>Loss</h4><p>In the paper, the authors present a loss function they call NT-Xent loss which is a normalized temperature-scaled cross-entropy loss. For calculating this loss, we start by sampling a mini-batch of N images, and for each image, we apply two randomly sampled augmentations (as we saw earlier). We consider each pair of augmented images (which came from the same original image) as a positive pair, while all other 2(N-1) images are considered negative examples relative to the two pictures in the positive pair.</p><p>Using cosine similarity (which is a dot product between L2 normalized <strong>u</strong> and <strong>v</strong>) we define</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/242/0*bNOhR-Q7uMzcQg-M.png" /></figure><p>Given a positive pair (<em>i, j</em>) we define a loss as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/620/1*gKWJ4gAFSohYCTuv9CFHbA.png" /></figure><p>where</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/197/1*o2ZplI3uLb0Th_2bZBsReg.png" /></figure><p>is an indicator function that gets one only if k≠i, τ is a temperature parameter and <em>z</em> is the projection result of the network (see the architecture scheme in the architecture section above).</p><p>For the final loss, we compute the above loss for all positive pairs in the mini-batch, for both (<em>i, j</em>) and (<em>j, i</em>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/599/1*216p4w8YA1g7tOAGcBYjdQ.png" /></figure><p>Let’s write it in code.</p><pre>class NTXentLoss(nn.Module):<br><br>    def __init__(self, batch_size, temperature, device=&#39;cuda&#39;):<br>      super().__init__()<br>      self.batch_size = batch_size<br>      self.device = device<br>      self.temperature = temperature<br>      self.similarity_function = nn.CosineSimilarity(dim=-1)<br>      self.negative_pairs_mask = self.get_negative_mask()<br>      self.criterion = nn.CrossEntropyLoss(reduction=&quot;sum&quot;).to(device)<br><br>    def calculate_similarity_matrix(self, x, y):<br>      &quot;&quot;&quot;<br>      Calculate the similary between each vector in the first<br>      argument with each vector in the second one, to get a matrix<br>      of similarity scores between all the vectors in the two arguments<br>      &quot;&quot;&quot;<br>      return self.similarity_function(x.unsqueeze(1), y.unsqueeze(0))<br><br>    def get_negative_mask(self):<br>      &quot;&quot;&quot;<br>      Create a mask with True for negative elements, i.e.<br>      pairs from views that don&#39;t come from the same<br>      original image<br>      &quot;&quot;&quot;<br>      mat_1 = np.eye(2 * self.batch_size)<br>      mat_2 = np.eye(2 * self.batch_size, k=-self.batch_size)<br>      mat_3 = np.eye(2 * self.batch_size, k=self.batch_size)<br>      mask = torch.from_numpy((mat_1 + mat_2 + mat_3))<br>      mask = (1 - mask).type(torch.bool)<br>      return mask<br><br>    def forward(self, zis, zjs):<br>      &quot;&quot;&quot;<br>      Args:<br>      zis (torch.Tensor): projections of the first view of the images<br>      zis (torch.Tensor): projections of the second view of the images<br>      &quot;&quot;&quot;<br>      projections = torch.cat([zjs, zis], dim=0)<br><br>      similarity_matrix = self.calculate_similarity_matrix(projections, projections)  #[2N, 2N]<br>      # The positive pairs are on the batch_size-th diagonals of the similarity matrix<br>      positive_pairs_1 = torch.diag(similarity_matrix, self.batch_size)  # [N]<br>      positive_pairs_2 = torch.diag(similarity_matrix, -self.batch_size)  # [N]<br>      positives_pairs = torch.cat([positive_pairs_1, positive_pairs_2]).view(2 * self.batch_size, 1)  # [2N, 1]<br><br>      negatives_pairs = similarity_matrix[self.negative_pairs_mask].view(2 * self.batch_size, -1)  # [2N, 2N-2]<br><br>      logits = torch.cat((positives_pairs, negatives_pairs), dim=1).to(self.device)  # [2N, 2N-1]<br>      logits /= self.temperature<br><br>      labels = torch.zeros(2 * self.batch_size).long().to(self.device)<br>      loss = self.criterion(logits, labels)<br><br>      return loss / (2 * self.batch_size)</pre><h4>Self-Supervised Training</h4><p>In this section, we will build the SimCLR network for our self-supervised training. We will use the ResNET-50 as the backbone for the representation and add an MLP for the projection.</p><p>For the training itself, we will use <a href="https://lightning.ai/docs/pytorch/stable/">Pytorch Lightning</a> which by implementing only three simple methods, training_step, validation_step<strong>, </strong>and configure_optimizers, in our model, lets us train the model without writing the training (and validation) loop by ourselves.</p><pre>class ResNetSimCLR(L.LightningModule):<br>  def __init__(self, backbone, output_dim, batch_size, freeze=True, device=&#39;cuda&#39;):<br>    super().__init__()<br>    self.save_hyperparameters()  # save the parameters of model into check points<br>    self.represntation_num_of_features = backbone.fc.in_features<br>    self.features_extractor = nn.Sequential(*list(backbone.children())[:-1])  #  ResNet without the classifier<br><br>    if freeze:<br>      # freeze all parameters of the features extractor except to the 2 last blocks<br>      num_blocks = len(list(self.features_extractor.children()))<br>      for index_block, block in enumerate(list(self.features_extractor.children()), 1):<br>        if index_block &lt;= num_blocks-2:<br>          for param in block.parameters():<br>            param.requires_grad = False<br><br>    self.mlp = nn.Sequential(<br>        nn.Linear(self.represntation_num_of_features, 2 * self.represntation_num_of_features),<br>        nn.BatchNorm1d(2 * self.represntation_num_of_features),<br>        nn.SELU(inplace=True),<br>        nn.Linear(2*self.represntation_num_of_features, self.represntation_num_of_features),<br>        nn.BatchNorm1d(self.represntation_num_of_features),<br>        nn.SELU(inplace=True),<br>        nn.Linear(self.represntation_num_of_features, output_dim)<br>    )<br><br>    self.loss = NTXentLoss(batch_size, 0.05, device)<br><br>  def forward(self, x):<br>    h = self.features_extractor(x)<br>    h = h.squeeze()<br>    z = self.mlp(h)<br>    return h, z<br><br>  def training_step(self, batch, batch_idx):<br>    x, _ = batch<br>    xis, xjs = x  # two augmented views for each image<br>    _, zis = self(xis)<br>    _, zjs = self(xjs)<br><br>    loss = self.loss(zis, zjs)<br>    return loss<br><br>  def validation_step(self, batch, batch_idx):<br>    x, _ = batch<br>    xis, xjs = x  # two augmented views for each image<br>    _, zis = self(xis)<br>    _, zjs = self(xjs)<br><br>    loss = self.loss(zis, zjs)<br>    self.log(&#39;val_loss&#39;, loss)  # log to tensorboard<br>    return loss<br><br>  def configure_optimizers(self):<br>    optimizer = optim.Adam(self.parameters(), lr=1e-4)<br>    scheduler = OneCycleLR(optimizer, anneal_strategy=&#39;cos&#39;, max_lr=0.1, total_steps=1500)<br>    return [optimizer], [scheduler]</pre><p>In the code above we create a ResNet-based SimCLR model. We take a ResNet network as a backbone (without the last block which is a classifier), and use it as the feature extractor (the part that generates the representation vectors). The dimension of the representation is the output of the feature extractor, and it equals the number of features in the input of the original classifier of the ResNet.</p><p>To make the training simpler, we added an option to freeze most of the ResNet (we can do it since the ImageNet dataset the ResNet already trained on is similar to the STL-10). If one wants to use it, one must get a pre-trained ResNet to use the SimCLR class. Here, we will train all from scratch.</p><p>Finally, we create an MLP for the projection head and create an NTXentLoss object as the loss.</p><p>We define the forward method of the network, and the training step and the validation step for Pytorch Lightning. Pytorch Lighting will create the training loop for us and use these two methods for the training step and validation step for each mini-batch. We also ask Pytorch Lighting to track the validation loss. We also need to create the configure_optimizers method with the optimizer we want to use, and we also use a learning scheduler. Again, Pytorch Lightning uses this method to handle the updates of the weights by itself.</p><p>We are ready now to train the network, but first, we need to create dataloaders.</p><pre>batch_size = 128<br>train_dataloader = DataLoader(unlabeled_data, batch_size=batch_size, shuffle=True, drop_last=True, pin_memory=True, num_workers=num_worker)<br>val_dataloader = DataLoader(train_data, batch_size=batch_size, shuffle=False, drop_last=True, pin_memory=True, num_workers=num_worker)</pre><p>In the paper, you can see that the SimCLR network should be trained with a large batch size, but as most of us don’t have powerful GPUs, we will use a smaller batch size.</p><p>And now the training itself.</p><pre>resnet50 = models.resnet50(weights=None)<br>simclr = ResNetSimCLR(resnet50, 128, batch_size, device=device, freeze=False)<br><br># Pytorch Lightning Trainer<br>checkpoint_callback = ModelCheckpoint(monitor=&#39;val_loss&#39;, mode=&#39;min&#39;, save_last=True)<br>trainer = L.Trainer(accelerator=&quot;auto&quot;, max_epochs=100, callbacks=checkpoint_callback)<br>trainer.fit(model=simclr, train_dataloaders=train_dataloader, val_dataloaders=val_dataloader)</pre><p>First, we set the device to use. Next, we download ResNet-50 with torchvision without pre-trained weights (but you can use weights=’DEFAULT’ to get the pre-trained weights), and create the SimCLR network. We create a checkpoint callback that says to Pytorch Lightning to monitor the validation loss and save a checkpoint anytime the validation loss gets its minimum value, and also save the last step for a case we want to continue the training. Then, we define a Trainer object that will use a GPU if there is one (else use the CPU) for 100 epochs, and use the fit method to start the training.</p><h4>Classification</h4><p>With the SimCLR network ready, we can move to train a classifier based on it. We start by creating supervised datasets and dataloaders.</p><pre>train_data = STL10(<br>    root=DATASET_PATH,<br>    split=&quot;train&quot;,<br>    download=True,<br>    transform=transforms.ToTensor())<br><br>test_data = STL10(<br>    root=DATASET_PATH,<br>    split=&quot;test&quot;,<br>    download=True,<br>    transform=transforms.ToTensor()<br>)<br><br>batch_size = 128<br>train_dataloader = DataLoader(train_data, batch_size=batch_size, shuffle=True, drop_last=False, pin_memory=True, num_workers=num_worker)<br>test_dataloader = DataLoader(test_data, batch_size=batch_size, shuffle=False, drop_last=False, pin_memory=True, num_workers=num_worker)</pre><p>And then create a simple classifier network with Pytorch Lightning.</p><pre>class Classifier(L.LightningModule):<br>    def __init__(self, features_extractor, representation_dim, num_classes):<br>        super().__init__()<br>        self.save_hyperparameters()<br>        self.features_extractor = features_extractor<br>        self.classifier = nn.Sequential(<br>            nn.Linear(representation_dim, 2 * representation_dim),<br>            nn.BatchNorm1d(2 * representation_dim),<br>            nn.SELU(inplace=True),<br>            nn.Linear(2 * representation_dim, num_classes)<br>          )<br>        self.accuracy_metric = Accuracy(task=&#39;multiclass&#39;, num_classes=num_classes)<br>    <br>    def forward(self, x):<br>        x = self.features_extractor(x)<br>        x = x.squeeze()<br>        x = self.classifier(x)<br>        return x<br>    <br>    def training_step(self, batch, batch_idx):<br>        x, y = batch<br>        y_predicted = self(x)<br>        loss = F.cross_entropy(y_predicted, y)<br>        return loss<br>    <br>    def validation_step(self, batch, batch_idx):<br>        x, y = batch<br>        y_predicted = self(x)<br>        loss = F.cross_entropy(y_predicted, y)<br>        self.accuracy_metric.update(y_predicted, y)<br>        self.log(&#39;accuracy&#39;, self.accuracy_metric, on_step=False, on_epoch=True, prog_bar=True)<br>        return loss<br>    <br>    def configure_optimizers(self):<br>        optimizer = optim.Adam(self.parameters(), lr=3e-4)<br>        return optimizer</pre><p>We also use <a href="https://medium.com/python-in-plain-english/introduction-to-torch-metrics-5aff01f4731f">TorchMetrics</a> to track the accuracy of the classification.</p><p>And now we can train the classifier based on the SimCLR network.</p><pre>simclr = ResNetSimCLR.load_from_checkpoint(checkpoint_callback.best_model_path)<br>features_extractor = simclr.features_extractor<br>representation_dim = simclr.represntation_num_of_features<br>classifier = Classifier(features_extractor=features_extractor, representation_dim=representation_dim, num_classes=10)<br><br># Pytorch Lightning Trainer<br>checkpoint_callback_classifier = ModelCheckpoint(dirpath=&#39;Classifier&#39;, monitor=&#39;accuracy&#39;, mode=&#39;max&#39;, save_last=True)<br>trainer = L.Trainer(accelerator=&quot;auto&quot;, max_epochs=200, callbacks=checkpoint_callback_classifier)<br>trainer.fit(model=classifier, train_dataloaders=train_dataloader, val_dataloaders=test_dataloader)</pre><p>We load the best model to the SimCLR network (the weights that have the minimum validation loss) and take only the features extractor part of the SimCLR network for the classifier. Finally, we train it in a similar way to the SimCLR training, but this time we track the accuracy for saving checkpoints.</p><p>After the train finished, we can check the best accuracy we got using the checkpoint callback that tracks the accuracy.</p><pre>print(f&#39;Best accuracy is {checkpoint_callback_classifier.best_model_score}&#39;)<br># Best accuracy is 0.7772499918937683</pre><p>We succeeded in getting 77.7% accuracy with our self-supervised using only 5000 labeled images for 10 labels.</p><p>Let’s do it again, but this time we will use only 500 labeled images (10% of the total labeled images). First, we need to create a subset dataset of the training data.</p><pre>labels = [x for _, x in train_data]<br>indexes = list(range(len(labels)))<br><br>indexes_subset, *_ = train_test_split(indexes, labels, train_size=0.1, shuffle=True, stratify=labels)<br>train_data_subset = Subset(train_data, indexes_subset)<br>train_dataloader_subset = DataLoader(train_data_subset, batch_size=batch_size, shuffle=True, drop_last=False, pin_memory=True, num_workers=num_worker)</pre><p>We create a list of all the labels in the dataset, and a list of indexes with the same length as the dataset. Then, we use the train_test_split from scikit-learn to take 10% of the data by choosing 10% of the indexes. We use the stratify argument with the labels list, so in the result, we get the same distribution of labels as in the original dataset. Using the Subset class we create a subset of the original train dataset with the indexes we have chosen and create a dataloader from this subset dataset.</p><p>Now we can retrain the classification network with the new dataset.</p><pre>simclr = ResNetSimCLR.load_from_checkpoint(checkpoint_callback.best_model_path)<br>features_extractor = simclr.features_extractor<br>representation_dim = simclr.represntation_num_of_features<br>classifier_subset = Classifier(features_extractor=features_extractor, representation_dim=representation_dim, num_classes=10)<br><br># Pytorch Lightning Trainer<br>checkpoint_callback_classifier_subset = ModelCheckpoint(dirpath=&#39;ClassifierSubset&#39;,monitor=&#39;accuracy&#39;, mode=&#39;max&#39;, save_last=True)<br>trainer = L.Trainer(accelerator=&quot;auto&quot;, max_epochs=200, callbacks=checkpoint_callback_classifier_subset)<br>trainer.fit(model=classifier_subset, train_dataloaders=train_dataloader_subset, val_dataloaders=test_dataloader)</pre><p>And check the accuracy</p><pre>print(f&#39;Best accuracy is {checkpoint_callback_classifier_subset.best_model_score}&#39;)<br># Best accuracy is 0.6567500233650208</pre><p>We got 65.6% accuracy with our self-supervised using only 500 labeled images for 10 labels.</p><p>However, the interesting question is what accuracy we will get if we use supervised training only. Let’s check it.</p><h4>Supervised Learning</h4><p>In this section, we will build a similar network based on ResNet as a features extractor and a classification head similar to the previous network.</p><pre>class SupervisedClassifier(L.LightningModule):<br>    def __init__(self, resnet, num_classes, freeze=True):<br>        super().__init__()<br>        self.features_extractor = nn.Sequential(*list(resnet.children())[:-1])  #  ResNet without the classifier<br>        self.num_classes = num_classes<br>        self.classifier_head_input_dim = resnet.fc.in_features<br>        <br>        if freeze:<br>            # freeze all parameters of the features extractor except to the 2 last blocks<br>            num_blocks = len(list(self.features_extractor.children()))<br>            for index_block, block in enumerate(list(self.features_extractor.children()), 1):<br>                if index_block &lt;= num_blocks-2:<br>                  for param in block.parameters():<br>                    param.requires_grad = False<br>        <br>        self.classifier_head = nn.Sequential(<br>            nn.Linear(self.classifier_head_input_dim, 2 * self.classifier_head_input_dim),<br>            nn.BatchNorm1d(2 * self.classifier_head_input_dim),<br>            nn.SELU(inplace=True),<br>            nn.Linear(2 * self.classifier_head_input_dim, num_classes)<br>          )<br>        self.accuracy_metric = Accuracy(task=&#39;multiclass&#39;, num_classes=num_classes)<br>    <br>    def forward(self, x):<br>        x = self.features_extractor(x)<br>        x = x.squeeze()<br>        x = self.classifier_head(x)<br>        return x<br>    <br>    def training_step(self, batch, batch_idx):<br>        x, y = batch<br>        y_predicted = self(x)<br>        loss = F.cross_entropy(y_predicted, y)<br>        return loss<br>    <br>    def validation_step(self, batch, batch_idx):<br>        x, y = batch<br>        y_predicted = self(x)<br>        loss = F.cross_entropy(y_predicted, y)<br>        self.accuracy_metric.update(y_predicted, y)<br>        self.log(&#39;accuracy&#39;, self.accuracy_metric, on_step=False, on_epoch=True, prog_bar=True)<br>        return loss<br>    <br>    def configure_optimizers(self):<br>        optimizer = optim.Adam(self.parameters(), lr=3e-4)<br>        return optimizer</pre><p>And again use ResNet-50, and train the network with the full labeled dataset.</p><pre>resnet50 = models.resnet50(weights=None)<br>supervised_classifier = SupervisedClassifier(resnet50, 10, freeze=False)<br><br># Pytorch Lightning Trainer<br>checkpoint_callback_supervised_classifier = ModelCheckpoint(dirpath=&#39;SupervisedClassifier&#39;,monitor=&#39;accuracy&#39;, mode=&#39;max&#39;, save_last=True)<br>trainer = L.Trainer(accelerator=&quot;auto&quot;, max_epochs=200, callbacks=checkpoint_callback_supervised_classifier)<br>trainer.fit(model=supervised_classifier, train_dataloaders=train_dataloader, val_dataloaders=test_dataloader)</pre><p>And look at the accuracy.</p><pre>print(f&#39;Best accuracy is {checkpoint_callback_supervised_classifier.best_model_score}&#39;)<br># Best accuracy is 0.5256249904632568</pre><p>We got only 52.5% this time.</p><p>Let’s repeat the training we 10% of the labeled data.</p><pre>resnet50 = models.resnet50(weights=None)<br>supervised_classifier_subset = SupervisedClassifier(resnet50, 10, freeze=False)<br><br># Pytorch Lightning Trainer<br>checkpoint_callback_supervised_classifier_subset = ModelCheckpoint(dirpath=&#39;SupervisedClassifierSubset&#39;,monitor=&#39;accuracy&#39;, mode=&#39;max&#39;, save_last=True)<br>trainer = L.Trainer(accelerator=&quot;auto&quot;, max_epochs=200, callbacks=checkpoint_callback_supervised_classifier_subset)<br>trainer.fit(model=supervised_classifier_subset, train_dataloaders=train_dataloader_subset, val_dataloaders=test_dataloader)</pre><pre>print(f&#39;Best accuracy is {checkpoint_callback_supervised_classifier_subset.best_model_score}&#39;)<br># Best accuracy is 0.30937498807907104</pre><p>We got a poor result of 30.9% accuracy.</p><h3>Conclusion</h3><p>Let’s repeat the results we got for accuracy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IzcA9Mb2bJi4Ji5V1Qswsg.png" /><figcaption>Accuracy scores for self-supervised and full supervised training</figcaption></figure><p>Using self-supervision we got much higher accuracy than by using supervised learning. Even with just 500 labeled images (only 50 images per class), we got a higher score than fully supervised learning using 5000 images.</p><p>This experiment shows us the potential of self-supervised learning, and that we can leverage unlabelled data (which is easier to collect than labeled data) to improve the performance of our data.</p><h3>In Plain English 🚀</h3><p><em>Thank you for being a part of the </em><a href="https://plainenglish.io/"><strong><em>In Plain English</em></strong></a><em> community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a> | <a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our other platforms: <a href="https://cofeed.app/"><strong>CoFeed</strong></a> | <a href="https://differ.blog/"><strong>Differ</strong></a></li><li>More content at <a href="https://plainenglish.io/"><strong>PlainEnglish.io</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b242bde77241" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/self-supervised-learning-for-visual-representation-b242bde77241">Self-Supervised Learning for Visual Representation</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Visual Transformer (ViT)]]></title>
            <link>https://python.plainenglish.io/visual-transformer-vit-4eaa90fb936b?source=rss-aaf4831d9b34------2</link>
            <guid isPermaLink="false">https://medium.com/p/4eaa90fb936b</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Dor Zvulun]]></dc:creator>
            <pubDate>Fri, 30 Aug 2024 09:24:38 GMT</pubDate>
            <atom:updated>2024-09-02T01:07:36.453Z</atom:updated>
            <content:encoded><![CDATA[<h4>An introduction to the Visual Transformer architecture, its main ideas and implementation.</h4><p>The Visual Transformer (ViT) architecture is a groundbreaking advancement in computer vision, offering a new way for machines to understand images. Unlike traditional methods that use convolutional neural networks (CNNs), Visual Transformers use self-attention mechanisms to process entire images at once. This approach captures global context and details more effectively, leading to improved performance in tasks like image classification and object detection.</p><p>In previous posts, we talked about the<a href="https://medium.com/@dzdata/attention-what-is-it-and-how-it-works-9ee0dcb97e53"> attention</a> and the <a href="https://medium.com/@dzdata/from-attention-to-self-attention-and-transformers-a-mathematical-introduction-45ed053a1ed3">self-attention</a> mechanisms. In this article, we explore the basics of the Visual Transformer architecture, which is based on the self-attention idea, by following the paper “<a href="https://arxiv.org/pdf/2010.11929">An image is worth 16X16 words</a>”.</p><h3>The Visual Transformer Architecture</h3><p>Let’s start by reviewing the architecture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zAaSZMaIXPbB7coT033ipw.png" /><figcaption>Model overview of ViT. Source: <a href="https://arxiv.org/pdf/2010.11929">An image is worth 16X16 words</a></figcaption></figure><p>In the first step, we take an image and split it into patches of 16x16 pixels. More formally, we start with an image, which is a tensor of size [H×W×C], where H is the height of the image, W is the width and C is the number of channels (3 in the case of an RGB image), divide it into patches, and flatten each of them so we get N vectors from the original image, each of them of size [P²⋅C], where [P×P] is the size of the patch (for example, 16×16). The number of patches, N, is given by N = HW/P².</p><p>Now we have N vectors of size [P²⋅C] that contain the same content as the original image (we just divided the image into patches and flattened each one of them). In the next step, we change the representation of the new vector by learning a new embedding representation. We set a dimension for the new patch embedding, D, and use a learnable linear transformation to map the flattened image patches into the new embedding representation. That means we map each of the N vectors from the original flattened image patch of dimension [P²⋅C] into a new vector of dimension D (which can be the same dimension, [P²⋅C]).</p><p>Mathematically, we mark the flattened patch as vector <strong><em>x</em></strong>ₚⁱ and the linear transformation as <strong><em>E</em></strong>, so the embedding is given by</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/689/0*nBr2jd3gv-58JrW6.png" /></figure><p>In addition to the image patch embedding, an additional embedding is added as class embedding. This embedding should represent at the output the content of the image (e.g., an image of a dog or a cat). In the final step, we use its output as input to the classification head to determine the image class, as seen in the figure above. Like the other embedding, we will learn it during training.</p><p>If you recall from the previous <a href="https://dzdata.medium.com/from-attention-to-self-attention-and-transformers-a-mathematical-introduction-45ed053a1ed3">post</a>, the self-attention mechanism, upon which the transformer is built, is permutation equivariant, meaning it doesn&#39;t aware of the order of the inputs. But in an image the order matters. So, we want to add <strong>positional embedding</strong> to the patches, so the transformers will be aware of the order of the sequence of patches.</p><p>For the positional embedding we can start with some random matrix Eₚₒₛ with size [(N+1)× D] (N patches + 1 class embedding) and learn the representation for the position of the patches.</p><p>All we described so far is formulated in the paper in equation (1) as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Um9Dkep5HwC8BZ2Rr555Cw.png" /></figure><p>The matrix z₀ (which is the concatenation of all the image patches embedding and the class embedding with the addition of the positional embedding) is the input we will use for the transformer encoder as in the figure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/722/1*GwmzBxi2V4JlAWMcPmaPaA.png" /></figure><h4>The Transformer Encoder</h4><p>The main processing of the data is performed inside the transformer encoder. This block is composed of transformer blocks with the following structure:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YaCJMGhuHBQdgMDtkVMC4Q.png" /></figure><p>The inputs start with a layer norm that normalizes each of the inputs independently and is then processed by a multi-head self-attention block (MSA). We add the input of the transformer block to the output of the multi-head self-attention using skip connection and again apply layer norm. The output of the layer norm goes through an MLP and we use a skip connection once again to get the output of the whole transformer block. This process repeats itself for L such blocks. If we mark the number of the transformer block as <em>l</em>, where <em>l</em> goes from 1 to L we can write the following formulas to describe the transformer block.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*xWnr9f_eR1rgbDCMwjoAvA.png" /></figure><p>After all L transformer blocks, we will take the output of the 0-th index and apply once more layer normalization to get the final output of the transformer encoder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/221/1*FOhbgI2DmtceodiAbEp1lA.png" /></figure><p>Finally, we insert the output y into the classification MLP head to determine the class of the image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/294/1*nb0OxR_tais-gBbR6-cVEw.png" /></figure><p>And that’s it! Now it is time to code it by ourselves.</p><h3>Let’s Code It</h3><p>In this section, we will implement and train a visual transformer. We will be based on the original paper “<a href="https://arxiv.org/pdf/2010.11929">An image is worth 16X16 words</a>” together with this <a href="https://www.learnpytorch.io/08_pytorch_paper_replicating/#1-get-data">blog</a>.</p><p>A full notebook can be found <a href="https://github.com/dorzv/ComputerVision/blob/master/vision_transformer/ViT.ipynb">here</a>.</p><p>Let’s start with some basic imports.</p><pre>import torch<br>from torch import nn<br>from torch.utils.data import DataLoader<br>from torchvision import transforms<br>from torchvision.datasets import ImageFolder<br>import matplotlib.pyplot as plt</pre><p>and we also install torchinfo which is a tool that let us see the structure of the network easily.</p><pre>! pip install -q torchinfo<br>import torchinfo</pre><p>Also, we set the device to use (either a GPU or a CPU in a case where a GPU isn’t available).</p><pre>device = &#39;cuda&#39; if torch.cuda.is_available() else &#39;cpu&#39;</pre><h4>The Data</h4><p>Next, we want to get images to work with and arrange them so we can use them to train the network. The first step will be to get the data, and for that, we will write a function that downloads the data and unzips it.</p><pre>import os<br>import zipfile<br>import requests <br><br>def download_data(source_link, destination_folder, remove_zip=True):<br>  &quot;&quot;&quot;<br>  A function to get a zip of data from a web, download it and unzip it into a<br>  folder. Can remove the zip after extraction.<br><br>  Args:<br>    source_link (str): a limk to the zip file.<br>    destination_folder (str): the name of the folder where the zip will be <br>      extracted.<br>    remove_zip (bool): whether to remove the zip after extraction or not. <br>      Default is True.<br>  <br>  Return:<br>    (str): path to the destination folder<br>  &quot;&quot;&quot;<br><br>  image_path = os.path.join(&#39;data&#39;, destination_folder)<br>  target_path = os.path.join(&#39;data&#39;, os.path.basename(source_link))<br>  if os.path.isdir(image_path):<br>    print(&#39;Folder is already exists. Skipping donwload&#39;)<br>    return image_path<br>  <br>  # Download zip<br>  os.makedirs(image_path)<br>  with open(target_path, &#39;wb&#39;) as file:<br>    request = requests.get(source_link)<br>    print(&#39;Downloading zip file...&#39;)<br>    file.write(request.content)<br>  <br>  with zipfile.ZipFile(target_path, &#39;r&#39;) as zip_file:<br>    print(&#39;Unzipping...&#39;)<br>    zip_file.extractall(image_path)<br><br>  if remove_zip:<br>    os.remove(target_path)<br>  <br>  return image_path</pre><p>As in the original post, we are going to use a dataset that contains images of pizzas, steaks, and sushi, and can be found <a href="https://github.com/mrdbourke/pytorch-deep-learning/tree/main/data">here</a>. Using the previous function we can download it and set variables for the train and test data folders.</p><pre>image_folder = download_data(&quot;https://github.com/mrdbourke/pytorch-deep-learning/raw/main/data/pizza_steak_sushi.zip&quot;, &quot;pizza_steak_sushi&quot;)<br>train_data_path = os.path.join(image_folder, &#39;train&#39;)<br>test_data_path = os.path.join(image_folder, &#39;test&#39;)</pre><p>As we work with Pytorch, we want to use a Dataset to handle our data. Because we use a very simple dataset we are not required to implement a Dataset class by ourselves, but can use instead the ImageFolder class from torchvision. This class is a generic dataset for images organized in a directory structure, where images are stored in subdirectories representing different classes.</p><pre>root/<br>├── class1/<br>│   ├── img1.png<br>│   ├── img2.png<br>│   └── ...<br>├── class2/<br>│   ├── img1.png<br>│   ├── img2.png<br>│   └── ...</pre><p>The ImageFolder class will return us an image together with its label (class) according to the folder&#39;s structure, without the need for us to define anything.</p><p>We want to apply some transformations to the images, so we define transforms and pass them too as input to the ImageFolder class.</p><pre>transformation = transforms.Compose([<br>    transforms.Resize((224, 224)),<br>    transforms.ToTensor()<br>])<br><br># create Datasets<br>train_dataset = ImageFolder(train_data_path, transformation)<br>test_dataset = ImageFolder(test_data_path, transformation)</pre><p>The labels will be sorted alphabetically and can be seen using</p><pre>classes_name = train_dataset.classes</pre><p>In our case label 0 is pizza, 1 is steak, and 2 is sushi.</p><p>After creating datasets for the train and the test data, we can create a DataLoader based on them.</p><pre>train_dataloader = DataLoader(train_dataset, batch_size=32, shuffle=True, <br>                              num_workers=os.cpu_count(), pin_memory=True)<br>test_dataloader = DataLoader(test_dataset, batch_size=32, shuffle=False,<br>                             num_workers=os.cpu_count(), pin_memory=True)</pre><p>We can check ourselves and visualize a sample from the dataset.</p><pre># see an example of image<br>image_batch, label_batch = next(iter(train_dataloader))<br>image = image_batch[0]<br>label = label_batch[0]<br>print(f&#39;image size = {image.shape}, label = {label}&#39;)<br>image_permute = image.permute(1, 2, 0)<br>plt.imshow(image_permute)<br>plt.title(classes_name[label])<br>plt.axis(False)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/483/1*CD4DlW13zIP7eD5Jug3UsQ.png" /></figure><p>To better understand the patching we previously discussed, we can visualize the image as patches of 16 by 16 pixels.</p><pre># visualize image as patches<br>image_size = image_permute.shape[0]<br>patch_size = 16<br>num_of_patches = image_size // patch_size<br>if image_size % patch_size != 0:<br>  raise ValueError(&quot;Image size must be divisible by patch size&quot;)<br><br>fig, axes = plt.subplots(nrows=num_of_patches, ncols=num_of_patches, <br>                       figsize=(num_of_patches, num_of_patches), sharex=True,<br>                       sharey=True)<br><br># plot each patch in. a subplot<br>for ii, patch_height in enumerate(range(0, image_size, patch_size)):<br>  for jj, patch_width in enumerate(range(0, image_size, patch_size)):<br>    axes[ii, jj].imshow(image_permute[patch_height:patch_height+patch_size,<br>                                          patch_width:patch_width+patch_size,<br>                                          :])<br>    <br>    # set labels<br>    axes[ii, jj].set_ylabel(ii+1, rotation=&quot;horizontal&quot;,<br>                            horizontalalignment=&quot;right&quot;, <br>                            verticalalignment=&quot;center&quot;) <br>    axes[ii, jj].set_xlabel(jj+1)<br>    axes[ii, jj].set_xticks([])<br>    axes[ii, jj].set_yticks([])<br>    axes[ii, jj].label_outer()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mWXoij5CzFZ1oR-MtJu_Ww.png" /></figure><h4>The Network</h4><p>Now, it’s the time to build the network. We will build building blocks first, and then use them to build the visual transformer. Let’s start with the block that takes an image, divides it into patches, and transforms them into embedded representation.</p><pre>  class PatchEmbedding(nn.Module):<br>  &quot;&quot;&quot;<br>  Convert a 2D image into a squence of 1D vector of embedding represntation<br>  of patches on the image.<br>  <br>  Args:<br>    in_channel (int): the number of channels of the input image (default 3)<br>    patch_size (int): the size of each patch of the image (default 16)<br>    embedding_size (int): the size of the embedding vector (defualt 768)<br>  &quot;&quot;&quot;<br>  def __init__(self, in_channels=3, patch_size=16, embedding_size=768):<br>    super().__init__()<br><br>    self.patch_size = patch_size<br><br>    self.patcher = nn.Conv2d(in_channels, embedding_size, patch_size, stride=patch_size)<br>    self.flatten = nn.Flatten(start_dim=2, end_dim=3)<br><br>  def forward(self, x):<br>    image_size = x.shape[-1]<br>    if image_size % self.patch_size != 0:<br>      raise ValueError(f&#39;Patch size {self.patch_size} is not compatible with the image size {image_size}&#39;)<br>    <br>    x_patched = self.patcher(x)<br>    x_flatten = self.flatten(x_patched)<br><br>    return x_flatten.permute((0, 2, 1))</pre><p>We use 2D convolution with a kernel of the size of the patch (with the same stride size) to divide the image into patches, and set the output number of channels to be the embedding size. Then, we flatten the spatial shape of the result to get 1D vectors of the embedding size. Before we return the result, we permute the result so the embedding dimension will be the last one.</p><p>We can check our block with the following code.</p><pre>patch_size = 16<br>print(f&#39;Image tensor shape is {image.shape}&#39;)<br>height, width = image.shape[1:3]<br><br># Add batch dimension<br>x = image.unsqueeze(0)<br><br>patch_embedding_layer = PatchEmbedding(in_channels=3, patch_size=patch_size, embedding_size=768)<br>patch_embedding = patch_embedding_layer(x)<br>print(f&#39;Patch embedding shape is {patch_embedding.shape}&#39;)<br><br><br>###########  Print ###########<br>&quot;&quot;&quot;<br>Image tensor shape is torch.Size([3, 224, 224])<br>Patch embedding shape is torch.Size([1, 196, 768])<br>&quot;&quot;&quot;</pre><p>In addition to the patch embedding, we want to add the class embedding. We initialize this embedding randomly and set it as Pytorch parameters so our network can learn it by itself. Finally, we concatenate the class embedding and the patch embedding.</p><pre>class_token = nn.Parameter(torch.randn(patch_embedding.shape[0], 1, patch_embedding.shape[-1]),<br>                          requires_grad=True)<br>print(f&#39;Class token embedding shape is {class_token.shape}&#39;)<br><br># Append the class token embedding to the patches embedding<br>patch_embedding_with_class_embedding = torch.cat((class_token, patch_embedding), dim=1)<br>print(f&#39;Class token  embedding with patch embedding shape is {patch_embedding_with_class_embedding.shape}&#39;)<br><br>###########  Print ###########<br>&quot;&quot;&quot;<br>Class token embedding shape is torch.Size([1, 1, 768])<br>Class token  embedding with patch embedding shape is torch.Size([1, 197, 768])<br>&quot;&quot;&quot;</pre><p>Similarly, we initialize the position embedding, but this time we add it to the other embeddings.</p><pre># Position embedding<br>position_embedding = nn.Parameter(torch.randn_like(patch_embedding_with_class_embedding),<br>                                  requires_grad=True)<br><br># Add the position embedding to the patches embedding<br>patch_and_position_embedding = patch_embedding_with_class_embedding + position_embedding<br>print(f&quot;Patch and position embedding shape is {patch_and_position_embedding.shape}&quot;)<br><br>###########  Print ###########<br>&quot;&quot;&quot;<br>Patch and position embedding shape is torch.Size([1, 197, 768])<br>&quot;&quot;&quot;</pre><p>Next, we write the multihead self-attention block. Pytorch already implements this kind of block, so we don’t need to write it by ourselves. Instead, we just wrap the Pytorch block with our class so it also includes layer norm block.</p><pre>class MultiheadSelfAttentionBlock(nn.Module):<br>  &quot;&quot;&quot;<br>  Create a multihead self attention block with layer normalization<br>  Args:<br>    embedding_dim (int): the hidden size - D<br>    num_heads (int): number of heads for the multihead self attention<br>    dropout (float): dropout probability <br>  &quot;&quot;&quot;<br><br>  def __init__(self, embedding_dim=768, num_heads=12, dropout=0):<br>      super().__init__()<br>      self.layer_norm = nn.LayerNorm(normalized_shape=embedding_dim)<br>      self.multihead_attention = nn.MultiheadAttention(embed_dim=embedding_dim,<br>                                                       num_heads=num_heads,<br>                                                       dropout=dropout,<br>                                                       batch_first=True)<br>  <br>  def forward(self, x):<br>    x = self.layer_norm(x)<br>    attention_output, _ = self.multihead_attention(query=x, key=x, value=x, need_weights=False)<br><br>    return attention_output</pre><p>We also need to write the MLP block that will be used in the transformer encoder. Pay attention it also contains the second layer norm of the encoder.</p><pre>class MLPBlock(nn.Module):<br>  &quot;&quot;&quot;<br>  Create a multilayer preceptron block with layer normalization<br><br>  Args:<br>    embedding_dim (int): the hidden size - D, used as input and output size<br>    mlp_size (int): the dimenstion of the hidden layer<br>    dropout (float): dropout probability <br>  &quot;&quot;&quot;<br><br>  def __init__(self, embedding_dim=768, mlp_size=3072, dropout=0.1):<br>    super().__init__()<br>    self.layer_norm = nn.LayerNorm(normalized_shape=embedding_dim)<br>    self.mlp = nn.Sequential(<br>        nn.Linear(embedding_dim, mlp_size),<br>        nn.GELU(),<br>        nn.Dropout(dropout),<br>        nn.Linear(mlp_size, embedding_dim),<br>        nn.Dropout(dropout)<br>    )<br><br>  def forward(self, x):<br>    x = self.layer_norm(x)<br>    x = self.mlp(x)<br>    return x</pre><p>With the self-attention block and the MLP, we can write the transformer encoder block.</p><pre>class TransformerEncoderBlock(nn.Module):<br>  &quot;&quot;&quot;<br>  Create the transformer encoder block<br><br>  Args:<br>    embedding_dim (int): the hidden size - D<br>    num_heads (int): number of heads for the multihead self attention<br>    mlp_size (int): the dimenstion of the hidden layer<br>    attention_dropout (float): dropout probability for the MSA <br>    mlp_dropout (float): dropout probability for the MLP<br>  &quot;&quot;&quot;<br><br>  def __init__(self, embedding_dim=768, num_heads=12, mlp_size=3072,<br>               attention_dropout=0, mlp_dropout=0.1):<br>    super().__init__()<br>    self.msa_block = MultiheadSelfAttentionBlock(embedding_dim=embedding_dim,<br>                                                 num_heads=num_heads,<br>                                                 dropout=attention_dropout)<br>    self.mlp_block =  MLPBlock(embedding_dim=embedding_dim,<br>                               mlp_size=mlp_size,<br>                               dropout=mlp_dropout)<br>    <br>  def forward(self, x):<br>    x =  self.msa_block(x) + x <br>    x = self.mlp_block(x) + x<br>    return x</pre><p>And now we are finally ready to write the visual transformer.</p><pre>class ViT(nn.Module):<br>  &quot;&quot;&quot;<br>  Create a Visuale Transformer. The defualt values of the hyperparameters are<br>  as of ViT-Base<br><br>  Args:<br>    img_size (int): image resolution. Default is 224 as in Table 3 in ViT paper<br>    in_channel (int): the number of channels of the input image. Default 3<br>    patch_size (int): the size (height and width) of the patch. Default 16<br>    num_transformer_layers (int): The number of transformers layers. Default is<br>      12 as in Table 1 in ViT paper for ViT base<br>    embedding_dim (int): hidden size D. Default is 768 as in Table 1 in ViT<br>      paper for ViT-Base<br>    mlp_size (int): MLP size. Default is 3072 as in Table 1 in ViT paper for ViT<br>      -Base<br>    num_heads (int): number of heads for the multihead self attention blocks.<br>      Default is 12 as in Table 1 in ViT paper for ViT-Base<br>    attention_dropout (float): dropout probability for the MSA block<br>    mlp_dropout (float): dropout probability for the MLP block<br>    embedding_dropout (float): dropout probability for patch and position<br>      embeddings<br>    num_classes (int): number of classes in the output. Default is 1000 as in <br>      ImageNet<br>  &quot;&quot;&quot;<br><br>  def __init__(self, <br>               img_size=224,<br>               in_channels=3,<br>               patch_size=16,<br>               num_transformer_layers=12,<br>               embedding_dim=768, <br>               mlp_size=3072,<br>               num_heads=12, <br>               attention_dropout=0,<br>               mlp_dropout=0.1, <br>               embedding_dropout=0.1,<br>               num_classes=1000):<br>    super().__init__()<br><br>    if img_size % patch_size != 0:<br>      raise ValueError(f&quot;Image size must be divisible by patch size, image size: {img_size}, patch size: {patch_size}.&quot;)<br><br>    self.num_patches = img_size ** 2 // patch_size ** 2<br>    <br>    # Create learnable class embedding (needs to go at front of sequence of patch embeddings)<br>    self.class_embedding = nn.Parameter(data=torch.randn(1, 1, embedding_dim),<br>                                        requires_grad=True)<br>    <br>    # Create learnable position embedding<br>    self.position_embedding = nn.Parameter(<br>        data=torch.randn(1, self.num_patches+1, embedding_dim),<br>        requires_grad=True)<br>    <br>    self.embedding_dropout = nn.Dropout(embedding_dropout)<br><br>    self.patch_embedding = PatchEmbedding(in_channels=in_channels,<br>                                              patch_size=patch_size,<br>                                              embedding_size=embedding_dim)<br>    <br>    self.transformer_encoder = nn.Sequential(<br>        *[TransformerEncoderBlock(embedding_dim=embedding_dim,<br>                                  num_heads=num_heads,<br>                                  mlp_size=mlp_size,<br>                                  mlp_dropout=mlp_dropout) <br>         for _ in range(num_transformer_layers)]<br>         )<br>    <br>    self.classifier = nn.Sequential(<br>            nn.LayerNorm(normalized_shape=embedding_dim),<br>            nn.Linear(in_features=embedding_dim, <br>                      out_features=num_classes)<br>        )<br>    <br>  def forward(self, x):<br>    batch_size = x.shape[0]<br>    class_token = self.class_embedding.expand(batch_size, -1, -1)<br>    <br>    x = self.patch_embedding(x)<br>    x = torch.cat((class_token, x), dim=1)<br>    x = self.position_embedding + x<br>    x = self.embedding_dropout(x)<br>    x = self.transformer_encoder(x)<br>    x = self.classifier(x[:, 0])<br><br>    return x</pre><p>The visual transformer is built as we describe in the first section, but you can see we also added dropout to the embedding to improve the generalization of the network.</p><p>If you want to better understand the structure of the visual transformer we wrote, you can use torchinfo to output the structure</p><pre>from torchinfo import summary<br>vit = ViT(num_classes=len(classes_name))<br><br>summary(model=vit, <br>        input_size=(32, 3, 224, 224), # (batch_size, color_channels, height, width)<br>        col_names=[&quot;input_size&quot;, &quot;output_size&quot;, &quot;num_params&quot;, &quot;trainable&quot;],<br>        col_width=20,<br>        row_settings=[&quot;var_names&quot;]<br> )</pre><pre>=================================================================================================================================================<br>Layer (type (var_name))                                           Input Shape          Output Shape         Param #              Trainable<br>=================================================================================================================================================<br>ViT (ViT)                                                         [32, 3, 224, 224]    [32, 3]              152,064              True<br>├─PatchEmbedding (patch_embedding)                                [32, 3, 224, 224]    [32, 196, 768]       --                   True<br>│    └─Conv2d (patcher)                                           [32, 3, 224, 224]    [32, 768, 14, 14]    590,592              True<br>│    └─Flatten (flatten)                                          [32, 768, 14, 14]    [32, 768, 196]       --                   --<br>├─Dropout (embedding_dropout)                                     [32, 197, 768]       [32, 197, 768]       --                   --<br>├─Sequential (transformer_encoder)                                [32, 197, 768]       [32, 197, 768]       --                   True<br>│    └─TransformerEncoderBlock (0)                                [32, 197, 768]       [32, 197, 768]       --                   True<br>│    │    └─MultiheadSelfAttentionBlock (msa_block)               [32, 197, 768]       [32, 197, 768]       2,363,904            True<br>│    │    └─MLPBlock (mlp_block)                                  [32, 197, 768]       [32, 197, 768]       4,723,968            True<br>│    └─TransformerEncoderBlock (1)                                [32, 197, 768]       [32, 197, 768]       --                   True<br>│    │    └─MultiheadSelfAttentionBlock (msa_block)               [32, 197, 768]       [32, 197, 768]       2,363,904            True<br>│    │    └─MLPBlock (mlp_block)                                  [32, 197, 768]       [32, 197, 768]       4,723,968            True<br>│    └─TransformerEncoderBlock (2)                                [32, 197, 768]       [32, 197, 768]       --                   True<br>│    │    └─MultiheadSelfAttentionBlock (msa_block)               [32, 197, 768]       [32, 197, 768]       2,363,904            True<br>│    │    └─MLPBlock (mlp_block)                                  [32, 197, 768]       [32, 197, 768]       4,723,968            True<br>│    └─TransformerEncoderBlock (3)                                [32, 197, 768]       [32, 197, 768]       --                   True<br>│    │    └─MultiheadSelfAttentionBlock (msa_block)               [32, 197, 768]       [32, 197, 768]       2,363,904            True<br>│    │    └─MLPBlock (mlp_block)                                  [32, 197, 768]       [32, 197, 768]       4,723,968            True<br>│    └─TransformerEncoderBlock (4)                                [32, 197, 768]       [32, 197, 768]       --                   True<br>│    │    └─MultiheadSelfAttentionBlock (msa_block)               [32, 197, 768]       [32, 197, 768]       2,363,904            True<br>│    │    └─MLPBlock (mlp_block)                                  [32, 197, 768]       [32, 197, 768]       4,723,968            True<br>│    └─TransformerEncoderBlock (5)                                [32, 197, 768]       [32, 197, 768]       --                   True<br>│    │    └─MultiheadSelfAttentionBlock (msa_block)               [32, 197, 768]       [32, 197, 768]       2,363,904            True<br>│    │    └─MLPBlock (mlp_block)                                  [32, 197, 768]       [32, 197, 768]       4,723,968            True<br>│    └─TransformerEncoderBlock (6)                                [32, 197, 768]       [32, 197, 768]       --                   True<br>│    │    └─MultiheadSelfAttentionBlock (msa_block)               [32, 197, 768]       [32, 197, 768]       2,363,904            True<br>│    │    └─MLPBlock (mlp_block)                                  [32, 197, 768]       [32, 197, 768]       4,723,968            True<br>│    └─TransformerEncoderBlock (7)                                [32, 197, 768]       [32, 197, 768]       --                   True<br>│    │    └─MultiheadSelfAttentionBlock (msa_block)               [32, 197, 768]       [32, 197, 768]       2,363,904            True<br>│    │    └─MLPBlock (mlp_block)                                  [32, 197, 768]       [32, 197, 768]       4,723,968            True<br>│    └─TransformerEncoderBlock (8)                                [32, 197, 768]       [32, 197, 768]       --                   True<br>│    │    └─MultiheadSelfAttentionBlock (msa_block)               [32, 197, 768]       [32, 197, 768]       2,363,904            True<br>│    │    └─MLPBlock (mlp_block)                                  [32, 197, 768]       [32, 197, 768]       4,723,968            True<br>│    └─TransformerEncoderBlock (9)                                [32, 197, 768]       [32, 197, 768]       --                   True<br>│    │    └─MultiheadSelfAttentionBlock (msa_block)               [32, 197, 768]       [32, 197, 768]       2,363,904            True<br>│    │    └─MLPBlock (mlp_block)                                  [32, 197, 768]       [32, 197, 768]       4,723,968            True<br>│    └─TransformerEncoderBlock (10)                               [32, 197, 768]       [32, 197, 768]       --                   True<br>│    │    └─MultiheadSelfAttentionBlock (msa_block)               [32, 197, 768]       [32, 197, 768]       2,363,904            True<br>│    │    └─MLPBlock (mlp_block)                                  [32, 197, 768]       [32, 197, 768]       4,723,968            True<br>│    └─TransformerEncoderBlock (11)                               [32, 197, 768]       [32, 197, 768]       --                   True<br>│    │    └─MultiheadSelfAttentionBlock (msa_block)               [32, 197, 768]       [32, 197, 768]       2,363,904            True<br>│    │    └─MLPBlock (mlp_block)                                  [32, 197, 768]       [32, 197, 768]       4,723,968            True<br>├─Sequential (classifier)                                         [32, 768]            [32, 3]              --                   True<br>│    └─LayerNorm (0)                                              [32, 768]            [32, 768]            1,536                True<br>│    └─Linear (1)                                                 [32, 768]            [32, 3]              2,307                True<br>=================================================================================================================================================<br>Total params: 85,800,963<br>Trainable params: 85,800,963<br>Non-trainable params: 0<br>Total mult-adds (G): 5.52<br>=================================================================================================================================================<br>Input size (MB): 19.27<br>Forward/backward pass size (MB): 3292.20<br>Params size (MB): 229.20<br>Estimated Total Size (MB): 3540.67<br>=============================================================================================================================================</pre><p>All we have to do now is to train the model. For that, we will build a training loop.</p><pre># train the model<br>optimizer = torch.optim.Adam(vit.parameters(), lr=3e-3, betas=(0.9, 0.999),<br>                             weight_decay=0.3)<br>loss_fn = torch.nn.CrossEntropyLoss()<br><br>epochs = 10<br><br>vit.to(device)<br><br>train_loss = []<br>train_acc = []<br>test_loss = []<br>test_acc = []<br><br>for epoch in range(epochs):<br>  vit.train()<br>  train_step_loss = 0<br>  train_step_acc = 0<br><br>  for batch_ii, (images_batch, labels_batch) in enumerate(train_dataloader):<br>    images_batch = images_batch.to(device)<br>    labels_batch = labels_batch.to(device)<br><br>    labels_predictions = vit(images_batch)<br>    loss = loss_fn(labels_predictions, labels_batch)<br>    train_step_loss += loss.item()<br>    train_step_acc += torch.sum(<br>        torch.argmax(labels_predictions, dim=1) == labels_batch<br>        ) / len(labels_batch)<br><br>    optimizer.zero_grad()<br>    loss.backward()<br>    optimizer.step()<br><br>  train_step_loss = train_step_loss / len(train_dataloader)<br>  train_step_acc = train_step_acc / len(train_dataloader)<br><br>  test_step_loss = 0<br>  test_step_acc = 0<br><br>  vit.eval()<br>  with torch.inference_mode():<br>    for batch_ii, (images_batch, labels_batch) in enumerate(test_dataloader):<br>      images_batch = images_batch.to(device)<br>      labels_batch = labels_batch.to(device)<br><br>      labels_predictions = vit(images_batch)<br>      loss = loss_fn(labels_predictions, labels_batch)<br>      test_step_loss += loss.item()<br>      test_step_acc += torch.sum(<br>        torch.argmax(labels_predictions, dim=1) == labels_batch<br>        ) / len(labels_batch)<br>    <br>  test_step_loss = test_step_loss / len(train_dataloader)<br>  test_step_acc = test_step_acc / len(test_dataloader)<br><br>  print(<br>  f&quot;Epoch: {epoch+1} | &quot;<br>  f&quot;train_loss: {train_step_loss:.4f} | &quot;<br>  f&quot;train_accuracy: {train_step_acc:.4f} | &quot;<br>  f&quot;test_loss: {test_step_loss:.4f} | &quot;<br>  f&quot;test_accuracy: {test_step_acc:.4f}&quot;<br>  )<br><br>  train_loss.append(train_step_loss)<br>  test_loss.append(test_step_loss)</pre><h3>Conclusion</h3><p>The Visual Transformer architecture introduces a powerful new approach to computer vision by using self-attention mechanisms instead of traditional CNNs. This allows for a better understanding of global context and details in images. By treating images as sequences of patches, ViT enhances performance in tasks like image classification. We saw the architecture of the visual transformer, understood its basic ideas and components, and finally implemented and trained the basic version of it.</p><h3>In Plain English 🚀</h3><p><em>Thank you for being a part of the </em><a href="https://plainenglish.io/"><strong><em>In Plain English</em></strong></a><em> community! Before you go:</em></p><ul><li>Be sure to <strong>clap</strong> and <strong>follow</strong> the writer ️👏<strong>️️</strong></li><li>Follow us: <a href="https://twitter.com/inPlainEngHQ"><strong>X</strong></a> | <a href="https://www.linkedin.com/company/inplainenglish/"><strong>LinkedIn</strong></a> | <a href="https://www.youtube.com/channel/UCtipWUghju290NWcn8jhyAw"><strong>YouTube</strong></a> | <a href="https://discord.gg/in-plain-english-709094664682340443"><strong>Discord</strong></a> | <a href="https://newsletter.plainenglish.io/"><strong>Newsletter</strong></a></li><li>Visit our other platforms: <a href="https://cofeed.app/"><strong>CoFeed</strong></a> | <a href="https://differ.blog/"><strong>Differ</strong></a></li><li>More content at <a href="https://plainenglish.io/"><strong>PlainEnglish.io</strong></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4eaa90fb936b" width="1" height="1" alt=""><hr><p><a href="https://python.plainenglish.io/visual-transformer-vit-4eaa90fb936b">Visual Transformer (ViT)</a> was originally published in <a href="https://python.plainenglish.io/">Python in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>