<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yutong Zhang on Medium]]></title>
        <description><![CDATA[Stories by Yutong Zhang on Medium]]></description>
        <link>https://medium.com/@zhangyt2000?source=rss-7111da92ec7e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Wr_VsPvfQKaFEVuFPHPdOQ.jpeg</url>
            <title>Stories by Yutong Zhang on Medium</title>
            <link>https://medium.com/@zhangyt2000?source=rss-7111da92ec7e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:59:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@zhangyt2000/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Diffusion Policy: Applying Diffusion Model to Policy Learning]]></title>
            <link>https://medium.com/correll-lab/diffusion-policy-applying-diffusion-model-to-policy-learning-8ee16fc056bb?source=rss-7111da92ec7e------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ee16fc056bb</guid>
            <category><![CDATA[paper-review]]></category>
            <category><![CDATA[manipulation]]></category>
            <category><![CDATA[diffusion-models]]></category>
            <category><![CDATA[robotics]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Yutong Zhang]]></dc:creator>
            <pubDate>Thu, 14 Nov 2024 17:15:05 GMT</pubDate>
            <atom:updated>2024-11-14T17:15:05.256Z</atom:updated>
            <content:encoded><![CDATA[<h3>Paper Review: Diffusion Policy: Visuomotor Policy Learning via Action Diffusion</h3><h4>Applying generative models for image generation to create robot trajectories</h4><p>In the past few years, generative AI has shown great progress in producing realistic images and artworks. Behind this boom is a technique called denoising diffusion probabilistic models (DDPM) introduced by <a href="https://arxiv.org/pdf/2006.11239">Ho et al. in 2020</a> based on work by <a href="https://proceedings.mlr.press/v37/sohl-dickstein15.pdf">Sohl-Dickstein et al. in 2015</a> and improved by <a href="https://arxiv.org/pdf/2102.09672">Nichol et al. in 2021</a>. You can refer to this article to learn more about text-conditioned image generation using DDPM.</p><p><a href="https://medium.com/correll-lab/dall-e-2-from-scratch-c055bf881b9a">DALL-E 2 from Scratch</a></p><p>In this article, we are going to discuss how diffusion could be applied to policy learning in robotics context. Specifically, we will follow the paper <a href="https://diffusion-policy.cs.columbia.edu/diffusion_policy_ijrr.pdf"><em>Diffusion Policy: Visuomotor Policy Learning via Action Diffusion</em></a>, which provides a solid theoretical foundation and reference implementation of using diffusion to learn robot actions. You can find videos and code examples on their website.</p><p><a href="https://diffusion-policy.cs.columbia.edu/">Diffusion Policy: Visuomotor Policy Learning via Action Diffusion</a></p><h3>Background</h3><h4>Diffusion Generative Model</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dxjAuipOXCxJ9rkt" /><figcaption>Source: <a href="https://dl.acm.org/doi/10.1145/3626235">Yang, Ling, et al. “Diffusion models: A comprehensive survey of methods and applications.” <em>ACM Computing Surveys</em> 56.4 (2023): 1–39.</a></figcaption></figure><p>This figure provides a good illustration of diffusion models applying on image generation. In the training stage, we add noise to an image to gradually “corrupt” it at each step, and train the neural network to predict the noise from the noisy image. The neural network learns to reverse this noise-adding process, step by step, to recover the original data distribution. In the generation stage, we start from pure random noise. The neural network predicts and removes a small amount of noise at each step, progressively refining the image until it is mostly noise-free.</p><p>If we think there is a score function describing the validity of the image at each step, the neural network is essentially predicting the gradient of the score function. By subtracting the predicted noise, we increase the validity score of the image as it becomes less noisy.</p><h4>Policy Representation</h4><p>Before we try to learn a policy, we first need to define how we represent a policy. We can represent the policy explicitly, where the policy is a function that takes in an observation and outputs an action. Conversely, we can represent the policy implicitly. We can either model or learn an energy function with respect to observation and action. Then, the optimal action could be obtained by minimizing the energy function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kpjBXfEFp5GhwsyZ" /><figcaption>An illustration of explicit policy, implicit policy, and diffusion policy. A diffusion policy resembles an implicit policy. However, rather than learning the energy function, it learns the gradient of the energy function. Source: <a href="https://diffusion-policy.cs.columbia.edu/diffusion_policy_ijrr.pdf">Chi, Cheng, et al. “Diffusion policy: Visuomotor policy learning via action diffusion.” <em>The International Journal of Robotics Research</em> (2023)</a></figcaption></figure><p>Drawing intuition from previous image generation example, we can obtain a policy through diffusion generation. Instead of learning the energy function like the implicit policy, we learn the gradient of the energy function and get our action through denoising.</p><h3>Contribution</h3><p>The diffusion policy framework introduced in the paper utilizes the diffusion method to produce action sequences with multi-modal distributions. Instead of diffusing in the 2D image space, the paper formulates the diffusion process in the space of end-effector trajectories. The paper further designs the architecture to incorporate state and visual observations to the noise prediction neural network. Both CNN-based and transformer-based networks are proposed and compared in different manipulation tasks.</p><p>One interesting choice the paper makes is to plan a trajectory with a certain horizon, execute part of it, and replan until the task is complete. This close-loop action sequence handles outside disturbance robustly. The paper does extensive experiments in both simulated environment and real-world setup. In almost all experiments, diffusion policy outperforms other policy learning methods.</p><h3>Methodology</h3><p>In this section, we are going to cover each component of diffusion policy.</p><h4>DDPM For Policy Learning</h4><p>First, let us review how DDPM works in image generation case.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/961/0*JDTlKKGvl00LQA3e" /><figcaption>Source: <a href="https://dl.acm.org/doi/10.1145/3626235">Yang, Ling, et al. “Diffusion models: A comprehensive survey of methods and applications.” <em>ACM Computing Surveys</em> 56.4 (2023): 1–39.</a></figcaption></figure><p>In the training step, we would sample a noise-free training image, sample a random iteration count, sample the noise and let the neural network try to predict the noise given noisy input and iteration count. The loss function is thus the mean square error between sampled noise and the predict output from the network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/0*bN42_TrhUsDum6P2" /><figcaption>Loss function for DDPM training. Source: <a href="https://diffusion-policy.cs.columbia.edu/diffusion_policy_ijrr.pdf">Chi, Cheng, et al. “Diffusion policy: Visuomotor policy learning via action diffusion.” <em>The International Journal of Robotics Research</em> (2023)</a></figcaption></figure><p>In the generation step, we start from a random Gaussian noise image and keep subtracting the noise predicted by the network scaled by a learning rate. We also add a small noise at each iteration to have some randomness for exploration. The process is expressed this way:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/699/0*C761qJAcaL5Z8Xsa" /><figcaption>Update rule for DDPM generation. Source: <a href="https://diffusion-policy.cs.columbia.edu/diffusion_policy_ijrr.pdf">Chi, Cheng, et al. “Diffusion policy: Visuomotor policy learning via action diffusion.” <em>The International Journal of Robotics Research</em> (2023)</a></figcaption></figure><p>To apply DDPM to policy learning, the changes we need to make are actually very minimal. First, in the training step, we replace the sample in image space with sample from the action space, in our case the end-effector trajectory. We also let the noise prediction neural network be conditioned on observation of current state. In practice, this is just an additional input to the neural network. The modified loss function becomes the following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/0*lWxYAJa6LAakO6vn" /><figcaption>Loss function for diffusion policy training. Source: <a href="https://diffusion-policy.cs.columbia.edu/diffusion_policy_ijrr.pdf">Chi, Cheng, et al. “Diffusion policy: Visuomotor policy learning via action diffusion.” <em>The International Journal of Robotics Research</em> (2023)</a></figcaption></figure><p>Similarly, in the generation step we are denoising the end-effector trajectory from a random sampled trajectory, and our noise prediction neural network takes an additional input of current observation. The modified process is expressed as following:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/795/0*8DRRRjFegRBdonO2" /><figcaption>Update rule for diffusion policy generation. Source: <a href="https://diffusion-policy.cs.columbia.edu/diffusion_policy_ijrr.pdf">Chi, Cheng, et al. “Diffusion policy: Visuomotor policy learning via action diffusion.” <em>The International Journal of Robotics Research</em> (2023)</a></figcaption></figure><h4>Close-loop Action Sequence</h4><p>To robustly execute the generated policy and handle changes in the environment, the paper suggests using close-loop action sequence. That is, we plan a trajectory, execute only a subsequence of actions, and then replan. We keep doing this “plan, execute, replan” cycles until the manipulation task is complete. Below, the GIF shows how this close-loop action sequence is executed on a real robot.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*YOmqL47JIOyLepz-" /><figcaption>Close-loop action sequence where we plan, execute and replan. Source: <a href="https://diffusion-policy.cs.columbia.edu/diffusion_policy_ijrr.pdf">Chi, Cheng, et al. “Diffusion policy: Visuomotor policy learning via action diffusion.” <em>The International Journal of Robotics Research</em> (2023)</a></figcaption></figure><h4>Noise Prediction Network</h4><p>Now, it is time to dive deeper into the noise prediction network design. The paper proposes two architectures: CNN-based and transformer-based. The difference is how we process the observation, which is made up of states or images from the past few time steps.</p><p>The CNN-based noise prediction network is a modified U-net architecture. Traditional U-net subsamples the input through convolution and then upscales through convolution, with some skipping connections. In the modified version, we need to condition the input on the observation. The conditional input is added between each convolution layer. The transformer-based noise prediction, on the other hand, computes the cross-attention between the input action embedding and the observation embedding. The two architectures are illustrated in the following figure, and the actual implementation can be found in the Jupyter notebook from the paper’s website.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/933/0*IDzScXVOc-FAGiKq" /><figcaption>CNN-based and transformer-based noise prediction neural network. Source: <a href="https://diffusion-policy.cs.columbia.edu/diffusion_policy_ijrr.pdf">Chi, Cheng, et al. “Diffusion policy: Visuomotor policy learning via action diffusion.” <em>The International Journal of Robotics Research</em> (2023)</a></figcaption></figure><p>In practice, the CNN-based network is a good baseline to start with, and performs reasonably well. In comparison, the transformer-based network performs better as it reduces the over-smoothing effect due to the locality bias of any CNN formulation. Thus, the transformer-based network could be more suitable for learning uneven trajectories or trajectories with sharp corners.</p><h4>Observation Encoder</h4><p>Lastly, to take in visual observations, we need a visual encoder to encode the images then feed into the noise prediction network discussed above. In practice, the visual encoder is trained end-to-end together with the noise prediction network.</p><h4>Full Architecture</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kGPOjhr0l0JS9wfFANefPQ.png" /><figcaption>Full architecture. Source: <a href="https://diffusion-policy.cs.columbia.edu/diffusion_policy_ijrr.pdf">Chi, Cheng, et al. “Diffusion policy: Visuomotor policy learning via action diffusion.” <em>The International Journal of Robotics Research</em> (2023)</a></figcaption></figure><p>We have now covered each component of diffusion policy. We take in past observations, diffuse an action sequence, execute part of it and diffuse the next action sequence with new observations. This gives us a robust way to learn manipulation tasks from demonstration trajectories.</p><h3>Analysis</h3><h4>Multi-Modal Action Sequence</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ltKwnmGPGdPi9Az2" /><figcaption>Comparison between different policy learning methods. Source: <a href="https://diffusion-policy.cs.columbia.edu/diffusion_policy_ijrr.pdf">Chi, Cheng, et al. “Diffusion policy: Visuomotor policy learning via action diffusion.” <em>The International Journal of Robotics Research</em> (2023)</a></figcaption></figure><p>Compared to other policy learning methods, diffusion policy learns multi-modal policies very well. In the above example, diffusion policy successfully learns to push T in two directions, while other methods struggle. Intuitively, this is because the denoising process explores different multi-modal basins in the global distribution of trajectories by randomly sampling starting point and denoising paths.</p><p>Another benefit is that by generating the action sequence once, diffusion policy exhibits great temporal consistency and is more tolerant to random pauses in demonstration. In contrast, RNN-based methods draw actions one after another and the action could be from different policy modes. Also, the one-by-one sampling may overfit to pause actions.</p><h4>Position vs. Velocity</h4><p>Another difference is that diffusion policy performs better with position control, while previous methods perform better with velocity control. The intuition is that action modality is more pronounced in position inputs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/963/0*4wzkN89M49bQny6E" /><figcaption>Comparison between velocity and position control. Source: <a href="https://diffusion-policy.cs.columbia.edu/diffusion_policy_ijrr.pdf">Chi, Cheng, et al. “Diffusion policy: Visuomotor policy learning via action diffusion.” <em>The International Journal of Robotics Research</em> (2023)</a></figcaption></figure><h3>Evaluation</h3><p>The paper evaluates diffusion policy extensively with numerous experiments. Make sure to check out their website to see videos of all the impressive experiments.</p><h4>Simulated Experiment</h4><p>The paper tests diffusion policy in the following 2D and 3D simulated experiments:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QkXuN2OdmIkvbjVY" /><figcaption>Simulated experiments. Source: <a href="https://diffusion-policy.cs.columbia.edu/diffusion_policy_ijrr.pdf">Chi, Cheng, et al. “Diffusion policy: Visuomotor policy learning via action diffusion.” <em>The International Journal of Robotics Research</em> (2023)</a></figcaption></figure><p>There are two demonstration datasets: single proficient human (ph) and multi human (mh). In all experiments, diffusion policy performs equally well or out-performs other methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XTpJJu_ip2tf_ZcX" /><figcaption>Experiment results. Source: <a href="https://diffusion-policy.cs.columbia.edu/diffusion_policy_ijrr.pdf">Chi, Cheng, et al. “Diffusion policy: Visuomotor policy learning via action diffusion.” <em>The International Journal of Robotics Research</em> (2023)</a></figcaption></figure><h4>Real-World Experiment</h4><p>The paper further evaluates diffusion policy in real-world experiments. One is pushing a T shaped object, where the end-effector trajectory is simply the 3D coordinate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*a96oWQewTa5vddQY" /><figcaption>Real push-T experiment. Source: <a href="https://diffusion-policy.cs.columbia.edu/diffusion_policy_ijrr.pdf">Chi, Cheng, et al. “Diffusion policy: Visuomotor policy learning via action diffusion.” <em>The International Journal of Robotics Research</em> (2023)</a></figcaption></figure><p>A more complex experiment is flipping a cup, where we need full 6D information on the end-effector for both position and orientation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/0*KA4c1KIffdXe43_v" /><figcaption>Cup flipping experiment. Source: <a href="https://diffusion-policy.cs.columbia.edu/diffusion_policy_ijrr.pdf">Chi, Cheng, et al. “Diffusion policy: Visuomotor policy learning via action diffusion.” <em>The International Journal of Robotics Research</em> (2023)</a></figcaption></figure><p>Lastly, the paper also designs a sauce pouring and spreading task, where diffusion policy handles complex sauce manipulation robustly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/650/1*dGf6yVcs1CxxdBPEwZndPA.png" /><figcaption>Sauce spreading experiment. Source: <a href="https://diffusion-policy.cs.columbia.edu/diffusion_policy_ijrr.pdf">Chi, Cheng, et al. “Diffusion policy: Visuomotor policy learning via action diffusion.” <em>The International Journal of Robotics Research</em> (2023)</a></figcaption></figure><h4>Ablation Study on Visual Encoder</h4><p>The paper also studies the choice of visual encoder. When using CNN-based vision network like Resnet, we can either train from scratch or finetune it together with the noise prediction network. On the other hand, for ViT based vision network like CLIP, we must finetune it with the noise prediction network to get good results. This indicates diffusion policy probably prefers its own way for visual representation, making it necessary to finetune the visual encoder with the noise prediction network.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/0*eTqYkC1zDN0vx_6C" /><figcaption>Ablation study on visual encoder. Source: <a href="https://diffusion-policy.cs.columbia.edu/diffusion_policy_ijrr.pdf">Chi, Cheng, et al. “Diffusion policy: Visuomotor policy learning via action diffusion.” <em>The International Journal of Robotics Research</em> (2023)</a></figcaption></figure><h3>Future Work</h3><p>Overall, the paper presents the formulation of diffusion method for policy learning tasks. The architecture is explained clearly and the performance is evaluated comprehensively on numerous experiments. While diffusion policy performs better, it also requires higher computation costs. In the context of image generation, there are many methods of accelerating diffusion models. We could draw inspirations to improve the inference time of diffusion policy. Another recent trend is to incorporate diffusion policy as a component for generating final action outputs, while leveraging large foundation models to interpret vision and language instructions into actionable steps.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ee16fc056bb" width="1" height="1" alt=""><hr><p><a href="https://medium.com/correll-lab/diffusion-policy-applying-diffusion-model-to-policy-learning-8ee16fc056bb">Diffusion Policy: Applying Diffusion Model to Policy Learning</a> was originally published in <a href="https://medium.com/correll-lab">Toward Humanoids</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Classifying Galaxies Using CLIP Model]]></title>
            <link>https://medium.com/correll-lab/classifying-galaxies-using-clip-model-edf8ef8739cc?source=rss-7111da92ec7e------2</link>
            <guid isPermaLink="false">https://medium.com/p/edf8ef8739cc</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Yutong Zhang]]></dc:creator>
            <pubDate>Tue, 08 Oct 2024 07:12:11 GMT</pubDate>
            <atom:updated>2024-10-16T19:38:41.736Z</atom:updated>
            <content:encoded><![CDATA[<h4>Training CLIP from scratch on the Galaxy10 dataset</h4><p>In recent years, transformer-based architectures have shown great results in various machine learning applications. CLIP (Contrastive Language-Image Pretraining) is one of those transformer-based models that links visual and text information. It has an intuitive and simple architecture with good generalizability on unseen images and texts. CLIP’s versatility makes it valuable for a wide range of applications where we want to integrate both visual and textual information into our learning pipeline. It has been widely adopted as a building block for larger foundation models, where it functions as a visual encoder for downstream network layers.</p><p>In this tutorial, we implement the CLIP model and use it to classify galaxy images to understand how CLIP works. This tutorial and corresponding Jupyter Notebook rely heavily on this sequence of tutorials created by <a href="https://medium.com/u/b66aa802e02f">Matt Nguyen</a>. Check out these articles to understand more about attentions, vision transformers and the CLIP model.</p><ul><li><a href="https://medium.com/correll-lab/building-a-vision-transformer-model-from-scratch-a3054f707cc6">Building a Vision Transformer Model From Scratch</a></li><li><a href="https://medium.com/correll-lab/building-clip-from-scratch-68f6e42d35f4">Building CLIP From Scratch</a></li></ul><h3>CLIP Model Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*c26wRt7NjAYHSzON.png" /><figcaption>CLIP Model Architecture from <a href="https://arxiv.org/pdf/2103.00020">CLIP Paper</a></figcaption></figure><p>CLIP trains a text encoder and an image encoder jointly, aiming to align their outputs in a shared embedding space. During training, both an image and the text caption are passed through encoders, and the model compares the cosine similarity between the resulting embeddings. The model maximizes similarity for correct pairs and minimizes it for mismatched ones. This helps the encoders learn the relationships between images and their captions. The result is a model that captures the contextual relationships between texts and images.</p><p>Below, we will briefly go over some modules of the CLIP model to have a clear understanding of the structure. The full implementation of the CLIP model and experiments can be found in this Jupyter Notebook:</p><p><a href="https://colab.research.google.com/drive/1CAHUkUJrCMQc2-wemHycrjOQ9kMPJir-?usp=drive_link">Google Colab</a></p><p>CLIP</p><pre>class CLIP(nn.Module):<br>    def __init__(self,<br>                 emb_dim, vit_width, img_size,<br>                 patch_size, n_channels, vit_layers,<br>                 vit_heads, vocab_size, text_width,<br>                 max_seq_length, text_heads, text_layers):<br>        super().__init__()<br><br>        self.image_encoder = ImageEncoder(vit_width, img_size, patch_size,<br>                                          n_channels, vit_layers, vit_heads, emb_dim)<br><br>        self.text_encoder = TextEncoder(vocab_size, text_width, max_seq_length,<br>                                        text_heads, text_layers, emb_dim)<br><br>        self.temperature = nn.Parameter(torch.ones([]) * np.log(1 / 0.07))<br><br>        self.device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br><br>    def forward(self,image,text, mask=None):<br>        I_e = self.image_encoder(image)<br>        T_e = self.text_encoder(text, mask=mask)<br><br>        # scaled pairwise cosine similarities [n, n]<br>        logits = (I_e @ T_e.transpose(-2,-1)) * torch.exp(self.temperature)<br><br>        # symmetric loss function<br>        labels = torch.arange(logits.shape[0]).to(self.device)<br><br>        loss_i = nn.functional.cross_entropy(logits.transpose(-2,-1), labels)<br>        loss_t = nn.functional.cross_entropy(logits, labels)<br><br>        loss = (loss_i + loss_t) / 2<br><br>        return loss</pre><p>In our CLIP model, we define a text encoder and an image encoder. In the forward function, the input image and text are passed through their respective encoders to generate embeddings. The model then computes the cosine similarities between the image and text embeddings, scaled by a learned temperature parameter.</p><h4>Text Encoder</h4><pre>class TextEncoder(nn.Module):<br>    def __init__(self,<br>                 vocab_size, width, max_seq_length,<br>                 n_heads, n_layers, emb_dim):<br>        super().__init__()<br><br>        self.max_seq_length = max_seq_length  # Maximum length of input sequence<br><br>        self.encoder_embedding = nn.Embedding(vocab_size, width) # Embedding Table<br><br>        self.positional_embedding = PositionalEmbedding(width, max_seq_length)<br><br>        self.encoder = nn.ModuleList([TransformerEncoder(width,n_heads) for _ in range(n_layers)])<br><br>        # learned proj of image to embed<br>        self.projection = nn.Parameter(torch.randn(width, emb_dim))<br><br>    def forward(self, text, mask=None):<br>        # Text Embedding<br>        x = self.encoder_embedding(text)<br><br>        # Positional Embedding<br>        x = self.positional_embedding(x)<br><br>        # Transformer Encoder<br>        for encoder_layer in self.encoder:<br>            x = encoder_layer(x, mask=mask)<br><br>        # Takes features from the EOT Embedding<br>        x = x[torch.arange(text.shape[0]),torch.sub(torch.sum(mask[:,0],dim=1),1)]<br><br>        # joint multimodal embedding<br>        if self.projection is not None:<br>            x = x @ self.projection<br><br>        x = x / torch.norm(x, dim=-1, keepdim=True)<br>        return x</pre><p>We use a transformer encoder as the core of our text encoder. To jointly train the image and text encoders, we project the text embeddings into the joint embedding space. The projection parameters handle this conversion, ensuring the text embeddings match the dimensions of the joint embedding space for effective alignment with the image embeddings.</p><h4>Image Encoder</h4><pre>class ImageEncoder(nn.Module):<br>    def __init__(self,<br>                 width, img_size, patch_size,<br>                 n_channels, n_layers, n_heads, emb_dim):<br>        super().__init__()<br><br>        assert img_size[0] % patch_size[0] == 0 and img_size[1] % patch_size[1] == 0, &quot;img_size dimensions must be divisible by patch_size dimensions&quot;<br>        assert width % n_heads == 0, &quot;width must be divisible by n_heads&quot;<br><br>        self.n_patches = (img_size[0] * img_size[1]) // (patch_size[0] * patch_size[1])<br><br>        self.max_seq_length = self.n_patches + 1<br><br>        # Patch Embedding<br>        self.linear_project = nn.Conv2d(n_channels, width, kernel_size=patch_size, stride=patch_size)<br><br>        # Classification Token<br>        self.cls_token = nn.Parameter(torch.randn(1, 1, width))<br><br>        self.positional_embedding = PositionalEmbedding(width,self.max_seq_length)<br><br>        self.encoder = nn.ModuleList([TransformerEncoder(width,n_heads) for _ in range(n_layers)])<br><br>        # learned proj of image to embed<br>        self.projection = nn.Parameter(torch.randn(width, emb_dim))<br><br><br>    def forward(self,x):<br>        # Patch Embedding<br>        x = self.linear_project(x)<br>        x = x.flatten(2).transpose(1, 2)<br><br>        # Positional Embedding<br>        x = torch.cat((self.cls_token.expand(x.size()[0], -1, -1),x), dim=1)<br>        x = self.positional_embedding(x)<br><br>        # Transformer Encoder<br>        for encoder_layer in self.encoder:<br>            x = encoder_layer(x)<br><br>        # Takes Class Tokens<br>        x = x[:, 0, :]<br><br>        # joint multimodal embedding<br>        if self.projection is not None:<br>            x = x @ self.projection<br><br>        x = x / torch.norm(x, dim=-1, keepdim=True)<br><br>        return x</pre><p>Similar to the text encoder, we use a vision transformer as our image encoder. We divide the input image into patches and encode them into a sequence of tokens. A classification token is added to capture global information. The patches and tokens are then passed through transformer layers for feature extraction. Finally, the output is projected into the joint embedding space, allowing the image encoding to align with the text encoding. The projection parameters ensure that the image embeddings are mapped to the correct dimensions in the joint embedding space.</p><h3>Galaxy10 Dataset</h3><p>We use the Galaxy10 DECaLS dataset to train and test our CLIP model. The dataset is provided by the astroNN library. More details can be found <a href="https://astronn.readthedocs.io/en/latest/galaxy10.html#">here</a>. A sample galaxy image is of size (256, 256, 3) and it looks like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/1*8-iJ1U6jkWlys3nny9jEcQ.png" /><figcaption>A sample galaxy image</figcaption></figure><p>The galaxies are grouped and labeled into 10 categories:</p><pre>Galaxy10 dataset (17736 images)<br>├── Class 0 (1081 images): Disturbed Galaxies<br>├── Class 1 (1853 images): Merging Galaxies<br>├── Class 2 (2645 images): Round Smooth Galaxies<br>├── Class 3 (2027 images): In-between Round Smooth Galaxies<br>├── Class 4 ( 334 images): Cigar Shaped Smooth Galaxies<br>├── Class 5 (2043 images): Barred Spiral Galaxies<br>├── Class 6 (1829 images): Unbarred Tight Spiral Galaxies<br>├── Class 7 (2628 images): Unbarred Loose Spiral Galaxies<br>├── Class 8 (1423 images): Edge-on Galaxies without Bulge<br>└── Class 9 (1873 images): Edge-on Galaxies with Bulge</pre><p>We plot a sample for each category:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xeA1zePyHCDqb3dl8xVcKA.png" /></figure><p>We then define our dataset class that divides the Galaxy10 dataset into training set and testing set. We also create a map between labels and captions for our CLIP training.</p><pre>class GalaxyDataset(Dataset):<br>    def __init__(self, images, labels, train=True):<br>        # hold 80% for traning, 20% for testing<br>        train_size = int(0.8 * images.shape[0])<br><br>        # split slicing for traning and testing set<br>        if train:<br>            self.split = slice(0, train_size)<br>        else:<br>            self.split = slice(train_size, None)<br><br>        # img-tensor transform function<br>        self.transform = T.ToTensor()<br>        # store images and labels<br>        self.images = images<br>        self.labels = labels<br><br>        # captions based on class label<br>        self.captions = {<br>            0: &quot;An image of disturbed galaxies&quot;,<br>            1: &quot;An image of merging galaxies&quot;,<br>            2: &quot;An image of round smooth galaxies&quot;,<br>            3: &quot;An image of in-between round smooth galaxies&quot;,<br>            4: &quot;An image of cigar shaped smooth galaxies&quot;,<br>            5: &quot;An image of barred spiral galaxies&quot;,<br>            6: &quot;An image of unbarred tight spiral galaxies&quot;,<br>            7: &quot;An image of unbarred loose spiral galaxies&quot;,<br>            8: &quot;An image of edge-on galaxies without bulge&quot;,<br>            9: &quot;An image of edge-on galaxies with bulge&quot;<br>        }<br><br>    def __len__(self):<br>        return self.images[self.split].shape[0]<br><br>    def __getitem__(self,i):<br>        img = self.images[self.split][i].copy()<br>        img = self.transform(img)<br>        # tokenize captions and get masks<br>        cap, mask = tokenizer(self.captions[self.labels[self.split][i]], )<br><br>        mask = mask.repeat(len(mask),1)<br><br>        return {&quot;image&quot;: img, &quot;caption&quot;: cap, &quot;mask&quot;: mask}</pre><h3>Experiment</h3><h4>Training and Testing</h4><p>We use 80% of the entire dataset as the training set, and use the rest 20% as the testing set. Furthermore, we hold 10% of the training set as the validation set to monitor the loss performance and save the best-performing model during the training time. The following code shows how we divide our dataset into train, val and data.</p><pre># load train and test dataset<br>astro_train_set = GalaxyDataset(images, labels, train = True)<br>astro_test_set = GalaxyDataset(images, labels, train = False)<br><br># further split train and val dataset<br>train_size = int(0.9 * len(astro_train_set))<br>val_size = len(astro_train_set) - train_size<br>astro_train_set, astro_val_set = random_split(astro_train_set, [train_size, val_size])<br><br>astro_train_loader = DataLoader(astro_train_set, shuffle=True, batch_size=batch_size)<br>astro_val_loader = DataLoader(astro_val_set, shuffle=True, batch_size=batch_size)<br>astro_test_loader = DataLoader(astro_test_set, shuffle=False, batch_size=batch_size)</pre><p>The model size influences the classification accuracy a lot. By increasing the number of attention heads in the visual encoder, the encoder could capture image features better. But we need to balance model size and performance. We end up with following hyperparameters:</p><pre>emb_dim = 64<br>vit_width = 36<br>img_size = (256,256)<br>patch_size = (16,16)<br>n_channels = 3<br>vit_layers = 6<br>vit_heads = 6<br>vocab_size = 256<br>text_width = 64<br>max_seq_length = 64<br>text_heads = 8<br>text_layers = 4<br>lr = 1e-3<br>epochs = 40<br>batch_size = 512</pre><p>The final prediction accuracy is 71.62%.</p><h4>Zero-shot Classification</h4><p>One advantage of the CLIP model is its strong generalization to unseen text and images. To demonstrate this, we curate a set of classes that do not fully match the captions used during training. We then use the pre-trained text and image encoders to encode these new classes and an image. By comparing the cosine similarity between the class label embeddings and the image embedding, we can compute the classification scores of the image for each class.</p><pre>class_names = [&quot;disturbed galaxies&quot;,<br>               &quot;mergings galaxies&quot;,<br>               &quot;round smooth galaxies&quot;,<br>               &quot;in-between round smooth galaxies&quot;,<br>               &quot;cigar shaped smooth galaxies&quot;,<br>               &quot;barred spiral galaxies&quot;,<br>               &quot;unbarred tight spiral galaxies&quot;,<br>               &quot;unbarred loose spiral galaxies&quot;,<br>               &quot;edge-on galaxies without bulge&quot;,<br>               &quot;edge-on galaxies with bulge&quot;]<br># tokenize captions and create masks<br>text = torch.stack([tokenizer(x)[0] for x in class_names]).to(device)<br>mask = torch.stack([tokenizer(x)[1] for x in class_names])<br>mask = mask.repeat(1,len(mask[0])).reshape(len(mask),len(mask[0]),len(mask[0])).to(device)<br><br># get an imgage and decoded caption<br>idx = 1<br>img = astro_test_set[idx][&quot;image&quot;][None,:]<br>cap = tokenizer(astro_test_set[idx][&quot;caption&quot;], encode=False, mask=astro_test_set[idx][&quot;mask&quot;][0])[0]<br># visualize<br>plt.imshow(img[0].permute(1, 2, 0))<br>plt.title(cap)<br>plt.show()<br><br># feed the image through the network<br>img = img.to(device)<br>model.eval()<br>with torch.no_grad():<br>  image_features = model.image_encoder(img)<br>  text_features = model.text_encoder(text, mask=mask)<br><br># normalize image and text encoding<br>image_features /= image_features.norm(dim=-1, keepdim=True)<br>text_features /= text_features.norm(dim=-1, keepdim=True)<br># compare similarity<br>similarity = (100.0 * image_features @ text_features.T).softmax(dim=-1)<br>values, indices = similarity[0].topk(5)<br><br># print the result<br>print(&quot;\nTop predictions:\n&quot;)<br>for value, index in zip(values, indices):<br>    print(f&quot;{class_names[int(index)]:&gt;16s}: {100 * value. Item():.2f}%&quot;)</pre><p>Here are some sample classification results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jIAMYqUOUgWls7BWgrtwaw.jpeg" /><figcaption>Zero-shot Classification Results</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IBgJV3AlVbFFKXSGRS1wLQ.jpeg" /><figcaption>More Zero-shot Classification Results</figcaption></figure><h4>Evaluation</h4><p>The classification accuracy is not very high, likely due to the complexity of this dataset. While some classes of galaxies appear distinctly different, others, like “edge-on galaxies with bulge” and “edge-on galaxies without bulge,” are closely related. Additionally, spiral galaxies are divided into three finer classes with subtle differences, making the dataset challenging and possibly requiring more hyperparameter tuning for improved performance. We also note that features in galaxies are highly localized. This suggests that traditional CNN-based networks may perform better on this specific dataset.</p><p>In the zero-shot classification experiments, the results are quite promising. Our CLIP model successfully classified images in the test set according to the modified class labels. This demonstrates the great generalizability of the CLIP architecture.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=edf8ef8739cc" width="1" height="1" alt=""><hr><p><a href="https://medium.com/correll-lab/classifying-galaxies-using-clip-model-edf8ef8739cc">Classifying Galaxies Using CLIP Model</a> was originally published in <a href="https://medium.com/correll-lab">Toward Humanoids</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>