<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lekai Chen on Medium]]></title>
        <description><![CDATA[Stories by Lekai Chen on Medium]]></description>
        <link>https://medium.com/@lekai?source=rss-1682ebfabfa5------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*bS3_t6sxZ1TPmZQV.jpg</url>
            <title>Stories by Lekai Chen on Medium</title>
            <link>https://medium.com/@lekai?source=rss-1682ebfabfa5------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:26:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lekai/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Paper Review: Robot Fleet Learning via Policy Merging]]></title>
            <link>https://medium.com/correll-lab/paper-review-robot-fleet-learning-via-policy-merging-a1e8df483d02?source=rss-1682ebfabfa5------2</link>
            <guid isPermaLink="false">https://medium.com/p/a1e8df483d02</guid>
            <dc:creator><![CDATA[Lekai Chen]]></dc:creator>
            <pubDate>Tue, 05 Nov 2024 06:59:10 GMT</pubDate>
            <atom:updated>2024-11-05T19:17:57.669Z</atom:updated>
            <content:encoded><![CDATA[<h4>How to train locally and merge the resulting weights, not the raw data itself, to learn using a team of robots</h4><p>The paper, <a href="https://openreview.net/pdf?id=IL71c1z7et">Robot Fleet Learning via Policy Merging</a>, explores how fleets of robots can effectively learn policies for diverse and complex tasks through a method called policy merging. With robots generating extensive, heterogeneous data from their interactions in various environments, it becomes increasingly challenging to consolidate these experiences without overwhelming communication channels or central data storage requirements as well as violating the privacy constraint. The authors address this by introducing <strong>FLEET-MERGE</strong>, a policy merging approach that focuses on transmitting only the trained neural network weights, rather than entire datasets, thereby reducing the data load while still enabling effective skill sharing among robots.</p><h3>Motivations</h3><p>As robot fleets expand in scale and diversity, traditional centralized training models become impractical due to the vast and continuously growing amounts of data generated by each robot’s unique experiences. This decentralized pool of data requires a policy learner that can learn a robust, comprehensive policy across heterogeneous sources while meeting real-world constraints, such as limited network bandwidth, latency requirements, and data privacy concerns. Transmitting the full scope of diverse data from each robot for centralized training would strain network resources and hinder real-time decision-making.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c3VggVvuY_WarkhDqQhBXA.png" /><figcaption>The image describes a more efficient approach to fleet learning through <strong>policy merging</strong>. Instead of gathering vast amounts of data centrally (requiring terabytes of data transfer daily, represented by the red arrow), this <strong>bottom-up approach</strong> (green arrow) merges policies trained locally by individual robots. This method allows for creating general, shared policies across the fleet without massive data transfers. By sharing learned weights rather than raw data, this approach is better suited for robots that continuously generate new data, especially in complex tasks like robotic control. The goal is to enable knowledge transfer across the fleet efficiently: “One robot learns, the entire fleet learns.”</figcaption></figure><p>To address these challenges in data communication, heterogeneity, and privacy, a common practical solution is model merging. Rather than training a single model over a fully centralized dataset, this approach trains models locally on each sub-dataset and then merges them into a single model by averaging their weights. For example, if we have an LLM optimized for mathematical reasoning (θ1​) and another for programming (θ2​), we can merge them by averaging their weights:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/572/1*S0fBDxEryjOon2Y0jJqP-w.png" /></figure><p>However, traditional weight averaging in neural network models often results in performance degradation due to parameter misalignment issues, stemming from <strong>Permutation Invariance</strong>. This property of networks — particularly feedforward or recurrent networks — means that swapping the positions of two neurons within a hidden layer, as long as their associated weights are adjusted accordingly, does not alter the network’s functionality. Consequently, multiple weight configurations can represent the same function, and reordering (or permuting) units in hidden layers does not change the network’s overall behavior. For example, if neurons h1​ and h2​ in a layer are swapped (with the corresponding input and output weights adjusted), the network will still produce the same output for any given input.</p><p>This permutation invariance complicates model merging since naively averaging weights from two functionally identical networks with different internal configurations can yield a model that poorly approximates either function. Essentially, without alignment, the hidden layers’ parameters from two networks cannot be directly merged, even if the networks produce identical outputs.</p><p>To overcome these challenges, this paper builds on recent insights into the permutation invariance property, proposing a novel approach that allows networks with identical functionality but different parameter configurations to be effectively aligned and merged. By leveraging permutation invariance, the authors aim to develop a flexible and precise merging method that enhances learning efficiency and robustness across diverse robot fleets.</p><h3>Methodology</h3><p>To address the challenges posed by data heterogeneity, communication constraints, and privacy concerns in distributed robotic fleets, this paper introduces a framework that leverages <strong>model merging</strong> with a focus on <strong>parameter alignment</strong> based on permutation invariance. The methodology consists of 3 main stages: local model training, alignment of hidden layers, parameter merging.</p><p>Firstly, each robot in the fleet generates unique, heterogeneous data based on its specific environment and tasks. Rather than centralizing all data, which would be infeasible and inefficient, each robot trains a model locally on its dataset. This decentralized approach respects bandwidth and privacy limitations by avoiding the need to transmit raw data to a central location. The result is a collection of specialized models, each fine-tuned to the specific distribution and context of its respective sub-dataset. They conduct their experiments on a behaviour cloning setting, where they use a typical BC loss function for each datset:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X_IDi3pPEKYcY-XfrGFECA.png" /></figure><p>Ideally, if the robot designer can pool all the data together, then the objective is to minimize the following imitation loss across datasets:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*f7cwtqAxe0ArE4UJ_SyQKA.png" /></figure><p>which is an upper bound for the performance to be compared with later. For policy merging, given the invariance properties introduced in motivations, merging by naive averaging the parameters:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/486/1*SOrtGLRBUtstr6gOj0vbJw.png" /></figure><p>may not perform well. Peña et al. (2022) propose an approach to merging two neural network models. Their method involves a <strong>gradient-based alignment</strong> that softens the typical rigid requirement of using a “hard permutation matrix” (a strict matrix that only allows specific, fixed weight arrangements) for merging.</p><p>In this approach, given two models (θ and θ’), the merging process involves using data trajectories 𝜏 from a shared dataset D. For each iteration, the alignment parameters P are updated based on the gradient of a loss function L_{bc}, applied to a blend of the two models’ parameters. Specifically, this blend is controlled by a parameter 𝛼, which adjusts the contribution from each model:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/460/1*fOXX7Idr3hi6WVBnmzClEw.png" /></figure><p>The goal is to find an optimal alignment that minimizes L_{bc}, ensuring the merged model performs effectively while retaining important features from both original models. This iterative process continues for successive steps s ≥ 1, refining the alignment gradually.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R5oJYNKfD4bIltXH1CGWAA.png" /></figure><p>Then the authors proposes a technique called <strong>soft permutation projection</strong> which is used to align or merge neural network parameters in a way that allows more flexibility than strict (or “hard”) permutation matrices. In essence, it permits gradual adjustments to the parameters, aligning two or more models without enforcing rigid constraints on the arrangement of weights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hWtJnzoW47I6N4rCZH-DJw.png" /></figure><p>In a standard “hard” permutation matrix, each row and column has exactly one entry as 1, with all others as 0, effectively reordering elements without any blending. In contrast, a soft permutation matrix allows entries to take values between 0 and 1, introducing a “soft” reordering where weights can be interpolated rather than fully swapped. This matrix is doubly stochastic, meaning each row and column sums to 1, which ensures that the weights are still normalized but allows them to “blend” rather than strictly reorder.</p><p>To encourage entries of the soft permutation matrix to gradually approach 0 or 1, the method often includes an entropy term in the optimization process. This entropy term rewards more uniform distributions across the matrix entries, preventing sudden, drastic changes. The amount of “softness” is controlled by a hyperparameter 𝜏, which balances between rigid and flexible permutations.</p><p>During each optimization step, the algorithm adjusts the entries of the soft permutation matrix by calculating the gradient of a loss function, often using techniques like Sinkhorn normalization to keep the matrix doubly stochastic. After each update, the matrix is re-projected to maintain its structure, preserving row and column sums while gradually converging towards a usable alignment.</p><p>By using a soft projection, the models’ weights can adapt iteratively, converging to an alignment that maintains important learned features from each model, especially useful when combining models trained on different data or tasks.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9EIgMZQntk-89IqtPaKQfg.png" /></figure><h3>Fleet-Tools</h3><p>To evaluate the merging of policies across distributed robot fleets in complex, tool-use tasks, they design the Fleet-Tools, a new robotic manipulation benchmark. Built on the Drake simulator, it emphasizes contact-rich tasks that require high levels of dexterity and adaptability, such as hammering, screwing, and cutting. By requiring robots to apply diverse tools to various objects, Fleet-Tools challenges the merged policies to generalize learned skills across multiple tool-object combinations, which is ideal for testing robustness and compositionality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3D4YExEWNzUuYLuoky5MQg.png" /></figure><p>Fleet-Tools’ scalable design allows the creation of numerous combinatorial tasks by pairing different tools and objects, which is especially relevant for fleet learning in heterogeneous environments. This setup provides a systematic way to simulate diverse local datasets across a fleet, mimicking real-world privacy constraints and data variability across distributed robots.</p><h3>Empirical Results</h3><p>The results across three merging experiments — Fleet-Tools, Meta-World, and linear control tasks — demonstrate the effectiveness of permutation-aligned policy merging strategies, particularly the proposed Fleet-Merge algorithm, in various contexts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dN631v7_vk2eQsFJ-gBvvQ.png" /></figure><p>For one-shot merging of policies trained on distinct tasks, Fleet-Merge outperformed naive averaging and other alignment methods like Git-Rebasin and SoftUpdate. The results showed that alignment through permutation significantly reduced the performance barrier between merged models, indicating that the merged policy successfully integrated knowledge without compromising task-specific behaviors. Additionally, Fleet-Merge provided a strong initialization for further fine-tuning on downstream tasks, leading to superior performance compared to other baselines.</p><p>Testing on Meta-World, where ResNet features were used as inputs, revealed minimal performance barriers for specific tasks, supporting the viability of permutation-aligned merging in multi-task scenarios. Fleet-Merge consistently outperformed baselines in multi-task settings, especially when local datasets varied due to the Dirichlet parameter (α) controlling heterogeneity, affirming its robustness across heterogeneous task settings.</p><p>For linear control tasks in high-dimensional settings, the gradient-based alignment approach was most effective, outperforming direct averaging and permutation-based alignment. Notably, control policies showed invariances to general linear transformations, making gradient-based alignment ideal, contrasting with RNN-based policies where projecting “soft permutations” to hard permutations was necessary to prevent performance degradation. This finding highlights the importance of considering the group structure of policy invariances when designing merging algorithms for different policy architectures.</p><p>In iterative merging experiments across Fleet-Tools and Meta-World, Fleet-Merge demonstrated strong performance under multi-task conditions, even with infrequent merging and low participation ratios in weight alignment. In Fleet-Tools, performance remained robust as the merging frequency was reduced, indicating adaptability to the communication constraints typical in fleet-learning setups. In Meta-World, Fleet-Merge achieved high performance and resilience to dataset heterogeneity, showing it could tolerate lower merging frequencies and selective agent participation without substantial degradation. This efficiency allows multi-task policies to be learned across diverse tasks without requiring pooled data, aligning well with federated learning protocols by minimizing centralized communication.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KzvPcfFuTbo868TbEZPd-A.png" /></figure><h3>Conclusion</h3><p>This paper introduces Fleet-Merge, a promising approach to policy merging tailored for robotic fleet learning, leveraging the inherent permutation invariance of RNNs. The methodology addresses critical practical issues in distributed robotics, such as data privacy and communication efficiency, by merging policy parameters instead of raw data. This approach is innovative as it builds on prior works in model merging by adapting techniques for the unique demands of fleet learning and by integrating permutation matrices to align the internal representation space of RNNs. Fleet-Merge shows robust performance on multi-task benchmarks like Meta-World and the newly introduced Fleet-Tools, and achieves notable improvements over baselines. The paper is well-structured, with strong experimental results that validate the method’s effectiveness across both simulated and real-world tasks.</p><p>Despite these strengths, some aspects could be improved. The paper’s presentation, particularly of Fleet-Tools. Its role and significance in policy merging need stronger motivation. Additionally, while the multi-task setting is emphasized, many experiments still focus on single-task scenarios, limiting the demonstration of Fleet-Merge’s potential in more complex, distributed settings. Another point raised is that, in the context of behavior cloning, the added complexity of Fleet-Merge may not be fully justified if simpler methods such as averaging can yield competitive performance on specific tasks.</p><p>Overall, Fleet-Merge’s most impactful contribution is the novel framework for aligning and merging policies across a fleet of robots, which holds significant promise for distributed, data-sensitive robotic applications. By introducing Fleet-Tools as a benchmark for complex, compositional tasks, the paper also provides a valuable resource for advancing policy learning in real-world robotics.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a1e8df483d02" width="1" height="1" alt=""><hr><p><a href="https://medium.com/correll-lab/paper-review-robot-fleet-learning-via-policy-merging-a1e8df483d02">Paper Review: Robot Fleet Learning via Policy Merging</a> was originally published in <a href="https://medium.com/correll-lab">Toward Humanoids</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Training a CLIP Model on Extremly Large Dataset with Limited Resources]]></title>
            <link>https://medium.com/correll-lab/training-a-clip-model-on-extremly-large-dataset-with-limited-resources-d69bd4aada7e?source=rss-1682ebfabfa5------2</link>
            <guid isPermaLink="false">https://medium.com/p/d69bd4aada7e</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[contrastive-learning]]></category>
            <dc:creator><![CDATA[Lekai Chen]]></dc:creator>
            <pubDate>Sat, 12 Oct 2024 21:53:36 GMT</pubDate>
            <atom:updated>2024-10-13T05:33:03.022Z</atom:updated>
            <content:encoded><![CDATA[<h3>Training CLIP for Chinese Captions from scratch</h3><h4>从头开始训练 CLIP 制作中文字幕</h4><p>The goal of this project is to train a CLIP (Contrastive Language-Image Pretraining) model for matching Chinese text descriptions with corresponding images using the <strong>Chinese-CLIP</strong> dataset. Initially, we conducted a preliminary experiment building CLIP from scratch following</p><p><a href="https://medium.com/correll-lab/building-clip-from-scratch-68f6e42d35f4">Building CLIP From Scratch</a></p><p>We faced several challenges, primarily due to computational limitations.</p><h4>Initial Experiment: Building CLIP from scratch</h4><p>In our first experiment, we used a smaller subset of the <strong>Chinese-CLIP</strong> dataset and a narrower model architecture. We trained the model for only a few epochs, which resulted in suboptimal performance. The limitations included:</p><ul><li><strong>Smaller Dataset</strong>: We could not fully utilize the large-scale dataset due to GPU memory constraints.</li><li><strong>Narrower Architecture</strong>: To accommodate the memory limits, we chose a smaller model, sacrificing some representational capacity.</li><li><strong>Limited Epochs</strong>: With only a few epochs of training, the model lacked sufficient exposure to the data, resulting in lower accuracy and poor generalization.</li></ul><p>The results from this initial experiment reflected these constraints, and we did not achieve the desired performance. The jupyter notebook of the initial experiment can be found at <a href="https://colab.research.google.com/drive/1QOlLplI0MMBFkZr1l7FoyfsJ0yB_Vt_O?usp=sharing">here</a>. The notebook that we use in this blog is <a href="https://colab.research.google.com/drive/1tsmYFJimPPJvnEP6x6zHt33vg6UMelXC?usp=sharing">this one</a>.</p><h4>Optimizing CLIP by using pretrained encoders and other optimizations</h4><p>Building upon the lessons learned from the initial experiment, we implemented several key optimizations in this current experiment:</p><ol><li><strong>Fine-Tuning Pretrained Models</strong>: Instead of training both text and image encoders from scratch, we fine-tuned pretrained models. This significantly sped up training while reducing the number of epochs required to achieve good results.</li><li><strong>Efficient Use of Resources</strong>: By precomputing text embeddings and storing them in memory, we avoided repetitive inferences and reduced memory consumption.</li><li><strong>Smaller ViT Model</strong>: We used a ViT-tiny model for the image encoder to reduce GPU memory requirements, which allowed us to train on a larger batch size.</li><li><strong>Optimization with SGD</strong>: To further reduce memory usage, we switched to SGD without momentum, which is more memory efficient than Adam. We also applied advanced optimization techniques such as freezing specific layers and using a cosine learning rate schedule with a warm-up phase.</li><li><strong>Expanded Dataset</strong>: This time, we expanded the dataset size and processed more samples to provide the model with more varied training examples.</li></ol><h4><strong>Comparison</strong>:</h4><ul><li>The <strong>initial experiment</strong> focused on exploring the feasibility of training a CLIP model with limited resources but resulted in underwhelming performance with less than <strong>2% accuracy</strong> due to the constraints mentioned above.</li><li>The <strong>optimized experiment</strong> allowed us to scale up both the dataset and training process while making efficient use of the available resources. As a result, the model achieved <strong>87.39% accuracy</strong> on the test set, a significant improvement over the initial results.</li></ul><p>This notebook details both the <strong>model architecture</strong> and the <strong>training process</strong> used in the current experiment, including the optimizations implemented to overcome previous limitations.</p><h4>1. <strong>Import Libraries and Modules</strong></h4><p>In this section, we import essential libraries for building, training, and evaluating the CLIP model, including torch for deep learning, transformers for pre-built models, and PIL for image processing.</p><pre>import matplotlib.pyplot as plt<br>import numpy as np<br>import os<br>import pandas as pd<br>import requests<br>import multiprocessing<br>import pandas as pd<br>import imghdr<br>import math<br>from loguru import logger<br>from glob import glob<br>from os.path import join<br>from tqdm import tqdm<br>from multiprocessing import cpu_count<br>from PIL import Image<br><br>import torch<br>torch.cuda.empty_cache()<br>import torch.nn as nn<br>import torch.optim as optim<br>import torchvision.transforms as T<br><br>from torch.optim import Adam<br>from torch.utils.data import Dataset, DataLoader<br>from datasets import load_dataset<br>from transformers import ViTFeatureExtractor, AutoTokenizer</pre><pre>/tmp/ipykernel_3309083/3887304629.py:8: DeprecationWarning: &#39;imghdr&#39; is deprecated and slated for removal in Python 3.13<br>  import imghdr<br>/home/josep/miniconda3/envs/clip/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html<br>  from .autonotebook import tqdm as notebook_tqdm</pre><h4>2. <strong>Define Text Encoder</strong></h4><p>The text encoder transforms tokenized text into a fixed-length vector representation, crucial for comparing text and images in the CLIP model. You can explore various text-embedding models via the <a href="https://huggingface.co/spaces/mteb/leaderboard">MTEB leaderboard</a> to select the one that best suits your needs.</p><pre>import torch.nn as nn<br>from transformers import AutoModel, AutoTokenizer<br>from huggingface_hub import hf_hub_download<br><br>class TextEncoder(nn.Module):<br>    def __init__(self, projection_dim, train=False):<br>        super().__init__()<br>        vector_dim = 1792<br><br>        if not train:<br>            text_model_dir = &quot;TencentBAC/Conan-embedding-v1&quot;<br>            # Load pretrained Stella model<br>            self.text_model = AutoModel.from_pretrained(text_model_dir, trust_remote_code=True).cuda().eval()<br>            self.text_model.requires_grad_(False)<br>            vector_linear_directory = &quot;2_Dense&quot;<br>            model_file = hf_hub_download(repo_id=text_model_dir, filename=f&quot;{vector_linear_directory}/pytorch_model.bin&quot;)<br>            self.vector_linear = torch.nn.Linear(in_features=self.text_model.config.hidden_size, out_features=vector_dim)<br>            vector_linear_dict = {<br>                k.replace(&quot;linear.&quot;, &quot;&quot;): v for k, v in torch.load(model_file).items()<br>            }<br>            self.vector_linear.load_state_dict(vector_linear_dict)<br>            self.vector_linear.requires_grad_(False)<br><br>        self.projection = nn.Linear(vector_dim, projection_dim)<br><br>    def inference_text_embedding(self, input_data):<br>        with torch.no_grad():<br>            attention_mask = input_data[&quot;attention_mask&quot;]<br>            <br>            last_hidden_state = self.text_model(**input_data)[0]<br>            last_hidden = last_hidden_state.masked_fill(~attention_mask[..., None].bool(), 0.0)<br>            pooled_output = last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]<br>            vector_linear_output = self.vector_linear(pooled_output)<br>            return vector_linear_output<br>        <br>    def inference(self, input_data):<br>        with torch.no_grad():<br>            attention_mask = input_data[&quot;attention_mask&quot;]<br><br>            last_hidden_state = self.text_model(**input_data)[0]<br><br>            last_hidden = last_hidden_state.masked_fill(~attention_mask[..., None].bool(), 0.0)<br>            pooled_output = last_hidden.sum(dim=1) / attention_mask.sum(dim=1)[..., None]<br>            vector_linear_output = self.vector_linear(pooled_output)<br>            projected_output = self.projection(vector_linear_output)<br>            normalized_output = projected_output / torch.norm(projected_output, dim=-1, keepdim=True)<br><br>            return normalized_output<br><br>    def forward(self, text_embedding):<br>        projected_output = self.projection(text_embedding)<br>        normalized_output = projected_output / torch.norm(projected_output, dim=-1, keepdim=True)<br><br>        return normalized_output</pre><h4>3. <strong>Define Image Encoder</strong></h4><p>The image encoder converts input images into vector representations, enabling direct comparison with text representations in the multimodal CLIP framework. Since we will fine-tune a full ViT model, selecting a small version reduces memory requirements. In this experiment, we use a ViT-tiny model from the WinKawaks/vit-tiny-patch16-224 series.</p><pre>import torch<br>import torch.nn as nn<br>from transformers import ViTModel, ViTImageProcessor, ViTConfig<br><br>class ImageEncoder(nn.Module):<br>    def __init__(self, emb_dim, train=False):<br>        super().__init__()<br><br>        vit_model_dir = &quot;WinKawaks/vit-tiny-patch16-224&quot;  # Specify the pretrained ViT model<br>        # Load a small pretrained ViT model from Hugging Face<br>        vision_config = ViTConfig.from_pretrained(vit_model_dir)<br>        <br>        self.projection_dim = emb_dim<br>        self.vision_embed_dim = vision_config.hidden_size<br>        self.vit_model = ViTModel.from_pretrained(vit_model_dir)<br><br>        self.freeze_patch_embedding_and_layernorm()<br><br>        self.visual_projection = nn.Linear(self.vision_embed_dim, self.projection_dim, bias=False)<br><br>    def freeze_patch_embedding_and_layernorm(self):<br>        &quot;&quot;&quot;<br>        Freeze the patch-to-token embedding layer and all LayerNorm layers<br>        in the ViT model.<br>        &quot;&quot;&quot;<br>        # Freeze the patch embedding layer<br>        patch_embedding = self.vit_model.embeddings.patch_embeddings<br>        for param in patch_embedding.parameters():<br>            param.requires_grad = False<br><br>    def forward(self, pixel_values):<br>        vision_outputs = self.vit_model(<br>            pixel_values=pixel_values,<br>            output_attentions=False,<br>            output_hidden_states=False,<br>            return_dict=False,<br>        )<br>        pooled_output = vision_outputs[1]  # pooled_output<br>        projected_output = self.visual_projection(pooled_output)<br><br>        normalized_output = projected_output / torch.norm(projected_output, dim=-1, keepdim=True)<br><br>        return normalized_output</pre><h4>4. <strong>Assemble Complete CLIP Model</strong></h4><p>The CLIP model (Contrastive Language-Image Pretraining) combines text and image encoders to learn associations between images and their corresponding text descriptions by comparing their encoded representations.</p><pre>class CLIP(nn.Module):<br>    def __init__(self, emb_dim, train=False):<br>        super().__init__()<br>        self.image_encoder = ImageEncoder(emb_dim, train)<br>        self.text_encoder = TextEncoder(emb_dim, train)<br>        self.temperature = nn.Parameter(torch.ones([]) * np.log(1 / 0.07))<br>        self.device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br><br>    def forward(self,image, input_data):<br>        <br>        I_e = self.image_encoder(image)<br>        T_e = self.text_encoder(input_data)<br><br>        # scaled pairwise cosine similarities [n, n]<br>        logits = (I_e @ T_e.transpose(-2,-1)) * torch.exp(self.temperature)<br><br>        # symmetric loss function<br>        labels = torch.arange(logits.shape[0]).to(self.device)<br><br>        loss_i = nn.functional.cross_entropy(logits.transpose(-2,-1), labels)<br>        loss_t = nn.functional.cross_entropy(logits, labels)<br><br>        loss = (loss_i + loss_t) / 2<br><br>        return loss</pre><h4>5. <strong>Dataset: Chinese-CLIP</strong></h4><p>We use the Chinese-CLIP dataset, which contains 1.37 million training images and 10,000 testing images. This dataset allows us to train and evaluate the model’s ability to match Chinese text to images. The dataset can be accessed <a href="https://o365coloradoedu-my.sharepoint.com/:f:/g/personal/lech8988_colorado_edu/Ev7eHmZ7n8JEs0UE-cFGF3EB9nmAB-0N07Dxx8DfDqTWYA?e=BV5rdj">here</a>.</p><pre>train_csv_file = &#39;./data/train-137w.csv&#39;<br>test_csv_file = &#39;./data/test-2.6w.csv&#39;<br>image_path = &#39;./data/images&#39;<br>train_csv_out_file = &#39;./data/train-filter.csv&#39;<br>test_csv_out_file = &#39;./data/test-filter.csv&#39;<br><br>train_set_num=-1<br>test_set_num=1000</pre><p>We first download the images,</p><pre># def download(file, url):<br>#     try:<br>#         headers = {&quot;User-Agent&quot;: &quot;Chrome/68.0.3440.106&quot;}<br>#         response = requests.get(url, headers=headers, timeout=(2, 0.5))<br>#         status_code = response.status_code<br>#         content_type = response.headers.get(&#39;Content-Type&#39;)<br>#         # NO GIF<br>#         if status_code == 200 and content_type != &#39;image/gif&#39;:<br>#             image = response.content<br>#             with open(file, &#39;wb&#39;) as f:<br>#                 f.write(image)<br>#     except Exception as e:<br>#         logger.info(&#39;downloading image error, file:{}&#39;.format(file.split(&#39;/&#39;)[-1]))<br>#         logger.info(e)<br><br># def download_images(csv_file, n=-1):<br>#     thread_num = cpu_count()  # cpu_count()<br>#     if not os.path.exists(image_path):<br>#         os.makedirs(image_path)<br><br>#     df = pd.read_csv(csv_file)<br>#     df = df[int(9e5):]<br>#     print(len(df))<br><br>#     # Initialize thread pool<br>#     pool = multiprocessing.Pool(processes=thread_num)<br>#     for _, row in tqdm(df.iterrows()):<br>#         filename = row[&#39;filename&#39;]<br>#         url = row[&#39;url&#39;]<br>#         file = join(image_path, filename)<br>#         if os.path.exists(file):<br>#             continue<br>#         pool.apply_async(download, (file, url))<br>#     pool.close()<br>#     pool.join()<br><br># download_images(csv_file=train_csv_file, n=train_set_num)<br># download_images(csv_file=test_csv_file, n=test_set_num)</pre><h4>6. <strong>Filter Downloaded Images (Remove GIFs)</strong></h4><p>We filter out GIFs and other unsupported image formats from the dataset and update the CSV file accordingly to include only valid images.</p><pre># def filter_gif(csv_in_file, csv_out_file, min_size=(10, 10)):<br>#     result = []<br>#     df = pd.read_csv(csv_in_file)<br>#     for _, row in tqdm(df.iterrows(), desc=&#39;filter gif and small images&#39;):<br>#         filename = row[&#39;filename&#39;]<br>#         file = join(image_path, filename)<br><br>#         if os.path.exists(file):<br>#             img_type = imghdr.what(file)<br>#             if img_type is None or img_type == &#39;gif&#39;:<br>#                 os.remove(file)<br>#             else:<br>#                 try:<br>#                     with Image.open(file) as img:<br>#                         width, height = img.size<br>#                         if width &lt; min_size[0] or height &lt; min_size[1]:<br>#                             os.remove(file)<br>#                         else:<br>#                             result.append(row)  # Only keep non-GIF, non-small images<br>#                 except Exception as e:<br>#                     pass<br>#     print(&#39;len of filtered data: {}&#39;.format(len(result)))<br>#     df = pd.DataFrame(result)<br>#     df.to_csv(csv_out_file, index=False)<br><br># filter_gif(train_csv_file, train_csv_out_file)<br># filter_gif(test_csv_file, test_csv_out_file)</pre><p>Define clip dataset for text only.</p><pre>class CLIPTextDataset(Dataset):<br><br>    def __init__(self, csv_path):<br>        df = pd.read_csv(csv_path, usecols=[&#39;text&#39;, &#39;filename&#39;])<br>        data_list = df.to_dict(&#39;records&#39;)<br>        print(&#39;len of data:{}&#39;.format(len(data_list)))<br>        self.data_list = data_list<br><br>    def __len__(self):<br>        return len(self.data_list)<br><br>    def __getitem__(self, index):<br>        row = self.data_list[index]<br>        text = row[&#39;text&#39;].strip()<br><br>        return text</pre><p>Convert all the text data to embeddings to avoid repeat computations.</p><pre>import pickle<br><br>text_model_dir = &quot;TencentBAC/Conan-embedding-v1&quot;<br>pkl_file_name = f&quot;text_embeddings_{text_model_dir.replace(&#39;/&#39;, &#39;_&#39;)}.pkl&quot;<br>if not os.path.exists(pkl_file_name):<br>    device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br>    text_model = TextEncoder(1024).to(device)<br>    train_set = CLIPTextDataset(csv_path=train_csv_out_file)<br>    embd_train_loader = DataLoader(train_set, shuffle=False, batch_size=256)<br>    qwen_tokenizer = AutoTokenizer.from_pretrained(text_model_dir)<br>    embedding_dict = {}<br><br>    for i, texts in tqdm(enumerate(embd_train_loader)):<br>        inputs = qwen_tokenizer(texts, padding=&quot;max_length&quot;, truncation=True, max_length=40, return_tensors=&quot;pt&quot;)<br>        inputs = {k: v.cuda() for k, v in inputs.items()}<br>        embd = text_model.inference_text_embedding(inputs)<br>        # TODO: save embedding dict {&#39;text&#39;: embd}<br>        for idx, txt in enumerate(texts):<br>            embedding_dict[txt] = embd[idx].cpu().numpy()<br><br>    with open(pkl_file_name, &quot;wb&quot;) as f:<br>        pickle.dump(embedding_dict, f)<br>else:<br>    with open(pkl_file_name, &quot;rb&quot;) as f:<br>        embedding_dict = pickle.load(f)<br><br>    # Now `embedding_dict` is a dictionary with your text embeddings<br>    print(f&quot;Loaded {len(embedding_dict)} embeddings from the file.&quot;)<br>Loaded 676251 embeddings from the file.</pre><p>Images and embeddings only dataset.</p><pre>class CLIPDataset(Dataset):<br><br>    def __init__(self, csv_path, image_path, embd_dict=None):<br>        df = pd.read_csv(csv_path, usecols=[&#39;text&#39;, &#39;filename&#39;])<br>        data_list = df.to_dict(&#39;records&#39;)<br>        print(&#39;len of data:{}&#39;.format(len(data_list)))<br>        self.data_list = data_list<br>        self.embd_dict = embd_dict<br>        self.image_path = image_path<br>        self.feature_extractor = ViTImageProcessor.from_pretrained(&quot;WinKawaks/vit-tiny-patch16-224&quot;)<br><br>    def __len__(self):<br>        return len(self.data_list)<br><br>    def __getitem__(self, index):<br>        row = self.data_list[index]<br>        text = row[&#39;text&#39;].strip()<br>        filename = row[&#39;filename&#39;]<br>        file = join(self.image_path, filename)<br>        try:<br>            image = Image.open(file).convert(&#39;RGB&#39;)<br>        except Exception as e:<br>            logger.info(&#39;open image error, text: {}, filename:{}&#39;.format(text, filename))<br>            logger.info(e)<br>            image = None<br><br>        image = self.feature_extractor(images=image, return_tensors=&quot;pt&quot;)[&#39;pixel_values&#39;].squeeze(0)<br>        embd = torch.zeros(2) if self.embd_dict is None else self.embd_dict[text]<br>        data = {&#39;image&#39;: image, &#39;text&#39;: text, &#39;embd&#39;: embd}<br>        return data</pre><h4>7. <strong>Set Hyperparameters</strong></h4><p>Hyperparameters, like learning rate, batch size, and number of layers, control model training. These are set based on experimentation to optimize the model’s performance.</p><pre>from torch.optim.lr_scheduler import LambdaLR<br><br>sgd_learning_rate = 1e-3<br>emb_dim = 1024<br><br>epochs = 50<br>batch_size = 640<br>warmup_steps = 1000</pre><h4>8. <strong>Load Dataset</strong></h4><p>This step involves loading the preprocessed dataset in batches, enabling the model to efficiently process and learn from the data during training.</p><pre>train_set = CLIPDataset(csv_path=train_csv_out_file, image_path=image_path, embd_dict=embedding_dict)<br>test_set = CLIPDataset(csv_path=test_csv_out_file, image_path=image_path)<br><br>train_loader = DataLoader(train_set, shuffle=True, batch_size=batch_size, num_workers=4)<br>test_loader = DataLoader(test_set, shuffle=False, batch_size=16)</pre><pre>len of data:681665<br>len of data:904</pre><p>Visualize a few samples from the training dataset (showing images, captions, and embeddings)</p><pre>import matplotlib.pyplot as plt<br>import torch<br>import numpy as np<br>import warnings<br><br>warnings.filterwarnings(&quot;ignore&quot;, category=UserWarning, module=&quot;IPython.core.pylabtools&quot;)<br><br>def visualize_samples_from_dataset(dataset, num_samples=3):<br>    &quot;&quot;&quot;<br>    Visualize a few samples (images, captions, and embeddings) from the training dataset.<br>    <br>    Args:<br>    dataset (Dataset): The dataset object.<br>    num_samples (int): The number of samples to visualize.<br>    &quot;&quot;&quot;<br>    # Get a few samples from the dataset<br>    for i in range(num_samples):<br>        sample = dataset[i]<br>        image = sample[&#39;image&#39;]  # The image tensor<br>        caption = sample[&#39;text&#39;]  # The text/caption<br>        embedding = sample[&#39;embd&#39;]  # The embedding<br><br>        # Convert the image tensor to a numpy array for visualization<br>        image_np = image.permute(1, 2, 0).numpy()  # Change dimensions from (C, H, W) to (H, W, C)<br>        image_np = (image_np - image_np.min()) / (image_np.max() - image_np.min())  # Normalize to [0, 1]<br><br>        # Plot the image and caption<br>        plt.figure(figsize=(5, 5))<br>        plt.imshow(image_np)<br>        print(f&quot;Caption: {caption}&quot;)<br>        plt.axis(&#39;off&#39;)<br>        plt.show()<br><br>        # Print the corresponding embedding<br>        print(f&quot;Embedding (sample {i+1}): {embedding}\n&quot;)<br><br># Visualize 3 samples from the training dataset<br>visualize_samples_from_dataset(train_set, num_samples=3)</pre><pre>Caption: 今年能跑赢96不?备战坦克两项俄军开始选拔参赛队员</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*t5S4dKWSgB3BJCtu2IfQ8g.png" /></figure><pre>Embedding (sample 1): [-0.5752107  -0.32569852  0.14942744 ...  0.5165749  -0.7796664<br> -0.16230711]<br><br>Caption: 13/14赛季 英超第5轮 曼城 vs 曼联 13.09.22</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*BM-tYMuYkCSbv2JQ3F7d0Q.png" /></figure><pre>Embedding (sample 2): [-0.9595318  -0.31669042 -0.75011516 ...  0.5824591   0.01416251<br> -0.0221024 ]<br><br>Caption: 一是持续加大&quot;双一流&quot;建设支持力度.</pre><figure><img alt="png" src="https://cdn-images-1.medium.com/proxy/1*ebQh199COlLtXkHYLSewRg.png" /></figure><pre>Embedding (sample 3): [-1.1964369  -1.4531888  -0.57500637 ...  0.12495439  0.12432495<br> -0.33586353]</pre><p>Define learning rate scheduler</p><pre>total_steps = epochs * len(train_loader)<br>def lr_lambda(step):<br>    if step &lt; warmup_steps:<br>        return step / warmup_steps<br>    else:<br>        return 0.5 * (1 + torch.cos(torch.tensor((step - warmup_steps) / (total_steps - warmup_steps) * 3.1416)))</pre><h4>9. <strong>Train the Model</strong></h4><p>Minimize contrastive loss, which encourages the model to correctly match images with their corresponding text descriptions. View loss an dlr information using tensorbaord.</p><pre>import torch<br>import torch.optim as optim<br>from torch.optim.lr_scheduler import CosineAnnealingLR<br>from torch.utils.tensorboard import SummaryWriter<br><br># Initialize TensorBoard writer<br>writer = SummaryWriter()<br><br>device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br>print(&quot;Using device: &quot;, device, f&quot;({torch.cuda.get_device_name(device)})&quot; if torch.cuda.is_available() else &quot;&quot;)<br><br>model = CLIP(emb_dim, train=True).to(device)<br><br>optimizer = optim.SGD(model.parameters(), lr=sgd_learning_rate)<br>scheduler = scheduler = LambdaLR(optimizer, lr_lambda)<br><br>best_loss = np.inf<br>losses = []<br>steps = []<br><br>step = 0<br><br>for epoch in range(epochs):<br>    for i, data in enumerate(train_loader, 0):<br>        img, embd = data[&quot;image&quot;].to(device), data[&quot;embd&quot;].to(device)<br><br>        # Forward pass and compute loss<br>        loss = model(img, embd)<br>        optimizer.zero_grad()<br>        loss.backward()<br>        optimizer.step()<br><br>        scheduler.step()<br><br>        # Store loss and step<br>        losses.append(loss.item())<br>        step += img.size(0)  # Track the total number of images processed<br>        steps.append(step)<br><br>        # Log metrics to TensorBoard<br>        writer.add_scalar(&#39;Loss/train&#39;, loss.item(), step)<br>        writer.add_scalar(&#39;Learning Rate&#39;, scheduler.get_last_lr()[0], step)<br><br>    loss_ep = np.mean(losses[-len(train_loader):])<br>    if loss_ep &lt; best_loss:<br>        best_loss = loss_ep<br>        torch.save(model.state_dict(), &quot;./clip_pretrain.pt&quot;)<br>        print(&quot;Model Saved.&quot;)<br><br>writer.close()</pre><pre>from IPython.display import display, HTML<br><br>def add_dark_background_to_svg(svg_content):<br>    # Insert a black background rectangle at the start of the &lt;svg&gt; element<br>    svg_content = svg_content.replace(<br>        &#39;&lt;svg&#39;,<br>        &#39;&lt;svg style=&quot;background-color:black;&quot;&#39;<br>    )<br>    return svg_content<br><br># Read and display the SVG with dark background for &quot;Loss_train.svg&quot;<br>with open(&#39;./Loss_train.svg&#39;, &#39;r&#39;) as svg_file:<br>    svg_content = svg_file.read()<br>    modified_svg_content = add_dark_background_to_svg(svg_content)<br>display(HTML(modified_svg_content))<br><br># Read and display the SVG with dark background for &quot;Learning Rate.svg&quot;<br>with open(&#39;./Learning Rate.svg&#39;, &#39;r&#39;) as svg_file:<br>    svg_content = svg_file.read()<br>    modified_svg_content = add_dark_background_to_svg(svg_content)<br>display(HTML(modified_svg_content))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qFf9FboSImr5idXrWHJRBA.png" /><figcaption>Training Loss</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0WEH1l2PqHwYtUMXNpJqGA.png" /><figcaption>Learning Rate</figcaption></figure><h4>10. <strong>Test the Model</strong></h4><p>After training, we evaluate the model’s performance on the test set by measuring its accuracy in matching images to their corresponding text descriptions.</p><pre>device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br>print(&quot;Using device: &quot;, device, f&quot;({torch.cuda.get_device_name(device)})&quot; if torch.cuda.is_available() else &quot;&quot;)<br><br># Loading Best Model<br>model = CLIP(emb_dim).to(device)<br>model.load_state_dict(torch.load(&quot;./clip_pretrain.pt&quot;, map_location=device), strict=False)<br>qwen_tokenizer = AutoTokenizer.from_pretrained(text_model_dir)<br>correct, total = 0,0<br>with torch.no_grad():<br>    for data in test_loader:<br>        images, texts, masks = data[&quot;image&quot;].to(device), data[&quot;text&quot;], data[&quot;embd&quot;]<br>        inputs = qwen_tokenizer(texts, padding=&quot;max_length&quot;, truncation=True, max_length=40, return_tensors=&quot;pt&quot;)<br>        inputs = {k: v.cuda() for k, v in inputs.items()}<br><br>        image_features = model.image_encoder(images)  # [batch_size, emb_dim]<br>        text_features = model.text_encoder.inference(inputs)  # [batch_size, emb_dim]<br><br>        image_features /= image_features.norm(dim=-1, keepdim=True)<br>        text_features /= text_features.norm(dim=-1, keepdim=True)<br>        similarity = (100.0 * image_features @ text_features.T)  # [batch_size, batch_size]<br><br>        # Extract diagonal elements for correct pairs (corresponding image-text pairs)<br>        correct_pair_similarity = torch.diag(similarity)<br><br>        # For each image, compare its similarity to the correct text vs other texts in the batch<br>        for i in range(len(correct_pair_similarity)):<br>            correct_similarity = correct_pair_similarity[i].item()<br>            other_similarities = similarity[i].tolist()<br>            other_similarities.pop(i)<br><br>            if correct_similarity &gt; max(other_similarities):<br>                correct += 1<br>            total += 1 <br><br>accuracy = 100 * correct / total<br>print(f&#39;\nModel Accuracy: {accuracy:.2f}%&#39;)</pre><pre>Using device:  cuda (NVIDIA GeForce RTX 4090)<br><br><br>Some weights of ViTModel were not initialized from the model checkpoint at WinKawaks/vit-tiny-patch16-224 and are newly initialized: [&#39;vit.pooler.dense.bias&#39;, &#39;vit.pooler.dense.weight&#39;]<br>You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.<br>/tmp/ipykernel_8119/1231573180.py:20: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don&#39;t have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.<br>  k.replace(&quot;linear.&quot;, &quot;&quot;): v for k, v in torch.load(model_file).items()<br>/tmp/ipykernel_8119/3340573946.py:6: FutureWarning: You are using `torch.load` with `weights_only=False` (the current default value), which uses the default pickle module implicitly. It is possible to construct malicious pickle data which will execute arbitrary code during unpickling (See https://github.com/pytorch/pytorch/blob/main/SECURITY.md#untrusted-models for more details). In a future release, the default value for `weights_only` will be flipped to `True`. This limits the functions that could be executed during unpickling. Arbitrary objects will no longer be allowed to be loaded via this mode unless they are explicitly allowlisted by the user via `torch.serialization.add_safe_globals`. We recommend you start setting `weights_only=True` for any use case where you don&#39;t have full control of the loaded file. Please open an issue on GitHub for any issues related to this experimental feature.<br>  model.load_state_dict(torch.load(&quot;./clip_pretrain.pt&quot;, map_location=device), strict=False)<br><br><br><br>Model Accuracy: 87.39%</pre><p>In this experiment, we trained a CLIP model to match Chinese text descriptions to images using the Chinese-CLIP dataset. However, computational limitations affected the process. Here are some key techniques that were used to address these challenges:</p><ol><li><strong>Fine-tuning Pretrained Models</strong>: Instead of training from scratch, we fine-tune the text and image encoders from pre-trained models, significantly reducing the number of required epochs.</li><li><strong>Focus on Image Encoder</strong>: In our experiments, the image encoder had the most significant impact on the final performance. We added a trainable linear layer to the pre-trained LLM-based text-embedding models and fine-tuned the entire ViT encoder.</li><li><strong>Embedding Precomputation</strong>: To reduce memory requirements and speed up training, we precompute and store the text embeddings in a dictionary. This avoids the need for repeated inference on the text encoder during training.</li><li><strong>Small ViT Model</strong>: Using a small model like vit-tiny-patch16-224 reduces memory consumption and makes training more efficient.</li><li><strong>Optimizers</strong>: We opted for SGD without momentum, as it consumes less memory (8 bytes per parameter) compared to optimizers like Adam, which require 16 bytes.</li><li><strong>Advanced Optimization Techniques</strong>: As a common sence, the ViT trained by Adam is usually better than the ones trained by SGD. This <a href="https://arxiv.org/pdf/2211.09359">paper</a> tells us how to train a even better ViT using SGD. They mentioned that when using SGD, the final performence will be worse if the first patch2token layer has too large gradient in the beginning of training. To optimize ViT training with SGD, we applied the following tricks:</li></ol><ul><li>Freezing the patch-to-token layer and associated LayerNorm layers (if exists).</li><li>Cosine learning rate scheduler.</li><li>1000-step warm-up phase.</li></ul><p>The combination of these techniques allowed us to achieve <strong>87.39% accuracy</strong> on the test set, which contains 700k images and text pairs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d69bd4aada7e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/correll-lab/training-a-clip-model-on-extremly-large-dataset-with-limited-resources-d69bd4aada7e">Training a CLIP Model on Extremly Large Dataset with Limited Resources</a> was originally published in <a href="https://medium.com/correll-lab">Toward Humanoids</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a CLIP Model with Limited Resources: An Experiment]]></title>
            <link>https://lekai.medium.com/building-a-clip-model-with-limited-resources-an-experiment-b6ccd949aaf6?source=rss-1682ebfabfa5------2</link>
            <guid isPermaLink="false">https://medium.com/p/b6ccd949aaf6</guid>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Lekai Chen]]></dc:creator>
            <pubDate>Tue, 08 Oct 2024 05:57:03 GMT</pubDate>
            <atom:updated>2024-10-08T06:15:40.724Z</atom:updated>
            <content:encoded><![CDATA[<p>In this blog, we will walk through the process of building a simplified CLIP (Contrastive Language-Image Pretraining) model using a limited dataset and computational resources. The model is designed to match Chinese text descriptions with corresponding images, and we use a subset of the Chinese-CLIP dataset for training and testing.</p><p>Due to the constraints of this experiment, the model architecture is narrow, and the training was done for only a few epochs, which resulted in limited performance. Let’s dive into the details.</p><p>The notebook on colab: <a href="https://colab.research.google.com/drive/1QOlLplI0MMBFkZr1l7FoyfsJ0yB_Vt_O?usp=sharing">https://colab.research.google.com/drive/1QOlLplI0MMBFkZr1l7FoyfsJ0yB_Vt_O?usp=sharing</a></p><p>The dataset can be found here: <a href="https://o365coloradoedu-my.sharepoint.com/:f:/g/personal/lech8988_colorado_edu/Ev7eHmZ7n8JEs0UE-cFGF3EB9nmAB-0N07Dxx8DfDqTWYA?e=BV5rdj">https://o365coloradoedu-my.sharepoint.com/:f:/g/personal/lech8988_colorado_edu/Ev7eHmZ7n8JEs0UE-cFGF3EB9nmAB-0N07Dxx8DfDqTWYA?e=BV5rdj</a></p><h3>Import Libraries and Modules</h3><p>First, we import the necessary libraries for constructing and training the CLIP model, along with modules for dataset management and image processing.</p><pre>import matplotlib.pyplot as plt<br>import numpy as np<br>import os<br>import pandas as pd<br>import requests<br>import multiprocessing<br>import pandas as pd<br>import imghdr<br>from loguru import logger<br>from glob import glob<br>from os.path import join<br>from tqdm import tqdm<br>from multiprocessing import cpu_count<br>from PIL import Image<br><br>import torch<br>torch.cuda.empty_cache()<br>import torch.nn as nn<br>import torch.optim as optim<br>import torchvision.transforms as T<br><br>from torch.optim import Adam<br>from torch.utils.data import Dataset, DataLoader<br>from datasets import load_dataset<br>from transformers import ViTFeatureExtractor, AutoTokenizer</pre><h3>Defining Positional Encoding</h3><p>Positional encoding allows the transformer model to understand the order of tokens in the input sequence. This is critical since transformers themselves are order-agnostic.</p><pre>class PositionalEmbedding(nn.Module):<br>  def __init__(self, width, max_seq_length):<br>    super().__init__()<br>    pe = torch.zeros(max_seq_length, width)<br>    for pos in range(max_seq_length):<br>      for i in range(width):<br>        if i % 2 == 0:<br>          pe[pos][i] = np.sin(pos/(10000 ** (i/width)))<br>        else:<br>          pe[pos][i] = np.cos(pos/(10000 ** ((i-1)/width)))<br>    self.register_buffer(&#39;pe&#39;, pe.unsqueeze(0))<br><br>def forward(self, x):<br>    x = x + self.pe<br>    return x</pre><h3>Defining the Attention Head</h3><p>An attention head is a core part of the transformer architecture that helps the model focus on different parts of the input sequence.</p><pre>class AttentionHead(nn.Module):<br>  def __init__(self, width, head_size):<br>    super().__init__()<br>    self.head_size = head_size<br><br>    self.query = nn.Linear(width, head_size)<br>    self.key = nn.Linear(width, head_size)<br>    self.value = nn.Linear(width, head_size)<br><br>  def forward(self, x, mask=None):<br>    # Obtaining Queries, Keys, and Values<br>    Q = self.query(x)<br>    K = self.key(x)<br>    V = self.value(x)<br><br>    # Dot Product of Queries and Keys<br>    attention = Q @ K.transpose(-2,-1)<br><br>    # Scaling<br>    attention = attention / (self.head_size ** 0.5)<br><br>    # Applying Attention Mask<br>    if mask is not None:<br>        attention = attention.masked_fill(mask == 0, float(&quot;-inf&quot;))<br><br>    attention = torch.softmax(attention, dim=-1)<br><br>    attention = attention @ V<br><br>    return attention</pre><h3>Multi-Head Attention</h3><p>Multi-head attention allows the model to focus on different representations of the input sequence at the same time.</p><pre>class MultiHeadAttention(nn.Module):<br>  def __init__(self, width, n_heads):<br>    super().__init__()<br>    self.head_size = width // n_heads<br>    self.W_o = nn.Linear(width, width)<br>    self.heads = nn.ModuleList([AttentionHead(width, self.head_size) for _ in range(n_heads)])<br><br>  def forward(self, x, mask=None):<br>    # Combine attention heads<br>    out = torch.cat([head(x, mask=mask) for head in self.heads], dim=-1)<br>    out = self.W_o(out)<br>    return out</pre><h3>Creating the Transformer Encoder</h3><p>We create the encoder based on the transformer architecture, which will process both text and image inputs.</p><pre>class TransformerEncoder(nn.Module):<br>    def __init__(self, width, n_heads, r_mlp=4):<br>        super().__init__()<br>        self.width = width<br>        self.n_heads = n_heads<br><br>        # Sub-Layer 1 Normalization<br>        self.ln1 = nn.LayerNorm(width)<br>        # Multi-Head Attention<br>        self.mha = MultiHeadAttention(width, n_heads)<br>        # Sub-Layer 2 Normalization<br>        self.ln2 = nn.LayerNorm(width)<br>        # Multilayer Perception<br>        self.mlp = nn.Sequential(<br>            nn.Linear(self.width, self.width*r_mlp),<br>            nn.GELU(),<br>            nn.Linear(self.width*r_mlp, self.width)<br>        )<br><br><br>    def forward(self, x, mask=None):<br>        x = x + self.mha(self.ln1(x), mask=mask)<br>        x = x + self.mlp(self.ln2(x))<br>        return x</pre><h3>Tokenizing the Input</h3><p>Here, we define a tokenizer using the Qwen’s tokenizer to efficiently encode Chinese text. The tokenizer converts text into token IDs that the model can process.</p><pre>qwen_tokenizer = AutoTokenizer.from_pretrained(&quot;Qwen/Qwen2.5-0.5B&quot;)<br><br>def tokenizer(text, encode=True, mask=None, max_seq_length=32):<br>    if encode:<br>        # Tokenize input text using Qwen tokenizer<br>        encoding = qwen_tokenizer(<br>            text,<br>            padding=&#39;max_length&#39;,  # Automatically pads to max_seq_length<br>            truncation=True,        # Truncates if the text exceeds max_seq_length<br>            max_length=max_seq_length,<br>            return_tensors=&#39;pt&#39;     # Returns a PyTorch tensor<br>        )<br>        <br>        # Extract token IDs and attention mask from the encoding<br>        out = encoding[&#39;input_ids&#39;].squeeze(0)  # input_ids tensor<br>        mask = encoding[&#39;attention_mask&#39;].squeeze(0)  # attention mask tensor<br><br>    else:<br>        # Convert token IDs back to text<br>        decoded_text = qwen_tokenizer.decode(text, skip_special_tokens=True)<br>        out = decoded_text<br>        mask = None<br><br>    return out, mask</pre><h3>Defining the Text Encoder</h3><p>The text encoder processes the tokenized input text and converts it into an embedding that can be compared with image embeddings.</p><pre>class TextEncoder(nn.Module):<br>    def __init__(self, vocab_size, width, max_seq_length, n_heads, n_layers, emb_dim):<br>        super().__init__()<br>        self.max_seq_length = max_seq_length<br>        self.encoder_embedding = nn.Embedding(vocab_size, width)<br>        self.positional_embedding = PositionalEmbedding(width, max_seq_length)<br>        self.encoder = nn.ModuleList([TransformerEncoder(width,n_heads) for _ in range(n_layers)])<br>        # learned proj of image to embed<br>        self.projection = nn.Parameter(torch.randn(width, emb_dim))<br><br>    def forward(self, text, mask=None):<br>        # Text Embedding<br>        x = self.encoder_embedding(text)<br>        # Positional Embedding<br>        x = self.positional_embedding(x)<br>        # Transformer Encoder<br>        for encoder_layer in self.encoder:<br>            x = encoder_layer(x, mask=mask)<br>        # Takes features from the EOT Embedding<br>        x = x[torch.arange(text.shape[0]), torch.sub(torch.sum(mask[:,0],dim=1),1)]<br>        # joint multimodal embedding<br>        if self.projection is not None:<br>            x = x @ self.projection<br>        x = x / torch.norm(x, dim=-1, keepdim=True)<br>        return x</pre><h3>Defining the Image Encoder</h3><p>The image encoder processes input images and generates an embedding that can be compared to the text embedding.</p><pre>class ImageEncoder(nn.Module):<br>    def __init__(self, width, img_size, patch_size, n_channels, n_layers, n_heads, emb_dim):<br>        super().__init__()<br><br>        assert img_size[0] % patch_size[0] == 0 and img_size[1] % patch_size[1] == 0, &quot;img_size dimensions must be divisible by patch_size dimensions&quot;<br>        assert width % n_heads == 0, &quot;width must be divisible by n_heads&quot;<br><br>        self.n_patches = (img_size[0] * img_size[1]) // (patch_size[0] * patch_size[1])<br><br>        self.max_seq_length = self.n_patches + 1<br><br>        self.linear_project = nn.Conv2d(n_channels, width, kernel_size=patch_size, stride=patch_size)<br><br>        self.cls_token = nn.Parameter(torch.randn(1, 1, width))<br><br>        self.positional_embedding = PositionalEmbedding(width,self.max_seq_length)<br><br>        self.encoder = nn.ModuleList([TransformerEncoder(width,n_heads) for _ in range(n_layers)])<br><br><br>        # learned proj of image to embed<br>        self.projection = nn.Parameter(torch.randn(width, emb_dim))<br><br><br>    def forward(self,x):<br>        # Patch Embedding<br>        x = self.linear_project(x)<br>        x = x.flatten(2).transpose(1, 2)<br><br>        # Positional Embedding<br>        x = torch.cat((self.cls_token.expand(x.size()[0], -1, -1),x), dim=1)<br>        x = self.positional_embedding(x)<br><br>        # Transformer Encoder<br>        for encoder_layer in self.encoder:<br>            x = encoder_layer(x)<br><br>        # Getting Class Tokens<br>        x = x[:, 0, :]<br><br>        # joint multimodal embedding<br>        if self.projection is not None:<br>            x = x @ self.projection<br><br>        x = x / torch.norm(x, dim=-1, keepdim=True)<br><br>        return x</pre><h3>The Complete CLIP Model</h3><p>The CLIP model (Contrastive Language-Image Pretraining) combines text and image encoders to learn associations between images and corresponding text descriptions by comparing their encoded representations.</p><pre>class CLIP(nn.Module):<br>    def __init__(self, emb_dim, vit_width, img_size, patch_size, n_channels, vit_layers, vit_heads, vocab_size, text_width, max_seq_length, text_heads, text_layers):<br>        super().__init__()<br><br>        self.image_encoder = ImageEncoder(vit_width, img_size, patch_size, n_channels, vit_layers, vit_heads, emb_dim)<br><br>        self.text_encoder = TextEncoder(vocab_size, text_width, max_seq_length, text_heads, text_layers, emb_dim)<br><br>        self.temperature = nn.Parameter(torch.ones([]) * np.log(1 / 0.07))<br><br>        self.device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br><br><br>    def forward(self,image,text, mask=None):<br>        I_e = self.image_encoder(image)<br>        T_e = self.text_encoder(text, mask=mask)<br><br>        # scaled pairwise cosine similarities [n, n]<br>        logits = (I_e @ T_e.transpose(-2,-1)) * torch.exp(self.temperature)<br><br>        # symmetric loss function<br>        labels = torch.arange(logits.shape[0]).to(self.device)<br><br>        loss_i = nn.functional.cross_entropy(logits.transpose(-2,-1), labels)<br>        loss_t = nn.functional.cross_entropy(logits, labels)<br><br>        loss = (loss_i + loss_t) / 2<br><br>        return loss</pre><h3>Dataset: Chinese-CLIP</h3><p>We define a subset of the Chinese-CLIP dataset, which contains 1.37 million training images and 10,000 testing images. First, we download the images.</p><pre>train_csv_file = &#39;./data/train-137w.csv&#39;<br>test_csv_file = &#39;./data/test-2.6w.csv&#39;<br>image_path = &#39;./data/images&#39;<br>train_csv_out_file = &#39;./data/train-filter.csv&#39;<br>test_csv_out_file = &#39;./data/test-filter.csv&#39;<br><br>train_set_num=10000<br>test_set_num=1000<br><br>def download(file, url):<br>    try:<br>        headers = {&quot;User-Agent&quot;: &quot;Chrome/68.0.3440.106&quot;}<br>        response = requests.get(url, headers=headers, timeout=2)<br>        status_code = response.status_code<br>        content_type = response.headers.get(&#39;Content-Type&#39;)<br>        # NO GIF<br>        if status_code == 200 and content_type != &#39;image/gif&#39;:<br>            image = response.content<br>            with open(file, &#39;wb&#39;) as f:<br>                f.write(image)<br>    except Exception as e:<br>        logger.info(&#39;downloading image error, url:{}&#39;.format(url))<br>        logger.info(e)<br><br><br>def download_images(csv_file, n=-1):<br>    thread_num = cpu_count()  # cpu_count()<br>    if not os.path.exists(image_path):<br>        os.makedirs(image_path)<br><br>    df = pd.read_csv(csv_file)<br>    df = df[:n]<br>    print(len(df))<br><br>    # Initialize thread pool<br>    pool = multiprocessing.Pool(processes=thread_num)<br>    for _, row in tqdm(df.iterrows()):<br>        filename = row[&#39;filename&#39;]<br>        url = row[&#39;url&#39;]<br>        file = join(image_path, filename)<br>        if os.path.exists(file):<br>            continue<br>        pool.apply_async(download, (file, url))<br>    pool.close()<br>    pool.join()<br><br>download_images(csv_file=train_csv_file, n=train_set_num)<br>download_images(csv_file=test_csv_file, n=test_set_num)</pre><h3>Filtering Downloaded Images (Removing GIFs)</h3><p>The downloaded dataset contains GIFs and unsupported formats. We filter out GIFs and remove the corresponding entries from the CSV file to ensure valid images are used.</p><pre>def filter_gif(csv_in_file, csv_out_file, min_size=(10, 10)):<br>    result = []<br>    df = pd.read_csv(csv_in_file)<br>    for _, row in tqdm(df.iterrows(), desc=&#39;filter gif and small images&#39;):<br>        filename = row[&#39;filename&#39;]<br>        file = join(image_path, filename)<br><br>        if os.path.exists(file):<br>            img_type = imghdr.what(file)<br>            if img_type is None or img_type == &#39;gif&#39;:<br>                print(&#39;remove file (GIF or invalid): {}&#39;.format(file))<br>                os.remove(file)<br>            else:<br>                try:<br>                    with Image.open(file) as img:<br>                        width, height = img.size<br>                        if width &lt; min_size[0] or height &lt; min_size[1]:<br>                            print(f&#39;remove file (small image): {file}, size: {img.size}&#39;)<br>                            os.remove(file)<br>                        else:<br>                            result.append(row)  # Only keep non-GIF, non-small images<br>                except Exception as e:<br>                    print(f&#39;Error processing file {file}: {e}&#39;)<br>    print(&#39;len of filtered data: {}&#39;.format(len(result)))<br>    df = pd.DataFrame(result)<br>    df.to_csv(csv_out_file, index=False)<br><br><br>filter_gif(train_csv_file, train_csv_out_file)<br>filter_gif(test_csv_file, test_csv_out_file)</pre><h3>Defining Hyperparameters</h3><p>We define the key hyperparameters for training the model, such as the embedding dimension, model width, and the number of layers.</p><pre>emb_dim = 32<br>vit_width = 9<br>img_size = (224, 224)<br>patch_size = (16, 16)<br>n_channels = 1<br>vit_layers = 3<br>vit_heads = 3<br>vocab_size = 151643<br>text_width = 32<br>max_seq_length = 32<br>text_heads = 8<br>text_layers = 4<br>lr = 5e-5<br>epochs = 50<br>batch_size = 128</pre><h3>Loading the Dataset</h3><p>We load the preprocessed dataset into the model and prepare it for training and evaluation.</p><pre>train_set = CLIPDataset(csv_path=train_csv_out_file, image_path=image_path, max_seq_length=max_seq_length)<br>test_set = CLIPDataset(csv_path=test_csv_out_file, image_path=image_path, max_seq_length=max_seq_length)<br><br>train_loader = DataLoader(train_set, shuffle=True, batch_size=batch_size)<br>test_loader = DataLoader(test_set, shuffle=False, batch_size=64)</pre><h3>Training the Model</h3><p>We train the CLIP model on the limited subset of the dataset using the defined hyperparameters.</p><pre>device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br>print(&quot;Using device: &quot;, device, f&quot;({torch.cuda.get_device_name(device)})&quot; if torch.cuda.is_available() else &quot;&quot;)<br><br>model = CLIP(emb_dim, vit_width, img_size, patch_size, n_channels, vit_layers, vit_heads, vocab_size, text_width, max_seq_length, text_heads, text_layers).to(device)<br><br>optimizer = optim.AdamW(model.parameters(), lr=lr)<br><br>best_loss = np.inf<br>for epoch in range(epochs):<br>    for i, data in enumerate(train_loader, 0):<br>        img, cap, mask = data[&quot;image&quot;].to(device), data[&quot;caption&quot;].to(device), data[&quot;mask&quot;].to(device)<br>        loss = model(img,cap,mask)<br>        optimizer.zero_grad()<br>        loss.backward()<br>        optimizer.step()<br><br>        print(f&quot;Epoch [{epoch+1}/{epochs}], Batch Loss: {loss.item():.3f}&quot;)<br><br>    # Saves model if it performed better than the previous best<br>    if loss.item() &lt;= best_loss:<br>        best_loss = loss.item()<br>        torch.save(model.state_dict(), &quot;./clip.pt&quot;)<br>        print(&quot;Model Saved.&quot;)</pre><h3>Testing the Model</h3><p>After training, we evaluate the model’s performance on the testing set by computing the accuracy of matching images with their corresponding text descriptions.</p><pre># Loading Best Model<br>model = CLIP(emb_dim, vit_width, img_size, patch_size, n_channels, vit_layers, vit_heads, vocab_size, text_width, max_seq_length, text_heads, text_layers).to(device)<br>model.load_state_dict(torch.load(&quot;./clip.pt&quot;, map_location=device))<br><br>correct, total = 0,0<br>with torch.no_grad():<br>    for data in test_loader:<br>        images, labels, masks = data[&quot;image&quot;].to(device), data[&quot;caption&quot;].to(device), data[&quot;mask&quot;].to(device)<br><br>        image_features = model.image_encoder(images)  # [batch_size, emb_dim]<br>        text_features = model.text_encoder(labels, mask=masks)  # [batch_size, emb_dim]<br><br>        image_features /= image_features.norm(dim=-1, keepdim=True)<br>        text_features /= text_features.norm(dim=-1, keepdim=True)<br>        similarity = (100.0 * image_features @ text_features.T)  # [batch_size, batch_size]<br><br>        # Extract diagonal elements for correct pairs (corresponding image-text pairs)<br>        correct_pair_similarity = torch.diag(similarity)<br><br>        # For each image, compare its similarity to the correct text vs other texts in the batch<br>        for i in range(len(correct_pair_similarity)):<br>            correct_similarity = correct_pair_similarity[i].item()<br>            other_similarities = similarity[i].tolist()<br>            other_similarities.pop(i)<br><br>            if correct_similarity &gt; max(other_similarities):<br>                correct += 1<br>            total += 1 <br><br>accuracy = 100 * correct / total<br>print(f&#39;\nModel Accuracy: {accuracy:.2f}%&#39;)</pre><h3>Conclusion</h3><p>The goal of this experiment was to train a CLIP model for matching Chinese text descriptions with images using a subset of the Chinese-CLIP dataset. Due to limited computational resources, we trained the model on a smaller subset, used a narrower model architecture, and trained for only a few epochs. As a result, the model’s performance was limited and did not reach full potential. In future experiments, we plan to expand the dataset, use a wider architecture, and train the model for a longer period to improve accuracy and generalization.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b6ccd949aaf6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>