<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alexey Kravets on Medium]]></title>
        <description><![CDATA[Stories by Alexey Kravets on Medium]]></description>
        <link>https://medium.com/@alexml0123?source=rss-cf3e4a05b535------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Sk0BsQAVvq8-jQhT9iVrwQ.jpeg</url>
            <title>Stories by Alexey Kravets on Medium</title>
            <link>https://medium.com/@alexml0123?source=rss-cf3e4a05b535------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 23:50:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@alexml0123/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Continual Learning — A Deep Dive Into Elastic Weight Consolidation Loss]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/continual-learning-a-deep-dive-into-elastic-weight-consolidation-loss-7cda4a2d058c?source=rss-cf3e4a05b535------2"><img src="https://cdn-images-1.medium.com/max/601/1*3Ntzj-T3qKXwITawF3vAIA.png" width="601"></a></p><p class="medium-feed-snippet">With PyTorch Implementation</p><p class="medium-feed-link"><a href="https://medium.com/data-science/continual-learning-a-deep-dive-into-elastic-weight-consolidation-loss-7cda4a2d058c?source=rss-cf3e4a05b535------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/continual-learning-a-deep-dive-into-elastic-weight-consolidation-loss-7cda4a2d058c?source=rss-cf3e4a05b535------2</link>
            <guid isPermaLink="false">https://medium.com/p/7cda4a2d058c</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Alexey Kravets]]></dc:creator>
            <pubDate>Tue, 02 Jul 2024 19:52:35 GMT</pubDate>
            <atom:updated>2024-07-02T19:52:35.124Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Improving CLIP performance in training-free manner with few-shot examples]]></title>
            <link>https://medium.com/data-science/improving-clip-performance-in-training-free-manner-with-few-shot-examples-a59f6b29cdc8?source=rss-cf3e4a05b535------2</link>
            <guid isPermaLink="false">https://medium.com/p/a59f6b29cdc8</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[clip]]></category>
            <category><![CDATA[computer-vision]]></category>
            <dc:creator><![CDATA[Alexey Kravets]]></dc:creator>
            <pubDate>Tue, 30 Jan 2024 17:54:54 GMT</pubDate>
            <atom:updated>2024-01-30T17:54:54.252Z</atom:updated>
            <content:encoded><![CDATA[<h3>Improving CLIP Performance in Training-Free Manner with Few-Shot Examples</h3><h4>Part 3 — A simple extension to zero-shot classification with Tip-Adapter.</h4><p>This is the 3rd article on how to improve CLIP performance on classification. You can find the first <a href="https://medium.com/towards-data-science/simple-way-of-improving-zero-shot-clip-performance-4eae474cb447">here </a>and the second one <a href="https://medium.com/towards-data-science/improving-performance-and-explainability-of-zero-shot-clip-33e579d3f4bb">here</a>. In the first two articles our focus was on zero-shot classification where we discovered that leveraging a large language model (LLM) to tailor prompts could enhance CLIP’s zero-shot classification performance. In this article we will explore how CLIP’s classification performance can be further enhanced when provided with a few visual examples for each class. Before proceeding, I recommend refreshing your understanding of CLIP from my first article of this series.</p><h3>Introduction</h3><p>The zero-shot classification abilities of CLIP are constrained by the knowledge it acquires during pre-training. Consequently, if we aim to classify data that are rare or absent in CLIP’s pre-training data the classification performance may be not satisfactory. While assembling an extensive dataset can be challenging, obtaining a few examples for each class is typically feasible. One approach to enhance CLIP’s performance is to incorporate small adapters on top and train them with the few-shot images while keeping CLIP’s original weights frozen. However, there are instances where training even small adapters may not always be viable. Alternatively, we can leverage CLIP in a training-free manner while still benefiting from the new knowledge in the few-shot examples. In this article, we will explore how to achieve this using a method called Tip-Adapter [1].</p><h3>Cached model</h3><p>The main idea behind performing training-free classification is to utilize a cached model. This approach entails encoding the available few-shot training images using CLIP’s visual encoder and storing these encodings. During testing, this cached model, which includes associated labels, can be employed to calculate similarities between the test image and the cached images within the image space. As we have multiple images available per class, we can aggregate the similarities across different labels and utilize them as indicators to determine which labels in the embedding space the test image is closest to. This is illustrated in the figure below. Note how this process resembles the k-nearest neighborhood model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/796/1*hHOpUl0hqD9DpRR0izimsg.png" /><figcaption>Cached model at test time (figure by author)</figcaption></figure><p>In formulas we have:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/266/1*x9yQFfYaxq0ekNPtrHcU1Q.png" /></figure><p>Here the <em>A</em> (affinity matrix) is basically what we have shown in the diagram, however it’s slightly reformulated adding a modulating hyperparameter <em>beta </em>that controls the sharpness of the similarities and an exponential function to make the affinities not negative. At the end, however, it is like calculating the cosine similarities between the images. <br>For example, if the images have a similarity of 0.1 and 0.8, with the above formula they become: <br>for beta=2: np.exp(-beta * (1–0.1)) = 0.16, np.exp(-beta * (1–0.8)) = 0.67. <br>for beta=5: np.exp(-2 * (1–0.1)) = 0.01, np.exp(-2 * (1–0.8)) = 0.36.<br>for beta=1: np.exp(-2 * (1–0.1)) = 0.40, np.exp(-2 * (1–0.8)) = 0.81.</p><p>Thus, if we increase beta, the similarities get further apart while for smaller beta they get closer.</p><p>The matrix <em>A</em> is of shape (1 x NK) where N is the number of classes and K number of examples (shots) for each class. If we then sum the similarities (affinity matrix) by class, which can be simply done by multiplying the Affinity matrix by one-hot encoded matrix L_train (NK x N) with value of 1 at the class index and 0 otherwise, we get a (1 x N) matrix where each entry is the total similarity of the test image to the images belonging to a certain class.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/67/1*vX2wK_FGLBIwPoXydJyGuQ.png" /></figure><p>Cached model by itself is however less useful as we still aim to leverage CLIP’s zero-shot multimodal capabilities. To achieve this, we merge the logits from the cached model in the image space with the logits from CLIP’s zero-shot predictions in the image-text space. As we discussed in previous articles, zero-shot prediction involves embedding all textual prompts into a matrix <em>W</em> of shape (N x C) where C represents the number of CLIP hidden dimensions. Multiplying this matrix by the test image of shape (1 x C) yields the similarities between the image and each textual prompt, resulting in a similarity matrix with the same shape as the similarities by class mentioned earlier (1 x N). Therefore, we combine these two matrices in a residual manner as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/247/1*ZziliO_4eVY-Ph-16FjrlA.png" /></figure><p>With <em>alpha </em>being<em> </em>a parameter that controls how much importance give to the cached model.</p><h3>Implementation</h3><h4>Dataset download steps</h4><p>First of all we need to download the data and have a procedure to create few-shot examples. We are not going to do this from scratch and will reuse the available code from other work.</p><ul><li>Let’s start by downloading Oxford pets dataset together with train test splits on from <a href="https://github.com/gaopengcuhk/Tip-Adapter/blob/main/DATASET.md#oxfordpets">here</a>. (License: The dataset is available to download for commercial/research purposes)</li><li>Install `Dassl` package to conveniently select few-shot examples from oxford pets dataset. The installation is described <a href="https://github.com/KaiyangZhou/Dassl.pytorch">here</a>.</li><li>Create a folder called “datasets” in your working directory with `oxford_pets.py` file and copy the code from <a href="https://medium.com/r?url=https%3A%2F%2Fgithub.com%2Fgeekyutao%2FTaskRes%2Fblob%2Fmain%2Fdatasets%2Foxford_pets.py">here</a>.</li></ul><p>By now you should have the following directory structure:</p><p>|— Data/<br>|— —oxford_pets/ <br>|— Dassl.pytorch/<br>|— Working_dir/<br>|— — datasets/<br>| — — — oxford_pets.py<br>|— — main.py (code below)</p><h4>Method</h4><p>First, we import some packages, the CLIP model and define the `config` file with the name of the dataset, the path to it, the number of shots per class and the batch size.</p><pre>from dassl.data.datasets.build import build_dataset<br>from dassl.config import get_cfg_default<br>from dassl.data.data_manager import build_data_loader<br>from dassl.data.transforms.transforms import build_transform<br><br>from tqdm import tqdm<br>import torch.nn.functional as F<br>from transformers import CLIPProcessor, CLIPModel<br>import numpy as np<br><br>import datasets.oxford_pets<br><br># import CLIP model<br>model = CLIPModel.from_pretrained(&quot;openai/clip-vit-base-patch32&quot;)<br>processor = CLIPProcessor.from_pretrained(&quot;openai/clip-vit-base-patch32&quot;)<br>model.eval().cuda()<br><br># define the config <br>cfg = get_cfg_default()<br><br># root to the images<br>cfg.DATASET.ROOT = &quot;/data/&quot;<br>cfg.DATALOADER.NUM_WORKERS = 0<br>cfg.DATASET.NUM_SHOTS = 8<br><br>cfg.DATALOADER.TRAIN_X.BATCH_SIZE = 16<br>cfg.DATALOADER.TEST.BATCH_SIZE = 128<br><br>cfg.DATASET.NAME = &#39;OxfordPets&#39; <br>cfg.DATASET.SUBSAMPLE_CLASSES = &quot;all&quot; <br>cfg.SEED = 1<br><br>dataset = build_dataset(cfg)<br><br># Transformation for the data to resize and center<br>tfm_train = build_transform(cfg, is_train=True)<br>tfm_test = build_transform(cfg, is_train=False)<br><br>train_loader_x = build_data_loader(<br>            cfg,<br>            data_source=dataset.train_x,<br>            batch_size=cfg.DATALOADER.TRAIN_X.BATCH_SIZE,<br>            tfm=tfm_train,<br>        )<br><br><br>test_loader = build_data_loader(<br>                cfg,<br>                sampler_type=cfg.DATALOADER.TEST.SAMPLER,<br>                data_source=dataset.test,<br>                batch_size=cfg.DATALOADER.TEST.BATCH_SIZE,<br>                tfm=tfm_test,<br>                is_train=False,<br>            )</pre><p>Now let’s create the cached model and extract the CLIP textual features with the prompt <em>“a photo of a {class}, a type of pet.”</em> for each type of pets in the dataset.</p><pre>def build_cache_model(cfg, clip_model, train_loader_cache):<br>    &quot;&quot;&quot; create the cached model with encoded images and one-hot labels &quot;&quot;&quot;<br>    cache_keys = []<br>    cache_values = []<br><br>    with torch.no_grad():<br>        train_features = []<br><br>        for i, batch in enumerate(tqdm(train_loader_cache)):<br>            images = batch[&#39;img&#39;].cuda()<br>            target = batch[&#39;label&#39;].cuda()<br>            <br>            image_features = clip_model.visual_projection(clip_model.vision_model(images).pooler_output)<br>            train_features.append(image_features)<br>            cache_values.append(target)<br>        cache_keys.append(torch.cat(train_features, dim=0).unsqueeze(0))<br><br>    cache_keys = torch.cat(cache_keys, dim=0).mean(dim=0)<br>    cache_keys /= cache_keys.norm(dim=-1, keepdim=True)<br>    cache_keys = cache_keys.permute(1, 0)<br>    cache_values = F.one_hot(torch.cat(cache_values, dim=0)).half()<br><br><br>    return cache_keys, cache_values<br><br>def clip_classifier(classnames, template, clip_model):<br>    &quot;&quot;&quot; encode textual prompts &quot;&quot;&quot;<br>    with torch.no_grad():<br>        clip_weights = []<br><br>        for classname in tqdm(classnames):<br>            # Tokenize the prompts<br>            classname = classname.replace(&#39;_&#39;, &#39; &#39;)<br>            texts = [t.format(classname) for t in template]<br>            texts = processor(text=texts, return_tensors=&quot;pt&quot;, padding=True)<br>            class_embeddings = clip_model.text_model(texts[&#39;input_ids&#39;].cuda()).pooler_output<br>            class_embeddings = clip_model.text_projection(class_embeddings)<br>            class_embeddings /= class_embeddings.norm(dim=-1, keepdim=True)<br>            clip_weights.append(class_embeddings)<br><br>        clip_weights = torch.stack(clip_weights, dim=1).cuda()[0].t()<br>    return clip_weights<br><br><br># Textual features<br>print(&quot;\nGetting textual features as CLIP&#39;s classifier.&quot;)<br>template = [&quot;a photo of a {}, a type of pet.&quot;]<br>clip_weights = clip_classifier(dataset.classnames, template, model)<br><br># cached model of the training data<br>print(&quot;\nConstructing cache model by few-shot visual features and labels.&quot;)<br>cache_keys, cache_values = build_cache_model(cfg, model, train_loader_x)<br><br><br># extract features for the test data<br>cache_keys_test, cache_values_test = build_cache_model(cfg, model, test_loader)</pre><p>Finally, let’s compute our results:</p><pre>def cls_acc(output, target, topk=1):<br>    # Get the topk predictions<br>    pred = np.argmax(output, axis=1)    <br>    # Check if predictions match the target<br>    correct = pred == target.reshape(1, -1)<br>    # Calculate accuracy<br>    acc = correct[:topk].reshape(-1).sum(0)<br>    acc = 100 * acc / target.shape[0]<br>    <br>    return acc<br><br>def pred_cached(cliplogits, feats, c_keys, c_values, labels, alpha, beta):<br>    <br>    affinity = feats @ c_keys<br>    logits_cache = ((-1) * (beta - beta * affinity)).exp() @ c_values<br><br>    tipa_logits = cliplogits + logits_cache * alpha<br>    acc = cls_acc(tipa_logits.detach().cpu().numpy(), labels.detach().cpu().numpy())<br>    return acc<br><br><br><br>clip_logits_test = 100. * cache_keys_test.t() @ clip_weights<br>acc_zeroshot = cls_acc(clip_logits_test.detach().cpu().numpy(), cache_values_test.argmax(1).detach().cpu().numpy())<br>print(acc_zeroshot) <br><br>acc_cached = pred_cached(clip_logits_test, cache_keys_test.t(), cache_keys, <br>                          cache_values.float(), cache_values_test.argmax(1), 1, 4)<br><br>print(acc_cached)</pre><p>From above we get 83.619 accuracy for zero-shot CLIP and 84.955 with the cached model of 8 examples per class. This might seem like not a big improvement, however it may vary depending on the dataset. In this case, the high accuracy of CLIP’s zero-shot predictions indicates that the model’s pre-trained knowledge already adeptly distinguishes between various types of pets. However, on a different dataset where CLIP’s zero-shot accuracy is lower, leveraging the cached model could potentially result in a much more significant performance enhancement.</p><h3>Conclusions</h3><p>In this third article of the series on CLIP zero-shot prediction we have seen how we can improve zero-shot capabilities of CLIP when we have some examples per class available in a training-free manner, without training any additional parameters. We have shown that on Oxford pets dataset we can get a boost of almost 1.5% with 8 examples per class.</p><h3>References</h3><p>[1] <a href="https://arxiv.org/pdf/2111.03930.pdf">2111.03930.pdf (arxiv.org)</a><br>[2] <a href="https://github.com/gaopengcuhk/Tip-Adapter">https://github.com/gaopengcuhk/Tip-Adapter</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a59f6b29cdc8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/improving-clip-performance-in-training-free-manner-with-few-shot-examples-a59f6b29cdc8">Improving CLIP performance in training-free manner with few-shot examples</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Improving performance and explainability of Zero-Shot CLIP]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/improving-performance-and-explainability-of-zero-shot-clip-33e579d3f4bb?source=rss-cf3e4a05b535------2"><img src="https://cdn-images-1.medium.com/max/875/0*c9YEu2t-eY4ub7wJ.png" width="875"></a></p><p class="medium-feed-snippet">Part 2&#x200A;&#x2014;&#x200A;Visual classification via Description from LLMs</p><p class="medium-feed-link"><a href="https://medium.com/data-science/improving-performance-and-explainability-of-zero-shot-clip-33e579d3f4bb?source=rss-cf3e4a05b535------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/improving-performance-and-explainability-of-zero-shot-clip-33e579d3f4bb?source=rss-cf3e4a05b535------2</link>
            <guid isPermaLink="false">https://medium.com/p/33e579d3f4bb</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Alexey Kravets]]></dc:creator>
            <pubDate>Sat, 25 Nov 2023 02:27:11 GMT</pubDate>
            <atom:updated>2023-11-25T02:27:11.873Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Simple way of improving Zero-Shot CLIP performance]]></title>
            <link>https://medium.com/data-science/simple-way-of-improving-zero-shot-clip-performance-4eae474cb447?source=rss-cf3e4a05b535------2</link>
            <guid isPermaLink="false">https://medium.com/p/4eae474cb447</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[clip]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Alexey Kravets]]></dc:creator>
            <pubDate>Fri, 03 Nov 2023 17:52:35 GMT</pubDate>
            <atom:updated>2023-11-03T17:52:35.548Z</atom:updated>
            <content:encoded><![CDATA[<h3>A Simple Way of Improving Zero-Shot CLIP Performance</h3><h4>Part 1 — Customized Prompts via Language Models (CuPL)</h4><p>Unimodal models are designed to work with data from a single mode, which can be either text or images. These models specialize in understanding and generating content specific to their chosen mode. For example, GPT are excellent at generating human-like text. They have been used for tasks like language translation, text generation, and answering questions. Convolutional Neural Networks (CNNs) are examples of image models that excel at tasks like image classification, object detection, and image generation. Currently, many interesting tasks such as Visual Question Answering (VQA) and Image-Text retrieval etc. require multimodal capabilities. Is it possible to combine both text and image processing? We can! CLIP stands out as one of the initial highly successful image-text models, demonstrating proficiency in both image recognition and text comprehension.</p><p>We will divide this article into the following sections:</p><ol><li>Introduction</li><li>Architecture</li><li>Training process and Contrastive loss</li><li>Zero-shot capability</li><li>CuPL</li><li>Conclusions</li></ol><h3>Introduction</h3><p>The CLIP model is an impressive zero-shot predictor, enabling predictions on tasks it hasn’t explicitly been trained for. As we will see more in detail in the next sections, by using natural language prompts to query images, CLIP can perform image classification without requiring task-specific training data. Nevertheless, its performance can be significantly enhanced with a few tricks. In this series of articles, we will explore methods that leverage additional prompts generated by Large Language Models (LLM) or a few-shot training examples without involving any parameter training. These approaches offer a distinct advantage as they are computationally less demanding and do not necessitate fine-tuning additional parameters.</p><h3>Architecture</h3><p>CLIP is a dual encoder model with two separate encoders for visual and textual modalities that encode images and texts independently. Such architecture is different from the fusion encoder that enables the interaction between visual and textual modalities through cross-attention which involves learning attention weights that help the model focus on specific regions of an image and corresponding parts of the text when processing both modalities. The idea is similar to self-attention, which allows each token to attend to other tokens within the same modality. Cross-attention extends this concept by allowing tokens in one modality (e.g., tokens or patches representing image features) to attend to tokens in another modality (e.g., tokens representing textual descriptions). The idea of dual and fusion encoders can be summarized as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZDVS480Jp3w5Klb0MQZ7kw.png" /><figcaption>Image by Author — Dual vs Fusion encoders</figcaption></figure><h4>Encoders</h4><p><strong>Text Encoder</strong>: Responsible for processing input text, the text encoder transforms it into a vector representation. Within CLIP, the model uses a standard that we thoroughly explored in <a href="https://medium.com/towards-data-science/deep-dive-into-the-code-of-bert-model-9f618472353e">this</a> article. The text encoder produces an embedding for the provided text, encapsulating semantic information associated with the input.<br><strong>Image Encoder:</strong> The image encoder processes images to derive their vector representations. The visual encoder can either be a Convolutional Neural Network like the ResNet model or a ViT Transformer (see <a href="https://medium.com/towards-data-science/a-deep-dive-into-the-code-of-the-visual-transformer-vit-model-1ce4cc05ca8d">here </a>to refresh your knowledge) that produces the image vector representations.</p><p>These two vectors share the same dimensions enabling the computation of similarity between a given text and image. If you have always worked with one modality you wonder how is it even possible that we can compare image and text embeddings? The key lies in the training process and the loss function which empower CLIP to learn a unified image-text space facilitating the comparison of vectors from different modalities.</p><h3>Training process and Contrastive loss</h3><p>CLIP was trained via a large-scale, multimodal objective on an extensive dataset of image-text pairs. When I say large-scale it means a LOT of data — approximately 0.4 billion image-text pairs. These were collected from publicly available sources on the internet and automatically filtered to ensure high quality.</p><p>Once image-text pairs are collected the model is trained via a contrastive loss. Contrastive loss enables the model to learn a shared image-text space by aligning the representations of images and text maximizing the similarity between the embeddings of matching pairs and minimizing the similarity between non-matching pairs. The process is presented in the image below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/503/1*NKzoIwp5u0Q7dgSeb9rCkA.png" /><figcaption>Image from <a href="https://arxiv.org/pdf/2103.00020.pdf">CLIP</a> paper — Contrastive loss</figcaption></figure><p>The image embedding I_i corresponds to the text embedding T_i (i.e., on the diagonal) forming the matching pair, while all other texts T_j (j ≠ i) (off-diagonal) are considered to be non-matching pairs. Similarly, for T_i only I_i is regarded as the matching image, and all other images I_j (j ≠ i) are not considered descriptions of T_i. However, this assumption might be limiting as there could be other text pairs that effectively describe an image and vice versa. Mining hard negative examples stands as a potential solution to this challenge. CLIP, despite this, manages to overcome this limitation attributed to its substantial batch size of 32,768.</p><p>After pre-training on this diverse dataset, CLIP’s learned embeddings can be used for many downstream applications — one of them that is truly impressive is the zero-shot image classification.</p><h3>Zero-Shot capability</h3><p>What does zero-shot mean in the first place? As mentioned in the introduction, zero-shot classification refers to the ability of a model to correctly classify unseen classes without the need for any specific examples or training data for those particular classes. CLIP was trained on a large dataset and it has learned to generalize across a broad range of concepts that enables it to recognize and classify classes based on their semantic relationships. Let’s see how this is done in practice:</p><p>Assume we only know the class names for a particular dataset like <br><em>[“dog”, “cat”, “horse”]</em>. As CLIP was trained to match image and text, we can compute the cosine similarity between a given test image and prompt <em>“Picture of a {class name}”</em> which becomes in our case:<br><em>“Picture of a {dog}”, “Picture of a {cat}”, “Picture of a {horse}”</em>.</p><p>The prompt with the highest cosine similarity represents the predicted class.</p><h3>Improving Zero-Shot CLIP via Customized Prompts via Language Models (CuPL)</h3><p>Now, Zero-Shot CLIP has achieved quite an impressive performance already, however, we can squeeze some more out of it with a few simple tricks. CLIP’s zero-shot performance is very sensitive to the textual prompts it’s fed with. This is why for different datasets such as ImageNet people have come up with different textual prompts such as <em>“an origami {class name}”</em>,<em> “a {class name} in a video game” </em>and similar. These manually designed prompts are better than a simple <em>“Picture of a {class name}”</em>, however, they still have some major limitations:</p><ol><li>Hand-written prompts require a lot of human effort</li><li>Hand-written prompts must be general — we cannot use a template like <em>“a photo of a {platypus}, a type of aquatic mammal”</em> as it would apply only to aquatic mammals and not to other categories. This is limiting, as descriptive details are useful for fine-grained classification.</li><li>Writing high-performing prompt templates requires prior information about the contents of the dataset. So in the case of ImageNet, we must know in advance that the dataset of interest contains origami, video game images, and so on.</li></ol><p>So what can we do? Let’s just ask a Large Language Model (LLM) to generate for us such prompts that can be easily scaled to any number of classes and datasets! We can ask a LLM the following questions:</p><ul><li><em>Describe what a/the {class name} looks like:</em></li><li><em>Describe a/the {class name}:</em></li><li><em>What are the identifying characteristics of a/the {class name}?</em></li></ul><p>Why should this be better than simple prompts? The hypothesis is that prompts that LLM generates will contain very detailed descriptions of the given classes and enable CLIP to place more importance on image regions that are most relevant for correct classification.</p><p>Let’s now jump into coding and see what we can get. We are going to use CLIP from the Transformers library from <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;ved=2ahUKEwiu6cr-kKGCAxXk_7sIHYoVCdMQFnoECB0QAQ&amp;url=https%3A%2F%2Fhuggingface.co%2Fdocs%2Ftransformers&amp;usg=AOvVaw1c0CbHgKUDJPUj-n5H3rT8&amp;opi=89978449">Hugging Face</a>. So let’s import the model — we are going to use ViT with a patch of size 32 and the processing pipeline that tokenizes text and pre-processes images:</p><pre>from transformers import CLIPProcessor, CLIPModel<br>import torch<br>import requests<br>from PIL import Image<br><br>model = CLIPModel.from_pretrained(&quot;openai/clip-vit-base-patch32&quot;)<br>processor = CLIPProcessor.from_pretrained(&quot;openai/clip-vit-base-patch32&quot;)</pre><p>Next, we are going to download an image of a <em>“tree frog”, </em>a<em> </em>class contained in the <a href="https://www.image-net.org/">ImageNet</a> dataset,<em> </em>from the <em>freeimages </em>website that has an open license for the image below. Then, we are going to predict whether it’s a <em>“tree frog” </em>or a <em>“tailed frog” </em>(they are visually similar and mainly differ by the large eye size) using CLIP with a simple prompt <em>“A photo of a {class}”</em>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jp6FidsW8Z4BJhdCaUoR4g.png" /><figcaption>Tree Frog image from <a href="https://images.freeimages.com/images/large-previews/342/green-tree-frog2-1616738.jpg">FreeImages</a> (license: <a href="https://www.freeimages.com/license">https://www.freeimages.com/license</a>)</figcaption></figure><pre>url = &quot;https://images.freeimages.com/images/large-previews/342/green-tree-frog2-1616738.jpg&quot;<br>image = Image.open(requests.get(url, stream=True).raw)<br><br>inputs = processor(text=[&quot;a photo of a tree frog&quot;, &quot;a photo of a tailed frog&quot;], images=image, return_tensors=&quot;pt&quot;, padding=True)<br><br>outputs = model(**inputs)<br>logits_per_image = outputs.logits_per_image  # this is the image-text similarity score<br>probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities<br>print(probs)<br><br>&quot;&quot;&quot;<br>Output:<br>tensor([[0.3164, 0.6836]], grad_fn=&lt;SoftmaxBackward0&gt;)<br>&quot;&quot;&quot;</pre><p>The model makes an incorrect prediction by selecting <em>“tailed frog”</em> with a probability of 0.68.</p><p>Let’s now ask a LLM (e.g. ChatGPT) to generate the prompts for us:</p><pre>prompts = {&quot;tree frog&quot;: [<br>        &quot;A tree frog is a small frog that typically has greenish coloration.&quot;,<br>        &quot;A tree frog is a small frog that typically has bright colors, long toes that help it climb, and suction cups on its feet.&quot;,<br>        &quot;A tree frog is small, typically green, frog that lives in trees.&quot;,<br>        &quot;A tree frog looks like a frog with special adaptations for living in trees.&quot;,<br>        &quot;A tree frog is a small, typically green frog with large adhesive pads on its feet that allow it to climb smooth surfaces like glass and plastic.&quot;,<br>        &quot;Tree frogs are small amphibians with big toes that help them climb.&quot;,<br>        &quot; Most tree frogs have bright colors.&quot;,<br>        &quot;Tree frogs are small frogs that live in trees.&quot;,<br>        &quot;A tree frog is a small frog that has large toe pads that help it climb trees.&quot;,<br>        &quot;A tree frog typically has green skin, although some species can be brown, gray, or yellow.&quot;,<br>        &quot;A tree frog is a small frog that lives in trees.&quot;,<br>        &quot;A tree frog is a small, tailless amphibian with large, powerful hind legs and webbed feet.&quot;,<br>        &quot;A tree frog is a amphibian that has well-developed hind legs which enable it to climb trees and other structures.&quot;,<br>        &quot;Avatar of the forest, the tree frog is a small amphibian with big eyes bulging out of its head.&quot;,<br>        &quot;A tree frog is a small amphibian that typically has bright green skin and lives in trees.&quot;,<br>        &quot;A tree frog is a small, slim frog that typically has brightly colored skin.&quot;,<br>        &quot;A tree frog is a small frog that typically has a bright green body and lives in trees.&quot;,<br>        &quot;A tree frog is a small, tailless amphibian that typically has bright green or yellowish skin and lives in trees or near bodies of water.&quot;,<br>        &quot;A tree frog is a small amphibian that typically has a green body and large eyes.&quot;,<br>        &quot;A tree frog is a small frog that typically has bright colors.&quot;,<br>        &quot;The identifying characteristics of a tree frog varies depending on the species, but some common features include large adhesive toes, protruding eyes, and bright colours.&quot;,<br>        &quot;Some identifying characteristics of a tree frog are that they have large toe pads, which help them grip onto tree branches, and their bodies are slim so that they can fit into small spaces.&quot;,<br>        &quot;Tree frogs are small frogs that live in trees and other high places.&quot;,<br>        &quot;The identifying characteristics of a tree frog are its long hind legs, which it uses to jump, and its adhesive pads, which it uses to stick to surfaces.&quot;,<br>        &quot;The identifying characteristics of a tree frog are that they have long, sticky toes that help them climb trees, and they have wrinkled skin that helps them absorb water.&quot;,<br>        &quot;Tree frogs have long, sticky toes that help them climb trees.&quot;,<br>        &quot;Tree frogs are small frogs that can climb trees.&quot;,<br>        &quot;Tree frogs have long hind legs that they use to jump.&quot;,<br>        &quot;There are over 6,300 species of tree frogs, so it is difficult to give one answer to this question.&quot;,<br>        &quot;The identifying characteristics of a tree frog are its long, sticky toes that help it climb trees, and its dark green or brown coloration that helps it blend in with leaves.&quot;<br>    ],<br>    &quot;tailed frog&quot;: [<br>        &quot; short, stout body; webbed hind feet with large, adhesive discs on the toes; long, muscular tail; small eyes located on top of the head; smooth or warty skin; and a small mouth.&quot;,<br>        &quot;A tailed frog has a long, skinny body and a long tail.&quot;,<br>        &quot;A tailed frog has a long, slender body with a tail that is about as long as its body.&quot;,<br>        &quot;A tailed frog is a frog with a long tail.&quot;,<br>        &quot;A tailed frog has a long tail and four legs.&quot;,<br>        &quot;A tailed frog is a small frog that has a long tail.&quot;,<br>        &quot;A tailed frog has a long tail that is often as long as its body.&quot;,<br>        &quot;A tailed frog has a long tail and webbed feet.&quot;,<br>        &quot;A tailed frog is a frog with a long tail.&quot;,<br>        &quot;A tailed frog is a small frog with a long tail.&quot;,<br>        &quot;A tailed frog (Asteriscus species) is a species of frog in the Asteriscidae family.&quot;,<br>        &quot;A tailed frog is a type of frog that has a long tail.&quot;,<br>        &quot;A tailed frog is a frog with a long tail, typically over 10 cm in length.&quot;,<br>        &quot;A tailed frog is a species of frog that has a long tail.&quot;,<br>        &quot;Tailed frogs are a type of frog that have a long tail.&quot;,<br>        &quot;A tailed frog is a type of frog that has a long, tail-like structure protruding from its back.&quot;,<br>        &quot;A tailed frog is a frog that has a long tail.&quot;,<br>        &quot;A tailed frog is a frog that has a long tail.&quot;,<br>        &quot;A tailed frog has a long, thin body with short legs.&quot;,<br>        &quot;A tailed frog is a small amphibian that has a long tail.&quot;,<br>        &quot;Some identifying characteristics of a tailed frog are that they have a long tail, they are good swimmers, and they live near water.&quot;,<br>        &quot;There are over 60 species of tailed frogs, so it is difficult to give a definitive answer.&quot;,<br>        &quot;There are over 100 species of tailed frogs, so it is difficult to give a general answer to this question.&quot;,<br>        &quot;Some identifying characteristics of a tailed frog are that they have a long tail, they are small, and they have webbed feet.&quot;,<br>        &quot;There are over 60 species of tailed frog, so identifying characteristics can vary.&quot;,<br>        &quot;Tailed frogs are a species of frog that are native to the western United States and northern Mexico.&quot;,<br>        &quot;The identifying characteristics of a tailed frog are its tail, which is used for swimming, and its webbed feet.&quot;,<br>        &quot;Tailed frogs are small, dark-colored frogs with long, slender hind legs and a long, thin tail.&quot;,<br>        &quot;Some tailed frogs have a tail that is about one-third the length of their body.&quot;,<br>        &quot;The identifying characteristics of a tailed frog are its long tail and its smooth, moist skin.&quot;<br>    ]}</pre><blockquote>These prompts are more informative about the classes which should guide the model in identifying the correct class. For example, many prompts for “tree frog” emphasize that it has “large eyes”, something that simple prompts like “Picture of a tree frog” do not capture.</blockquote><p>Using the above prompts and after some manipulation, we form the final prompt for a class with the average vector embedding:</p><pre>&quot;&quot;&quot;<br>First of all, we can verify that <br>outputs = model(**inputs)<br>logits_per_image = outputs.logits_per_image  # this is the image-text similarity score<br>probs = logits_per_image.softmax(dim=1)  # we can take the softmax to get the label probabilities<br>probs<br><br>Is the same as:<br>image_features = model.visual_projection(model.vision_model(inputs[&#39;pixel_values&#39;]).pooler_output)<br>text_features = model.text_projection(model.text_model(inputs[&#39;input_ids&#39;]).pooler_output)<br>image_features = image_features / image_features.norm(dim=-1, keepdim=True)<br>text_features = text_features / text_features.norm(dim=-1, keepdim=True)<br><br># cosine similarity as logits<br>logit_scale = model.logit_scale.exp()<br>logits_per_image = logit_scale * image_features @ text_features.t()<br>probs = logits_per_image.softmax(dim=1)<br>probs<br>&quot;&quot;&quot;<br><br>image_features = model.visual_projection(model.vision_model(inputs[&#39;pixel_values&#39;]).pooler_output)<br><br>tree_frog_vector = model.text_model(processor(prompts[&#39;tree frog&#39;], return_tensors=&quot;pt&quot;, padding=True)[&#39;input_ids&#39;]).pooler_output<br># take the mean prompt embedding<br>tree_frog_vector = tree_frog_vector.mean(dim=0, keepdims=True)<br># final projection <br>tree_frog_vector = model.text_projection(tree_frog_vector)<br><br>tailed_frog_vector = model.text_model(processor(prompts[&#39;tailed frog&#39;], return_tensors=&quot;pt&quot;, padding=True)[&#39;input_ids&#39;]).pooler_output<br># take the mean prompt embedding<br>tailed_frog_vector = tailed_frog_vector.mean(dim=0, keepdims=True)<br># final projection<br>tailed_frog_vector = model.text_projection(tailed_frog_vector)<br><br># concatenate <br>text_features = torch.cat([tree_frog_vector, tailed_frog_vector], dim=0)<br><br># normalize features<br>image_features = image_features / image_features.norm(dim=-1, keepdim=True)<br>text_features = text_features / text_features.norm(dim=-1, keepdim=True)<br><br># cosine similarity as logits<br>logit_scale = model.logit_scale.exp()<br>logits_per_image = logit_scale * image_features @ text_features.t()<br>probs = logits_per_image.softmax(dim=1)<br>print(probs)<br><br>&quot;&quot;&quot;<br>Output:<br>tensor([[0.6512, 0.3488]], grad_fn=&lt;SoftmaxBackward0&gt;)<br>&quot;&quot;&quot;</pre><p>Using prompts from LLM gives us the correct classification answer —<em> “tree frog”</em>.</p><h3>Conclusions</h3><p>In this article, we have seen how easily we can improve CLIP zero-shot prediction using a LLM. The advantage of this solution is not only higher accuracy but also its scalability as we do not need any human effort to generate prompts. In the next articles, we are going to explore other methods to improve CLIP’s zero-shot learning and training-free few-shot learning methods.</p><h3>References</h3><p>[1] <a href="https://arxiv.org/abs/2103.00020">[2103.00020] Learning Transferable Visual Models From Natural Language Supervision (arxiv.org)</a><br>[2] <a href="https://arxiv.org/abs/2209.03320">[2209.03320] What does a platypus look like? Generating customized prompts for zero-shot image classification (arxiv.org)</a><br>[3] <a href="https://huggingface.co/docs/transformers/model_doc/clip">CLIP (huggingface.co)</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4eae474cb447" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/simple-way-of-improving-zero-shot-clip-performance-4eae474cb447">Simple way of improving Zero-Shot CLIP performance</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Deep Dive into the LSTM-CRF Model]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/deep-dive-into-lstm-crf-model-423cf4dc42c?source=rss-cf3e4a05b535------2"><img src="https://cdn-images-1.medium.com/max/1167/1*Tv4_qn1A7jEvv4OhNm2vPg.png" width="1167"></a></p><p class="medium-feed-snippet">With PyTorch code</p><p class="medium-feed-link"><a href="https://medium.com/data-science/deep-dive-into-lstm-crf-model-423cf4dc42c?source=rss-cf3e4a05b535------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/deep-dive-into-lstm-crf-model-423cf4dc42c?source=rss-cf3e4a05b535------2</link>
            <guid isPermaLink="false">https://medium.com/p/423cf4dc42c</guid>
            <category><![CDATA[named-entity-recognition]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[deep-dives]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Alexey Kravets]]></dc:creator>
            <pubDate>Thu, 12 Oct 2023 20:50:57 GMT</pubDate>
            <atom:updated>2023-10-14T04:31:41.922Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[A Deep Dive into the Code of the Visual Transformer (ViT) Model]]></title>
            <link>https://medium.com/data-science/a-deep-dive-into-the-code-of-the-visual-transformer-vit-model-1ce4cc05ca8d?source=rss-cf3e4a05b535------2</link>
            <guid isPermaLink="false">https://medium.com/p/1ce4cc05ca8d</guid>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[vit]]></category>
            <category><![CDATA[transformers]]></category>
            <dc:creator><![CDATA[Alexey Kravets]]></dc:creator>
            <pubDate>Tue, 15 Aug 2023 19:30:48 GMT</pubDate>
            <atom:updated>2023-08-15T19:30:48.438Z</atom:updated>
            <content:encoded><![CDATA[<h4>Breaking down the HuggingFace ViT Implementation</h4><p>Vision Transformer (ViT) stands as a remarkable milestone in the evolution of computer vision. ViT challenges the conventional wisdom that images are best processed through convolutional layers, proving that sequence-based attention mechanisms can effectively capture the intricate patterns, context, and semantics present in images. By breaking down images into manageable patches and leveraging self-attention, ViT captures both local and global relationships, enabling it to excel in diverse vision tasks, from image classification to object detection and beyond. In this article, we are going to break down how ViT for classification works under the hood.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*9sBcMVMDGvhaX7_N" /><figcaption><a href="https://unsplash.com/photos/aVvZJC0ynBQ">https://unsplash.com/photos/aVvZJC0ynBQ</a></figcaption></figure><h3>Introduction</h3><p>The core idea of ViT is to treat an image as a sequence of fixed-size patches, which are then flattened and converted into 1D vectors. These patches are subsequently processed by a transformer encoder, which enables the model to capture global context and dependencies across the entire image. By dividing the image into patches, ViT effectively reduces the computational complexity of handling large images while retaining the ability to model complex spatial interactions.</p><p>First of all, we import the ViT model for classification from hugging face transformers library:</p><pre>from transformers import ViTForImageClassification<br>import torch<br>import numpy as np<br><br>model = ViTForImageClassification.from_pretrained(&quot;google/vit-base-patch16-224&quot;)</pre><p><em>patch16–224 </em>indicates that the model accepts images of size 224x224 and each patch has width and hight of 16 pixels.</p><p>This is what the model architecture looks like:</p><pre>ViTForImageClassification(<br>  (vit): ViTModel(<br>    (embeddings): ViTEmbeddings(<br>      (patch_embeddings): PatchEmbeddings(<br>        (projection): Conv2d(3, 768, kernel_size=(16, 16), stride=(16, 16))<br>      )<br>      (dropout): Dropout(p=0.0, inplace=False)<br>    )<br>    (encoder): ViTEncoder(<br>      (layer): ModuleList(<br>        (0): ViTLayer(<br>          (attention): ViTAttention(<br>            (attention): ViTSelfAttention(<br>              (query): Linear(in_features=768, out_features=768, bias=True)<br>              (key): Linear(in_features=768, out_features=768, bias=True)<br>              (value): Linear(in_features=768, out_features=768, bias=True)<br>              (dropout): Dropout(p=0.0, inplace=False)<br>            )<br>            (output): ViTSelfOutput(<br>              (dense): Linear(in_features=768, out_features=768, bias=True)<br>              (dropout): Dropout(p=0.0, inplace=False)<br>            )<br>          )<br>          (intermediate): ViTIntermediate(<br>            (dense): Linear(in_features=768, out_features=3072, bias=True)<br>          )<br>          (output): ViTOutput(<br>            (dense): Linear(in_features=3072, out_features=768, bias=True)<br>            (dropout): Dropout(p=0.0, inplace=False)<br>          )<br>          (layernorm_before): LayerNorm((768,), eps=1e-12, elementwise_affine=True)<br>          (layernorm_after): LayerNorm((768,), eps=1e-12, elementwise_affine=True)<br>        )<br>        <br>        .......<br><br>        (11): ViTLayer(<br>          (attention): ViTAttention(<br>            (attention): ViTSelfAttention(<br>              (query): Linear(in_features=768, out_features=768, bias=True)<br>              (key): Linear(in_features=768, out_features=768, bias=True)<br>              (value): Linear(in_features=768, out_features=768, bias=True)<br>              (dropout): Dropout(p=0.0, inplace=False)<br>            )<br>            (output): ViTSelfOutput(<br>              (dense): Linear(in_features=768, out_features=768, bias=True)<br>              (dropout): Dropout(p=0.0, inplace=False)<br>            )<br>          )<br>          (intermediate): ViTIntermediate(<br>            (dense): Linear(in_features=768, out_features=3072, bias=True)<br>          )<br>          (output): ViTOutput(<br>            (dense): Linear(in_features=3072, out_features=768, bias=True)<br>            (dropout): Dropout(p=0.0, inplace=False)<br>          )<br>          (layernorm_before): LayerNorm((768,), eps=1e-12, elementwise_affine=True)<br>          (layernorm_after): LayerNorm((768,), eps=1e-12, elementwise_affine=True)<br>        )<br>      )<br>    )<br>    (layernorm): LayerNorm((768,), eps=1e-12, elementwise_affine=True)<br>  )<br>  (classifier): Linear(in_features=768, out_features=1000, bias=True)<br>)</pre><h3>Embeddings</h3><h4>Patch Embedding</h4><p>Transformation of the image into patches is performed using a Conv2D layer. As we know, Conv2D layer does a 2-dimensional convolutional operations on input data to learn features and patterns from images. In this case though Conv2D layer is used to divide the image into NxN number of patches by using the stride parameter. Stride determines the step size at which the filter slides over the input data. In this case, because our images are 224x224 and the patch is of size 16, meaning that there are 224/16 = 14 patches in each dimension, if we choose stride=16 we effectively separate our image in 14 <strong>non-overlapping </strong>patches.</p><p>To be visual and assuming an image of shape 4x4 with and stride of 2:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/223/1*Ws3LLlRDogcogj3IqhW7yQ.gif" /><figcaption>Patches creation, Image by Author</figcaption></figure><p>So for example, the first &amp; the second patches are going to be :</p><pre>proj = model.vit.embeddings.patch_embeddings.projection<br>torch.allclose(torch.sum(image[0, :, 0:16, 0:16] * w[0]) + b[0],<br>               proj(image)[0][0][0, 0], atol=1e-6)<br># True<br><br><br>torch.allclose(torch.sum(image[0, :, 16:32, 0:16] * w[0]) + b[0],<br>                 proj(image)[0][0][1, 0], atol=1e-6)<br><br># True</pre><p>The pattern is clear — to compute each patch we skip 16 pixels to get non-overlapping patches. If we do this operation for the entire image we end up with 1 x 14 x 14 tensor where each patch is represented by one number computed using the first filter of Conv2D. However, there are <strong>768 filters</strong> which means that at the end we get a 768 x 14 x 14 dimensional tensor. So now we effectively have for each patch a 768 dimensional representation, that is our patch embedding. We also flatten and transpose the tensor, thus the embedding shape becomes <em>[batch_size, 196, 768] </em>where the second dimension is flattened 14 x 14 = 196 and we effectively have a sequence of 196 patches with embedding size of 768.</p><pre>embeddings = model.vit.embeddings.patch_embeddings.projection(image)<br># shape (batch_size, 196, 768)<br>embeddings = embeddings.flatten(2).transpose(1, 2)</pre><p>If we want to reproduce the layer entirely from scratch, this is the code:</p><pre><br>batch_size = 1 <br>F = 768 # number of filters<br>H1 = 14 # output dimension hight - 224/16<br>W1 = 14 # output dimension width - 224/16<br>stride = 16<br>HH = 16 # patch hight<br>WW = 16 # patch width<br>w = model.vit.embeddings.patch_embeddings.projection.weight<br>b = model.vit.embeddings.patch_embeddings.projection.bias<br><br>out = np.zeros((N, F, H1, W1))<br>chunks = []<br>for n in range(batch_size):<br>    for f in range(F):<br>        for i in range(H1):<br>            for j in range(W1):<br>                # perform convolution operation<br>                out[n, f, i, j] = torch.sum( image[n, :, i*stride:i*stride+HH, j*stride : j*stride + WW] * w[f] ) + b[f]<br>                <br>np.allclose(out[0], embeddings[0].detach().numpy(), atol=1e-5)<br># True</pre><p>Now, if you are familiar with the Language Transformer (check it out <a href="https://towardsdatascience.com/deep-dive-into-the-code-of-bert-model-9f618472353e">here</a> if needed) you should recall the [CLS] token, whose representation serves as a condensed and informative summary of the entire text, enabling the model to make accurate predictions based on the extracted features from the transformer encoder. Also in ViT we have the [CLS] token that has the same function as for text, and it’s appended to the representation computed above.</p><p>[CLS] token is a parameter that we are going to learn using back-propagation:</p><pre>cls_token = nn.Parameter(torch.randn(1, 1, 768))<br>cls_tokens = cls_token.expand(batch_size, -1, -1)<br># append [CLS] token<br>embeddings = torch.cat((cls_tokens, embeddings), dim=1)</pre><h4>Positional Embedding</h4><p>Just like in Language Transformer, to <strong>preserve the positional information of the patches</strong>, ViT includes positional embeddings. Positional embeddings help the model understand the spatial relationships between different patches, enabling it to capture the image’s structure. <br>Positional embedding is a Tensor of the same shape of the embeddings with [CLS] token compute before, i.e., <em>[batch_size, 197, 768]</em></p><pre>embeddings = embeddings + model.vit.embeddings.position_embeddings</pre><h4>Dropout</h4><p>Patch embedding is followed by a <a href="https://arxiv.org/abs/1706.06859">Dropout</a> layer. In dropout we replace with zero some of the values with certain dropout probability. Dropout helps to reduce overfitting as we randomly block signals from certain neurons so the network needs to find other paths to reduce the loss function, and thus it learns how to generalize better instead of relying on certain paths. We can also see dropout as a kind of models ensemble technique as during training at each step we randomly deactivate certain neurons ending up with “different” networks which we eventually ensemble during the evaluation time.</p><p>At the end of the Embeddings layer we have:</p><pre># compute the embedding<br>embeddings = model.vit.embeddings.patch_embeddings.projection(image)<br>embeddings = embeddings.flatten(2).transpose(1, 2)<br># append [CLS] token<br>cls_token = model.vit.embeddings.cls_token<br>embeddings = torch.cat((cls_tokens, embeddings), dim=1)<br># positional embedding<br>embeddings = embeddings + self.position_embeddings<br># droput<br>embeddings = model.vit.embeddings.dropout(embeddings)<br></pre><h3>Encoder</h3><p>ViT employs a stack of transformer encoder blocks, similar to those used in language models such as BERT. Each encoder block consists of multi-head self-attention and feed-forward neural networks. The self-attention mechanism enables the model to capture relationships between different patches, while the feed-forward neural networks perform non-linear transformations.</p><p>Specifically, each layer is composed of Self-Attention, Intermediate and Output modules.</p><pre>(0): ViTLayer(<br>  (attention): ViTAttention(<br>    (attention): ViTSelfAttention(<br>      (query): Linear(in_features=768, out_features=768, bias=True)<br>      (key): Linear(in_features=768, out_features=768, bias=True)<br>      (value): Linear(in_features=768, out_features=768, bias=True)<br>      (dropout): Dropout(p=0.0, inplace=False)<br>    )<br>    (output): ViTSelfOutput(<br>      (dense): Linear(in_features=768, out_features=768, bias=True)<br>      (dropout): Dropout(p=0.0, inplace=False)<br>    )<br>  )<br>  (intermediate): ViTIntermediate(<br>    (dense): Linear(in_features=768, out_features=3072, bias=True)<br>  )<br>  (output): ViTOutput(<br>    (dense): Linear(in_features=3072, out_features=768, bias=True)<br>    (dropout): Dropout(p=0.0, inplace=False)<br>  )<br>  (layernorm_before): LayerNorm((768,), eps=1e-12, elementwise_affine=True)<br>  (layernorm_after): LayerNorm((768,), eps=1e-12, elementwise_affine=True)<br>)</pre><h4>Self-Attention</h4><p>Self-attention is a pivotal mechanism within the Vision Transformer (ViT) model that enables it to capture relationships and dependencies between different patches in an image. It plays a crucial role in extracting contextual information and understanding long and short-range interactions among the patches.</p><p>Each patch is associated with three vectors: Key, Query, and Value. These vectors are learned through linear transformations of the original patch embeddings. The <strong>Key vector represents information from the current patches</strong>, the <strong>Query vector is used to ask questions about other patches</strong>, and the <strong>Value vector holds the information that is relevant to other patches</strong>.</p><p>As we have already computed the embeddings in the previous section, we compute the Key, Query and Value projecting the embeddings with the Key, Query and Value matrices:</p><pre>import math <br>import torch.nn as nn<br><br>torch.manual_seed(0)<br><br>hidden_size = 768<br>num_attention_heads = 12<br>attention_head_size = hidden_size // num_attention_heads # 64<br><br>hidden_states = embeddings<br><br># apply LayerNorm to the embeddings<br>hidden_states = model.vit.encoder.layer[0].layernorm_before(hidden_states)<br><br># take first layer of the Transformer<br>layer_0 = model.vit.encoder.layer[0]<br><br># shape (768, 64) <br>key_matrix = layer_0.attention.attention.key.weight.T[:, :attention_head_size]<br>key_bias = layer_0.attention.attention.key.bias[:attention_head_size]<br><br>query_matrix = layer_0.attention.attention.query.weight.T[:, :attention_head_size] <br>query_bias = layer_0.attention.attention.query.bias[:attention_head_size]<br><br>value_matrix = layer_0.attention.attention.value.weight.T[:, :attention_head_size]<br>value_bias = layer_0.attention.attention.value.bias[:attention_head_size]<br><br># compute key, query and value for the first head attention<br># all of shape (b_size, 197, 64)<br>key_1head = hidden_states @ key_matrix + key_bias<br>query_1head = hidden_states @ query_matrix + query_bias<br>value_1head = hidden_states @ value_matrix + value_bias</pre><p>Note that we skipped the LayerNorm operation, that we will cover later.</p><p>For each Query vector, attention scores are computed by measuring the compatibility or similarity between the Query and Key vectors of all other patches. This is done through a dot product operation and then applying the Softmax function to get normalized attention scores with the shape <em>[b_size, 197, 197]. </em>The attention matrix is square because all patches attend to each other, and this is why it’s called self-attention. These scores indicate how much focus or attention should be placed on each patch when processing the query patch. Because new embedding for the next layer of each patch is derived based on the attention scores and the values of all other patches, we get a <strong>contextual embedding</strong> for each patch as its derived based on all other patches in the image.</p><p>To clarify this further, recall that at the beginning we split the image into patches using the Conv2D layer to get a 768-dimensional embedding vector for each patch - these embedding are independent as there was no interaction (no overlap) between the patches. However, in the transformer layers the patches embeddings get mixed becoming a function of the embeddings of other patches. For example, the embedding in the first layer is:</p><pre># shape (b_size, 197, 197)<br># compute the attention scores by dot product of query and key<br>attention_scores_1head = torch.matmul(query_1head, key_1head.transpose(-1, -2))<br><br>attention_scores_1head = attention_scores_1head / math.sqrt(attention_head_size)<br>attention_probs_1head = nn.functional.softmax(attention_scores_1head, dim=-1)<br><br># contextualized embedding for this layer<br>context_layer_1head = torch.matmul(attention_probs_1head, value_1head)</pre><p>If we zoom in and look at the first patch:</p><pre>patch_n = 1<br># shape (, 197)<br>print(attention_probs_1head[0, patch_n])<br>[2.4195e-01, 7.3293e-01, ..,<br>        2.6689e-06, 4.6498e-05, 1.1380e-04, 5.1591e-06, 2.1265e-05],<br>     </pre><p>the new embeddings for it (token indexed at 0 is [CLS] token) is a combination of embeddings of different patches with most attention on the first patch itself (0.73), [CLS] token (0.24) and the remaining on all other patches. But this is not always the case. Indeed, in next layers the first patch might pay more attention to patches around it instead of the patch itself and [CLS] token or even to patches very far away — this depends on what the model thinks is useful to solve a certain task.</p><p>Also, you might have noticed that I selected only the first 64 columns from the weight matrices of query, key and value. These first 64 columns represent the <strong>first attention head</strong>, but actually there are 12 of them (in this model size). Each of these attention heads creates different representation of patches. Indeed, if we look at the third attention head for the first patch we can see that the first patch pays most attention (0.26) at the second patch rather than to itself like in the first attention head.</p><pre># shape (, 197)<br>[2.6356e-01, 1.2783e-03, 2.6888e-01, ... , 1.8458e-02]</pre><p>Thus, different attention heads will capture different types of relations among patches helping the model to see things from different prospective.</p><p>To compute all these heads in parallel we do as follows:</p><pre>def transpose_for_scores(x: torch.Tensor) -&gt; torch.Tensor:<br>    new_x_shape = x.size()[:-1] + (num_attention_heads, attention_head_size)<br>    x = x.view(new_x_shape)<br>    return x.permute(0, 2, 1, 3)<br><br>mixed_query_layer = layer_0.attention.attention.query(hidden_states)<br><br>key_layer = transpose_for_scores(layer_0.attention.attention.key(hidden_states))<br>value_layer = transpose_for_scores(layer_0.attention.attention.value(hidden_states))<br>query_layer = transpose_for_scores(mixed_query_layer)<br><br># Take the dot product between &quot;query&quot; and &quot;key&quot; to get the raw attention scores.<br>attention_scores = torch.matmul(query_layer, key_layer.transpose(-1, -2))<br>attention_scores = attention_scores / math.sqrt(attention_head_size)<br><br># Normalize the attention scores to probabilities.<br>attention_probs = nn.functional.softmax(attention_scores, dim=-1)<br><br># This is actually dropping out entire tokens to attend to, which might<br># seem a bit unusual, but is taken from the original Transformer paper.<br>attention_probs = layer_0.attention.attention.dropout(attention_probs)<br><br>context_layer = torch.matmul(attention_probs, value_layer)<br><br>context_layer = context_layer.permute(0, 2, 1, 3).contiguous()<br>new_context_layer_shape = context_layer.size()[:-2] + (hidden_size,)<br>context_layer = context_layer.view(new_context_layer_shape)</pre><p>After applying self-attention we apply another projection layer and Dropout — and here we go, we got through the self-attention layer!</p><pre>output_weight = layer_0.attention.output.dense.weight<br>output_bias = layer_0.attention.output.dense.bias<br><br>attention_output = context_layer @ output_weight.T + output_bias<br>attention_output = layer_0.attention.output.dropout(attention_output)</pre><p>Ops, wait a second, I promised I would explain the <em>LayerNorm </em>operation.</p><p>Layer Normalization is a normalization technique used to enhance the training and performance of deep learning models. It addresses the problem of internal covariate shifts — during training, as the weights of the neural network change, the distribution of inputs to each layer can change significantly, making it difficult for the model to converge. Layer Normalization addresses this by ensuring that the inputs to each layer have a consistent mean and variance, stabilizing the learning process. It’s implemented by standardizing each patch embedding by its mean and standard deviation so that it has zero mean and unit variance. We then apply a trained weights and bias so it can be shifted to have a different mean and variance for the model to adapt automatically during training. Because we compute mean and standard deviation across different examples independently from the others, it is different from <a href="https://en.wikipedia.org/wiki/Batch_normalization#:~:text=Batch%20normalization%20(also%20known%20as,and%20Christian%20Szegedy%20in%202015.">Batch Normalization</a> where the normalization is across the batch dimension and thus depends on other examples in the batch.</p><p>Let’s take the first patch embedding:</p><pre>first_patch_embed = embeddings[0][0]<br># compute first patch mean<br>first_patch_mean = first_patch_embed.mean()<br># compute first patch variance<br>first_patch_std = (first_patch_embed - first_patch_mean).pow(2).mean()<br># standardize the first patch<br>first_patch_standardized = (first_patch_embed - first_patch_mean) / torch.sqrt(first_patch_std + 1e-12)<br># apply trained weight and bias vectors<br>first_patch_norm = layer_0.layernorm_before.weight * first_patch_standardized + layer_0.layernorm_before.bias</pre><h4>Intermediate</h4><p>Before the Intermediate class we perform another layer normalization and a <strong>residual connection</strong>. By now it should be clear why we want to apply another layer normalization — we need to normalize the contextual embeddings coming from the self-attention to improve convergence, but what is that other residual thing I mentioned you are probably wondering? <br>Residual Connection is a critical component in deep neural networks that mitigates the challenges of training very deep architectures. As we increase the depth of a neural network by stacking more layers we bump into the problem of vanishing/exploding gradients, where in case of vanishing gradients the model is not able to learn anymore as the propagated gradients are close to zero and initial layers stop changing weights and improve (Check <a href="https://medium.com/towards-artificial-intelligence/backpropagation-and-vanishing-gradient-problem-in-rnn-clearly-explained-efce8824971b">this article</a> and <a href="https://medium.com/towards-artificial-intelligence/backpropagation-and-vanishing-gradient-problem-in-rnn-part-2-4fa4c0e27b54">this</a> if you want to learn more about the vanishing gradient). Opposite problem with exploding gradients when the weights cannot stabilize because of extreme updates which eventually explode (go to infinity). Now, proper initialisation of weights and normalization helps to address this problem but what has been observed is even if the network becomes more stable, the performance decreases as the optimization is harder. Adding these residual connections helps to improve performance and the network becomes easier to optimize even if we keep increasing depth.</p><p>How is it implemented? Simple — we just add the original input to the transformed output after some transformations of the original input:</p><pre>transformations = nn.Sequential([nn.Linear(), nn.ReLU(), nn.Linear()])<br>output = input + transformations(input)</pre><p>Another key insight is that if the <em>transformations</em> of a residual connection learn to approximate the identity function, the addition of the input with the learned features will not have any effect. In fact, the network can learn to modify or refine the features if needed.</p><p>In our case the residual connection is the sum between the initial <em>embeddings</em> and the <em>attention_output </em>which are <em>embeddings </em>after all the transformations in the <em>self-attention </em>layer.</p><pre># first residual connection - NOTE the hidden_states are the <br># `embeddings` here<br>hidden_states = attention_output + hidden_states<br><br># in ViT, layernorm is also applied after self-attention<br>layer_output = layer_0.layernorm_after(hidden_states)</pre><p>In the Intermediate class we perform a linear projection and apply a <strong>non-linearity</strong>:</p><pre>layer_output_intermediate = layer_0.intermediate.dense(layer_output)<br>layer_output_intermediate = layer_0.intermediate.intermediate_act_fn(layer_output_intermediate)</pre><p>The non-linearity used in ViT is GeLU activation function. It is defined as the cumulative distribution function of the standard normal distribution:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/255/0*w5Nfnh8WcRHVNYnq.png" /><figcaption><a href="https://arxiv.org/pdf/1606.08415v3.pdf">https://arxiv.org/pdf/1606.08415v3.pdf</a></figcaption></figure><p>It is normally approximated with the following formula for faster calculations:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/304/0*zPwkPJbw0_9y1uFY.png" /><figcaption><a href="https://arxiv.org/pdf/1606.08415v3.pdf">https://arxiv.org/pdf/1606.08415v3.pdf</a></figcaption></figure><p>Looking at the graph below we can see that if <em>ReLU,</em> that is given by the formula<em> max(input, 0), </em>is monotonic, convex and linear in the positive domain, <em>GeLU </em>is non-monotonic, non-convex and non-linear in the positive domain and thus can approximate more easily complicated functions.<br>Additionally, <em>GeLU </em>function is smooth — unlike the <em>ReLU </em>function, which is piecewise linear with a sharp transition at zero, <em>GeLU </em>provides a smooth transition across all values, making it more amenable to gradient-based optimization during training.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/399/0*V7UT0DCtQh7pfUjM.png" /><figcaption><a href="https://arxiv.org/pdf/1606.08415v3.pdf">https://arxiv.org/pdf/1606.08415v3.pdf</a></figcaption></figure><h4>Output</h4><p>The final bit remaining of the Encoder is the Output class. To compute it we already have all the elements we need — it is linear projection, Dropout and a residual connection:</p><pre># linear projection<br>output_dense = layer_0.output.dense(layer_output_intermediate)<br># dropout<br>output_drop = layer_0.output.dropout(output_dense)<br># residual connection - NOTE these hidden_states are computed in <br># Intermediate <br>output_res = output_drop + hidden_states # shape (b_size, 197, 768)</pre><p>Well, we went through the first layer ViT Layer, there are other 11 to go through and this is where the hard part comes …</p><p>Joking! We are actually done — all the other layers are exactly the same as the first, the only difference is that instead of starting from the embeddings like in the first layer the embeddings for the next layer are <em>output_res </em>we computed previously.</p><p>So the output after 12 layer of the encoder is:</p><pre>torch.manual_seed(0)<br># masking heads in a given layer<br>layer_head_mask = None<br># output attention probabilities<br>output_attentions = False<br><br>embeddings = model.vit.embeddings(image)<br>hidden_states = embeddings<br>for l in range(12):<br>    hidden_states = model.vit.encoder.layer[l](hidden_states, layer_head_mask, output_attentions)[0]<br><br>output = model.vit.layernorm(sequence_output)</pre><h3>Pooler</h3><p>Generally, in a Transformer model Pooler is a component used to aggregate information from the sequence of tokens embeddings after the transformer encoder blocks. Its role is to generate a fixed-size representation that captures the global context and summarizes the information extracted from the image patches, in case of ViT. The Pooler is essential for obtaining a compact and context-aware representation of the image, which can then be used for various downstream tasks such as image classification.</p><p>In this case Pooler is very simple — we take [CLS] token and use it as the compact and context-aware representation of the image.</p><pre>pooled_output = output[:, 0, :] # shape (b_size, 768)</pre><h3>Classifier</h3><p>Finally, we are ready to use the the <em>pooled_output </em>to classify the image. The classifier is a simple linear layer with output dimension equal to the number of classes:</p><pre>logits = model.classifier(pooled_output) # shape (b_size, num_classes)</pre><h3>Conclusions</h3><p>ViT fully revolutionized computer vision replacing Convolutional Neural Networks almost in every application, this is why it’s so important to understand how it works. Let’s not forget that the transformer architecture, which is the main component of ViT, originated in NLP, thus you should check out my previous article on BERT Transformer <a href="https://medium.com/p/9f618472353e">here</a>. Hope you enjoyed this read, see you next time!</p><p><a href="https://medium.com/@alexml0123/membership">Join Medium with my referral link - Alexey Kravets</a></p><h4>References</h4><p>[1] <a href="https://github.com/huggingface/transformers">https://github.com/huggingface/transformers</a><br>[2] <a href="https://arxiv.org/abs/2010.11929">[2010.11929] An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (arxiv.org)</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1ce4cc05ca8d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/a-deep-dive-into-the-code-of-the-visual-transformer-vit-model-1ce4cc05ca8d">A Deep Dive into the Code of the Visual Transformer (ViT) Model</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Variational Autoencoder (VAE) with Discrete Distribution using Gumbel Softmax]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/variational-autoencoder-vae-with-discrete-distribution-using-gumbel-softmax-b3f749b3417e?source=rss-cf3e4a05b535------2"><img src="https://cdn-images-1.medium.com/max/640/1*IB22-8qoNpIITYqKA034lQ.png" width="640"></a></p><p class="medium-feed-snippet">Theory and PyTorch Implementation</p><p class="medium-feed-link"><a href="https://medium.com/data-science/variational-autoencoder-vae-with-discrete-distribution-using-gumbel-softmax-b3f749b3417e?source=rss-cf3e4a05b535------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/variational-autoencoder-vae-with-discrete-distribution-using-gumbel-softmax-b3f749b3417e?source=rss-cf3e4a05b535------2</link>
            <guid isPermaLink="false">https://medium.com/p/b3f749b3417e</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[variational-autoencoder]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Alexey Kravets]]></dc:creator>
            <pubDate>Wed, 09 Aug 2023 19:45:08 GMT</pubDate>
            <atom:updated>2024-12-19T11:54:35.681Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Streamlining File Management: A Guide to Connecting and Modifying Files on a Server or a Docker…]]></title>
            <link>https://pub.towardsai.net/streamlining-file-management-a-guide-to-connecting-and-modifying-files-on-a-server-or-a-docker-841459872547?source=rss-cf3e4a05b535------2</link>
            <guid isPermaLink="false">https://medium.com/p/841459872547</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[sftp]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[ssh]]></category>
            <dc:creator><![CDATA[Alexey Kravets]]></dc:creator>
            <pubDate>Sat, 05 Aug 2023 12:01:45 GMT</pubDate>
            <atom:updated>2023-08-05T12:01:45.682Z</atom:updated>
            <content:encoded><![CDATA[<h3>Streamlining File Management: A Guide to Connecting and Modifying Files on a Server or a Docker Container Running on Server with SFTP in Sublime Text</h3><h4>Clearly Explained</h4><h4>Introduction</h4><p>Nowadays, many academic institutions, research labs, and cloud service providers offer GPU-equipped servers that people can access to run their experiments on. For example, my university offers access to high-performance computing clusters for research and educational purposes allowing students to experiment and learn without the need for physical hardware. The communication with the server is done via SSH (Secure Socket Shell) protocol through a command line. There are several ways one can modify files on the server, for example, <a href="https://en.wikipedia.org/wiki/Rsync">rsync</a> and SFTP. In this article, we are going to describe how to transfer files between a local machine and a remote server via SFTP using SublimeText.</p><h4>Installing the SFTP plugin to SublimeText</h4><p>Open SublimeText and click CTRL + Shift + P to open the Command Palette. Then search for <strong>Package Control: Install Package.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/763/1*vTgyupTfYiJqIfo6iPtyeg.png" /></figure><p>Click Enter and then look for the SFTP package:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/622/1*kgFPUIqLpR3GXzJh9JS6rQ.png" /></figure><p>Click on it, and the installation should start. After 10 seconds, go to the file and see if you have SFTP/FTP option there.</p><h4>Connection to the Server</h4><p>Once we have installed the SFTP package to the SublimeText, in File we should see SFTP/FTP option:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/347/1*g_fNHRTfackQSd-6JorOWw.png" /></figure><p>Click on it and then <em>Setup server. </em>A json file will open with different fields. You will need to fill in:</p><ul><li><em>host </em>— in my case, it’s <em>server.cs.university.ac.uk</em></li><li><em>user </em>— your username</li><li><em>password </em>— your password</li><li><em>remote_path </em>— starting path for navigation on the server, for example,/mnt/driver/user</li></ul><p>After that, we will create an empty file on the server called test.py to test whether it’s all set up and working. <br>Now we go back to sublime windows -&gt; file -&gt; SFTP/FTP -&gt; Browse Server. There, select the setup server file we created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/761/1*ImkqgZkRxoquE39spJJdsg.png" /></figure><p>Navigate to the directory where the file is stored and click on it. After that, click on <em>Edit </em>option<em> .</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/757/1*XLvZaKgKVCK77ZUk7TyD3A.png" /></figure><p>Write something and save it. If I now go back on the server console and display the file content there, we see that it has worked:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/333/1*Q87wQPgZqrf0oY_RzC5dgw.png" /></figure><h4>Connection to Docker running on the Server</h4><p>What if we have a Docker container running on the Server and we want to modify a file there? It’s best practice to mount the directory on the server into the Docker container so that any modifications can still be done like previously. However, what if we have some files only inside the container and not on the server? For example, we installed some packages on the container and we want to modify them. In this case, we need to do a double ssh connection. Let’s see how.</p><p>First, when writing a Docker installation file, we need to add the following code to it:</p><pre>RUN apt-get update &amp;&amp; apt-get install -y openssh-server<br>RUN mkdir /var/run/sshd<br><br># Edit &lt;password&gt; to be the desired root password. Keep it a secret!<br>RUN echo &#39;root:&lt;password&gt;&#39; | chpasswd<br>RUN sed -i &#39;s/#PermitRootLogin prohibit-password/PermitRootLogin yes/&#39; /etc/ssh/sshd_config<br>RUN apt install nano<br><br># SSH login fix. Otherwise, the user is kicked off after login.<br>RUN sed &#39;s@session\s*required\s*pam_loginuid.so@session optional pam_loginuid.so@g&#39; -i /etc/pam.d/sshd<br><br>ENV NOTVISIBLE &quot;in users profile&quot;<br>RUN echo &quot;export VISIBLE=now&quot; &gt;&gt; /etc/profile<br><br>EXPOSE 22<br>CMD [&quot;/usr/sbin/sshd&quot;, &quot;-D&quot;]</pre><p>After building the image and running the container it on the server, we need to find the container IP address using the following command:</p><pre>docker inspect -f &#39;{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}&#39; &lt;container name&gt;</pre><p>Next, go to the console and start an ssh tunnel:</p><pre>ssh -L 6000:IP-address:22 username@server.cs.university.ac.uk</pre><p>This allows us to double jump via SSH to the code in the Docker container — we first ssh to the server and, from there, SSH to the container. In other words, we establish a tunnel from port 22 of the target (the Docker container) to port 6000 on our local machine (localhost) using the proxy machine.</p><p>Now go back to Sublime and set up a new server like before but with some changes:</p><ul><li><em>host </em>— localhost</li><li><strong><em>user </em>— root</strong></li><li><em>password </em>— password from <em>RUN echo ‘root:&lt;password&gt;’ | chpasswd</em></li><li><strong><em>port </em>— 6000</strong></li><li><em>remote_path </em>— starting path for navigation on the server, for example /mnt/driver/user</li></ul><p>Then everything is the same as described before in the “Connection to the Server” section.</p><h4>Conclusions</h4><p>Establishing an SFTP connection via Sublime Text proves to be a transformative experience for developers and remote collaborators alike. The seamless integration of SFTP capabilities into this powerful code editor empowers users to effortlessly navigate, modify, and synchronize files on remote servers, all within a familiar and intuitive interface.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=841459872547" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/streamlining-file-management-a-guide-to-connecting-and-modifying-files-on-a-server-or-a-docker-841459872547">Streamlining File Management: A Guide to Connecting and Modifying Files on a Server or a Docker…</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Multiple GPU training in PyTorch and Gradient Accumulation as an alternative to it]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://medium.com/data-science/multiple-gpu-training-in-pytorch-and-gradient-accumulation-as-an-alternative-to-it-e578b3fc5b91?source=rss-cf3e4a05b535------2"><img src="https://cdn-images-1.medium.com/max/1000/0*jp3V1zwOdipUSAK7" width="1000"></a></p><p class="medium-feed-snippet">Code and Theory</p><p class="medium-feed-link"><a href="https://medium.com/data-science/multiple-gpu-training-in-pytorch-and-gradient-accumulation-as-an-alternative-to-it-e578b3fc5b91?source=rss-cf3e4a05b535------2">Continue reading on TDS Archive »</a></p></div>]]></description>
            <link>https://medium.com/data-science/multiple-gpu-training-in-pytorch-and-gradient-accumulation-as-an-alternative-to-it-e578b3fc5b91?source=rss-cf3e4a05b535------2</link>
            <guid isPermaLink="false">https://medium.com/p/e578b3fc5b91</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[pytorch]]></category>
            <dc:creator><![CDATA[Alexey Kravets]]></dc:creator>
            <pubDate>Mon, 24 Jul 2023 18:57:43 GMT</pubDate>
            <atom:updated>2023-08-01T21:14:28.864Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Backpropagation and Vanishing Gradient Problem in RNN (Part 2)]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://pub.towardsai.net/backpropagation-and-vanishing-gradient-problem-in-rnn-part-2-4fa4c0e27b54?source=rss-cf3e4a05b535------2"><img src="https://cdn-images-1.medium.com/max/600/1*hb9wcxZjjTUx12U0Lz4yhg.png" width="600"></a></p><p class="medium-feed-snippet">How it is reduced in LSTM</p><p class="medium-feed-link"><a href="https://pub.towardsai.net/backpropagation-and-vanishing-gradient-problem-in-rnn-part-2-4fa4c0e27b54?source=rss-cf3e4a05b535------2">Continue reading on Towards AI »</a></p></div>]]></description>
            <link>https://pub.towardsai.net/backpropagation-and-vanishing-gradient-problem-in-rnn-part-2-4fa4c0e27b54?source=rss-cf3e4a05b535------2</link>
            <guid isPermaLink="false">https://medium.com/p/4fa4c0e27b54</guid>
            <category><![CDATA[lstm]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[backpropagation]]></category>
            <dc:creator><![CDATA[Alexey Kravets]]></dc:creator>
            <pubDate>Wed, 19 Jul 2023 14:01:50 GMT</pubDate>
            <atom:updated>2023-08-01T21:15:02.253Z</atom:updated>
        </item>
    </channel>
</rss>