<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Emmanuel Olateju on Medium]]></title>
        <description><![CDATA[Stories by Emmanuel Olateju on Medium]]></description>
        <link>https://medium.com/@olatejuemmanuel?source=rss-8e9075c1d169------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*UNFCjtqMS4eIWoMI.jpg</url>
            <title>Stories by Emmanuel Olateju on Medium</title>
            <link>https://medium.com/@olatejuemmanuel?source=rss-8e9075c1d169------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 09:46:19 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@olatejuemmanuel/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Multi-Channel TF-Conformers for EEG Decoding — Part 1]]></title>
            <link>https://medium.com/@olatejuemmanuel/multi-channel-tf-conformers-for-eeg-decoding-part-1-56e71045e853?source=rss-8e9075c1d169------2</link>
            <guid isPermaLink="false">https://medium.com/p/56e71045e853</guid>
            <dc:creator><![CDATA[Emmanuel Olateju]]></dc:creator>
            <pubDate>Thu, 28 Aug 2025 03:57:30 GMT</pubDate>
            <atom:updated>2025-08-28T03:57:30.663Z</atom:updated>
            <content:encoded><![CDATA[<h3>Multi-Channel TF-Conformers for EEG Decoding — Part 1</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Yq6FC63ME2NgjJpDVN3ftQ.jpeg" /></figure><blockquote>“So Unrelated”: EEG Decoding Experiment Design</blockquote><p>Electroencephalogram (EEG) decoding in machine-learning is a tricky area with what I will refer to as weird experiment designs. For example, when working with EEG signals, and most other physiological signals in general: it is rather advisable to make use of unique instances of cross-validation for each set of data from subjects. This differs from the common train-test/validation split and cross-validation used in computer-vision and language modelling tasks.</p><p>When working with less complex physiological signals, some strategies like Group-Stratified-Splits, or Leave-One-Subject-Out (LOSO) validation do work. While the former creates splits such that the set of subjects&#39; data in the train and test sets are unique without overlap, the latter for each held-out validation subject trains a model on the data from all other subjects&#39;, validating on the held-out subject. These strategies provide insights into generalizability of models across different subjects or populations which is crucial in developing decoding models for physiological signals. But these approaches fail for training EEG decoding models, therefore the preponderance of within-subject cross-validation, focusing on intra-subject modelling.</p><blockquote>Why is EEG So Complicated?</blockquote><p>Let us first of all accept this as a law: <em>“hell, humans are too complex, only makes sense that our brain signals are overly complex”</em>. A foundational issue which expands is intrasubject variability. The human is at any point in time simultaneously in various states, these various states affect the expression of brain signals for each state; so, the probability of finding signals expressing a state being exactly equal at two points in time for a subject is infinitesimally small. This is for a single subject; in any single subject we cannot guarantee that the exact neural expression that occurred for a brain state at some time t = 0, will be the same neural expression we will experience for the same brain state at another time t = T. But we can be sure at both times, the neural expressions share certain characteristics.</p><p>Let us now try to understand the variability between two or more subjects. If at two time-points t = [0, T], the neural expressions for the same brain state in a subject have some variability due to differences in combinations of simultaneous brain states, it is safe to say the same will occur between two subjects. Between two or more subjects, we also expect more variability: a simpler explanation is perfect states synchrony between two or more subjects is extremely difficult, a reason to admire orchestra musicians. However, a better explanation for this is the anatomical variation of the brain among populations, variations in brain organization, and more.</p><blockquote>So why exactly is the title Multichannel TF-Conformers for EEG? I promise you it gets interesting from here, stick around a bit.</blockquote><h3>The Conformer Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N_w6eu-6w2OSpZ1WQof-1w.png" /><figcaption>The First EEG-Conformer: <a href="https://ieeexplore.ieee.org/abstract/document/9991178">Link</a></figcaption></figure><p>Above is the first conformer model for EEG decoding which was applied for motor-imagery decoding. The conformer model is basically a CNN with a transformer connected in a serial manner. But why use a CNN with a transformer, why not CNN + LSTMs.</p><blockquote>Motivation for Conformers</blockquote><p>CNNs are widely used in EEG decoding applications as they can be used as end-to-end models, processing raw EEG signals as 2D matrices, and can work with (2D) domain representations or features of the EEG such as time-frequency representations. However, CNNs make use of localized convolutional kernels capturing short-range patterns which may not be productive in applications where changes occur on a larger timescale.</p><p>A common work-around for this is the use of multi-scale convolution kernels, making use of various sizes of convolution kernels which capture short- and long-range patterns. The use of larger and multiple convolution kernels significantly increases model complexity and training time. Also, there are more parameters which make the model prone to overfitting, especially with relatively small EEG datasets in within-subject classification. The different kernel sizes potentially could capture overlapping information introducing unnecessary redundancy in information. CNNs inherently have local receptive fields and may still struggle to model long-range temporal dynamics in-spite of the use of multi-scale kernels without additional mechanisms</p><p>Another work-around, is the use of LSTMs with CNNs. The CNNs generate spatial or spectral features (across channels, frequency bands, or time windows), and LSTMs model the temporal dependencies across these features. While CNN+LSTM architectures are effective in EEG decoding, they also struggle with long-range dependencies due to vanishing/exploding gradients. Also, LSTMs are sensitive to sequence length and preprocessing, thus their performance may degrade if the sequence is too long or too short. Lastly, LSTMs are parameter-heavy and prevent parallelization taking long-time to train.</p><blockquote>Why Transformer Instead of LSTM?</blockquote><p>Transformers overcome many of the limitations of LSTMs by using self-attention mechanisms that can directly model long-range dependencies without relying on sequential recurrence. Unlike LSTMs, Transformers allow for parallelized training, making them far more efficient on modern hardware and scalable to large EEG datasets. The attention mechanism also provides a flexible way of weighting relevant temporal or spectral features, reducing the risk of vanishing gradients and improving interpretability. When combined with CNNs in a conformer model, the CNN layers capture localized spatial and spectral patterns from EEG, while the Transformer layers handle global temporal dependencies — resulting in a more balanced and powerful architecture for decoding complex brain signals.</p><blockquote>Now to the interesting part: Multi-Channel TF-Conformer</blockquote><h3>Multi-Channel TF-Conformer</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IdeakQLSZX545ywRkEg8AA.png" /><figcaption>Multi-Channel TF-Conformer: A first approach</figcaption></figure><p>From the diagram of the multi-channel TF-Conformer, we can see the only difference is it processing spectrograms from different channel electrodes as separate feature channels in the CNN, then stacking the resultant learnt representation from each feature channel across the frequency axis.</p><blockquote>Over-Ambitious Goals Makes a Man Aimless</blockquote><p>This approach was motivated by an over-ambitious goal being cut down like an overdesigned gown into a dress. The over-ambitious goals which motivated this were:</p><ol><li>Developing an architecture capable of processing large number of electrodes for general EEG decoding tasks.</li><li>Further fine-tuned on downstream tasks like motor-imagery with channel masking.</li><li>Interpretability on downstream tasks through attention matrices providing information on relevant temporal dependencies and frequency patterns.</li></ol><p>Too ambitious and very unclear. Then these over-ambitious goals were cut down.</p><blockquote>New Goals</blockquote><ol><li>Develop a conformer model that achieves SOTA performance on motor-imagery tasks using TF representation of central electrodes.</li><li>Explore cross-subject fine-tuning strategies through differntial learning rates.</li><li>Analysis to identify if more flexible CNN weights or more flexible transformer weights result in better cross-subject fine-tuning performance while preserving performance on past data.</li><li>Develop method to give insights into temporal relationships over frequency domain and spatial dimension that explain classification results.</li><li>Make a report on this.</li></ol><p>Well, these goals are still ambitious. But this is a write-up to keep track of my progress on this work. So, let’s see where we are at.</p><blockquote>Current State</blockquote><p>Currently, the conformer uses three layers of convolutions, each with kernel size of 3x3 (with LeakyReLU, batch-normalization and 2x2 average-pooling) to generate channels spectrogram representations. The representations are stacked along the frequency axis as shown in the Multi-Channel TF-Conformer diagram. Then a projector projects each time-point which is the frequency representation of all channels at that time to an embedding for the transformer. The best-performing conformer so far stacks two transformer encoders with eight attention heads for each.</p><p>Five-fold cross-validation is first carried out for intra-subject classification achieving slightly better performance than that of original conformer model. Then LOSO validation is implemented — with and without augmentation — as a first step towards cross-subject learning, and of course, the results are subpar. The table below shows the performance achieved on the BCI Competition IV-2a dataset. Augmentation explored include time-masking and frequency masking randomly masking 30 to 70% across time and frequency.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*wZ-0Afxaq-SrXayirvlr2A.png" /><figcaption>Current Results</figcaption></figure><p>The original conformer model achieves 78.66% across subjects&#39; accuracy on the BCI Competition IV-2a dataset</p><h3>Next Steps</h3><p>The immediate next step is to push the cross-subject performance closer to state-of-the-art levels. Right now, the conformer shows promise within subjects, but generalizing across subjects is still a bottleneck. I plan to experiment with more refined fine-tuning strategies, such as differential learning rates across CNN and Transformer layers, to better balance local feature extraction and global temporal modeling. I’ll also investigate data augmentation techniques tailored for EEG — like frequency masking or channel shuffling — to improve robustness and reduce overfitting on limited training data.</p><p>On the interpretability front, I want to explore how the attention maps from the Transformer can be analyzed not just for temporal relevance but also for frequency–channel interactions, giving more insight into what the model is “looking at” when making decisions. This may require refining the model architecture.</p><p>Finally, I intend to benchmark against simpler baselines to better highlight where the conformer truly adds value. This will keep the project grounded and ensure progress is measured, rather than just chasing ambitious abstractions.</p><blockquote>Thanks for reading through this long piece, and I hope you lookout for update articles on the Multi-Channel TF-Transformer</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=56e71045e853" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GPT4-Vision]]></title>
            <link>https://medium.com/rectlabs/gpt4-vision-6b70a6a48b6c?source=rss-8e9075c1d169------2</link>
            <guid isPermaLink="false">https://medium.com/p/6b70a6a48b6c</guid>
            <category><![CDATA[vidéo]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[transformation]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[vision]]></category>
            <dc:creator><![CDATA[Emmanuel Olateju]]></dc:creator>
            <pubDate>Mon, 14 Apr 2025 09:01:04 GMT</pubDate>
            <atom:updated>2025-04-14T09:01:04.982Z</atom:updated>
            <content:encoded><![CDATA[<p>That Vision Comprehension Model</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lFtJlmZU-iHSTX42XSFDWA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@ionfet?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Ion Fet</a> on <a href="https://unsplash.com/photos/blue-eye-photo-QRawWgV6gmo?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>You have probably heard about OpenAI’s latest model — GPT-4o. but actually, the latest release is GPT-4V, where the “V” stans for vision. Like previous version of openAI’s GPT, it is pre-trained on next work prediction tasks, however it is fine-tuned using reinforcement learning from human feedback (RLHF), but what sets it apart is its ability to understand and explain images. Let us break it down by comparing GPT models and their API usage, starting with GPT-3.5.</p><h3>GPT-3.5</h3><p>GPT-3.5 and subsequent models are based on the transformer architecture introduced in the paper: <a href="https://arxiv.org/abs/1706.03762"><em>“Attention Is All you Need”</em></a><em>. </em>While we won’t dive deep into the attention mechanism, we will give a high level overview on how the GPT architecture works, and you can read up on attention mechanism and the transformer architecture from <a href="https://medium.com/rectlabs">here</a>.</p><p>Generally, transformer architectures and the GPT-3.5 make use of an encoder which models the relationship between sequence elements in generating text through positional encoding of sequence and masked multi-head attention mechanisms. It also makes use of a decoder which processes immediate past outputs from the decoder itself using positional-encoding and masked multi-head attention mechanism, but with some token or sequence elements masked. Below we see the transformer architecture</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*AoAqO1FN_pXupD1UMwWPJQ.png" /><figcaption>Transformer architecture from the attention is all you need paper. Encoder on left, decoder on right.</figcaption></figure><p>One thing is obvious, the GPT-3.5 can only handle text inputs, positionally encoding each word in a sequence of texts and modelling the attention between this words in generating next word sequences. The GPT-3.5 is trained to predict the next word from previous words sequence input. Let’s see how the GPT-3.5 turbo is used</p><p><strong>Example GPT-3.5 API Usage</strong></p><pre>import openai<br><br>openai.api_key = &quot;your-api-key&quot;  # Replace with your actual API key<br><br>response = openai.ChatCompletion.create(<br>    model=&quot;gpt-3.5-turbo&quot;,  # Specify the model<br>    messages=[<br>        {&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;You are a helpful assistant.&quot;},<br>        {&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Can you explain how photosynthesis works?&quot;}<br>    ],<br>    temperature=0.7,  # Controls randomness; lower is more focused, higher is more creative<br>    max_tokens=200    # Limits the length of the response<br>)<br><br>print(response[&#39;choices&#39;][0][&#39;message&#39;][&#39;content&#39;])</pre><p>For GPT-3.5, we pass in a sentence, and we expect an output as a sentence or compilation of sentence forming paragraphs.</p><h3>GPT-4, GPT-4o</h3><p>Beyond general improvement in training and optimization, the key development in GPT-4 is support for multi-modal inputs — text, audio, and more. However, this full capability was not available until GPT-4o. earlier models like GPT-4 and GPT-4 Turbo improved performance and efficiency, but GPT-4o introduced a new approach using mixture-of-experts (MoE) models. Instead of a single model handling all inputs, GPT-4o uses specialized models for each modality. For example, audio input is converted to text by a model, then processed as text using transformers generating text outputs, the text outputs are then converted back to speech using TTS. While the core transformer remains, GPT-4o’s architecture enables true multi-modal interaction. Now let us see how to use it.</p><pre>import openai<br><br>openai.api_key = &quot;your-api-key&quot;  # Replace with your OpenAI API key<br><br># Use GPT-4o with image input<br>response = openai.chat.completions.create(<br>    model=&quot;gpt-4o&quot;,<br>    messages=[<br>        {<br>            &quot;role&quot;: &quot;user&quot;,<br>            &quot;content&quot;: [<br>                {&quot;type&quot;: &quot;text&quot;, &quot;text&quot;: &quot;What&#39;s happening in this image?&quot;},<br>                {<br>                    &quot;type&quot;: &quot;image_url&quot;,<br>                    &quot;image_url&quot;: {<br>                        &quot;url&quot;: &quot;https://example.com/path/to/your/image.jpg&quot;  # Replace with a real image URL<br>                    }<br>                }<br>            ]<br>        }<br>    ],<br>    max_tokens=300<br>)<br><br>print(response.choices[0].message.content)</pre><p>We see for GPT-4o, we can send an image alongside our text giving it instructions to describe what is happening in the image. Given this, then why GPT-4V.</p><h3>GPT-4 Vision (GPT-4V)</h3><p>While GPT-4o introduce multi-modal input and a mixture-of-experts (MoE) models that allows it to handle text, audio, and images simultaneously, making use of a mixture of models for processing he various input types, GPT-4V (Vision) is a specialized variant of GPT-4 that is explicitly optimized for visual reasoning. think of GPT-4V as they “eyes” of the GPT family — it does not just look at images; it understands, interprets, and reasons about them with remarkable depth.</p><p>GPT-4VV is based on GPT-4 architecture, but it is unquely trained and fine-tuned to process images alongside text in more integrated and deliberate way. this means that it can handle tasks like:</p><ol><li>Describing complex images.</li><li>Interpreting graphs and charts.</li><li>Explaining visual concepts.</li><li>Answering questions about what is happening in an image.</li></ol><p>Unlike GPT-4o, which routes input through different specialized models, GPT-4V is tightly integrated into a single transformer-based vision-language model enabling high fidelity and deep understanding of visual content. Now let us seed how the API of GPT-4V.</p><pre>import openai<br><br>openai.api_key = &quot;your-api-key&quot;  # Replace with your actual API key<br><br>response = openai.ChatCompletion.create(<br>    model=&quot;gpt-4-vision-preview&quot;,  # Vision model<br>    messages=[<br>        {<br>            &quot;role&quot;: &quot;user&quot;,<br>            &quot;content&quot;: [<br>                {&quot;type&quot;: &quot;text&quot;, &quot;text&quot;: &quot;Can you describe this diagram and explain what it shows?&quot;},<br>                {<br>                    &quot;type&quot;: &quot;image_url&quot;,<br>                    &quot;image_url&quot;: {<br>                        &quot;url&quot;: &quot;https://example.com/diagram.png&quot;  # Replace with actual image URL<br>                    }<br>                }<br>            ]<br>        }<br>    ],<br>    max_tokens=500<br>)<br><br>print(response[&#39;choices&#39;][0][&#39;message&#39;][&#39;content&#39;])</pre><p>GPT-4V is particularly useful in scenarios where visual context is critical — like educational content with diagrams, from analysis,medical imaging, or software interfaces. Its accuracy and reasoning on visual content set it apart from GPT-4o in tasks requiring deep image understanding.</p><h3>Comparing The GPTs</h3><pre>| Feature/Model           | GPT-3.5           | GPT-4o                           | GPT-4V (Vision)                        |<br>|-------------------------|-------------------|----------------------------------|----------------------------------------|<br>| **Input Types**         | Text only         | Text, audio, images              | Text + Image (Vision-specific)         |<br>| **Model Design**        | Transformer       | Mixture of Experts (multi-modal) | Vision-augmented Transformer           |<br>| **Image Understanding** |                   | (general capabilities)           | (deep visual reasoning)                |<br>| **Audio Support**       |                   | (TTS + STT supported)            |                                        |<br>| **Use Case Focus**      | Text generation   | Conversational multimodal AI     | Visual QA, OCR, diagram interpretation |<br>| **API Model Name**      | `gpt-3.5-turbo`   | `gpt-4o`                         | `gpt-4-vision-preview`                 |</pre><p>Well the GPT-4V model doe slack audio support, but it is obvious that when visual comprehension is the objective, then GPT-4V should be the go to model among the GPTs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6b70a6a48b6c" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rectlabs/gpt4-vision-6b70a6a48b6c">GPT4-Vision</a> was originally published in <a href="https://medium.com/rectlabs">Rectlabs Inc</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Video Classification Using SpatioTemporal Transformers]]></title>
            <link>https://medium.com/rectlabs/video-classification-using-spatiotemporal-transformers-17bad6b1d440?source=rss-8e9075c1d169------2</link>
            <guid isPermaLink="false">https://medium.com/p/17bad6b1d440</guid>
            <category><![CDATA[spatio-temporal]]></category>
            <category><![CDATA[action-recognition]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[vidéo]]></category>
            <dc:creator><![CDATA[Emmanuel Olateju]]></dc:creator>
            <pubDate>Fri, 04 Apr 2025 10:04:48 GMT</pubDate>
            <atom:updated>2025-04-04T10:04:48.152Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*APFem-5z2T1E4ETQEUvMKA.jpeg" /></figure><h3>Background</h3><p>In a previous series on transformers, we highlighted their advantages over RNNs and LSTMs for sequential data processing. Unlike sequential modelling in RNNs, transformers use attention mechanisms to capture long-range dependencies in a single pass, improving context retention and gradient flow via residual connections. Parallel processing also enables it in learning multiple attention contexts simultaneously, enhancing efficiency. This has led to architectures like BERT, enabling high-performance transfer learning across NLP, time-series forecasting, and vision tasks.</p><p>Speaking of vision tasks; the <a href="https://arxiv.org/abs/2010.11929"><strong><em>vision transformer architecture (ViT) </em></strong></a>improved image recognition performance with less computational complexity that state-of-the-art CNNs, becoming foundational for tasks like segmentation and detection. While BERTs and <a href="https://arxiv.org/abs/2010.11929"><strong><em>ViTs</em></strong></a><strong><em> </em></strong>handle 1D sequences, video tasks require 2D spatiotemporal modelling. Building on the <a href="https://arxiv.org/abs/2010.11929">ViT</a> architecture, a <a href="https://arxiv.org/abs/2102.05095"><strong><em>spatiotemporal transformer</em></strong></a> with sequential spatial and temporal attention outperforms state-of-the-art models for video classification like I3D and SlowFast while reducing training time.</p><h4>ViT &amp; Spatiotemporal Transformer</h4><p>The ViT processes images by dividing, say a 224x224 image into 256 patches (16x 16) 0f 196 pixels, the embedding each 196-pixel patch. The resultant embeddings are combined with positional encodings to retain spatial information. A CLS token is appended to aggregate information for prediction before feeding the sequence into a transformer encoder for final prediction.</p><p>The spatiotemporal transformer classifies image sequences by processing, say a Tx224x224 input, dividing each frame into 256 patches (16x16) of 196 pixels and embedding them. This forms a Tx16x16xE representation, combined with positional encoding to retain spatial and temporal positions. It applies temporal attention to model patch evolution over time for each patch using a time encoder, then spatial attention between patches within each frame using a spatial encoder. The result is flattened to Tx256xE, with a CLS token added to aggregate learned structures for final prediction.</p><p>Now that we understand how a spatiotemporal transformer works, let us implement one for the open source UCF50-Action-Recognition dataset.</p><h4>Implementation Details</h4><p>Rather than develop or spatiotemporal transformer from scratch, we make use of the <a href="https://huggingface.co/docs/transformers/en/model_doc/timesformer"><strong><em>huggingface TimesFormer</em></strong></a> package which implements the <a href="https://github.com/facebookresearch/TimeSformer"><strong><em>facebook spatiotemporal transformer</em></strong></a> described above and referred to as the <strong><em>TimesFormer.</em></strong></p><h4>Implementation</h4><p>We begin by defining or clip parameters: the number of frames for classification and the height and width of frames, then we prepare our dataset to match this.</p><pre>import os<br>import sys<br>import random<br>import math<br><br>import cv2<br><br>from sklearn.model_selection import StratifiedShuffleSplit<br><br>import numpy as np<br>import torch<br>import torch.nn as nn<br>import torch.optim as optim<br>from torch.utils.data import Dataset, Subset, DataLoader<br>import torch.nn.functional as F<br>from einops import rearrange<br><br>DEVICE = &quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;<br><br><br><br>T = SEQUENCE_LENGTH = 20<br>H, W = HEIGHT, WIDTH = 224, 224<br><br>ucf_path = &quot;/kaggle/input/ucf50-action-recognition-dataset/UCF50&quot;<br>actions = os.listdir(ucf_path)<br>N_CLASSES = 5<br>actions = [&#39;PlayingTabla&#39;, &#39;HulaHoop&#39;, &#39;Lunges&#39;, &#39;Swing&#39;, &#39;Skijet&#39;]<br># actions = random.sample(actions, N_CLASSES)<br>print(actions)<br><br>NC = 50<br><br><br>X = np.empty((0, T, H, W, 3))<br>y = []<br><br>for a, action in enumerate(actions):<br>    N = X.shape[0]<br>    clips = os.listdir(os.path.join(ucf_path, action))<br>    for clip in tqdm(clips, desc=f&quot;Extracting {action} Clip {a+1}/{len(actions)}&quot;, total=len(clips)):<br>        path = os.path.join(ucf_path, action, clip)<br>        cap = cv2.VideoCapture(path)<br><br>        if not cap.isOpened():<br>            continue<br><br>        frame_count = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))  # Total frames<br>        frames = []<br>        for f in range(frame_count):<br>            ret, frame = cap.read()<br>            if not ret:<br>                break<br><br>            frame = cv2.resize(frame, (HEIGHT, WIDTH))<br>            frame = frame/255<br>            frames.append(frame)<br><br>            if len(frames)==T:<br>                X = np.vstack((X, np.expand_dims(np.array(frames), axis=0)))<br>                y.append(a)<br>                frames = []<br><br>            if X.shape[0] &gt;= N + NC:<br>                break<br><br>        cap.release()<br><br>        if X.shape[0] &gt;= N + NC:<br>            break<br><br>X = torch.tensor(X).to(torch.float32)<br>Y = torch.tensor(y)<br><br>X = rearrange(X, &quot;b t h w c -&gt; b t c h w&quot;)<br><br>time.sleep(0.25)<br>clear_output()<br><br>print(X.size(), Y.size())</pre><p>Now let us make a PyTorch dataset and dataloader</p><pre>class UCF50Dataset(Dataset):<br><br>    def __init__(self, X, y):<br>        assert X.size(0) == len(y)<br><br>        self.X = X<br>        self.y = y<br><br>    def __len__(self):<br>        return self.X.size(0)<br><br>    def __getitem__(self, idx):<br>        if self.y != None:<br>            return self.X[idx], self.y[idx]<br>        return self.X[idx]<br><br>dataset = UCF50Dataset(X, Y)<br><br>train_batch_size = 5<br>val_batch_size = 3<br>print(train_batch_size, val_batch_size)<br><br>N = len(dataset)<br>split_ratio = 0.8<br>split_size = int(0.8*N)<br><br>split = StratifiedShuffleSplit(n_splits=1, test_size=1-split_ratio, random_state=42)<br>train_idx, val_idx = next(split.split(range(N), dataset.y))<br><br>train_dataset = Subset(dataset, train_idx)<br>val_dataset = Subset(dataset, val_idx)<br><br>train_loader = DataLoader(train_dataset, batch_size=train_batch_size, shuffle=True)<br>val_loader = DataLoader(val_dataset, batch_size=val_batch_size, shuffle=True)<br><br>print(f&quot;Train samples: {len(train_dataset)}, Test samples: {len(val_dataset)}&quot;)</pre><p>Now let us define our <a href="https://huggingface.co/docs/transformers/en/model_doc/timesformer"><strong><em>TimesFormer</em></strong></a><strong><em> </em></strong>hyperparameters</p><pre>from transformers import TimesformerConfig, TimesformerModel, TimesformerForVideoClassification<br>from transformers import VivitConfig, VivitModel<br><br>N_PATCHES = 64<br>N_EMBD = 768<br>N_ATTN_HEADS = 6<br>N_LAYERS = 2<br>FF_INTERMEDIATE = 3072<br>INIT_STD = 0.2<br><br>timesformer_config_dict = {<br>    &quot;image_size&quot;: HEIGHT,<br>    &quot;patch_size&quot;: N_PATCHES,<br>    &quot;num_channels&quot;: 3,<br>    &quot;num_frames&quot;: T,<br>    &quot;hidden_size&quot;: N_EMBD,<br>    &quot;num_hidden_layers&quot;:N_LAYERS,<br>    &quot;num_attention_heads&quot;: N_ATTN_HEADS,<br>    &quot;intermediate_size&quot;: FF_INTERMEDIATE,<br>    &quot;hidden_act&quot;: &quot;leaky_relu&quot;,<br>    &quot;hidden_dropout_prob&quot;: 0.05,<br>    &quot;attention_probs_dropout_prob&quot;: 0.1,<br>    &quot;initializer_range&quot;: INIT_STD,<br>    &quot;layer_norm_eps&quot;: 1E-6,<br>    &quot;qkv_bias&quot;: False,<br>    &quot;attention_type&quot;: &quot;divided_space_time&quot;,<br>    &quot;drop_path_rate&quot;: 0.2,<br>}<br><br>timesformer_config = TimesformerConfig(**timesformer_config_dict)</pre><p>Now we can define our model which creates a TimesFormer instance, and then adds an extra layer computes the logits for softmax classification using the learned CLS token.</p><pre>class ActionClassifier(nn.Module):<br><br>  def __init__(self, n_embd, n_classes, TimesFormer):<br>    super(ActionClassifier, self).__init__()<br>    self.timesformer = TimesFormer<br>    self.cls_layer = nn.Linear(N_EMBD, N_CLASSES)<br><br>  def forward(self, x, y=None, device=DEVICE):<br>    out = self.timesformer(x)<br>    embeddings = out.last_hidden_state<br>    cls_token = embeddings[:, 0, :]<br><br>    logits = F.leaky_relu(self.cls_layer(cls_token))<br>    if y==None:<br>      return logits<br>    elif isinstance(y, torch.Tensor):<br>      loss = F.cross_entropy(logits, y, label_smoothing=0.1)<br>      acc = accuracy_score(logits, y)<br>      return logits, loss, acc<br><br><br><br>model = ActionClassifier(N_EMBD, N_CLASSES, TimesformerModel(timesformer_config))<br>model =model.to(DEVICE)<br><br>def init_weights(m):<br>  if isinstance(m, nn.Linear):  # Apply only to Linear layers<br>    torch.nn.init.normal_(m.weight, mean=0.0, std=INIT_STD)<br>    if m.bias is not None:<br>      torch.nn.init.zeros_(m.bias)  # Set bias to 0<br><br>model.apply(init_weights)</pre><p>Then we can train our model</p><pre>torch.manual_seed(42)<br>np.random.seed(42)<br><br>n_iter = 3000<br>eval_inter = 5<br>n_evals = 3<br><br>lr_ = 3E-4<br>betas_ = (0.9, 0.999)<br>optimizer = optim.Adam(model.parameters(), lr=lr_, betas=betas_)<br>lr_scheduler = LRScheduler(lr_, patience=5, factor=1, margin=0.05, n_evals=n_iter//n_evals)<br><br>early_stop = EarlyStopping(patience=10, margin=0.1)<br><br>chk_pt = CheckpointModel(margin=0.05)<br><br>history = {<br>    &quot;train_loss&quot;: [],<br>    &quot;val_loss&quot;: [],<br>    &quot;train_acc&quot;: [],<br>    &quot;val_acc&quot;: [],<br>    &quot;iter&quot;: [],<br>    &quot;lr&quot;: [],<br>}<br><br>for i in tqdm(range(n_iter), total=n_iter):<br><br>    if i==0 or i%eval_inter==0 or i&gt;=n_iter-1:<br>        model = model.eval()<br>        with torch.no_grad():<br>            <br>            train_loss=torch.zeros((n_evals,))<br>            train_acc=torch.zeros((n_evals,))<br>            for j in range(n_evals):<br>                with torch.autocast(device_type=DEVICE):<br>                    x, y = next(iter(train_loader))<br>                    x = x.to(DEVICE)<br>                    y = y.to(DEVICE)<br>                    logits, loss, acc = model(x, y, device=DEVICE)<br>                train_loss[j] = loss<br>                train_acc[j] = acc<br><br>            val_loss = torch.zeros((n_evals,))<br>            val_acc = torch.zeros((n_evals,))<br>            for j in range(n_evals):<br>                with torch.autocast(device_type=DEVICE):<br>                    x, y = next(iter(val_loader))<br>                    x = x.to(DEVICE)<br>                    y = y.to(DEVICE)<br>                    logits, loss, acc = model(x, y, device=DEVICE)<br>                val_loss[j] = loss<br>                val_acc[j] = acc<br>                <br>            train_loss = train_loss.mean().item()<br>            train_acc = train_acc.mean()<br>            val_loss = val_loss.mean().item()<br>            val_acc = val_acc.mean()<br><br>            # lr_ = lr_scheduler.exponential_lr(val_loss)<br>            # for param_group in optimizer.param_groups:<br>            #     param_group[&quot;lr&quot;] = lr_<br><br>            history[&quot;train_loss&quot;].append(train_loss)<br>            history[&quot;val_loss&quot;].append(val_loss)<br>            history[&quot;train_acc&quot;].append(train_acc)<br>            history[&quot;val_acc&quot;].append(val_acc)<br>            history[&quot;iter&quot;].append(i)<br>            history[&quot;lr&quot;].append(lr_)<br><br>            early_stop_signal = early_stop.check(train_loss, val_loss)<br>            if early_stop_signal == True:<br>                break<br><br>            chk_pt.checkpoint(val_loss, model)<br><br>    model = model.train()<br>    optimizer.zero_grad()<br>    with torch.autocast(device_type=DEVICE):<br>        x, y = next(iter(train_loader))<br>        x, y = x.to(DEVICE), y.to(DEVICE)<br>        logits, loss, acc = model(x, y, device=DEVICE)<br>    loss.backward()<br>    torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)<br>    optimizer.step()   </pre><p>After training, let us see our training loss, accuracy, and confusion matrix, which tells us how well our model performs on the validation set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EqdGkzH-NEz8TbsVao5jXw.png" /><figcaption>Loss &amp; Accuracy</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/584/1*4SIkVX3HYMS41-XBJfwKEw.png" /><figcaption>Validation Set Confusion Matrix</figcaption></figure><h4>Real Time Inference</h4><p>Let us run inference on some clips with the resulting video shown</p><pre>model.load_state_dict(chk_pt.best_state)<br>model.eval()<br><br>fourcc = cv2.VideoWriter_fourcc(*&#39;mp4v&#39;)<br>output_video_path = &quot;continuous_inference.mp4&quot;<br>out = None<br><br>for a, action in enumerate(actions):<br>    clips = os.listdir(os.path.join(ucf_path, action))[:5]<br>    for clip in tqdm(clips, desc=action):<br>        path = os.path.join(ucf_path, action, clip)<br>        cap = cv2.VideoCapture(path)<br>    <br>        if not cap.isOpened():<br>            print(f&quot;Skipping file: {path}&quot;)<br>            continue<br>    <br>        frame_count = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))<br>        width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))<br>        height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))<br>        fps = int(cap.get(cv2.CAP_PROP_FPS))<br>        if out is None:<br>            out = cv2.VideoWriter(output_video_path, fourcc, fps, (width, height))<br>        frames = []<br>        <br>        for f in range(frame_count):<br>            ret, frame = cap.read()<br>            if not ret:<br>                break<br>    <br>            frame_resized = cv2.resize(frame, (HEIGHT, WIDTH))<br>            frame_resized = frame_resized/255<br>            if len(frames) == T:<br>                frames = frames[1:]<br>            frames.append(frame_resized)<br>    <br>            if len(frames)==T:<br>                X = torch.tensor(np.array(frames), dtype=torch.float32).to(DEVICE)<br>                X = X.unsqueeze(dim=0)<br>                X = rearrange(X, &quot;b t h w c -&gt; b t c h w&quot;)<br>    <br>                logits = model(X)<br>                probs = F.softmax(logits, dim=-1)<br>                pred = torch.argmax(probs, dim=-1)[0].item()<br>                pred = actions[pred]<br>    <br>                font = cv2.FONT_HERSHEY_SIMPLEX<br>                cv2.putText(frame, f&quot;Prediction: {pred}&quot;, (20,50), font, 0.5, (0, 255, 0), 2, cv2.LINE_AA)<br><br>            out.write(frame)<br>    <br>        cap.release()<br><br>if out is not None:<br>    out.release()<br><br>print(f&quot;Saved classified video: {output_video_path}&quot;)</pre><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FEzDjpJmlI68%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DEzDjpJmlI68&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FEzDjpJmlI68%2Fhqdefault.jpg&amp;type=text%2Fhtml&amp;schema=youtube" width="640" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/3a4dc3a58239ef27a22e033d3d5334ac/href">https://medium.com/media/3a4dc3a58239ef27a22e033d3d5334ac/href</a></iframe><h4>Conclusion</h4><p>In this write-up, we explored spatiotemporal transformers for video classification, building on the foundational ViT architecture. We implemented a TimesFormer model using Hugging Face’s package and applied it to the UCF50-Action-Recognition dataset. Our approach involved preprocessing video clips, defining a custom dataset, and training a transformer-based model capable of capturing both spatial and temporal dependencies.</p><p>The results highlight the potential of spatiotemporal transformers in video classification, potentially outperforming traditional CNN-based models while maintaining computational efficiency. Our validation metrics and real-time inference demonstrate the model’s ability to generalize well across diverse action categories.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=17bad6b1d440" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rectlabs/video-classification-using-spatiotemporal-transformers-17bad6b1d440">Video Classification Using SpatioTemporal Transformers</a> was originally published in <a href="https://medium.com/rectlabs">Rectlabs Inc</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CLIP — Contrastive Language-Image Pre-training]]></title>
            <link>https://medium.com/rectlabs/clip-contrastive-language-image-pre-training-dce66ae18fe1?source=rss-8e9075c1d169------2</link>
            <guid isPermaLink="false">https://medium.com/p/dce66ae18fe1</guid>
            <category><![CDATA[image]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[pre-trained-model]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[contrastive-learning]]></category>
            <dc:creator><![CDATA[Emmanuel Olateju]]></dc:creator>
            <pubDate>Tue, 28 Jan 2025 11:01:59 GMT</pubDate>
            <atom:updated>2025-01-28T11:01:59.577Z</atom:updated>
            <content:encoded><![CDATA[<h3>CLIP — Contrastive Language-Image Pre-training</h3><h4>Language-Image Context</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LBuXqalU8PPjKqnOSlOaHA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@rhondak?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">RhondaK Native Florida Folk Artist</a> on <a href="https://unsplash.com/photos/assorted-color-paintbrushes-_Yc7OtfFn-0?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><blockquote>Please, if you have no idea what contrastive learning is, you can review it <a href="https://medium.com/p/c2c557865402/edit">here</a> before reading this article.</blockquote><h3>A Light Joke</h3><p>During my undergraduate years, I worked on brain-computer interfaces, experimenting with non-invasive EEGs and EMGs. Back then, my classmates would often joke: <em>“Petron, can we copy ***_dove’s brain and paste it into ours?”</em> <em>***_dove</em> was the undisputed genius of our class — possibly the entire department — and their joke always made me laugh. I always wanted to explain and would reply saying, “It doesn’t work that way, but let me explain.”</p><p>Now, looking back, I almost wish it did work that way. Imagine the possibilities: replicating the mind of artists, then picking up a brush, making a single enigmatic dot on a blank canvas, and somehow selling that “masterpiece” for a piece of Tesla, SpaceX, or X shares. Nice, right? But, as a popular Nigerian song goes: <em>“vivid imagination.”</em></p><h4>Why The Joke?</h4><p>Artists possess a unique ability to create visually compelling works of art from prompts — whether emotional, verbal, or otherwise. For someone like me, who is, let’s say, artistically disabled, understanding these works often requires assistance from the artistically enabled. They can decipher the deeper meaning behind a painting or the inspiration driving its creation with ease.</p><p>But here’s what fascinates me most: how do these artistically enabled individuals intuitively interpret a painting, connecting it back to the underlying prompt? And more intriguingly, how do artists themselves manage to map an abstract prompt into tangible artwork that viewers can later “decode” and trace back to its essence?</p><p>I often wonder if I’ll ever join the ranks of the artistically enabled. Who knows — maybe I could become an art critic one day, trading STEM for a brush with artistic insight. <em>That African Van Gogh, init.</em></p><h4>Enough With The Jokes</h4><p>Or else, I just prepare for a career in comedy; would barely survive though. The reason for this joke is to point out the ability for the artist to associate prompts from a different domain to representations in a different domain, a visual domain. Also, the ability of the artistically enabled to map representations in the visual domain back to that of the prompt. How can we explain this?</p><p>First, we can assume that the artist and the artistically enabled — which we refer to as the art critic from this point henceforth — share similar prompt domains. Computationally or mathematically, the prompt distribution of the artist and art-critic are similar. While discussing transformers, we explained the concept of an embedding used in developing vector representation for items while preserving semantics by keeping vector representation of more similar items closer together in the embedding space: so the embedding for a knife and a metal-spoon are closer than a knife is to a spanner, though they are all metals. We can say this for the artist and art-critic: their prompt domain or space are semantically similar. Likewise we can assume the visual domains or space are similar for these two personalities. Then how do they relate these domains?</p><h3>Domain / Embedding Space &amp; Relationships</h3><h4>The Prompt Domain:</h4><p>To illustrate the concept of embedding text into a multi-dimensional space, let’s consider our example involving three everyday objects: a knife, a metal spoon, and a spanner. Suppose our embedding results in a 3D vector space. Each dimension of this space captures a specific attribute:</p><ul><li><strong>x-axis</strong>: The nature of the material (e.g., metal, wood) encoded as numerical values (e.g., 0, 1, …).</li><li><strong>y-axis</strong>: The area of application (e.g., kitchen, workshop, study), also mapped to numerical values.</li><li><strong>z-axis</strong>: The level of “flashiness” or visual appeal of the item.</li></ul><p>The <strong>z-component</strong> represents how flashy, ostentatious, or attractive an object is perceived to be. Now, let’s consider the relationships between our three items:</p><ol><li><strong>Material similarity (x-axis):</strong> All three items are made of metal, so their x-values will be relatively close, though not identical, as the quality of their materials may differ.</li><li><strong>Application overlap (y-axis):</strong> A metal spoon and a knife are primarily kitchen tools, so they’ll have similar y-values. A knife and a spanner, however, may also share some proximity here, as knives can occasionally be used in workshops.</li><li><strong>Flashiness (z-axis):</strong> A spanner is probably the least flashy of the three, while a metal spoon or knife (especially if it once belonged to someone like Mansa Musa) might be perceived as far more ornate or visually striking.</li></ol><p>In this 3D embedding space, the metal spoon and knife would cluster closely, particularly in the <strong>xy-plane</strong>, given their similar materials and applications. However, the spanner would be positioned farther away due to its distinct lack of flashiness and less overlap in application. The resulting embedding spaces helps capture nuanced relationships between the different items. This is the textual prompt domain or embedding space, also called the text-encoding. Let’s see the other domain.</p><h4>The Visual Domain:</h4><p>For the visual domain or embedding, we also assume a 3D embedding space where each axis quantifies specific visual elements. Here’s how we can define it:</p><ul><li><strong>x-axis</strong>: The number of circles required to construct the object.</li><li><strong>y-axis</strong>: The number of squares needed to represent the object.</li><li><strong>z-axis</strong>: The number of triangles necessary to form the object.</li></ul><p>This setup allows us to map visual objects into a structured representation based on their geometric components. However, a key challenge emerges: the relationships in the visual domain may not directly align with the semantics in the prompt domain.</p><h4>The Problem of Semantic &amp; Geometric Misalignment</h4><p>Consider the case of a “knife.” In the <strong>prompt domain</strong> (textual or conceptual understanding), the meaning and relationships of “knife” are intuitive and context-dependent. However, in the <strong>visual domain</strong>, the knife is represented based on geometric primitives (circles, squares, triangles), no information on semantic relationship.</p><p>This misalignment is due to the visual and prompt domains being defined independently. So therefore, relating the “knife” in one domain to its counterpart in the other becomes complex. So we need to align the representations of these domains, for that to happen, they have to be learnt and not predefined. Contrastive learning helps us learn these representations while aligning them.</p><h3>Contrastive Learning</h3><p>Contrastive learning offers a solution by optimizing the representations in both domains to align semantically. Instead of pre-defined embeddings, we enable the domains to learn from each other. By aligning the embeddings, the semantic relationships in the prompt domain can mirror those in the visual domain and vice versa.</p><p>In essence, contrastive learning bridges the gap between the conceptual understanding of the objects and their geometric representation. This alignment allows for more meaningful connection across the two domains. How does it work? Look at the two matrix below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/978/1*vpb4pMwM2uiLhZyW2SK0Iw.png" /></figure><p>Each row is an embedding for an object in the two domains. To align the embedding of these objects in both domains, we need to know how similar or far apart their current embeddings are: to do this we take the product of the matrix on the left, and the transpose of that on the right resulting in a matrix that tells us how the aligned the embedding of each object is in one domain to itself and other objects in the other domain. For explanation purposes, let us transpose the resultant matrix which we can call the similarity or alignment matrix. This image is shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*niATMGwLEpL50WzarENMzA.png" /><figcaption><a href="https://arxiv.org/pdf/2103.00020">https://arxiv.org/pdf/2103.00020</a></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/76/1*lkd5nr-_UdTpzhYmj3WzPQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/67/1*Sxjij_O-pbtlCAFS6-Lqiw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/377/1*52ceds8GrpL3FNQMNP1q-g.png" /></figure><p>The resultant matrix tells us how aligned each textual prompt embedding is to its corresponding visual embedding and vice-versa. Of course, non-diagonal values in the matrix can be significant i.e. a text prompt embedding is similar to a non-corresponding image embedding. This is why we optimize.</p><p>Taking the product of the text embeddings and the visual embeddings is proportional to the cosine similarity between the vectors in the two domains. We want to optimize such that only diagonals are significantly aligned, So we can generate a NxN matrix where the first row has its diagonal entry (the first ) being 1 and other entries zero, then subsequent rows are right shifted versions of the previous row. This matrix is described below</p><pre>   | T1 | T2 | ........... | TN|<br>I1 | 1  | 0  | ........... | 0 |<br>I2 | 0  | 1  | ........... | 0 |<br>.  | . .| .  | ........... | 0 |<br>.  | . .| .  | ........... | 0 |<br>I1 | 0  | 0  | ........... | 1 |</pre><p>We can then optimize the alignment matrix to follow this target matrix, and perform backpropagation, or alternatiely: generate a vector of 0 through N-1, and compute the cross-entropy between the alignment matrix and the vector. This way we perform contrastive learning, tuning the prompt domain to align to the visual domain and vice versa. The diagram explaining this is shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/735/1*lMT8PyB6UmxYE9tR86Tr0Q.png" /><figcaption><a href="https://arxiv.org/pdf/2103.00020">https://arxiv.org/pdf/2103.00020</a></figcaption></figure><p>We will not endeavor to explain contrastive learning itself in depth here. Please go through <a href="https://medium.com/p/c2c557865402/edit">this article</a> if you do not understand <a href="https://medium.com/p/c2c557865402/edit">constrastive learning</a>. Now let us see our implementation of the CLIP model.</p><h3>Code</h3><p>We will be using the Fashion_MNIST dataset.</p><pre>train = datasets.FashionMNIST(root=&quot;data/&quot;, train=True, transform=transforms.ToTensor(), download=True)<br>test = datasets.FashionMNIST(root=&quot;data/&quot;, train=False, transform=transforms.ToTensor(), download=True)<br><br>X_train, y_train = train.data.to(torch.long), train.targets<br>X_test, y_test = test.data.to(torch.long), test.targets<br><br>clear_output(wait=True)<br>time.sleep(0.25)<br><br>captions = {<br>    0: &quot; An image of a t-shirt/top &quot;,<br>    1: &quot; An image of trousers &quot;,<br>    2: &quot; An image of a pullover &quot;,<br>    3: &quot; An image of a dress &quot;,<br>    4: &quot; An image of a coat &quot;,<br>    5: &quot; An image of a sandal &quot;,<br>    6: &quot; An image of a shirt &quot;,<br>    7: &quot; An image of a sneaker &quot;,<br>    8: &quot; An image of a bag &quot;,<br>    9: &quot; An image of an ankle boot &quot;<br>}<br><br>NO_CLASSES = len(captions)<br><br>caption_fn = lambda idx: captions[idx].lower()<br>train_captions = list(map(caption_fn, y_train.tolist()))<br>test_captions = list(map(caption_fn, y_test.tolist()))<br><br>clear_output(wait=True)<br>time.sleep(0.25)</pre><p>Based on the labels of the images, we have attached captions or associated sentences serving as our prompt. Let us use byte-pair encoding to develop a vocabulary of our prompt.</p><pre>from collections import Counter<br><br>def byte_pair_encoding(data, num_merges):<br><br>    vocab = Counter()<br>    for char in data.split():<br>          vocab[&#39; &#39;.join(list(char))] += 1<br>    print(f&quot;Original Vocab: {vocab.items()}&quot;)<br><br>    for i in range(num_merges):<br>      pairs = Counter()<br>      for chord, freq in vocab.items():<br>        symbols = chord.split()<br>        for j in range(len(symbols)-1):<br>          pairs[symbols[j], symbols[j+1]] += freq<br>      most_frequent_pair = max(pairs, key=pairs.get)<br><br>      new_vocab = {}<br>      bigram = &#39;&#39;.join(most_frequent_pair)<br>      for chord in vocab:<br>        # print(f&quot;{&#39; &#39;.join(most_frequent_pair)}, {bigram}&quot;)<br>        chord_ = chord.replace(&#39; &#39;.join(most_frequent_pair), bigram)<br>        new_vocab[chord_] = vocab[chord]<br>      vocab = new_vocab<br><br>    print(f&quot;Byte Pairs: {vocab.items()}&quot;)<br>    return vocab<br><br>all_captions = &quot;&quot;<br>for idx in captions:<br>    all_captions += captions[idx]<br><br>VOCAB_MAPPING = byte_pair_encoding(all_captions, 53)</pre><p>From the image describing CLIP, for our text encoder we will use a BERT model. our special tokens are defined as:</p><pre># --&gt; FORM TOKEN EMBEDDING LOOK UP TABLE (LUT)<br>SPECIAL_TOKENS = [<br>    &quot;[PAD]&quot;, &quot;[UNK]&quot;, &quot;[CLS]&quot;, &quot;[MASK]&quot;, &quot;[SEP]&quot;, &quot;&lt;S&gt;&quot;, &quot;&lt;T&gt;&quot;<br>]<br>LST = len(SPECIAL_TOKENS) # --&gt; NO OF SPECIAL TOKENS<br>wtoi = {} # --&gt; WORD TO TOKEN EMBEDDING LUT<br>itow = {} # --&gt; TOKEN EMBEDDING TO WORD LUT<br>for idx, st in enumerate(SPECIAL_TOKENS): # --&gt; APPEND SPECIAL TOKESN TO LUTs<br>  wtoi[st] = idx<br>  itow[idx] = st<br>for idx, token in enumerate(VOCAB_MAPPING):<br>    wtoi[token] = idx + LST<br>    itow[idx + LST] = token<br><br>train_captions = [[wtoi[token] if token in wtoi else wtoi[&#39;[UNK]&#39;] for token in train_captions[idx].split(&quot; &quot;)] for idx in range(len(train_captions))]<br>test_captions = [[wtoi[token] if token in wtoi else wtoi[&#39;[UNK]&#39;] for token in test_captions[idx].split(&quot; &quot;)] for idx in range(len(test_captions))]</pre><p>Since we are using a BERT, we need to carry out the masked language model (MLM) objective, we can skip the Next Sentence Prediction (NSP) as this does not apply in this case and we use the [CLS] embedding vector as our prompt embedding. We extract masked prompts, their masks, and masked tokens according to the BERT paper, along with their corresponding images with the function below.</p><pre>def get_batch(x_images, x_captions, x_targets, batch_size):<br>    assert len(x_images) == len(x_captions) == len(x_targets)<br>    assert (batch_size) &lt;= len(x_images)<br><br>    samples_idx = torch.randint(0, len(x_images), (batch_size,))<br>    _captions = []<br>    masked_pos = []<br>    masked_tokens = []<br>    <br>    for s_idx in samples_idx:<br>        caption = x_captions[idx]<br>        caption = torch.tensor([wtoi[&#39;[CLS]&#39;]] + caption + [wtoi[&#39;[PAD]&#39;]]*(MAX_LEN - len(caption)) if len(caption)&lt;MAX_LEN else [wtoi[&#39;[CLS]&#39;]] + caption, dtype=torch.long)<br><br>        maskable_idx = [ idx for idx, i in enumerate(caption) if (i!=wtoi[&#39;[CLS]&#39;]) and (i!=wtoi[&#39;[PAD]&#39;]) and (i!=wtoi[&#39;[SEP]&#39;]) ]<br>        mask_idx = random.sample(maskable_idx, 4)<br>        masked_pos.append(mask_idx)<br>        masked_tokens.append(caption[mask_idx].tolist())<br>        <br>        for m_idx in mask_idx:<br>            if torch.rand(1).item() &lt; 0.2:<br>                r_idx = torch.randint(0, VOCAB_SIZE, (1,)).item()<br>                caption[m_idx] = wtoi[itow[r_idx]]<br>            elif torch.rand(1).item() &lt; 0.6:<br>                caption[m_idx] = wtoi[&#39;[MASK]&#39;]<br>            else:<br>                pass<br>        _captions.append(caption.tolist())<br><br>    _captions = torch.tensor(_captions)<br>    masked_tokens = torch.tensor(masked_tokens)<br>    masked_pos = torch.tensor(masked_pos)<br>    <br>    return (<br>        x_images[samples_idx], x_targets[samples_idx],<br>        _captions, masked_tokens, masked_pos<br>    )</pre><p>Parameters for the BERT:</p><pre># --&gt; BERT Parameters<br>ATTN_BLOCKS = L = 3<br>HEAD_SIZE = H = 12<br>ATTN_HEADS = A = 6<br>N_SEGMENTS = S = 1<br>N_EMBD = E = VOCAB_SIZE // 2<br><br><br>print(VOCAB_SIZE, MAX_LEN, N_EMBD, HEAD_SIZE)</pre><p>In a past <a href="https://medium.com/rectlabs/berts-f5fad6ba2806">article on BERT</a>, we implemented BERT, pre-trained and fine-tuned it. We just replicate that with slight changes to the forward method for the BERT class:</p><pre># --&gt; IMPLEMENT BERT TEXT ENCODER<br><br>def get_attn_mask(x):<br>  B, T = x.size()<br>  attn_mask = x.eq(wtoi[&#39;[PAD]&#39;]) | x.eq(wtoi[&#39;[UNK]&#39;])<br>  attn_mask = attn_mask.unsqueeze(1)<br>  attn_mask = attn_mask.expand(B, T, T)<br>  return attn_mask<br><br>class InputEmbedding(nn.Module):<br><br>  def __init__(self):<br>    super(InputEmbedding, self).__init__()<br>    self.tok_embd = nn.Embedding(VOCAB_SIZE, N_EMBD)<br>    self.segment_embd = nn.Embedding(N_SEGMENTS, N_EMBD)<br>    self.pos_embd = nn.Embedding(MAX_LEN+1, N_EMBD)<br>    self.norm = nn.LayerNorm(N_EMBD)<br><br>  def forward(self, x, seg, pos):<br>    B, T = x.size()<br>    tok_embd = self.tok_embd(x)<br>    seg_embd = self.segment_embd(seg)<br>    pos_embd = self.pos_embd(pos)<br>    embd = tok_embd + seg_embd + pos_embd<br>    return self.norm(embd)<br><br>class Head(nn.Module):<br><br>  def __init__(self):<br>    super(Head, self).__init__()<br>    self.dropout = nn.Dropout(0.4)<br>    self.query = nn.Linear(N_EMBD, HEAD_SIZE, bias=False)<br>    self.key = nn.Linear(N_EMBD, HEAD_SIZE, bias=False)<br>    self.value = nn.Linear(N_EMBD, HEAD_SIZE, bias=False)<br><br>  def forward(self, x, attn_mask):<br>    x = self.dropout(x)<br>    Q = self.query(x)<br>    K = self.key(x)<br>    V = self.value(x)<br><br>    scores = (Q @ K.transpose(-1, -2)) / HEAD_SIZE**0.5<br>    scores.masked_fill_(attn_mask, torch.iinfo(torch.long).min)<br>    attn = nn.Softmax(dim=-1)(scores)<br>    out = attn @ V<br><br>    return out<br><br>class MultiHeadAttention(nn.Module):<br><br>  def __init__(self):<br>    super(MultiHeadAttention, self).__init__()<br>    self.dropout = nn.Dropout(0.35)<br>    self.heads = nn.ModuleList([Head() for h in range(ATTN_HEADS)])<br>    self.proj = nn.Linear(ATTN_HEADS * HEAD_SIZE, N_EMBD, bias=False)<br>    self.norm = nn.LayerNorm(N_EMBD)<br><br>  def forward(self, x, attn_mask):<br>    out = self.dropout(x)<br>    out = torch.cat([h(x, attn_mask) for h in self.heads], dim=-1)<br>    out = self.proj(out)<br>    out = F.gelu(out)<br>    out = self.norm(x + out)  # --&gt; Layer Normalization Around MultiHead sub-layer of encoder<br>    return out<br><br>class EncoderLayer(nn.Module):<br><br>  def __init__(self):<br>    super(EncoderLayer, self).__init__()<br>    self.sa = MultiHeadAttention()<br>    self.fc1 = nn.Linear(N_EMBD, 2*N_EMBD, bias=False)<br>    self.fc2 = nn.Linear(2*N_EMBD, N_EMBD, bias=False)<br>    self.norm1 = nn.LayerNorm(N_EMBD)<br>  <br>  def forward(self, x, attn_mask):<br>    out = self.sa(x, attn_mask)<br>    out = F.gelu(self.fc1(out))<br>    out = F.gelu(self.fc2(out))<br>    out = self.norm1(x + out)<br>    return out<br><br>class BERT(nn.Module):<br><br>  def __init__(self):<br>    super(BERT, self).__init__()<br>    self.embedding = InputEmbedding()<br>    self.encoders = nn.ModuleList([EncoderLayer() for _ in range(ATTN_BLOCKS)])<br><br>    self.nsp_fc = nn.Linear(N_EMBD, NO_CLASSES, bias=False)<br>    self.mlm_fc = nn.Linear(N_EMBD, N_EMBD, bias=False)<br>    self.mlm = nn.Linear(N_EMBD, VOCAB_SIZE, bias=False)<br><br>  def forward(self, x, seg, pos, masked_pos=None):<br>    mask = get_attn_mask(x)<br>    embd = self.embedding(x, seg, pos)<br>    for enc in self.encoders:<br>      embd = enc(embd, mask)<br><br>    embedding = F.gelu(self.nsp_fc(embd[:, 0, ::]))<br><br>    masked_pos = masked_pos.unsqueeze(-1).expand(-1, -1, embd.size(-1)) #--&gt; Expand to 3D Adding N_EMBD as last dimension<br>    mlm = torch.gather(embd, 1, masked_pos)<br>    mlm = F.gelu(self.mlm_fc(mlm))<br>    logits_mlm = self.mlm(mlm)<br><br>    return embedding, logits_mlm<br><br>def init_weights(module):<br>  if isinstance(module, nn.Linear):<br>    torch.nn.init.normal_(module.weight, mean=0.0, std=0.02)<br>    if module.bias!=None:<br>      torch.nn.init.zeros_(module.bias)</pre><p>We make use of a CNN with three transformer encoder layers for our image or visual encoding:</p><pre># --&gt; IMAGE ENCOD, USE CNN<br><br>class CNN_Encoder(nn.Module):<br><br>    def __init__(self, channels=1, kernel_div=(4, 4), padding=(6, 6), dilation=(1, 1), height=28, width=28):<br>        super(CNN_Encoder, self).__init__()<br><br>        stride = kernel_div<br>        kernel_size = (height//kernel_div[0], width//kernel_div[1])<br>        self.conv_layer = nn.Conv2d(channels, channels, kernel_size, stride=kernel_div, bias=False)<br>        self.hout = ((height + 2*padding[0] - dilation[0]*(kernel_size[0] - 1) - 1) / stride[0]) + 1<br>        self.wout = ((width + 2*padding[1] - dilation[1]*(kernel_size[1] - 1) - 1) / stride[1]) + 1<br><br>        self.attn_dropout = nn.Dropout(0.4)<br>        self.b_norm1 = nn.BatchNorm1d(6*6, track_running_stats=True)<br><br>        self.pixel_embedding =nn.Embedding(6*6, N_EMBD)<br>        <br>        self.embd = nn.Linear(6*6, 6*6*N_EMBD, bias=False)<br>        self.encoders = nn.ModuleList([EncoderLayer() for i in range(L)])<br><br>        self.b_norm2 = nn.BatchNorm1d(6*6*N_EMBD, track_running_stats=True)<br><br>        self.proj = nn.Linear(6*6*N_EMBD, NO_CLASSES)<br><br>    def forward(self, x):<br>        x = x.unsqueeze(1).to(torch.float)<br>        out = self.conv_layer(x)<br>        <br>        out = self.attn_dropout(out)<br><br>        B, _, h, w =out.size() <br>        out = out.view(B, _*h*w)<br>        out = self.b_norm1(out)<br>        B, T = out.size()<br><br>        pos = torch.range(0, T-1, step=1).to(torch.long)<br>        pos = pos.to(device)<br>        pos_embd = self.pixel_embedding(pos)<br>        pos_embd = pos_embd.unsqueeze(dim=0)<br>        pos_embd = pos_embd.expand(B, -1, -1)<br><br>        embd = F.gelu(self.embd(out))<br>        embd = embd.view(B, T, N_EMBD)<br>        embd += pos_embd<br><br><br>        mask_matx = torch.ones((B, T, T), dtype=torch.bool)<br>        mask_matx = mask_matx.to(device)<br><br>        for enc in self.encoders:<br>            embd = enc(embd, mask_matx).to(device)<br><br>        B, T, E = embd.size()<br>        embd = embd.view(B, T*E)<br>        embd = self.b_norm2(embd)<br>        embd = F.gelu(self.proj(embd))<br><br>        return embd</pre><p>Now we can train our model</p><pre>device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br>device<br><br>batch_size = 50<br>eval_batch_size = 20<br><br>n_iter = 2000<br>eval_inter = 10<br>n_eval = 20<br><br>lr_ = 3E-4<br>best_loss = None<br># --&gt; Learning Rate Scheduling<br>lr_schedule_patience = 10<br>lr_schedule_factor = 0.5<br>lr_schedule_count = 0<br>lr_schedule_loss_threshold = 0.01<br>lr_schedule = lambda lr, factor: factor*lr<br># -- &gt; Early Stopping<br>es_patience = 10<br>es_count = 0<br>es_trigger_patience = 3<br>es_trigger_count = 0<br>es_threshold = 0.01<br><br>history = {<br>    &quot;train_loss&quot;: [],<br>    &quot;val_loss&quot;: [],<br>    &quot;lr&quot;: [],<br>    &quot;iter&quot;: []<br>}<br><br>text_encoder = BERT().to(device)<br>image_encoder = CNN_Encoder().to(device)<br>criterion = nn.CrossEntropyLoss()<br>optimizer = optim.Adam(list(text_encoder.parameters()) + list(image_encoder.parameters()), lr=lr_)<br><br>send_batch_to_device = lambda batch: [variable.to(device) for variable in batch]<br><br>for i in tqdm(range(n_iter), total=n_iter, desc=&quot;Training&quot;):<br><br>    text_encoder.train()<br>    image_encoder.train()<br>    <br>    batch = get_batch(X_train, train_captions, y_train, batch_size)<br>    image, label, text, masked_tokens, masked_pos = send_batch_to_device(batch)<br>    <br>    seg = torch.zeros((batch_size, MAX_LEN+1), dtype=torch.long).to(device)<br>    pos = torch.range(0, MAX_LEN, dtype=torch.long).to(device)<br>    pos = pos.unsqueeze(0).expand((batch_size, -1))<br>    <br>    image_enc = image_encoder(image)<br>    text_enc, logits_mlm = text_encoder(text, seg, pos, masked_pos)<br>    <br>    optimizer.zero_grad()<br>    logits = image_enc @ text_enc.T <br>    labels_N = torch.arange(logits.size(0)).to(device)<br>    loss_I = criterion(logits.transpose(-2,-1), labels_N)<br>    loss_T = criterion(logits, labels_N)<br>    loss_IT = (loss_I + loss_T)/2<br>    <br>    mlm_loss = criterion(logits_mlm.transpose(1, 2), masked_tokens)<br>    loss = mlm_loss + loss_IT<br>    <br>    loss.backward()<br>    optimizer.step()<br><br>    if i==0 or (i % eval_inter)==0 or i==(n_iter -  1):<br><br>        text_encoder.eval()<br>        image_encoder.eval()<br><br>        train_loss = torch.zeros((n_eval,))<br>        for j in range(n_eval):<br>            batch = get_batch(X_train, train_captions, y_train, eval_batch_size)<br>            image, label, text, masked_tokens, masked_pos = send_batch_to_device(batch)<br>    <br>            seg = torch.zeros((eval_batch_size, MAX_LEN+1), dtype=torch.long).to(device)<br>            pos = torch.range(0, MAX_LEN, dtype=torch.long).to(device)<br>            pos = pos.unsqueeze(0).expand((eval_batch_size, -1))<br><br>            image_enc = image_encoder(image)<br>            text_enc, logits_mlm = text_encoder(text, seg, pos, masked_pos)<br><br>            optimizer.zero_grad()<br>            logits = image_enc @ text_enc.T <br>            labels_N = torch.arange(logits.size(0)).to(device)<br>            loss_I = criterion(logits.transpose(-2,-1), labels_N)<br>            loss_T = criterion(logits, labels_N)<br>            loss_IT = (loss_I + loss_T)/2<br>            <br>            mlm_loss = criterion(logits_mlm.transpose(1, 2), masked_tokens)<br>            loss = mlm_loss + loss_IT<br>            train_loss[j] = loss.item()<br><br>        test_loss = torch.zeros((n_eval,))<br>        for j in range(n_eval):<br>            batch = get_batch(X_test, test_captions, y_test, eval_batch_size)<br>            image, label, text, masked_tokens, masked_pos = send_batch_to_device(batch)<br>    <br>            seg = torch.zeros((eval_batch_size, MAX_LEN+1), dtype=torch.long).to(device)<br>            pos = torch.range(0, MAX_LEN, dtype=torch.long).to(device)<br>            pos = pos.unsqueeze(0).expand((eval_batch_size, -1))<br><br>            image_enc = image_encoder(image)<br>            text_enc, logits_mlm = text_encoder(text, seg, pos, masked_pos)<br><br>            optimizer.zero_grad()<br>            logits = image_enc @ text_enc.T <br>            labels_N = torch.arange(logits.size(0)).to(device)<br>            loss_I = criterion(logits.transpose(-2,-1), labels_N)<br>            loss_T = criterion(logits, labels_N)<br>            loss_IT = (loss_I + loss_T)/2<br>            <br>            mlm_loss = criterion(logits_mlm.transpose(1, 2), masked_tokens)<br>            loss = mlm_loss + loss_IT<br>            test_loss[j] = loss.item()<br><br>    history[&quot;train_loss&quot;].append(train_loss.mean())<br>    history[&quot;val_loss&quot;].append(test_loss.mean())<br>    history[&quot;lr&quot;].append(lr_)<br>    history[&quot;iter&quot;].append(i)<br>    print(f&quot;{i/eval_inter}/{n_iter/eval_inter}: lr:{lr_:.2E} | train_loss: {train_loss.mean():.2f} | val_loss:{test_loss.mean():.2f}&quot;)<br><br>    # -&gt; Learning Rate Scheduling<br>    if (best_loss is None) or test_loss.mean() &lt; best_loss:<br>        lr_schedule_count = 0<br>        best_loss = test_loss.mean()<br>        encoders = {<br>            &quot;text_encoder&quot;: text_encoder.state_dict(),<br>            &quot;image_encoder&quot;: image_encoder.state_dict()<br>        }<br>        joblib.dump(encoders, &quot;encoders.sav&quot;)<br>        print(f&quot;MODEL CHECKPOINT, Loss: {best_loss}&quot;)<br>        time.sleep(0.1)<br>    elif test_loss.mean() &gt; best_loss + lr_schedule_loss_threshold:<br>        lr_schedule_count += 1<br><br>    if lr_schedule_count &gt;= lr_schedule_patience:<br>        lr_schedule_count = 0<br>        lr_ = lr_schedule(lr_, lr_schedule_factor)<br>        for param_group in optimizer.param_groups:<br>            param_group[&quot;lr&quot;] = lr_<br>    <br>    clear_output(wait=True)<br>    time.sleep(0.025)</pre><p>Below is our loss curve after training the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*Z7CSrmxnfQlnnTPtHelsZQ.png" /></figure><p>Now, let us see the pair matching accuracy of the CLIP model, testing with different batch sizes. We give the code first:</p><pre>image_encoder.load_state_dict(encoders[&quot;image_encoder&quot;])<br>text_encoder.load_state_dict(encoders[&quot;text_encoder&quot;])<br><br>image_encoder.eval()<br>text_encoder.eval()<br><br>send_batch_to_device = lambda batch: [variable.to(device) for variable in batch]<br><br>batch_size_acc = {<br>    &quot;iter&quot;: [],<br>    &quot;acc&quot;: []<br>}<br><br>start, end, step = (0, 1000, 1)<br>batch_size = 100<br>acc = 0<br><br>for i in list(range(start, end, step)):<br>    batch = get_batch(X_test, test_captions, y_test, batch_size)<br>    image, label, text, masked_tokens, masked_pos = send_batch_to_device(batch)<br>    <br>    with torch.no_grad():<br>    <br>        seg = torch.zeros((batch_size, MAX_LEN+1), dtype=torch.long).to(device)<br>        pos = torch.range(0, MAX_LEN, dtype=torch.long).to(device)<br>        pos = pos.unsqueeze(0).expand((batch_size, -1))<br>        text_features, mlm_loss = text_encoder(text, seg, pos, masked_pos)<br>        image_features = image_encoder(image)<br>    <br>        text_features = text_features / torch.norm(text_features, dim=-1, keepdim=True)<br>        image_features = text_features / torch.norm(image_features, dim=-1, keepdim=True)<br>    <br>        similarity = image_features @ text_features.T<br>        similarity = similarity.softmax(dim=-1)<br>        max_vals, max_indices = torch.max(similarity, dim=1)<br>    <br>        labels_N = torch.arange(similarity.size(0)).to(device)<br>        pair_match_acc = (torch.sum(max_indices==labels_N) / batch_size).item()<br>        acc += pair_match_acc<br><br>    batch_size_acc[&quot;iter&quot;].append(i)<br>    batch_size_acc[&quot;acc&quot;].append(pair_match_acc)<br>    print(f&quot;\n {(i+1)/end} | Batch Size:{batch_size} | Pair Matching Accuracy: {pair_match_acc}&quot;)<br>    clear_output(wait=True)<br>    time.sleep(0.025)<br><br>acc /= end<br>print(f&quot;Test Accuracy: {acc}&quot;)</pre><p>With this we have a pair matching accuracy (PMA) of 84.6%, pretty good. Now the plot:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*p5E4pbLw3ii0ymblcuOLQw.png" /></figure><p>Ideally, we would an approximately flat line of high value, on the y-axis, but an accuracy deviation of plus or minus 5 is no bad. To conclude, our implementation of the CLIP model using a BERT-based text encoder and a CNN with transformer layers for visual encoding demonstrates promising performance, achieving a pair matching accuracy of 84.6%. We could still improve the loss, but the results validate the approach, also, the CLIP model could be further fine-tuned to enhance alignment between textual and visual representations.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dce66ae18fe1" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rectlabs/clip-contrastive-language-image-pre-training-dce66ae18fe1">CLIP — Contrastive Language-Image Pre-training</a> was originally published in <a href="https://medium.com/rectlabs">Rectlabs Inc</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Contrastive Learning]]></title>
            <link>https://medium.com/rectlabs/contrastive-learning-c2c557865402?source=rss-8e9075c1d169------2</link>
            <guid isPermaLink="false">https://medium.com/p/c2c557865402</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[representation-learning]]></category>
            <category><![CDATA[learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[contrastive-learning]]></category>
            <dc:creator><![CDATA[Emmanuel Olateju]]></dc:creator>
            <pubDate>Fri, 24 Jan 2025 10:02:37 GMT</pubDate>
            <atom:updated>2025-01-24T10:02:37.686Z</atom:updated>
            <content:encoded><![CDATA[<h4>Learning Representations</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*69sHZIFnKYWOiqbvxOej1w.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@heijnsbroek_abstract_art?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Fons Heijnsbroek</a> on <a href="https://unsplash.com/photos/a-painting-of-a-hand-holding-a-blue-and-yellow-object-8vfknrtDrPU?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><h4>Embedding Spaces &amp; Representations</h4><p>If you’re reading about contrastive learning, you likely already know about embedding spaces. But here’s a quick recap: embedding spaces transform raw data (features) into structured, high-dimensional representations, where similar items are closer together, and dissimilar items are farther apart. Traditionally, supervised learning could be used to group labels to structure this space, but when group membership isn’t clear or discrete, things get tricky. This is where contrastive learning shines — encouraging similar items (positive pairs) to have higher similarity than dissimilar ones (negative pairs). Plus, it works in both supervised and unsupervised settings.</p><h4>Contrastive Learning</h4><p><strong>Contrastive learning</strong> is all about learning meaningful data representations by comparing <strong>positive pairs</strong> (similar samples) with <strong>negative pairs</strong> (dissimilar samples). The key idea is to map data into a <strong>feature space</strong> where semantically similar points are closer together, while dissimilar points are pushed farther apart.</p><p>This approach — bringing similar points closer in the embedding space and pushing dissimilar ones apart — has driven major breakthroughs in fields like <strong>computer vision</strong>, <strong>natural language processing</strong>, and beyond.</p><h4>Applications of Contrastive Learning</h4><p><strong>1. Computer Vision:</strong><br>Contrastive learning has revolutionized tasks like <strong>image classification</strong>, <strong>object detection</strong>, and <strong>image retrieval</strong> by learning robust visual representations.</p><p><strong>2. Natural Language Processing:</strong><br>In NLP, contrastive methods are foundational for models like <strong>Sentence-BERT</strong>, enabling tasks such as <strong>semantic text similarity</strong> and <strong>text clustering</strong>.</p><p><strong>3. Multimodal Learning:</strong><br>By aligning representations across different modalities, contrastive learning has paved the way for innovations in areas like <strong>text-visual understanding</strong>, <strong>audio-visual understanding</strong>, <strong>image-captioning</strong>, and even <strong>cross-lingual translation</strong>.</p><p><strong>4. Recommendation Systems:</strong><br>Contrastive learning helps capture user preferences by contrasting <strong>positive</strong> and <strong>negative interactions</strong>, leading to more accurate and personalized recommendations.</p><h3>Generic Loss Function for Contrastive Learning</h3><p>At the core of <strong>contrastive learning</strong> is the <strong>contrastive loss function</strong>. This function drives the learning process by encouraging <strong>positive pairs</strong> (related samples) to have higher similarity compared to <strong>negative pairs</strong> (unrelated samples) in the embedding space.</p><p>A commonly used formulation of this loss is based on a similarity metric, such as <strong>cosine similarity</strong>, and can be expressed as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*B7mtkSpqrW7UeNcmOe133g.png" /></figure><p>Let’s break down this <strong>loss function</strong>:</p><ul><li><strong>Numerator:</strong> It computes the exponential of the similarity between the embedding of a sample and its positive pair, divided by a temperature parameter (τ). This temperature controls how sharp or smooth the similarity distribution is.</li><li><strong>Denominator:</strong> It computes the sum of the exponentials of the similarities between the embedding of the sample and all other samples (both positives and negatives), also divided by τ.</li></ul><h4>What does this mean?</h4><p>The goal is to make the similarity between a sample and its <strong>positive pair</strong> as high as possible while minimizing the similarity with <strong>all other samples</strong>. This helps contrastive learning effectively separate items in the embedding space.</p><p>In contrastive learning, <strong>positive pairs</strong> are often generated using <strong>data augmentation</strong> techniques. For example, let’s say we have ten distinct items. The objective is to separate these ten items in the embedding space so that embeddings for the same item are closer together, while embeddings for different items are farther apart.</p><h4>The Softmax:</h4><p>Let’s focus on the arguments inside the logarithm — the ratio of the exponential of a value to the sum of exponentials. This is essentially the implementation of the <strong>softmax activation function</strong>. Here’s the process we follow for each item:</p><ol><li>Generate a <strong>positive example</strong> using data augmentation.</li><li>Compute the similarity between the item and its positive example.</li><li>Compute the similarity between the item and the other nine items.</li></ol><p>Next, we form a <strong>similarity matrix</strong>. Here’s how:</p><ul><li>For each item, create a <strong>similarity vector</strong> by stacking the similarity of the item to its positive example and its similarity with other items.</li><li>Stack the similarity vectors for all items to form a matrix.</li></ul><p>In this matrix, the similarity between an item and its positive example always appears on the <strong>diagonal </strong>(well, the diagonal rule, I do not think it is a rule of thumb). The similarity matrix looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/1*LJimNERP0RiPgx42zu-BIA.png" /></figure><p>By now, we’re probably familiar with the <strong>softmax activation function</strong> and can start to see the bigger picture. Softmax helps convert the similarity matrix (the similarity vector for each row/item) into a <strong>probability distribution</strong>, ensuring that the sum of probabilities in each row is equal to 1, and all probabilities are non-zero.</p><p>This means that for each row (or item), we get the probability of obtaining an embedding closer to its <strong>positive example</strong> or other items. Once we apply the exponential function, we arrive at the <strong>probability matrix</strong> shown below.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/241/1*aKNSqp80WeeaGx1KUPZm_A.png" /></figure><p>But why the log and the negative sign in the <strong>contrastive loss function</strong>?</p><p><strong>Logarithm &amp; Negative Sign:</strong></p><p>For each item, we calculate the ratio of the probability that its representation is similar to its positive sample compared to the sum of probabilities for all other items (including its positive sample).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/167/1*xQJJss1FPsSs2sBM93TSbA.png" /></figure><p>Now, let’s see what happens when we apply the logarithm to this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/231/1*xc1TTCv3bjlNyWHWWyUbXQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/329/1*gd5n13zwxuWli1tfvaftLw.png" /></figure><p>Not the following: each probability is close-bounded between 0 and 1, and the upper limit of the logarithm of the probabilities is bounded by 0, while the lower is open to large negative values.</p><p>Let’s break down the equation with two scenarios:</p><ol><li><strong>High probability:</strong> If the probability of the representation being similar to the positive sample is high (close to 1), the logarithm gives a value close to 0.</li><li><strong>Low probability:</strong> If the probability is lower, the logarithm gives a large negative value (less than 0).</li></ol><p>But here’s the catch: this is the <strong>loss</strong>, and we want the opposite. The loss should be <strong>larger</strong> when the probability of the positive sample representation is <strong>smaller</strong>, and <strong>smaller</strong> when the probability is <strong>higher</strong>. To make this happen, we use a <strong>negative sign</strong> in the contrastive loss formula, which brings us back to the contrastive loss equation shown below. Additionally, note that the second logarithm (the log of the sum of all probabilities) will always evaluate to <strong>zero</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/247/1*teToQJFjq9HCwgFEbhGMmA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/175/1*3zGJA9ikFKjf7Jx-y4fZ2w.png" /></figure><p>Repeating this for every item and summing up gives us the total contrastive loss:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/466/1*B7mtkSpqrW7UeNcmOe133g.png" /></figure><p>Now that we understand the beauty of <strong>contrastive learning</strong> and how <strong>contrastive loss</strong> works, we can see why it’s so powerful — it doesn’t even require labels to compute the loss from the neural network’s output. Pretty amazing, right? With this understanding, we can explore one of the most exciting applications of contrastive learning: <strong>text-visual understanding</strong> or <strong>image-captioning</strong>, which was the main motivation behind writing this article. One standout approach in this space is <strong>CLIP</strong> (Contrastive Language-Image Pre-training). Watch out for my next article on CLIP.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c2c557865402" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rectlabs/contrastive-learning-c2c557865402">Contrastive Learning</a> was originally published in <a href="https://medium.com/rectlabs">Rectlabs Inc</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[BERTs]]></title>
            <link>https://medium.com/rectlabs/berts-f5fad6ba2806?source=rss-8e9075c1d169------2</link>
            <guid isPermaLink="false">https://medium.com/p/f5fad6ba2806</guid>
            <dc:creator><![CDATA[Emmanuel Olateju]]></dc:creator>
            <pubDate>Fri, 17 Jan 2025 11:02:01 GMT</pubDate>
            <atom:updated>2025-01-17T11:02:01.030Z</atom:updated>
            <content:encoded><![CDATA[<h4>Bidirectional Encoder Representation From Transformers</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TbpFGEzRkmnH3Zqbe-0COQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@juice_world?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">JUICE</a> on <a href="https://unsplash.com/photos/a-black-screw-with-a-yellow-and-blue-background-oyGlH_K1CNg?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><blockquote>Do you get the pun of the image above? Look left and look right for clarity, never look in one direction — well they had a good career.</blockquote><p>We previously discussed <a href="https://medium.com/rectlabs/transformers-1-819152c2e68e">transformers</a> and their <a href="https://medium.com/rectlabs/transformers-2-c052036adf30">attention mechanisms</a>, including an example of a <a href="https://medium.com/rectlabs/transformers-3-d90a9b3f9a0e">masked self-attention transformer</a> for sequence generation. In that example, future items depended only on past items (left-to-right). However, understanding context in both directions is crucial for tasks beyond sequence generation.</p><p>For instance, in the statements <em>“The bank will likely liquidate its assets” </em>and <em>“The bank will likely be water-logged this month,”</em> left-to-right modeling struggles to distinguish whether “bank” refers to a financial institution or a riverbank, as it lacks right-to-left context. This highlights the need for BERT, which uses bidirectional representations (left-to-right, and right-to-left attention) to capture context in both directions. BERT outperforms existing models in sentence-level and token-level tasks and is widely adopted due to its ability to be pre-trained on unlabeled text and fine-tuned for various NLP tasks.</p><h4>Advantages of BERTs</h4><p>BERT models excel at learning bidirectional representations through the use of bidirectional self-attention. Unlike GPT models, which rely on unidirectional attention, BERT enables tokens in a sequence to attend to context from both directions (left-to-right and right-to-left) simultaneously. This bidirectional approach provides a more comprehensive understanding of the sequence’s context.</p><p>BERTs hold significant advantages over uni-directional transformer models like GPTs. Uni-directional attention restricts flexibility of architecuture during pre-training limiting adaptability to various tasks. Also, fine-tuned models based on pre-trained BERTs require minimal architectural changes as the base models are usually the same with limited fine-tuning for downstream tasks, making them efficient for a wide range of applications. Additionally, BERT input representation supports both single sentences and sentence pairs, with segment embeddings distinguishing sentence to which words belong and sentence relationships. This design allows BERT to be effectively pre-trained for diverse downstream tasks.</p><p>Bidirectional self-attention gives the model access to all elements of a sequence, including the item being predicted, which could lead to the item influencing its own prediction through its input representation, or strengthening weights which effectively route the item to the output in a multilayered context. To prevent this, BERT uses Masked Language Modeling (MLM), a technique that masks certain tokens in the input to be predicted, ensuring the model learns relationships between items without direct exposure to the predicted item. Let’s explore MLM next.</p><h4>Masked Language Modelling (MLM)</h4><p>What is masked language modelling (MLM). In our example statements, let us replace <em>“liquidate” </em>and the <em>“water-logged” </em>tokens with special tokens with the goal of predicting these tokens given bidirectional context: <em>“The bank will likely [MASK] its assets” </em>and <em>“The bank will likely be [MASK] this month.” </em>MLM requires us to learn to predict the masked tokens from these masked versions of the input.</p><p>In a GPT utilizing left-to-right attention for context, in both examples, the transformer will predict a similar token as it would be predicting the next token from : <em>“The bank will likely”</em> and <em>“The bank will likely be”</em>, it does not have a way to see that bank is used in two different contexts but a BERT does.</p><p>Using the masked examples in the paragraph above, and integrating both left-to-right and right-to-left attention, BERTs are able to see the words <em>“assets”</em> and <em>“this month” </em>in these examples, and that is enough to let the BERT know the two sentences imply different contexts. Not that <em>“this month”</em> intuitively says: “<em>hey we are referring to a river bank here”</em>, but knowing the sentences refer to different contexts, the BERT optimizes itself for more appropriate words to replace the [MASK] special token in each example.</p><p>A BERT pre-trained this way can then be fine-tuned for a task such as sentence generation, but can this be used for other downstream tasks which would utilize cross-attention in understanding relationship between two different sequences, such as question answering. For this reason, we also pre-train the BERT on another task simultaneously. This task is called Next Sentence Prediction (NSP). But first let us understand how BERT is able to represent both single sentences and sentence pairs.</p><h4>Input Representation &amp; Next Sentence Prediction (NSP)</h4><p>We will describe this quickly using the image from the <a href="https://arxiv.org/pdf/1810.04805">paper</a>. We should not spend so much time on this, it is a pretty straight forward concept.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/926/1*HHcbUiQFblsANnx_RPvaTA.png" /><figcaption><a href="https://arxiv.org/pdf/1810.04805">BERT: Pre-training of Deep Bidirectional Transformers for<br>Language Understanding</a></figcaption></figure><p>In the diagram above, we see there are two sentences: <em>“my dog is cute” </em>and <em>“he likes playing”</em>, we will not bother about separating playing into two tokens. Now we notice that a special token [CLS] for classification is added at the beginning of the first sentence, then a [SEP] special token separates the two sentences, and is added at the end of the second sentence forming one input sequence. While we mask certain tokens which are not special tokens employing the MLM objective, we would also like to optimize the BERT network to predict if the second sentence is a follow up sentence after the first sentence. This is the purpose of the classification token ([CLS]), the resulting vector representation for this token from the BERT is used in predicting if the next sentence after the first [SEP] token is a follow up to the sentence before the first [SEP] token.</p><p>From our <a href="https://medium.com/rectlabs/transformers-3-d90a9b3f9a0e">GPT implementation</a>, we understand token embeddings and position embeddings, but what do segment embeddings do? On forming the input sequence, we have no way of knowing what sentence each token comes from despite the position embeddings, thus the segment embedding adds unique identification tags which tells us to what sentence each token belongs to. While implementing, we employ a third segment embedding C apart from that of the two sentences A and B. The third segment embedding is used for [PAD] tokens in cases of the input sequence not meeting up with the predefined input length of the BERT model.</p><h4>Considerations For Fine-Tuning</h4><p>As a rule of thumb, for every input sequence consisting of a single sentence or a sentence pair, we select randomly 15% of the input sequence excluding special tokens to be masked for the MLM objective. When forming the sentence pairs, for NSP, we select 50% of sentence pairs in which the sentence from the second segment is the next sentence after that from the first segment, and the remaining 50% we select random sentences for the second segment.</p><p>Masking 15% of the input sequence works for pre-training MLM objective, but we are creating a mismatch for the fine-tuning phase after pre-training in that when fine-tuning, we are no longer utilizing the MLM objective and masked tokens would be absent. We compensate for this by replacing the 15% tokens with the [MASK] token 80% of the time, a random token 10% of the time and the actual target token itself 10% of the time.</p><p>As simple as it seems, we have all the ingredients for pre-training a BERT, so far as you understand the transformer architecture and attention, if not go through these write-ups:<a href="https://medium.com/rectlabs/transformers-1-819152c2e68e">transformers</a> , <a href="https://medium.com/rectlabs/transformers-2-c052036adf30">attention mechanisms</a>, and <a href="https://medium.com/rectlabs/transformers-3-d90a9b3f9a0e">masked self-attention transformer</a>. Let us craft a BERT recipe, and then pre-train a simple BERT we implement ourselves.</p><h4>Recipe For BERTs</h4><p>Given your dataset of sentences where the order of sentences is preserved:</p><ol><li>Select 50% of sentence pairs where, the second sentence in the pair comes after the first sentence in the pair, and another 50% where the second sentence in the pair is a random sentence not coming after the first.</li><li>For every sentence pair, merge as follows: add the [CLS] token followed by the first sentence followed by a [SEP] token, then the second sentence, followed by another [SEP] token, forming the input sequences.</li><li>Truncate or pad the input sequences accordingly.</li><li>Select randomly 15% of tokens from each input sequence, excluding special tokens for masking. You can select the [PAD] tokens for masking for programming sake, they will eventually be masked to zero by the attention mechanism</li><li>Replace selected tokens with [MASK] 80% of the time, a random token 10% of the time, and the original target token 10% of the time.</li><li>Pass the whole batch of input sequence to the network optimizing it to predict if the sentence pair is a subsequent pair not using the [CLS] representation, and to predict the masked tokens.</li></ol><p>Now let us implement our BERT.</p><h4>BERT Implementation</h4><p>First let us prepare our dataset:</p><pre># @title Dataset Loading, Preprocssing &amp; Vocabulary Definition<br><br># --&gt; LOAD SENTENCES<br><br>if &#39;text&#39; not in globals():<br>  sentences = load_dataset(&quot;cc_news&quot;, split=&quot;train&quot;)[&quot;text&quot;]<br>  clear_output(wait=True)<br>  time.sleep(0.001)<br>  print(&quot;DATASET LOADED&quot;)<br><br>  # --&gt; PREPROCESSING, TOKENIZATION<br>  text = [obj.lower() for obj in sentences]<br>  len_text = len(text)<br>  subset_size = 1E-2<br>  text = text[0:int(subset_size*len_text)]<br>  del sentences<br><br>  for idx, sentence in enumerate(text):<br>    text[idx] = re.split(r&quot;[ .]+&quot;, sentence)<br><br># --&gt; DEFINE VOCABULARY &amp; VOCABULARY SIZE<br>if &#39;VOCAB&#39; not in globals():<br>  VOCAB = []  # --&gt; VOCABULARY<br>  for s_idx , sentence in tqdm(enumerate(text), total=len(text), desc=&quot;Compiling Vocabulary&quot;):<br>        VOCAB += sentence<br>  VOCAB = list(set(VOCAB))<br><br># --&gt; FORM TOKEN EMBEDDING LOOK UP TABLE (LUT)<br>SPECIAL_TOKENS = [<br>    &quot;[PAD]&quot;, &quot;[UNK]&quot;, &quot;[CLS]&quot;, &quot;[MASK]&quot;, &quot;[SEP]&quot;, &quot;&lt;S&gt;&quot;, &quot;&lt;T&gt;&quot;<br>]<br>LST = len(SPECIAL_TOKENS) # --&gt; NO OF SPECIAL TOKENS<br>wtoi = {} # --&gt; WORD TO TOKEN EMBEDDING LUT<br>itow = {} # --&gt; TOKEN EMBEDDING TO WORD LUT<br>for idx, st in enumerate(SPECIAL_TOKENS): # --&gt; APPEND SPECIAL TOKESN TO LUTs<br>  wtoi[st] = idx<br>  itow[idx] = st<br>for idx, word in enumerate(VOCAB):  # --&gt; APPEND WORDS IN VOCABULARY TO LUTs<br>  wtoi[word] = idx + LST<br>  itow[idx + LST] = word<br><br>VOCAB = SPECIAL_TOKENS + VOCAB<br>VOCAB_SIZE = len(VOCAB) # --&gt; VOCABULARY SIZE<br><br>dataset = []<br>for sentence in text:<br>  dataset.append([wtoi[word] for word in sentence])<br><br>min_len_sentences = math.inf<br>max_len_sentences = 0<br>for sentence in dataset:<br>  min_len_sentences = min(min_len_sentences, len(sentence))<br>  max_len_sentences = max(max_len_sentences, len(sentence))<br><br>MAX_LEN = int(0.1*max_len_sentences)<br>del min_len_sentences, max_len_sentences</pre><p>We can then implement the recipe for creating a batch of input sequences:</p><pre># @title Make Dataloader<br><br>def get_batch(sentences, batch_size=10):<br><br>  assert (batch_size % 2) == 0<br>  datasize = len(sentences)<br>  batch = []<br><br>  no_pve, no_nve = 0, 0<br>  while no_pve != (batch_size / 2) or no_nve != (batch_size / 2):<br><br>    a_idx, b_idx = random.randint(0, datasize-1), random.randrange(0, datasize-1)<br>    segment_a, segment_b = sentences[a_idx], sentences[b_idx]<br>    input_sequence = [wtoi[&#39;[CLS]&#39;]] + segment_a + [wtoi[&#39;[SEP]&#39;]] + segment_b + [wtoi[&#39;[SEP]&#39;]]<br>    segment_embeddings = [0]*(1 + len(segment_a) + 1) + [1]*(len(segment_b) + 1)<br>    pos_embeddings = [idx for idx, _ in enumerate(input_sequence)]<br>    # TRUNCATE SEQUENCE TO MAXIMUM INPUT SEQUENCE LENGTH IF GREATER THAN<br>    if len(input_sequence) &gt; MAX_LEN:<br>      input_sequence = input_sequence[:MAX_LEN]<br>      segment_embeddings = segment_embeddings[:MAX_LEN]<br>      pos_embeddings = [idx for idx, _ in enumerate(input_sequence)]<br><br>    # --&gt; PAD OR SEQUENCE TO MAXIMUM INPUT SEQUENCE LENGTH<br>    if len(input_sequence) &lt; MAX_LEN:<br>      n_pad = MAX_LEN - len(input_sequence)<br>      input_sequence += [wtoi[&#39;[PAD]&#39;]] * n_pad<br>      segment_embeddings += [2] * n_pad<br>      pos_embeddings = [idx for idx, _ in enumerate(input_sequence)]<br><br>    # --&gt; 15% MASKING of INPUT SEQUENCE<br>      # --&gt; GET TOKENS TO MASK<br>    mask_pos = [idx for idx, token in enumerate(input_sequence) if (token != wtoi[&#39;[CLS]&#39;]) and <br>                (token != wtoi[&#39;[SEP]&#39;]) and (token != wtoi[&#39;[PAD]&#39;])]<br>    mask_pos = random.sample(mask_pos, int(0.15 * len(mask_pos))) # --&gt; INDEX OF TOKENS TO MASK<br>      # --&gt; OUT OF 15% TOKENS TO BE MASKED, MASK 80% OF IT, REPLACE 10 % WITH RANDOM TOKENS &amp; LEAVE 10% AS SAME<br>    masked_tokens, masked_pos = [], []  # --&gt; THE MASKED TOKENS AND THEIR POSITION IN THE INPUT SEQUENCE<br>    for pos in mask_pos:<br>      masked_tokens.append(input_sequence[pos])<br>      masked_pos.append(pos)<br>      if random.random()&lt;0.1: # --&gt; REPLACE WITH RANDOM TOKEN<br>        idx = random.randint(0, VOCAB_SIZE-1)<br>        input_sequence[pos] = wtoi[itow[idx]]<br>      elif random.random()&lt;0.8: # --&gt; REPLACE WITH [MASK] TOKEN<br>        input_sequence[pos] = wtoi[&#39;[MASK]&#39;]<br>      else: # --&gt; LEAVE WITHOUT CHANGE<br>        pass<br><br>    assert len(input_sequence) == MAX_LEN<br>    assert len(segment_embeddings) == MAX_LEN<br>    assert len(pos_embeddings) == MAX_LEN<br>    assert len(masked_tokens) == len(mask_pos)<br>    assert len(masked_pos) == len(mask_pos)<br><br>    # --&gt; CHECK FOR 50% IsNext &amp; 50% NotNext<br>    if (a_idx + 1) == b_idx and no_pve &lt; (batch_size/2):<br>      batch.append([input_sequence, segment_embeddings, pos_embeddings,<br>          masked_tokens, masked_pos, True<br>      ])<br>      no_pve += 1<br>    elif (a_idx + 1) != b_idx and no_nve &lt; (batch_size/2):<br>      batch.append([input_sequence, segment_embeddings, pos_embeddings,<br>          masked_tokens, masked_pos, False<br>      ])<br>      no_nve += 1<br><br>  return batch</pre><p>From the implementation above, we see that once an input sequence is longer than or shorter the predefined MAX_LEN of input to the BERT which we defined when preparing our dataset, we truncate or pad using the [PAD] token respectively.</p><p>Now we do not want these [PAD] tokens to be attended to in the bidirectional attention, so we mask them when computing our attention score, we do this using the function below. Please at this point, review the write-ups on transformers if you do not understand the code below, specifically this: <a href="https://medium.com/rectlabs/transformers-3-d90a9b3f9a0e">masked self-attention transformer</a>.</p><pre>def get_attn_mask(x):<br>  B, T = x.size()<br>  attn_mask = x.eq(wtoi[&#39;[PAD]&#39;])<br>  attn_mask = attn_mask.unsqueeze(1)<br>  attn_mask = attn_mask.expand(B, T, T)<br>  return attn_mask</pre><p>Now let us implement our BERT model, the underlying implementation is pretty much the same as our GPT implementation except the attention masking we use above, the MLM and NSP implementation. Review <a href="https://medium.com/rectlabs/transformers-3-d90a9b3f9a0e">this</a> as a reminder of implementing a GPT.</p><pre># @title Make BERT<br><br>class InputEmbedding(nn.Module):<br><br>  def __init__(self):<br>    super(InputEmbedding, self).__init__()<br>    self.tok_embd = nn.Embedding(VOCAB_SIZE, N_EMBD)<br>    self.segment_embd = nn.Embedding(N_SEGMENTS, N_EMBD)<br>    self.pos_embd = nn.Embedding(MAX_LEN, N_EMBD)<br>    self.norm = nn.LayerNorm(N_EMBD)<br><br>  def forward(self, x, seg, pos):<br>    B, T = x.size()<br>    tok_embd = self.tok_embd(x)<br>    seg_embd = self.segment_embd(seg)<br>    pos_embd = self.pos_embd(pos)<br>    embd = tok_embd +seg_embd + pos_embd<br>    return self.norm(embd)<br><br>class Head(nn.Module):<br><br>  def __init__(self):<br>    super(Head, self).__init__()<br>    self.dropout = nn.Dropout(0.2)<br>    self.query = nn.Linear(N_EMBD, HEAD_SIZE)<br>    self.key = nn.Linear(N_EMBD, HEAD_SIZE)<br>    self.value = nn.Linear(N_EMBD, HEAD_SIZE)<br><br>  def forward(self, x, attn_mask):<br>    x = self.dropout(x)<br>    Q = self.query(x)<br>    K = self.key(x)<br>    V = self.value(x)<br><br>    scores = (Q @ K.transpose(-1, -2)) / HEAD_SIZE**0.5<br>    scores.masked_fill_(attn_mask, torch.iinfo(torch.long).min)<br>    attn = nn.Softmax(dim=-1)(scores)<br>    out = attn @ V<br><br>    return out<br><br>class MultiHeadAttention(nn.Module):<br><br>  def __init__(self):<br>    super(MultiHeadAttention, self).__init__()<br>    self.dropout = nn.Dropout(0.35)<br>    self.heads = nn.ModuleList([Head() for h in range(ATTN_HEADS)])<br>    self.proj = nn.Linear(ATTN_HEADS * HEAD_SIZE, N_EMBD)<br>    self.norm = nn.LayerNorm(N_EMBD)<br><br>  def forward(self, x, attn_mask):<br>    out = self.dropout(x)<br>    out = torch.cat([h(x, attn_mask) for h in self.heads], dim=-1)<br>    out = self.proj(out)<br>    out = F.gelu(out)<br>    out = self.norm(x + out)<br>    return out<br><br>class EncoderLayer(nn.Module):<br><br>  def __init__(self):<br>    super(EncoderLayer, self).__init__()<br>    self.sa = MultiHeadAttention()<br>    self.fc1 = nn.Linear(N_EMBD, 2*N_EMBD)<br>    self.fc2 = nn.Linear(2*N_EMBD, N_EMBD)<br>    self.norm1 = nn.LayerNorm(N_EMBD)<br>    self.norm2 = nn.LayerNorm(N_EMBD)<br>  <br>  def forward(self, x, attn_mask):<br>    temp = self.norm1(x + self.sa(x, attn_mask))<br>    out = F.gelu(self.fc1(temp))<br>    out = F.gelu(self.fc2(out))<br>    out = self.norm2(temp + out)<br>    return out<br><br>class BERT(nn.Module):<br><br>  def __init__(self):<br>    super(BERT, self).__init__()<br>    self.embedding = InputEmbedding()<br>    self.encoders = nn.ModuleList([EncoderLayer() for _ in range(ATTN_BLOCKS)])<br><br>    self.nsp_fc = nn.Linear(N_EMBD, N_EMBD)<br>    self.nsp_classifier = nn.Linear(N_EMBD, 2)<br>    self.mlm_fc = nn.Linear(N_EMBD, N_EMBD)<br>    self.mlm = nn.Linear(N_EMBD, VOCAB_SIZE)<br><br>  def forward(self, x, seg, pos, masked_pos):<br>    mask = get_attn_mask(x)<br>    embd = self.embedding(x, seg, pos)<br>    for enc in self.encoders:<br>      embd = enc(embd, mask)<br>    <br>    nsp = F.gelu(self.nsp_fc(embd[:, 0, ::]))<br>    logits_nsp = self.nsp_classifier(nsp)<br><br>    masked_pos = masked_pos.unsqueeze(-1).expand(-1, -1, embd.size(-1)) #--&gt; Expand to 3D Adding N_EMBD as last dimension<br>    mlm = torch.gather(embd, 1, masked_pos)<br>    mlm = F.gelu(self.mlm_fc(mlm))<br>    logits_mlm = self.mlm(mlm)<br><br>    return logits_mlm, logits_nsp</pre><p>We see for the NSP, we only select the first token embedding which is passed to a layer having two outputs, logits of a valid sentence pair or an invalid one. Now let us pre-train our BERT.</p><pre># @title Train<br><br>device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br><br>batch_size = 30<br>eval_batch_size = 10<br><br>n_iter = 1000<br>eval_inter = 10<br>n_eval = 5<br><br>lr_ = 3E-4<br><br>model = BERT().to(device)<br>criterion = nn.CrossEntropyLoss()<br>optimizer = optim.Adam(model.parameters(), lr=lr_)<br><br>history = {<br>    &quot;train_loss&quot;: [],<br>    &quot;val_loss&quot;: [],<br>    &quot;iter&quot;: [],<br>    &quot;lr&quot;: []<br>}<br><br>for i in tqdm(range(n_iter), total=n_iter, desc=&quot;Training&quot;):<br><br>  batch = get_batch(train, batch_size=batch_size)<br>  # print([len(b) for b in batch[0][0:-1]])<br>  sequence, segment, pos,  masked_tokens, masked_pos, IsNext = map(torch.LongTensor, zip(*batch))<br>  sequence = sequence.to(device)<br>  segment = segment.to(device)<br>  pos = pos.to(device)<br>  masked_tokens = masked_tokens.to(device)<br>  masked_pos = masked_pos.to(device)<br>  IsNext = IsNext.to(device)<br><br>  model.train()<br>  logits_mlm, logits_nsp = model(sequence, segment, pos, masked_pos)<br>  optimizer.zero_grad()<br>  mlm_loss = criterion(logits_mlm.transpose(1,2), masked_tokens)<br>  nsp_loss = criterion(logits_nsp, IsNext)<br>  loss = mlm_loss + nsp_loss<br>  loss.backward()<br>  optimizer.step()<br><br>  if (i % eval_inter)==0 or i==(n_iter-1):<br><br>    model.eval()<br><br>    train_loss = torch.zeros((eval_batch_size))<br>    for j in range(eval_batch_size):<br><br>      batch = get_batch(train, batch_size=eval_batch_size)<br>      sequence, segment, pos,  masked_tokens, masked_pos, IsNext = map(torch.LongTensor, zip(*batch))<br>      sequence = sequence.to(device)<br>      segment = segment.to(device)<br>      pos = pos.to(device)<br>      masked_tokens = masked_tokens.to(device)<br>      masked_pos = masked_pos.to(device)<br>      IsNext = IsNext.to(device)<br><br>      logits_mlm, logits_nsp = model(sequence, segment, pos, masked_pos)<br>      mlm_loss = criterion(logits_mlm.transpose(1,2), masked_tokens)<br>      nsp_loss = criterion(logits_nsp, IsNext)<br>      loss = mlm_loss + nsp_loss<br><br>      train_loss[j] = loss.item()<br>    <br>    val_loss = torch.zeros((eval_batch_size,))<br>    for j in range(eval_batch_size):<br><br>      batch = get_batch(val, batch_size=eval_batch_size)<br>      sequence, segment, pos,  masked_tokens, masked_pos, IsNext = map(torch.LongTensor, zip(*batch))<br>      sequence = sequence.to(device)<br>      segment = segment.to(device)<br>      pos = pos.to(device)<br>      masked_tokens = masked_tokens.to(device)<br>      masked_pos = masked_pos.to(device)<br>      IsNext = IsNext.to(device)<br><br>      logits_mlm, logits_nsp = model(sequence, segment, pos, masked_pos)<br>      mlm_loss = criterion(logits_mlm.transpose(1,2), masked_tokens)<br>      nsp_loss = criterion(logits_nsp, IsNext)<br>      loss = mlm_loss + nsp_loss<br><br>      val_loss[j] = loss.item()<br><br>    print(f&quot;{i/eval_inter}/{n_iter/eval_inter} Train Loss: {train_loss.mean():.2f} | Val Loss: {val_loss.mean():.2f}&quot;)<br>    history[&quot;train_loss&quot;].append(train_loss.mean())<br>    history[&quot;val_loss&quot;].append(val_loss.mean())<br>    history[&quot;iter&quot;].append(i)</pre><p>We can see the MLM objective loss is computed between the logits of the vocabulary for each masked position and the actual masked tokens of those positions. That for the NSP is computed between the representation of the first token, [CLS] and the labels of sentence pairs being valid or not. Below we show our loss curve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/729/1*KPoPZiHvAIrJxZcIgevfIg.png" /></figure><p>We have pre-trained our BERT model on a subset from a larger document-corpus. How about fine-tuning? We will do that in the second part of this article in which we make inference with out pre-trained BERT, then fine-tune it on a larger version of the same dataset we pre-trained on on a specific task. We also compare the performance of BERT and GPT on that task</p><p>While GPTs mostly employ unidirectional attention, BERTs are able to provide better performance due to bidirectional attention conditioning, masked learning modelling objective and next sentence prediction. See you in the next article, Tschüss.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f5fad6ba2806" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rectlabs/berts-f5fad6ba2806">BERTs</a> was originally published in <a href="https://medium.com/rectlabs">Rectlabs Inc</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Variational AutoEncoders — Image Reconstruction/Generation]]></title>
            <link>https://medium.com/rectlabs/variational-autoencoders-image-reconstruction-generation-c74c1b9568e8?source=rss-8e9075c1d169------2</link>
            <guid isPermaLink="false">https://medium.com/p/c74c1b9568e8</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[variational-autoencoder]]></category>
            <category><![CDATA[autoencoder]]></category>
            <category><![CDATA[image-generation]]></category>
            <dc:creator><![CDATA[Emmanuel Olateju]]></dc:creator>
            <pubDate>Mon, 06 Jan 2025 11:17:08 GMT</pubDate>
            <atom:updated>2025-01-06T11:17:08.481Z</atom:updated>
            <content:encoded><![CDATA[<h3>AutoEncoders — Image reconstruction &amp; Generation.</h3><h4>Variational AutoEncoders (VAEs) &amp; Generative Modelling</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tTuxPshZauTRCUhvHDZsfA.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@santesson89?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Andrea De Santis</a> on <a href="https://unsplash.com/photos/black-and-white-robot-toy-on-red-wooden-table-zwd435-ewb4?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>In the previous section of this article, we introduced the concept of using autoencoders in generative modeling, stating the peculiarities of variational autoencoders (VAEs) in modelling probabilistic relationships.</p><p>VAEs are particularly well-suited for generative modelling due to their ability to model the generation of samples of similar distribution to<strong><em> </em></strong>that of some input distribution of samples <strong><em>x. </em></strong>This is done by modelling the relationship between the input distribution called the evidence to another probabilistic distribution <strong><em>p(z) </em></strong>called the prior — usually assumed to be a Gaussian — and making use of this prior in generating a distribution similar to the evidence. In this section, we will delve deeper into how VAEs work and implement an example using the MNIST dataset.</p><h4>The Big Idea of VAES &amp; Generative Modelling</h4><p>Given a dataset <strong><em>X</em></strong> of variables <strong><em>x</em></strong> with some distribution <strong><em>p(x)</em></strong>, we call this distribution the evidence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/129/1*jILtnCuUprCwHF-tmVhaTw.png" /></figure><p>For generative modelling, we sample from the distribution of the latent space — assumed as the prior — pass it to the decoder and generate new samples similar to the input distribution. So we want to learn a probabilistic mapping from a distribution of the latent space <strong><em>p(z)</em></strong>, where <strong><em>z</em></strong> is the latent space representation to one similar to the evidence<strong><em> p(x).</em></strong> But to do this we would need to understand how the evidence distribution maps to the prior, how do we come up with such mapping without actually knowing the latent space distribution? We make an assumption of the latent space distribution being a multi-variate Gaussian, and the probability distribution of its variables being independent (diagonal covariance matrix). We call this assumed latent space distribution the prior. So rather, we try to fit the latent representations of the evidence samples into the prior which is denoted as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/104/1*oLqXDRGl0Au9_sf5tvDsiQ.png" /></figure><p>So when we train a VAE, we are not just learning the latent space (<strong><em>z</em></strong>) vectors for reconstruction, we are actually learning parameters <em>θ</em> of an autoencoder network that fit each sample <strong><em>x</em></strong> and its representation <strong><em>z</em></strong> into the prior multivariate Gaussian through mean (<strong><em>μ</em></strong>) and deviation (<strong><em>σ</em></strong>) values. We are trying to learn these values for a prior, and each latent space variable is now described in terms of these values, where <strong><em>∈</em></strong> is a random small value for sampling a latent vector representation from the prior:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/139/1*c-jlYAIYeDrYK6Hp2FePZA.png" /></figure><p><strong>The Prior Is Not Enough</strong></p><p>We have assumed a prior distribution, and we can model our input distribution which is the evidence. Since the goal is generative modelling, we have to be able to describe the evidence and distributions similar to the evidence in terms of the prior. This can be done through another distribution called the likelihood which tells us the probability of observing a sample <strong><em>x</em></strong> given a latent representation <strong><em>z</em></strong>. The higher this likelihood, the closer the representation and distribution we get to <strong><em>x </em></strong>and<strong><em> p(x) </em></strong>respectively.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/162/1*oJyzma1biRBvKY3QKZk2Ag.png" /></figure><p>So this tells us the probability of observing an<strong><em> x </em></strong>or some representation of similar statistical value to (mean, deviation values)<strong><em> x </em></strong>from the evidence given a particular <strong><em>z</em></strong>. But we want our likelihood to account for other latent representations <strong><em>z </em></strong>and all <strong><em>x </em></strong>in our input distribution, not just one, so we want to know the probability of obtaining all<strong><em> x </em></strong>given all <strong><em>z</em></strong>, this is referred to as the marginal likelihood and is given as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/1*w5ukUrmgPFykv-By-Zc-HA.png" /></figure><p>Let us explain what the marginal likelihood does. Recall our prior is a multivariate Gaussian, but we plot the distribution of just one dimension in the latent space; we also see the various likelihood distribution of<strong><em> x</em></strong> given various dimensions of <strong><em>z</em></strong>, let us see the plot of these two below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/789/1*9dOfgwihttpNhrK8tTVAZg.png" /></figure><p>So for each dimension of <strong><em>z </em></strong>in our latent space, we multiply its distribution from the left graph, and the likelihood for that <strong><em>z</em></strong> from the right graph, then we sum up the result of these products for all the <strong><em>z</em></strong> to get the marginal likelihood. So we need an objective that optimizes the parameters <strong><em>θ</em></strong> of a network to maximize the likelihood of observing our evidence distribution <strong><em>p(x)</em></strong>, or a similar distribution, given our prior. Therefore our likelihood acts as a probabilistic decoder, making an estimate of the input distribution given the prior through the marginal likelihood; the objective would work to reduce the distance between the actual evidence <strong><em>p(x)</em></strong> and the evidence estimate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/221/1*wm_ihiNUrO512iCl0DZSZA.png" /></figure><p><strong><em>Probabilistic Decoder, What About A Probabilistic Encoder?</em></strong></p><p>Remember we are working with an autoencoder, and for all of our discussion so far, we have only described the assumption of the prior as a multivariate Gaussian, the likelihood, and marginal likelihood. We also said that the VAE fits the latent representations of the input distribution <strong><em>p(x) </em></strong>into the assumed prior, but how? It would need some probability distribution that describes the prior given some input <strong><em>x. </em></strong>We call this the posterior; the posterior gives the probability of some latent representation <strong><em>z, </em></strong>given some input <strong><em>x. </em></strong>Now this posterior is to act as a probabilistic encoder.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/156/1*U5z03DwDqTF4Uqad-XYU0Q.png" /></figure><p>But we face a problem with the posterior just as we face with the likelihood, we do not know its nature, is it Gaussian or uniform? We cannot make assumptions about it as we did about the prior already and computing it is difficult. What do we do about this?</p><p><strong>The Approximate Posterior</strong></p><p>When we look at the likelihood, it is also a measure of our ability to reconstruct the evidence from the prior distribution, which is not changing. Rather what changes is how the decoder maps the prior to our evidence estimate so our evidence estimate is as close to the actual evidence as possible, but we need a posterior to map the evidence to the prior first.</p><p>Since we are using an autoencoder, how the network fits each sample from the evidence into the prior can also change allowing us to variationally infer an approximate of the posterior. The encoder part with parameters <strong><em>Φ </em></strong>to be optimized, models the approximate posterior, attempting to make it as close as possible to the true posterior which is difficult to compute. Since the true posterior is difficult to compute, we know our prior which can be expressed in terms of the true posterior marginally. Since our true posterior would result in exactly the prior, the encoder parameters <strong><em>Φ</em></strong> can be optimized to produce a distribution close to the prior by optimizing the approximate posterior, and we can look at how close the approximate posterior is to the prior for each dimension of<strong><em> z — </em></strong>since we do not know the true posterior — as a measure of how well we fit the evidence into the prior.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/274/1*Y623U_nitMlc3iBy12_fgg.png" /></figure><p>Let’s use a graph to demonstrate this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/790/1*H6BnwOu_2WVC-q51X2vOug.png" /></figure><p>On the left, we have various approximate posteriors for various dimensions in the latent space. On the right we have the distribution of any particular dimension in the latent space. For each dimension in the latent space, we can average out the approximate posterior on the left for various <strong><em>x</em></strong>, and compare how similar the result is to the prior on the right as a measure of how well we fit the evidence into the prior for that dimension.</p><p><strong>Putting It All Together</strong></p><p>Now let us make sense of what the VAE does. The VAE given our evidence distribution <strong><em>p(x), </em></strong>assumes a multivariate Gaussian prior which describes the distribution of the latent space of the inputs <strong><em>x</em></strong>. Given the inputs <strong><em>x</em></strong>, the encoder outputs parameters (<strong><em>mean, deviation</em></strong>) of the posterior distribution with which we sample latent representations from the prior, then the decoder attempts to reconstruct the inputs which we refer to as the evidence estimate. So the encoder gives us the approximate posterior, the decoder the likelihood, and we have our prior which we are adjusting our latent representations to fit into.</p><blockquote>So two things are of interest: (1) how well do we reconstruct the inputs given the prior, and (2) how close is the approximate posterior to the real posterior, or alternatively to the prior? Since solving the real posterior is difficult we can focus on how close the resulting distribution from the approximate posterior is to the prior.</blockquote><p>How well we reconstruct the inputs from the sampled latent representations can be measured through the likelihoods. In practice, the mean-squared loss or binary cross-entropy is used between the input and the reconstructed version. A measure of the similarity of the approximate posterior to the true posterior or prior can be done using a formula called the <a href="https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence">Kullback Leibler divergence</a>. Having understood the working of the VAE, we now go on to discuss the measure which serve as the objective or loss functions on which the VAE is trained.</p><h4>The Mathematics — Evidence Lower Bound (ELBO)</h4><p>When we mentioned the marginal likelihood, we also discussed the probabilistic decoder, and the objective being to optimize the parameters <strong><em>θ </em></strong>of the probabilistic decoder (likelihood) to reproduce the input distribution from the prior.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/1*w5ukUrmgPFykv-By-Zc-HA.png" /></figure><p>Rather than using the marginal likelihood, we can take its logarithm, and incorporating the approximate posterior through a multiply by 1 trick, we will find that we can measure how well we reconstruct the inputs from the prior, and how close the approximated posterior is to the prior.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/275/1*4pXuj2DX-XN_cePET8n4XA.png" /></figure><p>Now the trick:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/337/1*8qZIxPKR2UucSIlNegNUYg.png" /></figure><p>Expectancy over the approximate posterior and then take logarithm:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/307/1*O__-o0Rq2osiSDgdxaW9_w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/443/1*cWZaHPzkBRzhR_f7StISuA.png" /></figure><p>The first term of the R.H.S of the equation above is a measure of how well we can reconstruct the inputs from the prior, while the second term gives us the <a href="https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence">Kullback-Leibler divergence</a> of the approximate posterior from the prior. This objective can now be rewritten as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*zxEDf_hCTzEHsdKloMEDRA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/451/1*EPUaw14CbpwavBaJTDbQiQ.png" /></figure><p>The first term is usually referred to as the variational lower bound on the marginal likelihood because the KL divergence is non-negative. Usually the KL divergence is computed as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/427/1*qBXMv-OlE6F6P6yKhriPRw.png" /></figure><p>We will not go into the proof of this, but we can find it in the <a href="https://arxiv.org/abs/1312.6114">autoencoding variational bayes paper</a>. We have just one concept we have not discussed which we referenced but did not define it, the reparametarization trick.</p><p><strong>The Reparametarization Trick</strong></p><p>In our VAE, the encoder learns to adjust the latent distribution parameters of each <strong><em>x</em></strong> to fit into the prior assumed to be a Gaussian unlike other autoencoders which learn the latent representations directly, so the encoder describes our latent representation as a distribution</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/102/1*0agfpkI7qwvCAKh6RWSfPw.png" /></figure><p>For the decoder to generate new samples similar to the input <strong><em>x</em></strong>, it needs samples of <strong><em>z </em></strong>(a latent representation). Also for back-propagation when training the VAE, the <strong><em>z </em></strong>as it is from the encoder would make back-propagation somewhat difficult being a random variable, thus we need to reparametarize it to make itdeterministic by sampling from the distribution, we do this as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/179/1*j4bIn0LqKYg9MxqP6Q2yHg.png" /></figure><p>We do this by sampling from the approximate posterior distribution which our objective function (<strong><em>ELBO</em></strong>) attempts to make similar to the assumed prior. This results in a deterministic <strong><em>z</em></strong> which makes back-propagation easier. The sampled <strong><em>z</em></strong> can then be used with the decoder while aiming to maximize the likelihood of obtaining <strong><em>x</em></strong> from the decoder through the <strong><em>ELBO</em></strong> function log-likelihood or minimizing the mean-squared error. Now that we have discussed the reparametarization trick, we can go ahead to implement our VAE.</p><h4>VAE Implementation</h4><p>First of we will be implementing our VAE using the MNIST datset, let’s get the dataset, make our train, test and validation splits.</p><pre>import torch<br>import torch.nn.functional as F<br>from torch import nn<br><br>import torchvision.datasets as datasets<br>from torchvision import transforms<br><br>import matplotlib.pyplot as plt<br><br><br><br>train = datasets.MNIST(root=&quot;data/&quot;, train=True, transform=transforms.ToTensor(), download=True)<br>test = datasets.MNIST(root=&quot;data/&quot;, train=False, transform=transforms.ToTensor(), download=True)<br>clear_output(wait=False)<br><br>y_test = test.targets<br>x_test = test.data.to(torch.float32)/255<br>print(f&quot;TEST  SIZE =&gt; X:{x_test.shape}, y:{y_test.shape}&quot;)<br>del test<br><br>targets = train.targets<br>train = train.data.to(torch.float32)/255<br><br>val_size, height, width = x_test.size()<br>H, W = height, width<br><br>idx = torch.randperm(train.size(0))<br>val_idx, train_idx = idx[:val_size], idx[val_size:]<br><br>x_val, y_val = train[val_idx], targets[val_idx]<br>print(f&quot;VAL   SIZE =&gt; X:{x_val.shape}, y:{y_val.shape}&quot;)<br><br>x_train, y_train = train[train_idx], targets[train_idx]<br>print(f&quot;TRAIN SIZE =&gt; X:{x_train.shape}, y:{y_train.shape}&quot;)</pre><p>Then we use the function below to fetch batches of data during training:</p><pre>train = datasets.MNIST(root=&quot;data/&quot;, train=True, transform=transforms.ToTensor(), download=True)<br>test = datasets.MNIST(root=&quot;data/&quot;, train=False, transform=transforms.ToTensor(), download=True)<br>clear_output(wait=False)<br><br>y_test = (test.targets).to(device)<br>x_test = (test.data.to(torch.float32)/255).to(device)<br>print(f&quot;TEST  SIZE =&gt; X:{x_test.shape}, y:{y_test.shape}&quot;)<br>del test<br><br>targets = train.targets<br>train = train.data.to(torch.float32)/255<br><br>val_size, height, width = x_test.size()<br>H, W = height, width<br><br>idx = torch.randperm(train.size(0))<br>val_idx, train_idx = idx[:val_size], idx[val_size:]<br><br>x_val, y_val = train[val_idx].to(device), targets[val_idx].to(device)<br>print(f&quot;VAL   SIZE =&gt; X:{x_val.shape}, y:{y_val.shape}&quot;)<br><br>x_train, y_train = train[train_idx].to(device), targets[train_idx].to(device)<br><br>print(f&quot;TRAIN SIZE =&gt; X:{x_train.shape}, y:{y_train.shape}&quot;)<br><br>def get_batch(x, y=None, batch_size=32):<br>  idx = torch.randint(0, x.size(0), (batch_size,)).to(device)<br>  x_ = x[idx].to(device)<br>  if y is None:<br>    return x_<br>  else:<br>    y_ = y[idx].to(device)<br>    return x_, y_</pre><p>Now we start implementing the VAE. First we implement our encoder, assuming our latent space is of ten dimensions, and an input layer of 200 neurons.</p><pre>z_dim = 10<br>n_hid = 200<br><br># Encoder<br>encoder = nn.Sequential(<br>    nn.Linear(H*W, n_hid, bias=linear_bias),<br>    nn.LeakyReLU(0.2),<br>    nn.BatchNorm1d(n_hid, track_running_stats=linear_bias),<br>    nn.Linear(n_hid, z_dim, bias=linear_bias),<br>    nn.LeakyReLU(0.2),<br>    nn.BatchNorm1d(z_dim, track_running_stats=True),<br>    ).to(device)</pre><p>Seems like the encoder maps to the latent space from the code above, but recall the encoder is learning the appropriate statistical parameters for our inputs to fit in the assumed prior, so we are trying to lean mean and deviation values, so we have two layers for this.</p><pre>mu_layer = nn.Linear(z_dim, z_dim, bias=False).to(device)<br>sigma_layer = nn.Linear(z_dim, z_dim, bias=False).to(device)</pre><p>Now let us implement our decoder:</p><pre>decoder = nn.Sequential(<br>    nn.Linear(z_dim, n_hid, bias=linear_bias),<br>    nn.Tanh(),<br>    nn.Linear(n_hid, 400, bias=linear_bias),<br>    nn.Tanh(),<br>    nn.Linear(400, 600, bias=False),<br>    nn.Tanh(),<br>    nn.Linear(600, H*W),<br>    nn.Sigmoid()<br>).to(device)</pre><p>But seems like the decoder is mapping from the latent space back to the input space, so do we really need the <strong><em>mu</em></strong> and <strong><em>sigma</em></strong> layers. We do, and we will see how with the reparametarize trick implementation which uses the mu and sigma layer outputs in sampling from the prior to get our latent space representation which the decoder uses.</p><pre>def forward(x, y=None, rtn_loss=True, loss_fn=nn.MSELoss(reduction=&quot;sum&quot;).to(device), beta=1):<br><br>  b, h, w = x.size()<br>  x = x.view(b, h*w)<br><br>  z = encoder(x).to(device)<br><br>  # REPARAMETARIZATION<br>  mu = F.tanh(mu_layer(z))<br>  sigma = F.tanh(sigma_layer(z))<br>  eps = torch.randn_like(sigma)<br>  z_rpm = mu + sigma*eps<br><br>  x_hat = decoder(z_rpm)<br><br>  try:<br>    if rtn_loss is True:<br>      l2_loss = loss_fn(x_hat, x)<br>      KL_div = -0.5*torch.sum(1 + torch.log(sigma.pow(2)) - mu.pow(2) - sigma.pow(2)) * beta<br>      loss = l2_loss + (KL_div*beta)<br>      return x_hat, loss<br>    else:<br>      x_hat = x_hat.view(b, h, w)<br>      return x_hat<br>  except Exception as e:<br>    print(mu, sigma)<br>    print((x_hat.min(), x_hat.max()))<br>    print((x.min(), x.max()))<br>    print(e)</pre><p>We can see we replace the likelihood in the <strong><em>ELBO</em></strong> with the <strong><em>MSE</em></strong> loss, and we can also see the KL divergence implemented just as we described to get the total loss. Now let’s see our training code:</p><pre># Set Trainable Parameters<br>encoder_params = [layer.weight for layer in encoder if isinstance(layer, nn.Linear)]<br>reparameterization_params = [mu_layer.weight, sigma_layer.weight]<br>decoder_params = [layer.weight for layer in decoder if isinstance(layer, nn.Linear)]<br>parameters = encoder_params + reparameterization_params + decoder_params<br>for p in parameters:<br>  p.require_grad = True<br><br># Initialize Weights<br>blocks = [encoder, [mu_layer, sigma_layer], decoder]<br>for block in blocks:<br>  for layer in block:<br>    if isinstance(layer, nn.Linear):<br>      torch.nn.init.kaiming_uniform_(layer.weight, nonlinearity=&quot;leaky_relu&quot;)<br>      # torch.nn.init.kaiming_normal_(layer.weight, nonlinearity=&quot;leaky_relu&quot;)<br>      # torch.nn.init.normal_(layer.weight, mean=0.0, std=2.5)<br>      if layer.bias is not None:<br>        torch.nn.init.zeros_(layer.bias)<br><br>def get_layer_weights(layer):<br>  weights = layer.weight.cpu()<br>  gradients = layer.weight.grad<br>  return weights.detach().numpy(), gradients<br><br>blocks = [encoder, [mu_layer, sigma_layer], decoder]<br>for b, block in enumerate(blocks):<br>  for l, layer in enumerate(block):<br>    if isinstance(layer, nn.Linear):<br>      print(f&quot;Layer {layer}: {layer}&quot;)<br>      weights, gradients = get_layer_weights(layer)<br>      plt.hist(weights.flatten(), bins=30, alpha=0.7, label=f&quot;Layer-{layer}&quot;)<br><br><br># TRAINING PARAMETERS<br>batch_ratio = 0.05<br>train_batch_size = int(batch_ratio*x_train.size(0))<br>val_batch_size = int(batch_ratio*x_val.size(0))<br>n_iters = 10000<br>eval_interval =10<br>eval_iters = 15<br>beta = 0.15<br>beta_reg = 1<br>lr = 1E-2<br><br><br># TRAINING<br>history = {<br>    &quot;train_loss&quot;: [],<br>    &quot;val_loss&quot;: [],<br>    &quot;iter&quot;: [],<br>    &quot;lr&quot;: []<br>}<br><br>plateau_lr_patience = 5<br>plateau_lr_count = 0<br>plateau_lr_factor = 0.7<br>plateau_lr_threshold = 5E-3<br><br>best_val_loss = None<br>best_model = None<br><br>model = {<br>    &quot;encoder&quot;: encoder.state_dict(),<br>    &quot;mu_layer&quot;: mu_layer.state_dict(),<br>    &quot;sigma_layer&quot;: sigma_layer.state_dict(),<br>    &quot;decoder&quot;: decoder.state_dict(),<br>    &quot;loss&quot;: None,<br>    &quot;epoch&quot;: None<br>}<br><br>def checkpoint(epoch, loss, path):<br>  model[&quot;encoder&quot;] = encoder.state_dict()<br>  model[&quot;mu_layer&quot;] = mu_layer.state_dict()<br>  model[&quot;sigma_layer&quot;] = sigma_layer.state_dict()<br>  model[&quot;decoder&quot;] = decoder.state_dict()<br>  model[&quot;loss&quot;] = loss<br>  model[&quot;epoch&quot;] = epoch<br>  torch.save(model, path)<br>  print(f&quot;Model Checkpoint @ Epoch {epoch} with loss: {loss}&quot;)<br><br>velocities = []<br>for idx, p in enumerate(parameters):<br>    velocities.append(0)<br>v = velocities<br><br>encoder.train()<br>mu_layer.train()<br>sigma_layer.train()<br>decoder.train()<br><br>for i in range(n_iters):<br><br>  x = get_batch(x_train, batch_size=train_batch_size)<br><br>  x_hat, loss = forward(x, beta=beta_reg)<br>  for idx, p in enumerate(parameters):<br>    p.grad = None<br>  loss.backward()<br>  for idx, p in enumerate(parameters):<br>    torch.nn.utils.clip_grad_norm_(p.grad, max_norm=1.0)<br>    # torch.nn.utils.clip_grad_value_(p.grad, clip_value=1.0)<br>    torch.clip(p.grad, min=-2, max=2)<br>    v[idx] = beta*v[idx] + lr*p.grad<br>    p.data -= v[idx]<br>    # p.data -= lr*p.grad<br><br>  if (i % eval_interval == 0) or (i == n_iters-1):<br>    with torch.no_grad():<br>      train_loss = torch.zeros(eval_iters)<br>      val_loss = torch.zeros(eval_iters)<br><br>      for j in range(eval_iters):<br><br>        x = get_batch(x_train, batch_size=train_batch_size).to(device)<br>        _, loss = forward(x, beta=beta_reg)<br>        train_loss[j] = loss.item()<br><br>        x = get_batch(x_val, batch_size=val_batch_size).to(device)<br>        _, loss = forward(x, beta=beta_reg)<br>        val_loss[j] = loss.item()<br><br>      history[&quot;train_loss&quot;].append(train_loss.mean())<br>      history[&quot;val_loss&quot;].append(val_loss.mean())<br>      history[&quot;iter&quot;].append(int((i/eval_interval)+1))<br>      history[&quot;lr&quot;].append(lr)<br><br>    print(f&quot;step {int((i/eval_interval)+1)}/{int(n_iters/eval_interval)} | lr {lr:E}, train_loss: {train_loss.mean():.2f}, val_loss: {val_loss.mean():.2f}&quot;)<br>    if (best_val_loss==None) or (val_loss.mean() &lt;= best_val_loss + plateau_lr_threshold):<br>      best_val_loss = val_loss.mean()<br>      checkpoint(int((i/eval_interval)+1), best_val_loss, &quot;vae_model.pt&quot;)<br>      plateau_lr_count = 0<br>    else:<br>      plateau_lr_count += 1<br>    <br>    if plateau_lr_count &gt; plateau_lr_patience:<br>      plateau_lr_count = 0<br>      lr *= plateau_lr_factor<br><br><br>best_model = torch.load(&quot;vae_model.pt&quot;)<br>best_model.keys()<br>print(f&quot;Best Loss: {best_model[&#39;loss&#39;]}&quot;)<br><br>encoder.load_state_dict(best_model[&quot;encoder&quot;])<br>mu_layer.load_state_dict(best_model[&quot;mu_layer&quot;])<br>sigma_layer.load_state_dict(best_model[&quot;sigma_layer&quot;])<br>decoder.load_state_dict(best_model[&quot;decoder&quot;])<br><br>encoder.eval()<br>mu_layer.eval()<br>sigma_layer.eval()<br>decoder.eval()<br><br>with torch.no_grad():<br>  test_loss = torch.zeros(10)<br>  for j in range(10):<br>    x = get_batch(x_test, batch_size=val_batch_size, ).to(device)<br>    _, loss = forward(x, beta=beta_reg)<br>    test_loss[j] = loss.item()<br>print(&quot;----------------------------------------------&quot;)<br>print(f&quot;TEST LOSS: {test_loss.mean()}&quot;)</pre><p>Let’s see the training results, then use the VAE in generating a similar example given an input, first our loss curve:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*ZkoxfvU7lJuHPcLqOcYJOw.png" /><figcaption>Loss Curve &amp; Learning Rate Reduce on Plateau</figcaption></figure><p>Now let us take an input and generate an output:</p><pre>x_hat = torch.zeros((1, H*W))<br><br>mu = torch.randn((10,))<br>sigma = torch.randn((10,))<br><br>samples = 1<br>for i in range(samples):<br>    with torch.no_grad():<br>        eps = torch.randn_like(sigma)<br>        z = (mu + sigma*eps).to(device)<br>        x_hat += decoder(z).cpu()<br>x_hat /= samples<br>B, _ = x_hat.size()<br>x_hat = x_hat.view(B, H, W)<br>x_hat = x_hat.cpu().squeeze(dim=0)<br><br>plt.figure(figsize=(3,3))<br><br>plt.imshow(x_hat, cmap=&quot;gray&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*CyyDYeaxHsCmZRL1sFgQRg.png" /><figcaption>Generates the Number 0 from random mean &amp; deviation paramaters</figcaption></figure><p>We see our VAE is able to generate some output similar to some input class in the inputs, despite not having trained to learn the latent space, but having trained to fit the evidence into the assumed prior distribution, generating new and similar variants of the input. You can view the notebook used in developing the VAE <a href="https://www.kaggle.com/code/emmanuelolateju/vae-mnist">here</a>. Our VAE generates new samples similar to the input sample.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*dCoVv9QTgNwkNKoYFl3qjA.png" /></figure><h3>Conclusion</h3><p>In conclusion, VAEs represent a powerful approach to generative modeling, enabling the reconstruction and generation of data by learning probabilistic mappings between input data distributions and latent representations. Through the use of the reparameterization trick, VAEs facilitate efficient training and backpropagation by approximating the posterior distribution and aligning it with the assumed Gaussian prior. The training process optimizes reconstruction accuracy while minimizing the divergence between the learned posterior and the prior using the Evidence Lower Bound (ELBO).</p><p>This ability has made VAEs valuable in various generative tasks beyond image generation, including text synthesis, music composition, and anomaly detection in medical imaging. Their versatility and capacity to model complex data distributions have solidified VAEs as a foundational tool in advancing generative modeling across diverse domains.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c74c1b9568e8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rectlabs/variational-autoencoders-image-reconstruction-generation-c74c1b9568e8">Variational AutoEncoders — Image Reconstruction/Generation</a> was originally published in <a href="https://medium.com/rectlabs">Rectlabs Inc</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AutoEncoders — Introduction]]></title>
            <link>https://medium.com/rectlabs/autoencoders-introduction-db7f314f428d?source=rss-8e9075c1d169------2</link>
            <guid isPermaLink="false">https://medium.com/p/db7f314f428d</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[autoencoder]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[variational-autoencoder]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Emmanuel Olateju]]></dc:creator>
            <pubDate>Mon, 06 Jan 2025 11:14:57 GMT</pubDate>
            <atom:updated>2025-01-06T11:14:57.720Z</atom:updated>
            <content:encoded><![CDATA[<h3>AutoEncoders — Introduction &amp; Application</h3><h4>That Simple &amp; Ubiquitous Concept</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5X01pfhvBqf8M48ohecKRw.jpeg" /><figcaption>Image by <a href="https://pixabay.com/users/geralt-9301/?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=8743259">Gerd Altmann</a> from <a href="https://pixabay.com//?utm_source=link-attribution&amp;utm_medium=referral&amp;utm_campaign=image&amp;utm_content=8743259">Pixabay</a></figcaption></figure><p>Say we have a set of inputs <strong><em>X = {…x3, x4, x5….}</em></strong>, we can represent each unique item of <strong><em>X</em></strong> with some unique identifier <strong><em>Z = {….z3, z4, z5…}</em></strong>, we can also translate the various identifiers back to their original data form X. This forward translation is called encoding and the backward decoding.</p><p>This is fairly straightforward and preconceived. What if we have a very large input space, do we come up with all of the z-representations ourselves. No, neural network comes and helps us with the encoding and decoding. A unique neural network architecture called the autoencoder exists for this purpose.</p><h4><strong>The AutoEncoder</strong></h4><p>Below we see and autoencoder, it has eight input neurons and eight output neurons so what does it do exactly?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*r_3RkuX3MBu_KLx841oCMQ.png" /><figcaption><a href="https://tikz.net/autoencoder/">https://tikz.net/autoencoder/</a></figcaption></figure><p>Assume we have some eight vector embedding for each <strong><em>x</em></strong>, from the image of the autoencoder above, we can infer that: the autoencoder attempts to learn an encoder A that maps those eight vectors for each <strong><em>x</em></strong> to some latent space vector representation of size three, and then a decoder model B attempts learning to map the latent space representation to a representation similar to the input, if not exactly the same (taking it too far). As we expect losses between the reconstructed representation and the original input, we label the reconstructed form as <strong><em>x_hat</em></strong>.</p><p><strong>It’s Loss Function</strong></p><p>Remember the goal is to make the input and its resultant representation as similar as possible. So for each vector entry, we aim to minimize the distance of its resultant representation from it. For this we can make use of the l2 norm loss or the squared loss, with the goal of minimizing the mean square loss across all of the inputs and their resultant representations, with respect to the weights of the encoder A and decoder B. Therefor the loss function of an autoencoder also referred to as the reconstruction loss is of the form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*IjNgQfDY4Uy2qugXp-Emhg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/1*-2Tj4Epe51HnULOpYJwq-w.png" /></figure><p>where the triangle indicates the l2 (squared) loss function. We are essentially optimizing the autoencoder network to make the resulting representation the same as the input.</p><blockquote>The interesting thing about the autoencoder is it has so many applications, it is like the ubiquitous formula for Newton’s second law of motion (F=ma), finding application in various areas of engineering. Maybe we can refer to it as machine learning’s string theory or quantum field theory, hell no.</blockquote><h4>AutoEncoder Applications</h4><p>The autoencoder architecture can be used in various supervised, unsupervised and semi-supervised learning problems, it can be used in building de-noising models, generative models, and more. Let us see some of these applications of the autoencoder network.</p><p><strong>Supervised — Unsupervised Learning</strong></p><p>Let us start with some unsupervised task: <strong>dimensionality reduction</strong>. The ability of the autoencoder to project data from a higher dimension to a lower dimension (latent space) representation makes it ubiquitous for various machine learning applications. This projection is done by the encoder under the primary reconstruction objective of the decoder. This alongside the non-linear activation often results in better reconstruction of the input data from the latent space compared to conventional dimensionality reduction techniques like principal component analysis.</p><p>The lower dimensional representation of the latent space also called the bottleneck can also be used as the input to a k-means <strong>clustering</strong> algorithm for <strong>clustering</strong> purposes. But this is done under the autoencoder’s reconstruction objective without any loss function optimizing the encoder network for optimal clustering results. This can be addressed by applying clustering on the latent space representations under the primary reconstruction objective to identify the cluster centers, then retraining the latent representations every iteration by adding a loss function which penalizes the distance between the latent representation and the cluster center as the secondary objective.</p><p>The latent space representations can also be used for <strong>classification</strong> under the assumption that samples in the same class would have similar latent representation. The autoencoder is trained for reconstruction, and then the latent representations can be of the encoder can be used as input to a classification algorithm. The two can also be done simultaneously in a parallel manner. If done in a parallel manner, the reconstruction loss can also be used as a form of regularization for the classifier.</p><p>Lastly we talk of <strong>anomaly detection</strong>. This does not make use of the latent space particularly for some objective, but uses the whole autoencoder. What happens here is that we train the autoencoder to reconstruct some positive examples, after which we expect the autoencoder to reconstruct positive examples with low reconstruction loss and negative examples with high reconstruction loss.</p><p><strong>Denoising</strong></p><p><strong>Denoising autoencoders</strong> are pretty straightforward. The goal is to train the autoencoder to reconstruct the original data from a noise contaminated version of the data. The reconstruction loss is computed between the noise free data and the autoencoder representation, while the autoencoder representation is generated from a version of the data contaminated with some form of noise usually gaussian white noise.</p><p>Another version of autoencoders called <strong>contractive autoencoders </strong>approaches denoising a bit differently. They add a regularization loss to the reconstruction loss which measures the gradient of neuron nodes to each input through a Jacobian matrix. This is imposed on the encoder part so the feature extraction generating latent space is less sensitive to small perturbations (of the input due to noise) which is not required for reconstruction. Latent representations resulting from noisy inputs not important for reconstruction will be diminished (have their gradients pushed towards zero), while those require for reconstruction retain some positive or negative gradient.</p><p><strong>Generative Modelling</strong></p><p>We have seen applications of the encoder and its latent space in various machine learning problems, but there is a scenario where we are interested in the decoder, this is generative modelling.</p><p>Assume we have a large set <strong><em>X </em></strong>of fairly homogeneous samples and assume its representation space is a set called the prior (<strong><em>Z</em></strong>), then lets extract a subset of the input that maps to a subset of the prior; we can train an autoencoder to reconstruct this subset of <strong><em>X, </em></strong>let’s assume it perfectly reconstructs this subset. What happens if we take just the decoder, sample form the larger prior set <strong><em>Z </em></strong>exclusive of the subset used in training, and pass the sampled <strong><em>z </em></strong>to the decoder? Then we would have generated some other samples similar to the those in the input subset we used in training. This is the big idea behind generative modelling, but the prior is actually assumed to be some distribution: uniform, normal, etc.</p><p><strong>Variational autencoders (VAEs)</strong><em>, </em>are used for this purpose. They are preferred as they are able to learn some encoder parameters that maps the input (evidence) distribution to some assumed prior distribution, and learn decoder parameters that map the prior distribution to some distribution similar to the evidence.</p><h3>Conclusion</h3><p>While other autoencoders work towards reconstructing the original input from a clean or noise contaminated version, optionally imposing a regularization loss which makes the latent space less sensitive to small signal noise; VAEs are not just learning reconstruction, but learning a probabilistic mapping which maps the input distribution to an assumed prior distribution, and generating distributions similar to the input by sampling from the assumed prior.</p><p>Our main goal is to talk about VAEs and generative modelling, so we would not implement an autoencoder just yet. In the next part of this article, we explicitly explain how a VAE works, then develop and train one for generative modeling. Let’s proceed to the second and final part of this article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=db7f314f428d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rectlabs/autoencoders-introduction-db7f314f428d">AutoEncoders — Introduction</a> was originally published in <a href="https://medium.com/rectlabs">Rectlabs Inc</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transformers — 2]]></title>
            <link>https://medium.com/rectlabs/transformers-2-c052036adf30?source=rss-8e9075c1d169------2</link>
            <guid isPermaLink="false">https://medium.com/p/c052036adf30</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Emmanuel Olateju]]></dc:creator>
            <pubDate>Mon, 16 Dec 2024 00:56:17 GMT</pubDate>
            <atom:updated>2024-12-16T00:56:17.385Z</atom:updated>
            <content:encoded><![CDATA[<h3>Hands on intro to LLMs: Attention mechanism (part 2)</h3><h4>Components Of A Transformer &amp; Attention Mechanisms</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gTS82qN2XI-LbpML3mw6Mg.jpeg" /><figcaption>`Photo by <a href="https://unsplash.com/@6690img?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Jona</a> on <a href="https://unsplash.com/photos/a-group-of-pink-and-blue-balls-on-a-black-background-UopUfxghnWo?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>In the first part of this series on LLMs, we discussed the concept of attention, and how the attention mechanism allows for parallel processing of sequences optimizing memory/time parameters of training. But the transformer architecture is not made up of just attention mechanism (commonly referred to as attention head) and neural network layers, but is just the main mechanism behind the functionality of transformers. In this part we discuss the other components of transformers. Any transformer is made up of, or makes use of the following concepts or components.</p><ol><li>Embedding</li><li>Positional Encoding</li><li>Residual Connections</li><li>Encoder and Decoders</li><li>Layer Normalization</li></ol><p>Now let’s discuss other components of a transformer, and then we dive into the various types of attention mechanisms used in transformer architectures.</p><h4>Embedding</h4><p>Neural network models cannot directly operate on words or characters but instead work with numbers. Remember encoding helps in that aspect, but encoding maps the chord (characters) to integers, so the representation of the chords does not contain information on how similar certain chords are.</p><p>For example, if we are dealing with a sequence of words, let us consider the following sequences: <em>“I am here”, “I am there”, “You are here”, “You are there”</em>; we could have encoding for each words, and intuitively we see that <em>“I”</em> and <em>“You”</em> can be used in the same sentence interchangeably, also <em>“here”</em> and <em>“there”</em> can be used interchangeably, but the encoding does not tell us that these word pairs are similar.</p><p>In actual practice, we come up with a linear layer which generates a vector representation of size <strong><em>E </em></strong>called the embedding size,<strong><em> </em></strong>for the encoding of each item; the representation can be re-learnt during training to make the vectors of similar items closer together. This results in what we call an embedding. An embedding maps items (such as words or characters) to points in a vector space while preserving their semantic meaning (keeping more similar items together). PyTorch easily helps us implement an embedding.</p><pre>embedding_layer = nn.Embedding(vocab_size, n_embd)</pre><h4>Positional Encoding</h4><p>When working with transformers, we process the entire sequence of words (or tokens) at once instead of one word at a time. This parallel processing makes transformers very efficient, but it also means they lose track of the order or position of words in the sequence. For example, without additional information, the model wouldn’t know if “I ate breakfast” is different from “Breakfast ate I.”</p><p>We use positional encoding to add order information to each token (item). Each token in a sequence is converted into a vector (embedding), forming a <strong><em>T</em></strong>x<strong><em>E</em></strong> matrix for <strong><em>T</em></strong> tokens and embedding size <strong><em>E</em></strong>. To encode position, a <strong><em>T</em></strong>x<strong><em>E</em></strong><em> </em>positional encoding matrix is generated using sine and cosine functions. Adding this to the token embedding helps the model understand both token meaning and position. Let us see an example heat map of a position encoding matrix, for <strong><em>100 tokens</em></strong> and an <strong><em>embedding size of 50</em></strong>. Also, the formula that generated it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5pEFwx33x-ID5O2JG-TeRQ.png" /></figure><p>Now let us look at the mathematical formula that generates the matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/333/1*FIZiq_OHq5cN_6tDI01wpw.png" /><figcaption>Positional Encoder Formula</figcaption></figure><p>Let us look at the formula for a sinusoidal function of position. This can be <strong><em>sin(kx)</em></strong> or <strong><em>cos(kx)</em></strong><em>,</em>where <strong><em>x</em></strong><em> </em>is the position (<strong><em>pos</em></strong>) and <strong><em>k</em></strong> is the spatial frequency. We can equate <strong><em>k</em></strong> and <strong><em>x</em></strong> to the following in our positional encoding formula.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/110/1*gSqZCKsXun835r2jHFxM0Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/72/1*9W7UYRYSkf2jcLnyi2GDPw.png" /></figure><p>n the sinusoidal positional encoding formula, <strong><em>x </em></strong>represents the position, and <strong><em>k </em></strong>corresponds to the frequency determined by the embedding dimension <strong><em>i</em></strong>. For each dimension <strong><em>i</em></strong>, <strong><em>k </em></strong>defines a unique frequency, and the positional encoding value at position <strong><em>x </em></strong>is calculated accordingly. This adds a unique vector to each token embedding. In simple terms, positional encoding is like giving each item or element a “badge” to indicate its position in the sequence, helping the model understand order and relationships. This allows the model to comprehend sequence structure, like understanding that “I ate breakfast” makes sense, but “Breakfast ate I” does not.</p><p>Simply, we can implement positional-encoding with an embedding layer where we indicate the number of items to generate embedding for, as the length of the sequence input for the network, and the same embedding size <strong><em>E.</em></strong></p><pre>position_embedding_table = nn.Embedding(block_size, n_embd)</pre><h4>Residual Connections</h4><p>What are residual connections? They are best described by the paper, but we describe them as a simple mathematical operation of the form:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/147/1*NezWVGx1mg32zrsTs3kGlA.png" /><figcaption>Mathematics of Residual Connection</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/330/1*rVejJp8Jj3-mXOrslFk3nQ.png" /><figcaption><a href="https://en.wikipedia.org/wiki/Residual_neural_network">https://en.wikipedia.org/wiki/Residual_neural_network</a></figcaption></figure><p>What this simply means is when we have a neural network layer <strong><em>f</em></strong> which is a convolution of many other sub-layers (a multi-layered network), given an input <strong><em>x</em></strong><em> </em>to the network, rather than making use of the direct output, then we make a connection between the input <strong><em>x</em></strong> and output <strong><em>f(x)</em></strong> to form a new output <strong><em>x + f(x) = g(x)</em></strong>. That is residual connections skips one or more layers in combining the input with the output of a particular layer. It offers various advantages such as:</p><ol><li>Improved gradient flow during back-propagation, helping to avoid layers where gradients may vanish or explode (mitigates vanishing/exploding gradients problem).</li><li>Leads to faster convergence as it makes training path smoother.</li><li>Helps avoid feature degradation as it helps skip layers as needed to retain essential features</li><li>They allow models to scale in depth while maintaining important features, thus they enable development of pre-trained models for transfer learning.</li></ol><h4>Encoder-Decoder Architecture</h4><p>Let’s say we want to translate <strong><em>“good morning”</em></strong> to its German variant: <strong><em>“guten morgen”</em></strong><em>, </em>using a neural network, the network has to be able to generate a representation of <strong><em>“good”</em></strong> which it also understands to be <strong><em>“guten”</em></strong><em>, </em>and another for <strong><em>“morning”</em></strong> which it also understands to be <strong><em>“morgen”</em></strong><em>. </em>The encoder-decoder architecture takes in all of these sequence, generates a latent representation called the context vector (vector of fixed length) from the encoder which is used by the decoder to translate the whole sequence</p><p>The encoder-decoder architecture is a foundational architectures for not just translation tasks, but for prediction of future values, other form of sequence transformation, and in the development of generative models. Its strength lies in its ability to represent and transform data from tow domains into a single context.</p><p>Of course the fixed-length of its context vector limits its performance on longer sequences, and it is highly sensitive to the quality of the input data. Also, many encoder-decoder architectures use recursive networks internally which handles the sequence in a sequential manner to generate the context vector. This leads to the same issues of relationship modelling and memory-time complexity during training.</p><p>Attention based encoder/decoders with feedfoward networks were introduced to overcome these limitations. Just as they addressed the issues with RNNs, they also improve encoder-decoder models by allowing the decoder to focus dynamically on different parts of the input. This enhances performance on long or complex sequences and is computationally more efficient. So rather than having recurrent networks in our encoder and decoder, we would have attention mechanisms just as we described in the first part of this article.</p><h4>Layer Normalization</h4><p>Layer normalization is a technique used in deep learning to stabilize and accelerate training by normalizing the inputs of a layer across the features, ensuring consistent scaling and reducing the risk of vanishing or exploding gradients. For an input of size <strong><em>B</em></strong>x<strong><em>d</em></strong>,<strong><em> </em></strong>where <strong><em>B</em></strong> is the batch size and <strong><em>d</em></strong> the feature size, layer-normalization normalizes each instance in the batch as opposed to each feature in batch-normalization, followed by scaling and shifting with learnable parameters. Its independence of batch size makes it more suitable for models like transformers that process sequences independently during inference, ensuring efficient gradient flow and stability in deep architectures. Typically in transformers, a layer is made up of an attention mechanism sub-layer, and a feedforward sub-layer, we employ residual connection on this layer and then apply layer normalization.</p><p>Now let us see how attention mechanism is implemented in transformer architectures, not as we did in the previous part where we aimed at understanding the importance of attention. Then we see the various types of attention mechanisms.</p><h4>Fundamental Concept Of Transformer Attention</h4><p>Previously, we described attention by computing an attention matrix and demonstrating its use in a sequence task, but in transformer architecture, attention is implemented differently as neural network layers learn to compute the attention matrix. For a transformer the key components of an attention mechanism are:</p><ol><li><strong>Query (Q)</strong>: The current element seeking context or information</li><li><strong>Key (K)</strong>: A representation used to compute relevance</li><li><strong>Value (V)</strong>: The actual information to be extracted</li><li><strong>Attention Scores</strong>: Computed by comparing queries and keys</li><li><strong>Attention Weights</strong>: Normalized scores determining the focus</li></ol><p>Let’s get an overview of how it works. Given an input sequence, one linear layer converts it into query vectors, another into key vectors, and another into value vectors. For an input sequence of length five, these linear layers produce three different column vectors, each of size <strong><em>5</em></strong>x<strong><em>E</em></strong>, corresponding to the query, key, and value vectors. We then compute attention scores by performing an operation between the query and key, apply an activation function to get attention weights, and project the value vectors onto these weights to obtain a new representation of the sequence.</p><p>So, what exactly happens here? Three layers learn different projections of the token embedding. The attention matrix is computed from the query and key vectors, and then the value vectors are weighted by the rows of the attention matrix. Through backpropagation, these layers (key, query, value) learn to model the appropriate relationships between the token embedding and adjust their projections to generate the desired output in various scenarios. This is the way of attention, the way of the force.</p><p>Now let’s see the various types of attention mechanisms, and how they are implemented using query, key and vectors.</p><h4>Attention Mechanisms</h4><p>Depending on how these components are used, there are various attention mechanisms used for various tasks.</p><ol><li>Scaled dot-product Attention — <em>Method of Computing Attention</em></li><li>Masked Attention — <em>Variant of Scaled Dot-Product Attention</em></li><li>Self vs. Cross Attention — <em>Usage context</em></li><li>Multi-Head Attention — <em>Combining Attention Heads</em></li></ol><p><strong>Scaled Dot-Product Attention: </strong>This is the most adopted method of attention computation. The key innovation lies in its ability to compute attention weights efficiently while addressing potential scaling issues. Given three matrices: <em>Query</em> (<em>Q), Key (K), Value (V), </em>the attention is computed as:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/380/1*dQrOvMcRJqNYApfP3oJjUw.png" /></figure><p>The dot-product of the query and key matrix is computed and scaled by the inverse square-root of the head size <em>(dimension of each vector in the query/key/value vectors: usually dimension space embedding is mapped to) </em>giving us the attention score, we then make use of the softmax activation function to normalize the attention scores so as to get the attention weights. we can then weight the value vectors with these attention weights.</p><p><strong>Masked Attention: </strong>Recall in the first part of this series, the upper-diagnoal of the attention matrix was zero. Well the scaled dot-product attention does not do that effectively allowing past elements of a sequence to depend on future elements. We address this through masking. Masked attention is a critical mechanism in transformer architectures that prevents models from accessing future information during sequence generation. This technique is fundamental in maintaining the causal nature of auto-regressive models, ensuring that predictions depend only on past and present inputs. It is computed as shown below where <em>M </em>is the masking matrix with zeros on and below diagonals for addition, or ones on and below diagonals for element-wise product, and negative infinity above diagonals.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/430/1*0SH130Z0HcH-OSu-Vfe4fw.png" /></figure><p>The primary goal of masked attention is to simulate a sequential generation process by restricting the attention mechanism to only consider previous tokens when generating the current token. This mimics the way humans process and generate language, where future context is not available during each step of generation. Just as negative infinity becomes zero when softmax is applied, the same applies here.</p><p><strong>Self-attention:</strong> allows elements within the same sequence to interact and compute relationships, capturing internal dependencies. It is used in tasks where the input and output are the same, such as in sequence generation and transformer encoder layers. In self-attention, the query, key, and value all come from the same sequence.</p><p><strong>Cross-attention:</strong> enables interaction between two different sequences, used in sequence-to-sequence tasks like translation. It allows one sequence to attend to another, crucial in encoder-decoder models. In cross-attention, the query comes from one sequence, while the key and value come from another.</p><p>In summary, self-attention captures within sequence relationships, while cross-attention captures between sequence interactions.</p><p><strong>Multi-head Attention: </strong>Conventionally, an attention mechanism is also called an attention head. We can have multiple attention heads operate on the same sequence input and they would learn different representation contributing different perspectives on the input sequence. Their resultant representation may be combined in some way, forming a multi-head attention.</p><p>Multi-head attention enhances standard attention by performing parallel computations across multiple “heads,” each learning different aspects of the input. It projects queries, keys, and values into different subspaces, capturing different views of the relationship within <em>(self-attention)</em> or between <em>(cross-attention)</em> sequences. The outputs from these heads can be concatenated, and linearly transformed, improving the model’s ability to understand complex interactions.</p><p>This mechanism is key to transformer models like BERT and GPT, enabling richer, more nuanced representations. Mathematically, multi-head attention computes <strong><em>h</em></strong> parallel attention functions, offering a more expressive approach to learning contextual information.</p><p>We have learnt of the various components of a transformer, and various attention mechanisms used in transformers. We can now move to implementation.</p><h3>Next</h3><p>In the next (final) part of this series on transformers and LLMs, we will implement a music chord generator. We would use masked attention heads to form multi-attention heads, which we would use in implementing each part of the transformer architecture, we would do this step by step, implementing the architecture from input to output.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c052036adf30" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rectlabs/transformers-2-c052036adf30">Transformers — 2</a> was originally published in <a href="https://medium.com/rectlabs">Rectlabs Inc</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transformers — 3]]></title>
            <link>https://medium.com/rectlabs/transformers-3-d90a9b3f9a0e?source=rss-8e9075c1d169------2</link>
            <guid isPermaLink="false">https://medium.com/p/d90a9b3f9a0e</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[deep-learning]]></category>
            <dc:creator><![CDATA[Emmanuel Olateju]]></dc:creator>
            <pubDate>Mon, 16 Dec 2024 00:55:22 GMT</pubDate>
            <atom:updated>2024-12-16T00:55:22.786Z</atom:updated>
            <content:encoded><![CDATA[<h3>Hands-on Intro To LLMs: Mini-GPTs (part 3)</h3><h4>A Music Chord GPT</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xvhztJ80_lf5ZQwB46YlyQ.jpeg" /><figcaption>Photo by <a href="https://unsplash.com/@gmaxstudios?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">GMax Studios</a> on <a href="https://unsplash.com/photos/white-star-wars-action-figure-CnJBqlEiYIU?utm_content=creditCopyText&amp;utm_medium=referral&amp;utm_source=unsplash">Unsplash</a></figcaption></figure><p>Now that we understand the functions of each part of the transformer model, let’s build one for music chord generation. To make it easier, we’ll use a dataset of chord sequences from popular Western pop songs which I compiled using OpenAI’s ChatGPT (yeah, there would be various issues with such data). We will implement the code step by step to understand how a transformer works in the simplest way possible. Now let us start building, for: <strong><em>“Do or Do Not, There Is No Try”.</em></strong></p><h4>The Dataset</h4><p>So lets compare a MLP without attention, to one with attention. We use chords for the following songs which I asked OpenAI’s ChatGPT.</p><pre>| **Song Titles**               | **Song Titles**                | **Song Titles**                 |<br>|-------------------------------|--------------------------------|---------------------------------|<br>| I&#39;m Not the Only One          | Let It Be                      | With or Without You             |<br>| Someone Like You              | Zombie                         | No Woman No Cry                 |<br>| All of Me                     | Rolling in the Deep            | Wonderwall                      |<br>| Yesterday                     | Shallow                        | Perfect                         |<br>| Stand by Me                   | Let Her Go                     | Hey Jude                        |<br>| The Scientist                 | Sweet Home Alabama             | Don’t Stop Believin’            |<br>| One Love                      | Time After Time                | Boulevard of Broken Dreams      |<br>| Tears in Heaven               | This Land Is Your Land         | Hallelujah                      |<br>| Killing Me Softly             | Thinking Out Loud              | Wonder                          |<br>| Imagine                       | Hotel California               | Blackbird                       |<br>| Fly Me to the Moon            | Country Roads                  | Can’t Help Falling in Love      |<br>| Amazing Grace                 | House of the Rising Sun        |                                |</pre><pre>data = &quot;F,A7,Dm,Bb, ,Gm,Bb,F,C, ,F,A7,Dm,Bb, ,Gm,Bb,F,C, ,F,A7,Dm,Bb, ,C,G,Am,F, ,G,D,Em,C, ,D,A,Bm,G, ,Em,C,G,D, ,F,G,C,Am, ,E,B,C#m,A, ,Am,F,C,G, ,G,D,Em,C, ,A,E,F#m,D, ,Bb,F,C,Dm, ,F,C,G,Am, ,D,A,Bm,F#m, ,C,F,G,C, ,F,Am,G,C, ,Dm,G,C,Am, ,F,G,Am,F, ,E,A,D,A, ,A,D,A,E, ,D,G,Bm,A, ,C,Am,F,G, ,G,Bm,Am,D, ,Am,G,F,C, ,E,B,A,C#m, ,G,C,D,G, ,A,E,D,A, ,C,G,Am,Em, ,F,C,Dm,G, ,G,F,C,G, ,F,Dm,Bb,C, ,C,Am,G,F, ,E,A,B7,A, ,C,Em,G,F, ,F,Dm,G,C, ,D,A,G,A, ,Bb,F,C,Dm, ,E,Am,C,G, ,C,F,Am,G, ,Am,C,G,D, ,G,D,C,D, ,A,F#m,D,E, ,C,Am,F,C, ,E,G,C,G, ,C,E7,Am,F, ,D,G,A,Bm, ,A,C#m,Bm,D, ,F,Bb,Dm,G, ,E,A,C#m,F#m, ,G,D,C,Bm, ,Am,G,F,G, ,F,C,G,C, ,D,G,A,F#m, ,Bb,F,G,C, ,A,E,F#m,B, ,Am,G,C,F, ,D,A,E,F#m, ,C,F,Dm,G, ,G,Bm,D,C, ,F,Dm,C,Am, ,E,A,G,D, ,A,E,A,E, ,G,C,G,D, ,Am,D,F,G, ,C,G,Bb,F, ,F,D,G,C, ,E,B,C#m,A, ,D,G,A,C, ,Am,C,G,F, ,C,D,G,C, ,A,E,G,D&quot;<br>sequence = data.replace(&quot; &quot;,&quot;.&quot;)<br>sequence = sequence.replace(&quot;,&quot;,&quot; &quot;)<br>sequence<br><br>&quot;&quot;&quot;<br>OUTPUT:<br>F A7 Dm Bb . Gm Bb F C . F A7 Dm Bb . Gm Bb F C . F A7 Dm Bb . C G Am F . <br>G D Em C . D A Bm G . Em C G D . F G C Am . E B C#m A . Am F C G . G D Em <br>C . A E F#m D . Bb F C Dm . F C G Am . D A Bm F#m . C F G C . F Am G C . <br>Dm G C Am . F G Am F . E A D A . A D A E . D G Bm A . C Am F G . G Bm Am D . <br>Am G F C . E B A C#m . G C D G . A E D A . C G Am Em . F C Dm G . G F C G . F <br>Dm Bb C . C Am G F . E A B7 A . C Em G F . F Dm G C . D A G A . Bb F C Dm . E <br>Am C G . C F Am G . Am C G D . G D C D . A F#m D E . C Am F C . E G C G . C E7 <br>Am F . D G A Bm . A C#m Bm D . F Bb Dm G . E A C#m F#m . G D C Bm . Am G F G . <br>F C G C . D G A F#m . Bb F G C . A E F#m B . Am G C F . D A E F#m . C F Dm G . <br>G Bm D C . F Dm C Am . E A G D . A E A E . G C G D . Am D F G . C G Bb F . F D <br>G C . E B C#m A . D G A C . Am C G F . C D G C . A E G D<br>&quot;&quot;&quot;</pre><p>Then we make use of the byte-pair encoding I mentioned in the first part of this series to identify unique chord items. We see our vocabulary size <strong><em>(vocab_size) </em></strong>is the number of unique chords we have in the data. This is often referred to as the<strong><em> vocabulary.</em></strong></p><pre>from collections import Counter<br><br>def byte_pair_encoding(data, num_merges):<br><br>    vocab = Counter()<br>    for char in data.split():<br>          vocab[&#39; &#39;.join(list(char))] += 1<br>    print(f&quot;Original Vocab: {vocab.items()}&quot;)<br><br>    for i in range(num_merges):<br>      pairs = Counter()<br>      for chord, freq in vocab.items():<br>        symbols = chord.split()<br>        for j in range(len(symbols)-1):<br>          pairs[symbols[j], symbols[j+1]] += freq<br>      most_frequent_pair = max(pairs, key=pairs.get)<br><br>      new_vocab = {}<br>      bigram = &#39;&#39;.join(most_frequent_pair)<br>      for chord in vocab:<br>        # print(f&quot;{&#39; &#39;.join(most_frequent_pair)}, {bigram}&quot;)<br>        chord_ = chord.replace(&#39; &#39;.join(most_frequent_pair), bigram)<br>        new_vocab[chord_] = vocab[chord]<br>      vocab = new_vocab<br><br>    print(f&quot;Byte Pairs: {vocab.items()}&quot;)<br>    return vocab<br><br># Perform BPE<br>num_merges = 12  # Adjust as needed<br>bpe_result = byte_pair_encoding(sequence, num_merges)<br><br>vocab_size = len(bpe_result)<br><br>print(f&quot;Vocabulary Size: {vocab_size}&quot;)<br><br>&quot;&quot;&quot;<br>Vocabulary Size: 19<br>&quot;&quot;&quot;</pre><p>We would obtain our unique chords as:</p><pre>Original Vocab: dict_items([(&#39;F&#39;, 5), (&#39;A 7&#39;, 3), (&#39;D m&#39;, 3), (&#39;B b&#39;, 5), (&#39;N&#39;, 4), (&#39;G m&#39;, 2), (&#39;C&#39;, 2)])<br>Byte Pairs: dict_items([(&#39;F&#39;, 5), (&#39;A7&#39;, 3), (&#39;Dm&#39;, 3), (&#39;Bb&#39;, 5), (&#39;N&#39;, 4), (&#39;Gm&#39;, 2), (&#39;C&#39;, 2)])</pre><p>Now let us implement our simple encoder and decoder for the data</p><pre>chords = [chord for chord in bpe_result]<br><br>ctoi = {ch:i+1 for i,ch in enumerate([chord for chord in chords if chord!=&quot;.&quot;])}<br>ctoi[&quot;.&quot;] = 0<br><br>itoc = {i:ch for i,ch in zip(ctoi.values(), ctoi.keys())}</pre><p>Our lookup table from chord to integer and vice-versa:</p><pre>Encoding: {&#39;F&#39;: 1, &#39;A7&#39;: 2, &#39;Dm&#39;: 3, &#39;Bb&#39;: 4, &#39;Gm&#39;: 5, &#39;C&#39;: 6, &#39;G&#39;: 7, &#39;Am&#39;: 8, &#39;D&#39;: 9, &#39;Em&#39;: 10, &#39;A&#39;: 11, &#39;Bm&#39;: 12, &#39;E&#39;: 13, &#39;B&#39;: 14, &#39;C#m&#39;: 15, &#39;F#m&#39;: 16, &#39;B7&#39;: 17, &#39;E7&#39;: 18, &#39;.&#39;: 0}<br>Decoding: {1: &#39;F&#39;, 2: &#39;A7&#39;, 3: &#39;Dm&#39;, 4: &#39;Bb&#39;, 5: &#39;Gm&#39;, 6: &#39;C&#39;, 7: &#39;G&#39;, 8: &#39;Am&#39;, 9: &#39;D&#39;, 10: &#39;Em&#39;, 11: &#39;A&#39;, 12: &#39;Bm&#39;, 13: &#39;E&#39;, 14: &#39;B&#39;, 15: &#39;C#m&#39;, 16: &#39;F#m&#39;, 17: &#39;B7&#39;, 18: &#39;E7&#39;, 0: &#39;.&#39;}</pre><p>Now we can make encoders and decoders for our tokens, then encode our original chords sequence, the original data we will train on:</p><pre>encoder = lambda chord_: ctoi[chord_]<br>decoder = lambda value_ : itoc[value_]<br><br>sequence_data = sequence.split(&quot; &quot;)</pre><p>Let us see our sequence data, encoded and decoded:</p><pre>ORIGINAL SEQUENCE: [&#39;F&#39;, &#39;A7&#39;, &#39;Dm&#39;, &#39;Bb&#39;, &#39;.&#39;, &#39;Gm&#39;, &#39;Bb&#39;, &#39;F&#39;, &#39;C&#39;, &#39;.&#39;, &#39;F&#39;, &#39;A7&#39;, &#39;Dm&#39;, &#39;Bb&#39;, &#39;.&#39;, &#39;Gm&#39;, &#39;Bb&#39;, &#39;F&#39;, &#39;C&#39;, &#39;.&#39;, &#39;F&#39;, &#39;A7&#39;, &#39;Dm&#39;, &#39;Bb&#39;, &#39;.&#39;, &#39;C&#39;, &#39;G&#39;, &#39;Am&#39;, &#39;F&#39;, &#39;.&#39;, &#39;G&#39;, &#39;D&#39;, &#39;Em&#39;, &#39;C&#39;, &#39;.&#39;, &#39;D&#39;, &#39;A&#39;, &#39;Bm&#39;, &#39;G&#39;, &#39;.&#39;, &#39;Em&#39;, &#39;C&#39;, &#39;G&#39;, &#39;D&#39;, &#39;.&#39;, &#39;F&#39;, &#39;G&#39;, &#39;C&#39;, &#39;Am&#39;, &#39;.&#39;, &#39;E&#39;, &#39;B&#39;, &#39;C#m&#39;, &#39;A&#39;, &#39;.&#39;, &#39;Am&#39;, &#39;F&#39;, &#39;C&#39;, &#39;G&#39;, &#39;.&#39;, &#39;G&#39;, &#39;D&#39;, &#39;Em&#39;, &#39;C&#39;, &#39;.&#39;, &#39;A&#39;, &#39;E&#39;, &#39;F#m&#39;, &#39;D&#39;, &#39;.&#39;, &#39;Bb&#39;, &#39;F&#39;, &#39;C&#39;, &#39;Dm&#39;, &#39;.&#39;, &#39;F&#39;, &#39;C&#39;, &#39;G&#39;, &#39;Am&#39;, &#39;.&#39;, &#39;D&#39;, &#39;A&#39;, &#39;Bm&#39;, &#39;F#m&#39;, &#39;.&#39;, &#39;C&#39;, &#39;F&#39;, &#39;G&#39;, &#39;C&#39;, &#39;.&#39;, &#39;F&#39;, &#39;Am&#39;, &#39;G&#39;, &#39;C&#39;, &#39;.&#39;, &#39;Dm&#39;, &#39;G&#39;, &#39;C&#39;, &#39;Am&#39;, &#39;.&#39;, &#39;F&#39;, &#39;G&#39;, &#39;Am&#39;, &#39;F&#39;, &#39;.&#39;, &#39;E&#39;, &#39;A&#39;, &#39;D&#39;, &#39;A&#39;, &#39;.&#39;, &#39;A&#39;, &#39;D&#39;, &#39;A&#39;, &#39;E&#39;, &#39;.&#39;, &#39;D&#39;, &#39;G&#39;, &#39;Bm&#39;, &#39;A&#39;, &#39;.&#39;, &#39;C&#39;, &#39;Am&#39;, &#39;F&#39;, &#39;G&#39;, &#39;.&#39;, &#39;G&#39;, &#39;Bm&#39;, &#39;Am&#39;, &#39;D&#39;, &#39;.&#39;, &#39;Am&#39;, &#39;G&#39;, &#39;F&#39;, &#39;C&#39;, &#39;.&#39;, &#39;E&#39;, &#39;B&#39;, &#39;A&#39;, &#39;C#m&#39;, &#39;.&#39;, &#39;G&#39;, &#39;C&#39;, &#39;D&#39;, &#39;G&#39;, &#39;.&#39;, &#39;A&#39;, &#39;E&#39;, &#39;D&#39;, &#39;A&#39;, &#39;.&#39;, &#39;C&#39;, &#39;G&#39;, &#39;Am&#39;, &#39;Em&#39;, &#39;.&#39;, &#39;F&#39;, &#39;C&#39;, &#39;Dm&#39;, &#39;G&#39;, &#39;.&#39;, &#39;G&#39;, &#39;F&#39;, &#39;C&#39;, &#39;G&#39;, &#39;.&#39;, &#39;F&#39;, &#39;Dm&#39;, &#39;Bb&#39;, &#39;C&#39;, &#39;.&#39;, &#39;C&#39;, &#39;Am&#39;, &#39;G&#39;, &#39;F&#39;, &#39;.&#39;, &#39;E&#39;, &#39;A&#39;, &#39;B7&#39;, &#39;A&#39;, &#39;.&#39;, &#39;C&#39;, &#39;Em&#39;, &#39;G&#39;, &#39;F&#39;, &#39;.&#39;, &#39;F&#39;, &#39;Dm&#39;, &#39;G&#39;, &#39;C&#39;, &#39;.&#39;, &#39;D&#39;, &#39;A&#39;, &#39;G&#39;, &#39;A&#39;, &#39;.&#39;, &#39;Bb&#39;, &#39;F&#39;, &#39;C&#39;, &#39;Dm&#39;, &#39;.&#39;, &#39;E&#39;, &#39;Am&#39;, &#39;C&#39;, &#39;G&#39;, &#39;.&#39;, &#39;C&#39;, &#39;F&#39;, &#39;Am&#39;, &#39;G&#39;, &#39;.&#39;, &#39;Am&#39;, &#39;C&#39;, &#39;G&#39;, &#39;D&#39;, &#39;.&#39;, &#39;G&#39;, &#39;D&#39;, &#39;C&#39;, &#39;D&#39;, &#39;.&#39;, &#39;A&#39;, &#39;F#m&#39;, &#39;D&#39;, &#39;E&#39;, &#39;.&#39;, &#39;C&#39;, &#39;Am&#39;, &#39;F&#39;, &#39;C&#39;, &#39;.&#39;, &#39;E&#39;, &#39;G&#39;, &#39;C&#39;, &#39;G&#39;, &#39;.&#39;, &#39;C&#39;, &#39;E7&#39;, &#39;Am&#39;, &#39;F&#39;, &#39;.&#39;, &#39;D&#39;, &#39;G&#39;, &#39;A&#39;, &#39;Bm&#39;, &#39;.&#39;, &#39;A&#39;, &#39;C#m&#39;, &#39;Bm&#39;, &#39;D&#39;, &#39;.&#39;, &#39;F&#39;, &#39;Bb&#39;, &#39;Dm&#39;, &#39;G&#39;, &#39;.&#39;, &#39;E&#39;, &#39;A&#39;, &#39;C#m&#39;, &#39;F#m&#39;, &#39;.&#39;, &#39;G&#39;, &#39;D&#39;, &#39;C&#39;, &#39;Bm&#39;, &#39;.&#39;, &#39;Am&#39;, &#39;G&#39;, &#39;F&#39;, &#39;G&#39;, &#39;.&#39;, &#39;F&#39;, &#39;C&#39;, &#39;G&#39;, &#39;C&#39;, &#39;.&#39;, &#39;D&#39;, &#39;G&#39;, &#39;A&#39;, &#39;F#m&#39;, &#39;.&#39;, &#39;Bb&#39;, &#39;F&#39;, &#39;G&#39;, &#39;C&#39;, &#39;.&#39;, &#39;A&#39;, &#39;E&#39;, &#39;F#m&#39;, &#39;B&#39;, &#39;.&#39;, &#39;Am&#39;, &#39;G&#39;, &#39;C&#39;, &#39;F&#39;, &#39;.&#39;, &#39;D&#39;, &#39;A&#39;, &#39;E&#39;, &#39;F#m&#39;, &#39;.&#39;, &#39;C&#39;, &#39;F&#39;, &#39;Dm&#39;, &#39;G&#39;, &#39;.&#39;, &#39;G&#39;, &#39;Bm&#39;, &#39;D&#39;, &#39;C&#39;, &#39;.&#39;, &#39;F&#39;, &#39;Dm&#39;, &#39;C&#39;, &#39;Am&#39;, &#39;.&#39;, &#39;E&#39;, &#39;A&#39;, &#39;G&#39;, &#39;D&#39;, &#39;.&#39;, &#39;A&#39;, &#39;E&#39;, &#39;A&#39;, &#39;E&#39;, &#39;.&#39;, &#39;G&#39;, &#39;C&#39;, &#39;G&#39;, &#39;D&#39;, &#39;.&#39;, &#39;Am&#39;, &#39;D&#39;, &#39;F&#39;, &#39;G&#39;, &#39;.&#39;, &#39;C&#39;, &#39;G&#39;, &#39;Bb&#39;, &#39;F&#39;, &#39;.&#39;, &#39;F&#39;, &#39;D&#39;, &#39;G&#39;, &#39;C&#39;, &#39;.&#39;, &#39;E&#39;, &#39;B&#39;, &#39;C#m&#39;, &#39;A&#39;, &#39;.&#39;, &#39;D&#39;, &#39;G&#39;, &#39;A&#39;, &#39;C&#39;, &#39;.&#39;, &#39;Am&#39;, &#39;C&#39;, &#39;G&#39;, &#39;F&#39;, &#39;.&#39;, &#39;C&#39;, &#39;D&#39;, &#39;G&#39;, &#39;C&#39;, &#39;.&#39;, &#39;A&#39;, &#39;E&#39;, &#39;G&#39;, &#39;D&#39;]<br>ENCODED SEQUENCE: [1, 2, 3, 4, 0, 5, 4, 1, 6, 0, 1, 2, 3, 4, 0, 5, 4, 1, 6, 0, 1, 2, 3, 4, 0, 6, 7, 8, 1, 0, 7, 9, 10, 6, 0, 9, 11, 12, 7, 0, 10, 6, 7, 9, 0, 1, 7, 6, 8, 0, 13, 14, 15, 11, 0, 8, 1, 6, 7, 0, 7, 9, 10, 6, 0, 11, 13, 16, 9, 0, 4, 1, 6, 3, 0, 1, 6, 7, 8, 0, 9, 11, 12, 16, 0, 6, 1, 7, 6, 0, 1, 8, 7, 6, 0, 3, 7, 6, 8, 0, 1, 7, 8, 1, 0, 13, 11, 9, 11, 0, 11, 9, 11, 13, 0, 9, 7, 12, 11, 0, 6, 8, 1, 7, 0, 7, 12, 8, 9, 0, 8, 7, 1, 6, 0, 13, 14, 11, 15, 0, 7, 6, 9, 7, 0, 11, 13, 9, 11, 0, 6, 7, 8, 10, 0, 1, 6, 3, 7, 0, 7, 1, 6, 7, 0, 1, 3, 4, 6, 0, 6, 8, 7, 1, 0, 13, 11, 17, 11, 0, 6, 10, 7, 1, 0, 1, 3, 7, 6, 0, 9, 11, 7, 11, 0, 4, 1, 6, 3, 0, 13, 8, 6, 7, 0, 6, 1, 8, 7, 0, 8, 6, 7, 9, 0, 7, 9, 6, 9, 0, 11, 16, 9, 13, 0, 6, 8, 1, 6, 0, 13, 7, 6, 7, 0, 6, 18, 8, 1, 0, 9, 7, 11, 12, 0, 11, 15, 12, 9, 0, 1, 4, 3, 7, 0, 13, 11, 15, 16, 0, 7, 9, 6, 12, 0, 8, 7, 1, 7, 0, 1, 6, 7, 6, 0, 9, 7, 11, 16, 0, 4, 1, 7, 6, 0, 11, 13, 16, 14, 0, 8, 7, 6, 1, 0, 9, 11, 13, 16, 0, 6, 1, 3, 7, 0, 7, 12, 9, 6, 0, 1, 3, 6, 8, 0, 13, 11, 7, 9, 0, 11, 13, 11, 13, 0, 7, 6, 7, 9, 0, 8, 9, 1, 7, 0, 6, 7, 4, 1, 0, 1, 9, 7, 6, 0, 13, 14, 15, 11, 0, 9, 7, 11, 6, 0, 8, 6, 7, 1, 0, 6, 9, 7, 6, 0, 11, 13, 7, 9]<br>DECODED SEQUENCE: [&#39;F&#39;, &#39;A7&#39;, &#39;Dm&#39;, &#39;Bb&#39;, &#39;.&#39;, &#39;Gm&#39;, &#39;Bb&#39;, &#39;F&#39;, &#39;C&#39;, &#39;.&#39;, &#39;F&#39;, &#39;A7&#39;, &#39;Dm&#39;, &#39;Bb&#39;, &#39;.&#39;, &#39;Gm&#39;, &#39;Bb&#39;, &#39;F&#39;, &#39;C&#39;, &#39;.&#39;, &#39;F&#39;, &#39;A7&#39;, &#39;Dm&#39;, &#39;Bb&#39;, &#39;.&#39;, &#39;C&#39;, &#39;G&#39;, &#39;Am&#39;, &#39;F&#39;, &#39;.&#39;, &#39;G&#39;, &#39;D&#39;, &#39;Em&#39;, &#39;C&#39;, &#39;.&#39;, &#39;D&#39;, &#39;A&#39;, &#39;Bm&#39;, &#39;G&#39;, &#39;.&#39;, &#39;Em&#39;, &#39;C&#39;, &#39;G&#39;, &#39;D&#39;, &#39;.&#39;, &#39;F&#39;, &#39;G&#39;, &#39;C&#39;, &#39;Am&#39;, &#39;.&#39;, &#39;E&#39;, &#39;B&#39;, &#39;C#m&#39;, &#39;A&#39;, &#39;.&#39;, &#39;Am&#39;, &#39;F&#39;, &#39;C&#39;, &#39;G&#39;, &#39;.&#39;, &#39;G&#39;, &#39;D&#39;, &#39;Em&#39;, &#39;C&#39;, &#39;.&#39;, &#39;A&#39;, &#39;E&#39;, &#39;F#m&#39;, &#39;D&#39;, &#39;.&#39;, &#39;Bb&#39;, &#39;F&#39;, &#39;C&#39;, &#39;Dm&#39;, &#39;.&#39;, &#39;F&#39;, &#39;C&#39;, &#39;G&#39;, &#39;Am&#39;, &#39;.&#39;, &#39;D&#39;, &#39;A&#39;, &#39;Bm&#39;, &#39;F#m&#39;, &#39;.&#39;, &#39;C&#39;, &#39;F&#39;, &#39;G&#39;, &#39;C&#39;, &#39;.&#39;, &#39;F&#39;, &#39;Am&#39;, &#39;G&#39;, &#39;C&#39;, &#39;.&#39;, &#39;Dm&#39;, &#39;G&#39;, &#39;C&#39;, &#39;Am&#39;, &#39;.&#39;, &#39;F&#39;, &#39;G&#39;, &#39;Am&#39;, &#39;F&#39;, &#39;.&#39;, &#39;E&#39;, &#39;A&#39;, &#39;D&#39;, &#39;A&#39;, &#39;.&#39;, &#39;A&#39;, &#39;D&#39;, &#39;A&#39;, &#39;E&#39;, &#39;.&#39;, &#39;D&#39;, &#39;G&#39;, &#39;Bm&#39;, &#39;A&#39;, &#39;.&#39;, &#39;C&#39;, &#39;Am&#39;, &#39;F&#39;, &#39;G&#39;, &#39;.&#39;, &#39;G&#39;, &#39;Bm&#39;, &#39;Am&#39;, &#39;D&#39;, &#39;.&#39;, &#39;Am&#39;, &#39;G&#39;, &#39;F&#39;, &#39;C&#39;, &#39;.&#39;, &#39;E&#39;, &#39;B&#39;, &#39;A&#39;, &#39;C#m&#39;, &#39;.&#39;, &#39;G&#39;, &#39;C&#39;, &#39;D&#39;, &#39;G&#39;, &#39;.&#39;, &#39;A&#39;, &#39;E&#39;, &#39;D&#39;, &#39;A&#39;, &#39;.&#39;, &#39;C&#39;, &#39;G&#39;, &#39;Am&#39;, &#39;Em&#39;, &#39;.&#39;, &#39;F&#39;, &#39;C&#39;, &#39;Dm&#39;, &#39;G&#39;, &#39;.&#39;, &#39;G&#39;, &#39;F&#39;, &#39;C&#39;, &#39;G&#39;, &#39;.&#39;, &#39;F&#39;, &#39;Dm&#39;, &#39;Bb&#39;, &#39;C&#39;, &#39;.&#39;, &#39;C&#39;, &#39;Am&#39;, &#39;G&#39;, &#39;F&#39;, &#39;.&#39;, &#39;E&#39;, &#39;A&#39;, &#39;B7&#39;, &#39;A&#39;, &#39;.&#39;, &#39;C&#39;, &#39;Em&#39;, &#39;G&#39;, &#39;F&#39;, &#39;.&#39;, &#39;F&#39;, &#39;Dm&#39;, &#39;G&#39;, &#39;C&#39;, &#39;.&#39;, &#39;D&#39;, &#39;A&#39;, &#39;G&#39;, &#39;A&#39;, &#39;.&#39;, &#39;Bb&#39;, &#39;F&#39;, &#39;C&#39;, &#39;Dm&#39;, &#39;.&#39;, &#39;E&#39;, &#39;Am&#39;, &#39;C&#39;, &#39;G&#39;, &#39;.&#39;, &#39;C&#39;, &#39;F&#39;, &#39;Am&#39;, &#39;G&#39;, &#39;.&#39;, &#39;Am&#39;, &#39;C&#39;, &#39;G&#39;, &#39;D&#39;, &#39;.&#39;, &#39;G&#39;, &#39;D&#39;, &#39;C&#39;, &#39;D&#39;, &#39;.&#39;, &#39;A&#39;, &#39;F#m&#39;, &#39;D&#39;, &#39;E&#39;, &#39;.&#39;, &#39;C&#39;, &#39;Am&#39;, &#39;F&#39;, &#39;C&#39;, &#39;.&#39;, &#39;E&#39;, &#39;G&#39;, &#39;C&#39;, &#39;G&#39;, &#39;.&#39;, &#39;C&#39;, &#39;E7&#39;, &#39;Am&#39;, &#39;F&#39;, &#39;.&#39;, &#39;D&#39;, &#39;G&#39;, &#39;A&#39;, &#39;Bm&#39;, &#39;.&#39;, &#39;A&#39;, &#39;C#m&#39;, &#39;Bm&#39;, &#39;D&#39;, &#39;.&#39;, &#39;F&#39;, &#39;Bb&#39;, &#39;Dm&#39;, &#39;G&#39;, &#39;.&#39;, &#39;E&#39;, &#39;A&#39;, &#39;C#m&#39;, &#39;F#m&#39;, &#39;.&#39;, &#39;G&#39;, &#39;D&#39;, &#39;C&#39;, &#39;Bm&#39;, &#39;.&#39;, &#39;Am&#39;, &#39;G&#39;, &#39;F&#39;, &#39;G&#39;, &#39;.&#39;, &#39;F&#39;, &#39;C&#39;, &#39;G&#39;, &#39;C&#39;, &#39;.&#39;, &#39;D&#39;, &#39;G&#39;, &#39;A&#39;, &#39;F#m&#39;, &#39;.&#39;, &#39;Bb&#39;, &#39;F&#39;, &#39;G&#39;, &#39;C&#39;, &#39;.&#39;, &#39;A&#39;, &#39;E&#39;, &#39;F#m&#39;, &#39;B&#39;, &#39;.&#39;, &#39;Am&#39;, &#39;G&#39;, &#39;C&#39;, &#39;F&#39;, &#39;.&#39;, &#39;D&#39;, &#39;A&#39;, &#39;E&#39;, &#39;F#m&#39;, &#39;.&#39;, &#39;C&#39;, &#39;F&#39;, &#39;Dm&#39;, &#39;G&#39;, &#39;.&#39;, &#39;G&#39;, &#39;Bm&#39;, &#39;D&#39;, &#39;C&#39;, &#39;.&#39;, &#39;F&#39;, &#39;Dm&#39;, &#39;C&#39;, &#39;Am&#39;, &#39;.&#39;, &#39;E&#39;, &#39;A&#39;, &#39;G&#39;, &#39;D&#39;, &#39;.&#39;, &#39;A&#39;, &#39;E&#39;, &#39;A&#39;, &#39;E&#39;, &#39;.&#39;, &#39;G&#39;, &#39;C&#39;, &#39;G&#39;, &#39;D&#39;, &#39;.&#39;, &#39;Am&#39;, &#39;D&#39;, &#39;F&#39;, &#39;G&#39;, &#39;.&#39;, &#39;C&#39;, &#39;G&#39;, &#39;Bb&#39;, &#39;F&#39;, &#39;.&#39;, &#39;F&#39;, &#39;D&#39;, &#39;G&#39;, &#39;C&#39;, &#39;.&#39;, &#39;E&#39;, &#39;B&#39;, &#39;C#m&#39;, &#39;A&#39;, &#39;.&#39;, &#39;D&#39;, &#39;G&#39;, &#39;A&#39;, &#39;C&#39;, &#39;.&#39;, &#39;Am&#39;, &#39;C&#39;, &#39;G&#39;, &#39;F&#39;, &#39;.&#39;, &#39;C&#39;, &#39;D&#39;, &#39;G&#39;, &#39;C&#39;, &#39;.&#39;, &#39;A&#39;, &#39;E&#39;, &#39;G&#39;, &#39;D&#39;]</pre><p>Now let us form the dataset from which we extract our train, validation and test subsets during training:</p><pre>block_size = 4<br>data = []<br><br>for i in range(block_size, len(sequence_data)):<br>  data.append(sequence_data[i-block_size:i] + sequence_data[i+1-block_size:i+1])<br><br>dataset = pd.DataFrame(data)<br><br>dataset = dataset.applymap(encoder)</pre><p>Please go through the code above again. Below, the first four columns are our input sequence and the last four columns are our targets, since we are using a sequence length <strong><em>(block_size)</em></strong> of <strong><em>4</em></strong>. Our dataset now looks like this:</p><pre>|     | 0  | 1  | 2  | 3  | 4  | 5  | 6  | 7  |<br>|-----|----|----|----|----|----|----|----|----|<br>| 0   | 1  | 2  | 3  | 4  | 2  | 3  | 4  | 0  |<br>| 1   | 2  | 3  | 4  | 0  | 3  | 4  | 0  | 5  |<br>| 2   | 3  | 4  | 0  | 5  | 4  | 0  | 5  | 4  |<br>| 3   | 4  | 0  | 5  | 4  | 0  | 5  | 4  | 1  |<br>| 4   | 0  | 5  | 4  | 1  | 5  | 4  | 1  | 6  |<br>| ... | ...| ...| ...| ...| ...| ...| ...| ...|<br>| 360 | 6  | 9  | 7  | 6  | 9  | 7  | 6  | 0  |<br>| 361 | 9  | 7  | 6  | 0  | 7  | 6  | 0  | 11 |<br>| 362 | 7  | 6  | 0  | 11 | 6  | 0  | 11 | 13 |<br>| 363 | 6  | 0  | 11 | 13 | 0  | 11 | 13 | 7  |<br>| 364 | 0  | 11 | 13 | 7  | 11 | 13 | 7  | 9  |</pre><p>Then we create a function to get train, validation and test subsets during training.</p><pre>train_ratio, val_ratio, test_ratio = 0.8, 0.1, 0.1<br><br>train = dataset.sample(frac=train_ratio, random_state=42).to_numpy()<br>val = dataset.sample(frac=val_ratio, random_state=42).to_numpy()<br>test = dataset.sample(frac=test_ratio, random_state=42).to_numpy()<br><br>def get_batch(data, batch_size=10, block_size=4):<br>  ix = torch.randint(0, data.shape[0], (batch_size,))<br>  X = torch.tensor(data[ix, :block_size].astype(np.int64), dtype=torch.long)<br>  y = torch.tensor(data[ix, block_size:].astype(np.int64), dtype=torch.int64)<br>  return X, y</pre><p>Okay all well and good, now lets implement the transformer.</p><h4>Parts Of The Transformer Implemented and Explained</h4><p>Ensure you to refer to this <a href="https://en.wikipedia.org/wiki/Transformer_(deep_learning_architecture)#/media/File:Transformer,_full_architecture.png">image</a> as you go through the remainder of this article. Now let’s implement various parts of the transformer. First, take note of the following variables:</p><pre>head_size = 5<br>n_heads = 10<br>block_size = 4<br>vocab_size = len(bpe_result)<br>n_embd = vocab_size<br>batch_size = 120</pre><h4>Attention Head:</h4><p>In the image of the transformer architecture found in the paper, the first component from the bottom is the embedding generation, then the positional encoding, layer-normalization and a masked multi-head attention. Given how these components can be easily implemented in PyTorch, we would like to implement a masked self-attention head, and then form a masked multi-head self-attention as classes so we can easily call them for use. Below, we implement a masked self-attention head using query, key and value layers, as we described in the second part of this article series.</p><pre>class AttnHead(nn.Module):<br><br>  def __init__(self, head_size, n_embd, block_size):<br>    super().__init__()<br>    self.head_size = head_size<br>    self.block_size = block_size<br>    self.n_embd = n_embd<br>    self.query_layer = nn.Linear(n_embd, head_size, bias=False)<br>    self.key_layer = nn.Linear(n_embd, head_size, bias=False)<br>    self.value_layer = nn.Linear(n_embd, head_size, bias=False)<br>    self.register_buffer(&quot;tril&quot;, torch.tril(torch.ones(self.block_size, self.block_size)))<br><br>  def forward(self, x):<br>    query = self.query_layer(x)<br>    key = self.key_layer(x)<br>    value = self.value_layer(x)<br><br>    T = self.block_size<br>    attn_weights_batch = query @ key.transpose(-2, -1) * head_size**-0.5<br>    attn_weights_batch = attn_weights_batch.masked_fill(self.tril[:T, :T]==0, float(&#39;-inf&#39;))<br>    attn_weights_batch = F.softmax(attn_weights_batch, dim=-1)<br><br>    x = attn_weights_batch @ value<br><br>    return x</pre><p>We see we implement three linear layers one for the key, one for the query, and one for the value generation. Rather than project the embedding to the same dimension of the embedding space (<strong><em>19</em></strong>), we project to a smaller space of <strong><em>5 </em></strong>which we refer to as the <strong><em>head_size</em></strong>. Also observe we save the lower-triangular matrix we would use for the masking operation in a <strong><em>register_buffer</em></strong>.</p><p>In the forward method, we multiply the resultant vectors of the query and key and scaling with the inverse square-root of the <strong><em>head_size</em></strong> as we showed in the previous part of this series. We apply the softmax to the resultant to get the attention weights/matrix, and then we project the value vectors on to the attention weights. Next let us see how a multi-head attention is formed.</p><h4><strong>Multi-Head Attention:</strong></h4><p>The code below implements a multi-head attention and how we pass data through it.</p><pre><br># The Multi-Head Attention<br>mthead_attn = nn.ModuleList([AttnHead(head_size, n_embd, block_size) for _ in range(n_heads)])<br><br># passing data through it<br>x = torch.cat([h(x) for h in self.mthead_attn], dim=-1)</pre><p>We had decided the number of multiple (parallel) heads (<strong><em>n_heads</em></strong>) we hope to concatenate to form a multi-head attention. So we make that number of PyTorch module instance of attention heads in what is called a <strong><em>ModuleList</em></strong> offered by PyTorch, and then we pass the data through each, then concatenate them along the last dimension. What does it mean to concatenate along the last dimension, say our data is of size <strong><em>B</em></strong>x<strong><em>T</em></strong>x<strong><em>E, </em></strong>passing it through each attention head we get <strong><em>B</em></strong>x<strong><em>T</em></strong>x<strong><em>h, h </em></strong>being the<strong><em> head_size</em></strong>. Since we have five self-attention heads, for every batch and every time (item of sequence) we concatenate the vector of size<strong><em> h </em></strong>from each attention head into a single vector of size<strong><em> h*5, </em></strong>so after the multi-head attention, we have a vector of size<strong><em> B</em></strong>x<strong><em>T</em></strong>x<strong><em>(h*5).</em></strong></p><h4><strong>FeedForward Network, Residual Connection and Layer Normalization:</strong></h4><p>After, the masked multi-headed self-attention, there is a layer-normalization and a multi-headed cross-attention, but in this case, we are not interested in the relationship between two sequences, so we use just the masked multi-headed self-attention and then proceed to implement the residual connection from the layer-normalization input prior to and its output. Take note that a complete multi-head attention has a linear layer after the concatenation of the attention heads output, to project back to the original embedding space.Let us see a straight forward implementation of this:</p><pre># Layer-Normalization, Multi-Head Attention, and Residual Connection<br>mthead_attn = nn.ModuleList([AttnHead(head_size, n_embd, block_size) for _ in range(n_heads)])<br><br>x_ln = nn.LayerNorm(n_embd)(x)<br>x_mta = torch.cat([h(x_ln) for h in mthead_attn], dim=-1<br>x_mta = nn.Linear(head_size * n_heads, n_embd, bias=False)(x_mta)<br>x = x + x_mta</pre><p>Remember we are skipping the multi-headed cross-attention. Next we implement a layer-normalization from the previous output, then pass the output to a feedforward network, and then make a residual connection from the input to the layer-normalization to the output of the feedforward network.</p><pre># FeedForward Netowrk, Residual Connection, and Layer-Normalization<br><br>ff = nn.Sequential(<br>  nn.Linear(n_embd, n_embd*2, bias=False),<br>  nn.Linear(n_embd*2, n_embd, bias=False)<br>)<br><br>x_ln = nn.LayerNorm(n_heads)(x)<br>x_ff = ff(x_ln)<br>x = x + x_ff</pre><p>Finally we apply layer-normalization to the previous output and then use a linear layer to get our predictions from the resultant embedding.</p><pre># Linear Head<br>linear_layer =  nn.Linear(n_embd, vocab_size, bias=False)<br><br>x = nn.LayerNorm(n_embd)(x)<br>logits = linear_layer(x)</pre><h4>Putting It All Together:</h4><p>Let us put it all together to form the chords generator GPT:</p><pre>class ChordsGPT(nn.Module):<br><br>  def __init__(self, block_size, n_embd, head_size, n_heads, vocab_size):<br>    super().__init__()<br>    self.embd = nn.Embedding(vocab_size, n_embd)<br>    self.pos_embd = nn.Embedding(block_size, n_embd)<br>    self.mthead_attn = nn.ModuleList([AttnHead(head_size, n_embd, block_size) for _ in range(n_heads)])<br>    self.mta_ff = nn.Linear(head_size * n_heads, n_embd, bias=False)<br>    self.ff = nn.Sequential(<br>        nn.Linear(n_embd, n_embd*2, bias=False),<br>        nn.Linear(n_embd*2, n_embd, bias=False)<br>      )<br>    self.linear_layer =  nn.Linear(n_embd, vocab_size, bias=False)<br><br>  def forward(self, x, y=None):<br><br>    # Embedding &amp; Positional Encoding<br>    B, T = x.shape<br>    embd = self.embd(x)<br>    pos_embd = self.pos_embd(torch.arange(T))<br>    x = embd + pos_embd<br><br>    # Layer-Normalization, Multi-Head Attention, and Residual Connection<br>    x_ln = nn.LayerNorm(n_embd)(x)<br>    x_mta = torch.cat([h(x_ln) for h in self.mthead_attn], dim=-1)<br>    x_mta = self.mta_ff(x_mta)<br>    x = x + x_mta<br><br>    # FeedForward Netowrk, Residual Connection, and Layer-Normalization<br>    x_ln = nn.LayerNorm(n_embd)(x)<br>    x_ff = self.ff(x_ln)<br>    x = x + x_ff<br><br>    # Linear Head<br>    x = nn.LayerNorm(n_embd)(x)<br>    logits = self.linear_layer(x)<br><br>    if y is None:<br>      loss = None<br>    else:<br>      B, T, C = logits.size()<br>      logits = logits.view(B*T, C)<br>      y = y.view(B*T)<br>      loss = F.cross_entropy(logits, y)<br><br>    return logits, lossclass ChordsGPT(nn.Module):<br><br>  def __init__(self, block_size, n_embd, head_size, n_heads, vocab_size):<br>    super().__init__()<br>    self.embd = nn.Embedding(vocab_size, n_embd)<br>    self.pos_embd = nn.Embedding(block_size, n_embd)<br>    self.mta_head = nn.ModuleList([AttnHead(head_size, n_embd, block_size) for _ in range(n_heads)])<br>    self.mta_ff = nn.Linear(head_size * n_heads, n_embd, bias=False)<br>    self.linear_layer =  nn.Linear(n_embd, vocab_size, bias=False),<br><br>  def forward(self, x, y=None):<br><br>    # Embedding &amp; Positional Encoding<br>    B, T = x.shape<br>    embd = self.embd(x)<br>    pos_embd = self.pos_embd(torch.arange(T))<br>    x_ = embd + pos_embd<br><br>    # Multi-Head Attention<br>    x = torch.cat([h(x_) for h in self.mta_head], dim=-1)<br>    x = nn.LayerNorm(n_embd)(x_ + self.mta_ff(x))     # RESIDUAL CONNECTION + LAYER NRMALIZATION<br><br>    # Linear Head<br>    logits = self.linear_layer(x)<br><br>    if y is None:<br>      loss = None<br>    else:<br>      B, T, C = logits.size()<br>      logits = logits.view(B*T, C)<br>      y = y.view(B*T)<br>      loss = F.cross_entropy(logits, y)<br><br>    return logits, loss</pre><p>Now let us go through the <strong><em>forward</em></strong> method of the <strong><em>ChordsGPT</em></strong> class to understand the transformer architecture.</p><p>First we get the current embedding of the chords input from the embedding layer, get position-encoding vectors for each item in the sequence, and add them together:</p><pre># Embedding &amp; Positional Encoding<br>B, T = x.shape<br>embd = self.embd(x)<br>pos_embd = self.pos_embd(torch.arange(T))<br>x_ = embd + pos_embd</pre><p>Next we apply layer normalization to our data, passing the result to a multi-headed self-attention with a linear layer, then make a residual connection from the input of layer-normalization to the output of the multi-headed self-attention linear layer:</p><pre># Layer-Normalization, Multi-Head Attention, and Residual Connection<br>x_ln = nn.LayerNorm(n_embd)(x)<br>x_mta = torch.cat([h(x_ln) for h in self.mthead_attn], dim=-1)<br>x_mta = self.mta_ff(x_mta)<br>x = x + x_mta</pre><p>Then we apply layer-normalization to the resultant output, and then a feedforward network with residual connection from the input to the layer-normalization to the output of the feedforward network:</p><pre># FeedForward Netowrk, Residual Connection, and Layer-Normalization<br>x_ln = nn.LayerNorm(n_embd)(x)<br>x_ff = self.ff(x_ln)<br>x = x + x_ff</pre><p>Finally, we apply layer-normalization to the resultant output, then a linear layer in projecting the resultant embedding into a vector of logits. The vector is of the same size as our vocabulary (number of unique chords) to which we can apply softmax to get the probability of each chord as the next chord, and then select the most likely chord.</p><pre># Linear Head<br>x = nn.LayerNorm(n_embd)(x)<br>logits = self.linear_layer(x)</pre><h4>Training Results:</h4><p>Rather than showing the whole code for training the <strong><em>ChordsGPT, </em></strong>we show the results of training and the result of the network generating chord sequences. The notebook for the training is available <a href="https://colab.research.google.com/drive/1PI8xWAMjTutr59ES26gr5dZgDKXeTev7#scrollTo=JxVi9N23Z6W1">here</a>. We end up with a test loss of 0.81 as shown below. We also show the loss curve.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*TASa_OHGB44mQY6Xm7Vf3g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/547/1*d2Fvz2EhypFx2tRU4EaQqg.png" /><figcaption>Loss Curve</figcaption></figure><p>You can also view the same <a href="https://colab.research.google.com/drive/1PI8xWAMjTutr59ES26gr5dZgDKXeTev7#scrollTo=Lxpq_gDby8At">notebook</a> to see the results of training a network without attention, with which we achieve a test loss of 1.11 despite the wide and fairly deep architecture of the network.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d90a9b3f9a0e" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rectlabs/transformers-3-d90a9b3f9a0e">Transformers — 3</a> was originally published in <a href="https://medium.com/rectlabs">Rectlabs Inc</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>