<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by samtoosoon on Medium]]></title>
        <description><![CDATA[Stories by samtoosoon on Medium]]></description>
        <link>https://medium.com/@samtoosoon?source=rss-3b87a8805b71------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*QkWjJnMufz9cAdWvOXbRhw.jpeg</url>
            <title>Stories by samtoosoon on Medium</title>
            <link>https://medium.com/@samtoosoon?source=rss-3b87a8805b71------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 18:36:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@samtoosoon/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[RAG vs. Fine-Tuning: Choosing the Right Approach for Dynamic AI Applications]]></title>
            <link>https://medium.com/@samtoosoon/rag-vs-fine-tuning-choosing-the-right-approach-for-dynamic-ai-applications-d0d5ae201654?source=rss-3b87a8805b71------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0d5ae201654</guid>
            <dc:creator><![CDATA[samtoosoon]]></dc:creator>
            <pubDate>Fri, 23 May 2025 04:54:04 GMT</pubDate>
            <atom:updated>2025-05-23T06:46:01.769Z</atom:updated>
            <content:encoded><![CDATA[<h3><strong>Introduction</strong></h3><p>In today’s fast-paced digital world, particularly in domains like real-time news updates, stock market predictions, or rapidly evolving product information, Artificial Intelligence models must remain agile and current. To achieve this, two primary strategies stand out: Retrieval-Augmented Generation (RAG) and fine-tuning. While both aim to enhance an AI model’s performance and knowledge, they operate fundamentally differently, each with its unique strengths, trade-offs, and ideal use cases. Understanding these distinctions is crucial for deploying effective and efficient AI solutions.</p><h3>What is Retrieval-Augmented Generation (RAG)?</h3><p>Retrieval-Augmented Generation (RAG) is an innovative approach that marries the generative power of large language models (LLMs) with the ability to retrieve up-to-date, external information.</p><blockquote>Think of it as giving an intelligent assistant access to a vast, constantly updated library before it answers your question. Instead of solely relying on its pre-trained knowledge (which can become outdated), the RAG system first searches for relevant documents or data snippets from a dedicated knowledge base.</blockquote><p>Once retrieved, this fresh, contextual information is then fed alongside your query to the language model, which uses both its inherent knowledge and the retrieved data to formulate a more accurate, relevant, and current response.</p><p><strong><em>Why it’s Crucial for Dynamic Data?</em></strong></p><p>The core strength of RAG lies in its ability to handle rapidly changing information. In fields like <strong>news reporting </strong>or <strong>stock market analysis</strong>, where facts and figures can shift by the minute, traditional models struggle to keep up without constant, expensive retraining. RAG elegantly solves this by decoupling the knowledge base from the model itself. When new information becomes available, you simply update the underlying database, not the complex AI model. This makes RAG exceptionally well-suited for applications demanding <strong>real-time accuracy</strong> and access to the very latest facts.</p><p><strong>Typical Workflow:</strong></p><ol><li><strong>User Query:</strong> A user submits a question or a prompt (e.g., “What’s the latest news on stocks today?” or “What were Tesla’s earnings last quarter?”).</li><li><strong>Information Retrieval:</strong> A specialized retrieval component (often powered by vector databases or sophisticated search algorithms) searches a vast, up-to-date external knowledge base (e.g., a news archive, a financial database, internal company documents) for information relevant to the user’s query.</li><li><strong>Contextual Augmentation:</strong> The retrieved documents or snippets are then bundled together with the original user query, forming an augmented prompt.</li><li><strong>Generation:</strong> This augmented prompt is fed into a pre-trained Large Language Model (LLM).8 The LLM processes both its inherent knowledge and the provided external context to generate a comprehensive, accurate, and up-to-date answer.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*suONyhqdeFdBIRqGGbvzjg.png" /><figcaption>Use Case of RAG(In PDF Summarization)</figcaption></figure><p><a href="https://github.com/Samtoosoon/Pdfquestion?tab=readme-ov-file">(Link to code)</a></p><p><strong>Benefits of RAG:</strong></p><ul><li><strong>Always Up-to-Date:</strong> The most significant advantage. New information is integrated by updating the retrieval database, not by retraining the LLM, ensuring responses are current.</li><li><strong>Reduced “Hallucination”:</strong> By grounding responses in retrieved facts, RAG significantly lessens the tendency of LLMs to generate plausible-sounding but incorrect or fabricated information.</li><li><strong>Cost-Effective for Dynamic Data:</strong> Avoids the high computational and time costs associated with frequent model retraining for new information.</li><li><strong>Explainability:</strong> In some RAG implementations, you can trace the answer back to the retrieved sources, providing a degree of transparency and verifiability.</li><li><strong>Versatile:</strong> Excellent for question-answering systems, dynamic content generation, summarization of new articles, and real-time data analysis.</li></ul><p><strong>Limitations of RAG:</strong></p><ul><li><strong>Retrieval Quality is Key:</strong> The effectiveness of RAG heavily depends on the quality and relevance of the retrieved documents. Poor retrieval can lead to poor answers.</li><li><strong>Latency:</strong> The retrieval step adds a slight latency to the overall response time, though this is often negligible for most applications.</li><li><strong>Coverage:</strong> If the information isn’t in your retrieval database, RAG cannot provide it.</li></ul><h3>What is Fine-Tuning?</h3><p>Fine-tuning involves taking a pre-trained language model (which has already learned a vast amount of general knowledge and language patterns from a diverse dataset) and further training it on a smaller, task-specific, or domain-specific dataset.This process “adjusts” the model’s internal parameters (weights) to better understand and generate text aligned with the nuances, jargon, style, or specific tasks of the new data.</p><blockquote>Think of it as teaching a brilliant generalist to become an expert in a very specific field.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/1*w3P8gJwq1IbFS2H6RridiA.png" /></figure><p>Traditional Fine-Tuning:</p><p>In its original form, fine-tuning involves updating all, or a significant portion, of a pre-trained model’s parameters.For models with hundreds of millions or even billions of parameters, this is a computationally intensive and resource-demanding process. It requires substantial GPU power, memory, and time, making it less practical for frequent updates or for users with limited hardware.The output is a new, specialized version of the entire model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2noaa68o61AEmfhgKBfqpA.jpeg" /><figcaption>Comparing models(Traditional, pre-trained, fine-tuned on custom dataset)</figcaption></figure><p><strong>Challenges of Traditional Fine-Tuning:</strong></p><ul><li><strong>High Compute Requirements:</strong> Demands significant computational resources (GPUs) and memory.</li><li><strong>Time-Consuming:</strong> Can take hours or days for large models and datasets.</li><li><strong>Costly:</strong> High operational costs due to compute usage.</li><li><strong>Storage:</strong> Each fine-tuned version is a full copy of the model, requiring substantial storage.</li></ul><h4>Adapters and LoRA: The Era of Parameter-Efficient Fine-Tuning (PEFT)</h4><p>Recognizing the challenges of full fine-tuning, researchers developed Parameter-Efficient Fine-Tuning (PEFT) methods, with <strong>Adapters</strong> and <strong>LoRA (Low-Rank Adaptation)</strong> being two prominent examples.22 These techniques significantly reduce the computational burden and storage requirements, making fine-tuning more accessible and practical.</p><p><strong>1. Adapters:</strong></p><ul><li><strong>Mechanism:</strong> Adapters introduce small, new, trainable layers (often just a few million parameters) into the pre-trained model’s existing architecture.During fine-tuning, the original, vast majority of the model’s parameters are “frozen” (i.e., they are not updated). Only these newly added, small adapter layers are trained.</li><li><strong>Analogy:</strong> Imagine a complex machine where you only need to change a few specific dials to adjust its output for a new task, rather than re-engineering the entire machine.</li><li><strong>Benefits:</strong> Substantially reduces the number of parameters to be trained, leading to faster training times and lower memory consumption compared to full fine-tuning.Different adapter layers can be swapped in and out for different tasks, without changing the base model.</li><li><strong>Consideration:</strong> Can introduce a slight increase in inference latency due to the additional layers during model execution.</li></ul><p><strong>2. LoRA (Low-Rank Adaptation):</strong></p><ul><li><strong>Mechanism:</strong> LoRA is an even more ingenious approach. Instead of adding new layers, LoRA injects small, low-rank matrices directly into the existing layers of the pre-trained model. When fine-tuning, only the parameters of these small low-rank matrices are updated, while the vast majority of the original model’s weights remain frozen. For instance, sometimes only 0.2% of the total model parameters are trained with LoRA, making it incredibly efficient.</li><li><strong>Analogy:</strong> It’s like subtly “nudging” a few key internal components of a large engine to slightly adjust its performance characteristics, rather than rebuilding the engine or bolting on new parts.</li></ul><p><strong>Benefits:</strong></p><ul><li><strong>Extremely Efficient:</strong> Achieves performance comparable to or even superior to full fine-tuning while training a minuscule fraction of parameters (e.g., 0.2% — 0.3%).</li><li><strong>Minimal Memory Footprint:</strong> Can often be performed on modest hardware (e.g., a single consumer-grade GPU).</li><li><strong>Faster Training:</strong> Significantly reduces training time.</li><li><strong>No Inference Latency:</strong> Unlike some adapter approaches, LoRA often incurs no additional inference latency because the trained low-rank matrices can be merged with the original model weights for deployment.</li><li><strong>Modular Deployment:</strong> Allows for multiple task-specific LoRA modules to be associated with a single base model, enabling easy switching between different specialized capabilities without loading entirely different models.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UnZoHrrk29B0kWnP3BYakw.jpeg" /><figcaption>Low Rank Adapters(Only adapter weights are updated)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*egMDe99GG_UqNl-9xLW0bw.jpeg" /><figcaption>Three scenarios compared (Without Adapters, With Traditional Adapters, LoRA, QLoRA)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CNtd11_6_AwbFtlJ_JIxmg.png" /></figure><h3>QLoRA Quantization: Floating Point Example, Bit Reduction, and BERT Case Study</h3><h3>1. Floating Point Quantization in QLoRA</h3><ul><li><strong>Standard Model Storage:</strong><br> Neural networks typically use 32-bit floating point (FP32) numbers to store weights. This allows for high precision but uses significant memory and compute resources.</li></ul><p><strong>Reducing Precision:</strong><br> To save memory and speed up training/inference, weights can be quantized to lower precisions:</p><ul><li><strong>FP32 (32 bits):</strong> Highest precision, largest memory.</li><li><strong>FP16/BFloat16 (16 bits):</strong> Good balance of speed and accuracy.</li><li><strong>INT8 (8 bits):</strong> Further reduction, some loss in accuracy.</li><li><strong>NF4 (4 bits, NormalFloat4):</strong> Extreme reduction, using only 16 possible values per weight.</li></ul><h4><strong>Why 16-bit is a Sweet Spot:</strong></h4><ul><li>16-bit (FP16 or BFloat16) is widely used because it offers a strong trade-off: nearly the same accuracy as FP32 but much faster and less memory-intensive.</li><li>Going to 8-bit or 4-bit (as in QLoRA) can introduce more quantization error, but with special techniques like NF4, accuracy loss can be minimized.</li></ul><h3>2. How QLoRA Quantizes Weights</h3><ul><li><strong>NF4 (NormalFloat4) Quantization:</strong></li><li>Uses quantile quantization: instead of dividing the range [-1, 1] into equally spaced bins, it creates bins so that each contains an equal number of weights (ideal for normally distributed data).</li><li>For 4 bits, there are 2⁴ = 16 bins.</li><li>This preserves more information where the weights are densest (around zero), reducing quantization error.</li></ul><p><strong>Double Quantization:</strong></p><ul><li>After quantizing weights to 4-bit, QLoRA further quantizes the scale/offset constants (used for dequantization) to 8-bit, saving even more memory</li></ul><p><strong>Computation:</strong></p><ul><li>During training, weights are dequantized to 16-bit (BFloat16 or FP16) for matrix multiplications, ensuring training remains accurate.</li></ul><h3>3. LoRA Formula and Parameter Count</h3><ul><li><strong>LoRA Parameter Formula:</strong></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ySHb1WfuP6YRNd1f8ZKtEQ.jpeg" /><figcaption>Weight Updation in Matrices (2 low rank matrices used)</figcaption></figure><ul><li><strong>Example with BERT:</strong></li><li>BERT-base has 345 million parameters (FP32).</li><li>After quantization (e.g., 4-bit), memory drops by 8x (since 32/4=832/4 = 832/4=8), so the storage is equivalent to ~43 million FP32 parameters.</li><li>LoRA adapters add a small number of trainable parameters (e.g., if r=8r=8, and applied to several layers, total LoRA parameters can be around 12 million or less, depending on configuration).</li></ul><h3>4. BERT(Bidirectional encoder representations from transformers) Quantization Example</h3><h3><strong>Original:</strong></h3><ul><li>BERT-base: 345 million parameters, all in FP32 (32-bit floats).</li></ul><p><strong>After QLoRA Quantization:</strong></p><ul><li>All weights quantized to 4-bit NF4.</li><li>Storage requirement drops to about 1/8th.</li><li>LoRA adapters (kept in 16-bit for training accuracy) add a small number of parameters.</li></ul><p><strong>Result:</strong></p><ul><li><strong>Total memory use:</strong> Dramatically reduced.</li><li><strong>Trainable parameters:</strong> Only LoRA adapters are updated (e.g., ~12 million), not the full 345 million.</li><li><strong>Accuracy:</strong> With careful quantization (NF4), accuracy remains close to full-precision models.</li></ul><h3>Key Takeaways:</h3><ul><li>QLoRA enables fine-tuning of massive models (like BERT) on consumer hardware by quantizing weights to 4 bits, using advanced quantization (NF4) to preserve accuracy.</li><li>LoRA adapters are kept in higher precision (16-bit) and are the only trainable parameters, making training efficient.</li><li>For BERT, quantization can reduce storage from 345M FP32 params to the equivalent of ~12M, with minimal accuracy loss.</li></ul><h4><strong>When to Use RAG vs. Fine-Tuning?</strong></h4><p>Choosing between RAG and fine-tuning (especially with LoRA/Adapters) boils down to the specific characteristics of your data, the desired outcome, and your operational constraints.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*9toDoXQxNbqUT9A7QZw1KQ.png" /></figure><ul><li><strong>For news, stock, or any scenario where factual data changes frequently and real-time accuracy is paramount, RAG is the superior choice.</strong> You update your database, and your AI is immediately current, avoiding the need for costly and time-consuming model retraining cycles.</li><li><strong>For applications requiring deep, stable, domain-specific expertise, a particular tone, or complex reasoning that doesn’t change daily, fine-tuning (especially with LoRA or adapters) is incredibly efficient and effective.</strong>This makes the model “smarter” in that specific niche. Examples include internal company knowledge bases, specialized customer support bots, or legal document summarization.</li></ul><h3>Benchmarks and Findings</h3><p>Recent research and practical implementations consistently highlight the strengths of both RAG and PEFT methods:</p><ul><li><strong>LoRA vs. Full Fine-Tuning:</strong> Numerous studies and benchmarks, demonstrate that LoRA often achieves comparable or even superior performance to full fine-tuning across a variety of tasks (e.g., classification, summarization, question answering). This is achieved while training an incredibly small fraction of the model’s parameters (typically 0.2–0.3%), making it a game-changer for accessibility and efficiency.</li><li><strong>Adapters:</strong> Similar to LoRA, adapters provide strong results with minimal compute compared to full fine-tuning. Their main trade-off can be a slight increase in inference latency due to the additional layers that need to be processed during prediction.</li><li><strong>RAG Performance:</strong> The efficacy of RAG systems is directly tied to the quality of the retrieval system and the freshness of the underlying knowledge base.When the retrieval is accurate and the data is up-to-date, RAG excels at providing current and factual information, outperforming static fine-tuned models on dynamic tasks.</li></ul><h3>Distillation: A Brief Note on Efficiency</h3><p>Beyond RAG and fine-tuning, <strong>knowledge distillation</strong> is another powerful technique focused on model efficiency.In distillation, a large, complex, and highly accurate model (often called the “teacher” model) is used to train a smaller, more efficient model (the “student” model).The student model learns to mimic the outputs and behaviors of the teacher, thereby inheriting much of the teacher’s accuracy and capabilities but at a fraction of the size and computational cost.</p><p><strong>Example: DeepSeek-Coder-V2-Lite for Local Machine Deployment</strong></p><p>A prime example of a distilled model designed for efficiency and local deployment is <strong>DeepSeek-Coder-V2-Lite</strong>. This model is a smaller, more efficient version of the larger DeepSeek-Coder-V2, which is known for its strong coding capabilities.</p><p>While the original DeepSeek-Coder-V2 might require powerful GPUs and significant RAM, the “Lite” version is specifically engineered to be more resource-friendly. This makes it a perfect candidate for running on consumer-grade hardware, such as a laptop with a modern GPU (even one with 8GB or 12GB of VRAM) or even just a CPU, albeit with slower inference.</p><p><strong>How DeepSeek-Coder-V2-Lite exemplifies distillation for local use:</strong></p><ol><li><strong>Teacher Model:</strong> The massive, highly performant <strong>DeepSeek-Coder-V2</strong> acts as the teacher. It possesses extensive knowledge of coding languages, syntax, and programming logic.</li><li><strong>Student Model:</strong> <strong>DeepSeek-Coder-V2-Lite</strong> is the student. Through distillation, it has learned to reproduce the coding abilities of its larger counterpart, but with far fewer parameters.</li><li><strong>Local Deployment:</strong> Because of its reduced size, DeepSeek-Coder-V2-Lite can be downloaded and run locally using popular open-source inference frameworks like:</li></ol><ul><li><strong>Ollama:</strong> A user-friendly tool that allows you to easily run large language models on your local machine. You simply download the model (e.g., ollama run deepseek-coder-v2-lite) and can start interacting with it via a command line or API.</li><li><strong>Transformers (Hugging Face):</strong> Developers can load the DeepSeek-Coder-V2-Lite model directly using the Hugging Face transformers library. This allows for fine-grained control over inference and integration into custom applications.</li><li><strong>GGML/GGUF:</strong> Often, these smaller, distilled models are quantized (a process that reduces the precision of the model’s weights to save memory) into formats like GGML or GGUF. These quantized versions can run on even more constrained hardware, including CPUs, making them highly accessible for local development and personal use.</li></ul><p><strong>Practical Application with DeepSeek-Coder-V2-Lite:</strong></p><p>Imagine a developer who needs an AI assistant for code completion, bug fixing, or generating small code snippets, but doesn’t want to rely on cloud APIs for privacy or cost reasons. By running DeepSeek-Coder-V2-Lite locally:</p><ul><li>They can get fast responses without internet latency.</li><li>Their code remains on their machine, enhancing data privacy.</li><li>They incur no API costs.</li></ul><p>This ability to run powerful, albeit distilled, models locally opens up new possibilities for personal AI assistants, offline applications, and development environments where data sovereignty or cost-efficiency are critical.</p><p><strong>When is Distillation Useful?</strong></p><ul><li><strong>Resource-Constrained Environments:</strong> Ideal for deploying AI on edge devices, mobile phones, or applications with strict latency requirements where a large LLM would be impractical.</li><li><strong>Cost Reduction:</strong> Lowers inference costs significantly due to the smaller model size.</li><li><strong>Production Deployment:</strong> Enables the deployment of highly capable models in real-world scenarios where efficiency is paramount.</li></ul><h3>Conclusion</h3><p>The choice between RAG and fine-tuning is not mutually exclusive; they are often complementary.</p><ul><li><strong>RAG is the champion for dynamic, frequently updated knowledge scenarios</strong> such as real-time news feeds, stock market analysis, or constantly evolving product FAQs.Its strength lies in ensuring information freshness without incurring retraining costs.</li><li><strong>Fine-tuning with LoRA or adapters is the optimal approach for embedding stable, domain-specific expertise efficiently.</strong> It allows models to internalize specific patterns, jargon, and nuances of a particular field, making them highly specialized and accurate within that domain.</li><li><strong>Distillation serves as a vital step for deploying highly efficient, smaller models</strong> that can perform well in resource-constrained production environments.</li></ul><p>In practice, a robust AI strategy often combines these methods: one might <strong>fine-tune</strong> a base language model for their specific industry domain (e.g., finance, healthcare) to embed core expertise. Then, <strong>RAG</strong> is layered on top to provide real-time updates and access to the very latest factual information.Finally, if deployment on resource-limited devices is necessary, the combined system or a specialized part of it might be <strong>distilled</strong> into a smaller, more efficient model.</p><p><strong>Tip:</strong> Always start by defining your primary use case and, critically, the <strong>frequency of information updates</strong>. If your data changes hourly or daily, lean heavily towards RAG. If you need deep, stable expertise that doesn’t change often, leverage LoRA or adapters for efficient fine-tuning.By strategically applying these techniques, organizations can build AI systems that are both powerful and practical.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0d5ae201654" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Seeing Images Through the Eyes of a Transformer: A Patch-by-Patch Perspective]]></title>
            <link>https://medium.com/@samtoosoon/seeing-images-through-the-eyes-of-a-transformer-a-patch-by-patch-perspective-481e13c8e574?source=rss-3b87a8805b71------2</link>
            <guid isPermaLink="false">https://medium.com/p/481e13c8e574</guid>
            <dc:creator><![CDATA[samtoosoon]]></dc:creator>
            <pubDate>Sun, 11 May 2025 17:45:34 GMT</pubDate>
            <atom:updated>2025-05-11T18:15:54.606Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Reffering the iconic research paper </em><a href="https://arxiv.org/pdf/2010.11929"><em>“</em></a><em>AN IMAGE IS WORTH 16X16 WORDS</em><a href="https://arxiv.org/pdf/2010.11929"><em>” (Dosovitskiy et al., 2021)</em></a><em>.</em></p><p>This was the intial flow of basic CNN(Convolutional Neural Network):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/754/1*I4CZwhbLnc82onuViMfWFg.png" /><figcaption>Basic CNN Architecture for Image Classification.</figcaption></figure><blockquote>The success of CNNs in image tasks paved the way for Vision Transformers (ViTs) by demonstrating the power of hierarchical feature learning, inspiring researchers to explore the application of the Transformer architecture, originally designed for sequence data, to image recognition by treating image patches as sequences.</blockquote><p>Imagine taking an <strong>image</strong>, say one that’s <strong>224 pixels</strong> wide and <strong>224 pixels</strong> tall (that’s a standard size many models work with). Now, instead of looking at the whole picture at once, what if we broke it down into smaller, equal-sized squares? That’s essentially what a <strong>Vision Transformer (ViT)</strong> does.</p><p>Think of it like tiling our <strong>224x224 image</strong> with little <strong>16x16 pixel squares</strong>. If you do the math, you’ll find you end up with (224/16) * (224/16) = 14 * 14 = 196 of these little <strong>patches</strong>.</p><p>Now, how do these <strong>patches</strong> move across the <strong>image</strong>? In this case, we’re using a “<strong>stride</strong>” of <strong>16 pixels</strong>. What that means in simple terms is that each new <strong>patch</strong> we pick doesn’t <strong>overlap</strong> with the previous one at all. It’s like placing the tiles perfectly next to each other.</p><p>Here’s where things get interesting. Each of these <strong>2D patches</strong>, which is a little <strong>16x16 grid of pixel information</strong>, is then treated as a separate “<strong>input token</strong>.” To make it easier for our <strong>transformer</strong> to process, we <strong>flatten</strong> each of these <strong>2D squares</strong> into a <strong>1D vector</strong>. So, a <strong>16x16 patch</strong> becomes a <strong>vector</strong> of 16 * 16 = <strong>256 numbers</strong> (or “<strong>tokens</strong>”).</p><p>Let’s take a simpler example to solidify this. If we had a smaller <strong>32x32 image</strong> and still used <strong>16x16 patches</strong> with a <strong>stride</strong> of <strong>16</strong>, we’d get just four distinct <strong>patches</strong> with no <strong>overlap</strong>. Each of these <strong>2D patches</strong> would then be <strong>flattened</strong> into a <strong>1D vector</strong> of <strong>256 elements</strong>.</p><p>Now that we have these <strong>flattened 1D vectors</strong>, we need to feed them into the <strong>transformer</strong>. The first step is a “<strong>linear projection</strong>.” Think of this as a way to transform each of these <strong>256-dimensional vectors</strong> into a <strong>lower-dimensional representation</strong>, say a <strong>128-dimensional vector</strong>.</p><p>This <strong>linear projection</strong> is actually quite similar to what happens in the <strong>convolutional layers of a CNN</strong>. It involves multiplying the <strong>input</strong> (our <strong>flattened patch vector</strong>) by a <strong>weight matrix</strong> and then adding a <strong>bias</strong>. This process is <strong>learned during training</strong> and helps the model <strong>extract the most important features</strong> from each <strong>patch</strong> while <strong>reducing the overall dimensionality</strong>. This <strong>reduction</strong> has several benefits, like using less <strong>memory</strong> and speeding up <strong>computations</strong>. It also helps the model focus on the <strong>essential features</strong> and discard some of the less important details or <strong>noise</strong>.</p><p>After the <strong>linear projection</strong>, we introduce “<strong>positional encoding</strong>.” Why is this important? Because the <strong>transformer</strong>, in its raw form, doesn’t inherently know the <strong>spatial arrangement</strong> of the original <strong>patches</strong> in the <strong>image</strong>. It just sees a sequence of <strong>1D vectors</strong>. <strong>Positional encoding</strong> adds information about the <strong>location</strong> of each <strong>patch</strong> in the original <strong>image</strong>. It tells the <strong>transformer</strong>, “Hey, this <strong>patch</strong> was in the top-left corner, and this one was next to it,” and so on.</p><p>Next comes the heart of the <strong>transformer</strong>: “<strong>self-attention</strong>.” This is a powerful mechanism that allows each <strong>patch</strong> to look at all the other <strong>patches</strong> in the <strong>image</strong> and figure out which ones are most <strong>relevant</strong> to it. It helps the model understand the <strong>relationships</strong> and <strong>dependencies</strong> between different parts of the <strong>image</strong>, capturing both <strong>local details</strong> and the overall <strong>global context</strong>.</p><p>Following the <strong>self-attention layer</strong>, each <strong>patch’s representation</strong> goes through a “<strong>feed-forward network</strong>.” This is a standard <strong>neural network</strong> that helps to capture more <strong>complex, non-linear relationships</strong> within the information learned from the <strong>self-attention</strong> mechanism.</p><p>Finally, for tasks like <strong>image classification</strong>, we have a “<strong>classification head</strong>,” which is often a simple <strong>Multi-Layer Perceptron (MLP)</strong>. This head takes the output from the <strong>transformer</strong> and maps it to the desired output format, like the different <strong>classes</strong> of objects in an <strong>image</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/888/1*1UfEv4e4kxSo55TYt9rGZw.png" /><figcaption><strong>Workflow of ViT</strong></figcaption></figure><p>Here’s a key difference between a <strong>Vision Transformer</strong> and the traditional <strong>Transformers</strong> used in <strong>Natural Language Processing (NLP)</strong>: the <strong>ViT</strong> typically only has an <strong>encoder</strong>. In <strong>NLP</strong>, tasks like <strong>translation</strong> or <strong>text generation</strong> require a <strong>decoder</strong> to generate output sequences based on the input. However, for understanding the <strong>spatial relationships</strong> within an <strong>image</strong> — which is the main goal of a <strong>ViT</strong> — the <strong>encoder</strong>, with its <strong>self-attention mechanism</strong>, is sufficient to capture the necessary <strong>local</strong> and <strong>global dependencies</strong> between the <strong>image patches</strong>. So, while <strong>NLP Transformers</strong> often have both an <strong>encoder</strong> and a <strong>decoder</strong>, the <strong>Vision Transformer</strong> focuses on the power of the <strong>encoder</strong> to learn directly from the <strong>image patches</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/930/1*Qzrkx5oxPdhVh_hvVj0NEA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/973/1*U-PCqyIetO2bgdr2UVtBTg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*PQd603vewYGLR7c1c38PDQ.png" /></figure><p><a href="https://arxiv.org/pdf/2010.11929"><strong>Challenges &amp; Limitations</strong></a></p><p>~Computational Cost: ViT-Huge requires 632M parameters . ~Interpretability: Hard to visualize why self-attention works .</p><p>~Deployment: Less efficient than CNNs on edge devices .</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=481e13c8e574" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>