<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Pratik on Medium]]></title>
        <description><![CDATA[Stories by Pratik on Medium]]></description>
        <link>https://medium.com/@pratik.vyas_10544?source=rss-d183e2545ed0------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*dmbNkD5D-u45r44go_cf0g.png</url>
            <title>Stories by Pratik on Medium</title>
            <link>https://medium.com/@pratik.vyas_10544?source=rss-d183e2545ed0------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 21:41:57 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@pratik.vyas_10544/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Architectural Evolution of Large Language Models in 2025]]></title>
            <link>https://medium.com/@pratik.vyas_10544/the-architectural-evolution-of-large-language-models-in-2025-fb01f111e2d7?source=rss-d183e2545ed0------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb01f111e2d7</guid>
            <category><![CDATA[mamba]]></category>
            <category><![CDATA[sms]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[mixture-of-experts]]></category>
            <category><![CDATA[gqa]]></category>
            <dc:creator><![CDATA[Pratik]]></dc:creator>
            <pubDate>Fri, 15 May 2026 17:31:43 GMT</pubDate>
            <atom:updated>2026-05-15T17:31:43.441Z</atom:updated>
            <content:encoded><![CDATA[<p>Since the introduction of the “Vanilla Transformer” in 2017, Large Language Models (LLMs) have undergone a radical transformation. As of early 2025, the industry paradigm has shifted away from simply scaling up dense parameters. Today’s frontier models — such as GPT-4, Mixtral, and DeepSeek-V3 — prioritize conditional computation, hardware-aware attention mechanisms, state-space hybrids, and inference-time reasoning. This article provides a comprehensive technical step-by-step breakdown of the state-of-the-art LLM architectures and techniques defining the current era of artificial intelligence.</p><h4><strong>What makes a 2025 LLM “Modern”?</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/830/1*GxcHrdHLQFo1qTK4ELoRjQ.jpeg" /></figure><h3>1. The Paradigm Shift to Sparse Activation: Mixture-of-Experts (MoE)</h3><p>For years, the standard approach to scaling LLMs was building larger “Dense” architectures, where every parameter is activated for every forward pass. The logic was brute-force but effective: more parameters meant more capacity for learning complex distributions. However, dense scaling faces severe diminishing returns regarding compute costs and memory bandwidth constraints. The modern solution to this scaling wall is the <strong>Mixture-of-Experts (MoE)</strong> architecture, which embraces sparse activation.</p><h4>Technical Mechanics: Routing and Specialization</h4><p>Instead of a monolithic feed-forward network (FFN) at each transformer block, an MoE layer contains multiple specialized sub-networks called “experts.” The critical component is the trainable routing mechanism — typically a softmax gating network — that evaluates each incoming token’s representation and dynamically computes a probability distribution over the available experts. The token is then routed only to the top-k experts (often k=1 or k=2 in modern implementations).</p><p>If we conceptualize a dense model as a generalist attempting to process every input through a single cognitive pathway, an MoE is a vast library of specialists. For example, a model might possess 1 trillion total parameters (its “sparse” parameter count), but the router only activates 20 billion parameters (its “active” parameter count) per token. The experts themselves are standard FFNs, but because they only see a subset of the training data, they implicitly specialize. One expert might become highly tuned for parsing Python syntax, while another becomes attuned to nuanced sentiment in French literature.</p><h4>Load Balancing and Training Dynamics</h4><p>Training MoE models introduces unique challenges. The most prominent is the risk of “representation collapse” or “router imbalance.” If the router initially favors a few experts (due to random initialization), those experts receive more gradient updates, become “smarter,” and are subsequently favored even more by the router. This creates a feedback loop where a small number of experts handle all the traffic while the rest atrophy, effectively reducing the model back to a smaller dense model.</p><p>To counter this, modern MoE architectures incorporate auxiliary load-balancing loss functions during training. These penalties force the router to distribute tokens more evenly across all available experts, ensuring the entire capacity of the model is utilized. Furthermore, advanced implementations like DeepSeek-V3 utilize “expert parallelism,” where different experts are distributed across different GPUs or even different nodes in a cluster, enabling training at an unprecedented scale.</p><h4>Why It’s a Breakthrough</h4><p>This sparse activation fundamentally decouples a model’s total knowledge capacity (parameter count) from its computational cost (FLOPs per token). It allows researchers to scale the model’s capacity drastically without a proportional increase in inference latency or active memory footprint. The result is a model that demonstrates the intelligence and nuanced understanding of a massive parameter space while maintaining the operational speed and cost-efficiency of a much smaller one.</p><h3>2. Evolving Attention: Tackling the Quadratic Bottleneck</h3><p>The original Self-Attention mechanism is notoriously “Quadratic” ($O(N²)$) with respect to sequence length. If sequence length doubles, the computational operations and memory requirements quadruple. This fundamental mathematical property was the primary barrier to processing long documents or maintaining extensive conversational histories. Two major innovations have become standard in 2025 to mitigate this bottleneck.</p><h4>Grouped-Query Attention (GQA): Optimizing the KV Cache</h4><p>During autoregressive generation, standard Multi-Head Attention (MHA) requires storing a massive Key-Value (KV) Cache for every token previously generated. As context windows grow to hundreds of thousands of tokens, this cache consumes enormous amounts of VRAM, quickly leading to memory bandwidth bottlenecks rather than compute limits. The GPU spends more time waiting for the KV cache to load from memory than it does actually performing calculations.</p><p><strong>Grouped-Query Attention (GQA)</strong>, popularized by models like Llama 3, acts as an optimal middle ground between the heavy MHA and the overly simplified Multi-Query Attention (MQA). In MQA, all query heads share a single Key and Value head, which saves massive amounts of memory but severely degrades the model’s ability to represent complex relationships.</p><p>In GQA, multiple “Query” heads are grouped together to share a single “Key” and “Value” projection. For instance, instead of 32 distinct Keys and Values for 32 Queries (as in MHA) or 1 Key/Value (as in MQA), there might be 8 Keys and Values shared among the 32 Queries (groups of 4). This architectural tweak drastically reduces the size of the KV Cache and the associated memory bandwidth requirements for loading it during generation. It facilitates significantly longer context windows without crippling inference speed, and empirical evidence shows it maintains nearly identical performance to full MHA.</p><h4>FlashAttention-3: Hardware-Aware Mathematics</h4><p>While GQA is an architectural change, <strong>FlashAttention-3</strong> is a crucial software-level, hardware-aware optimization. It fundamentally rethinks how the attention equation is calculated based on the physical constraints of GPU memory hierarchy.</p><p>Traditional attention requires repeatedly reading and writing large intermediate attention matrices ($Q \times K^T$) to the GPU’s High Bandwidth Memory (HBM). HBM is relatively slow compared to the GPU’s processing cores. FlashAttention recognizes that the GPU has a small pool of extremely fast memory located right next to the compute cores called SRAM.</p><p>FlashAttention utilizes a technique called “tiling” to compute exact attention in the ultra-fast SRAM. It breaks the large matrices into smaller blocks (tiles), loads them into SRAM, computes the attention score, applies the softmax function, and writes only the final output back to HBM. This prevents the GPU from constantly moving massive amounts of intermediate data back and forth. By dramatically reducing these memory reads/writes (smashing through the “memory wall”), FlashAttention-3 yields a 2x–4x speedup in both training and inference, making it an indispensable component of any modern LLM stack.</p><h3>3. Advanced Positional Encoding: Rotary Positional Embeddings (RoPE)</h3><p>Transformers are inherently permutation-invariant; without positional information, the model treats “The dog bit the man” exactly the same as “The man bit the dog.” The attention mechanism sees words as an unordered set. Early models relied on absolute positional embeddings (e.g., adding a fixed, learned vector or a sinusoidal curve for “Position 1”, “Position 2”). This method struggled significantly when models encountered sequences longer than those seen during training because the model didn’t know how to interpret “Position 2049” if it was only trained up to “Position 2048.”</p><p>Modern architectures universally utilize <strong>Rotary Positional Embeddings (RoPE)</strong> to solve this context length generalization problem.</p><h4>The Mechanics of Rotation in Complex Space</h4><p>RoPE encodes positional information not by addition, but by rotating the token embeddings in a complex plane. Instead of adding a fixed position vector, RoPE applies a rotation matrix where the angle of rotation is proportional to the token’s position in the sequence.</p><p>When the model computes the attention score (the dot product) between a Query at position $m$ and a Key at position $n$, the mathematical properties of RoPE ensure that the result is dependent <em>only</em> on the relative distance ($m — n$) between the tokens, rather than their absolute positions ($m$ or $n$). The attention score between “dog” at position 5 and “bit” at position 6 is the same as if they were at positions 500 and 501.</p><h4>Length Extrapolation and Infinite Context</h4><p>Because RoPE inherently encodes <em>relative distance</em>, it provides a critical mathematical property: <strong>length extrapolation</strong>. It allows models (like the latest iterations of Gemini with multi-million-token windows) to process massive contexts effectively. The model learns how tokens interact based on distance, so it can reliably understand the relationship between two tokens separated by 10,000 words, even if it was primarily trained on sequences of only 4,000 words.</p><p>This shift from absolute to relative encoding is what made the “long context revolution” possible. It transforms the model’s understanding from “where am I in the document?” to “how far away is this relevant piece of information?”</p><h3>4. Breaking the Transformer Monopoly: SSMs and Hybrid Architectures</h3><p>While Transformers dominate the landscape, their fundamental reliance on the KV cache for looking back at previous tokens creates an unavoidable “context bottleneck.” As sequences grow into the millions of tokens, the KV cache grows proportionally, inevitably slowing down generation and consuming terabytes of VRAM. A new paradigm has emerged to challenge this dominance: <strong>State Space Models (SSMs)</strong>, most notably architectures based on <strong>Mamba</strong>.</p><h4>The Mamba Solution: Hardware-Aware Linear Time</h4><p>Mamba solves the quadratic attention problem by moving away from explicit attention entirely. Instead, it processes sequences via a selective “hidden state” — conceptually similar to a continuous stream of consciousness or a highly advanced Recurrent Neural Network (RNN).</p><p>However, unlike older RNNs which were notoriously slow to train, Mamba is designed to be highly parallelizable during training using parallel scans and hardware-aware optimizations. As each new word arrives during inference, the internal state updates selectively. The model’s parameters dictate what information to prioritize and remember, and what to forget, based on the current context.</p><p>Because it compresses the past into a fixed-size state matrix, its inference computational complexity is <strong>Linear</strong> (O(N)) with respect to sequence length, and its memory footprint is constant (O(1)). It does not slow down, nor does it consume exponentially more memory, as the conversation history grows.</p><h4>Hybrid Architectures (e.g., Jamba)</h4><p>Despite their efficiency, pure SSMs face challenges. By compressing all history into a fixed-size state, they inherently lose some information. They can sometimes struggle with tasks requiring deep, complex associative recall over short contexts compared to the brute-force effectiveness of Transformer attention, where every past token is explicitly accessible.</p><p>The cutting-edge solution in 2025 is the <strong>Hybrid Architecture</strong>. Models like Jamba interleave standard Transformer Attention layers with Mamba layers. The Transformer layers handle rigorous, deep reasoning and exact recall over immediate contexts (perhaps processing the most recent 4,000 tokens), while the Mamba layers act as fast, efficient long-term memory processors for the millions of tokens that came before. By combining these, developers are achieving compute-optimal architectures that boast the best of both worlds: the high performance and accurate recall of attention alongside the linear scaling efficiency and massive context windows of state space models.</p><blockquote><strong>Check Appendix for more details : How State Space Models (SSMs), Mamba works</strong></blockquote><h3>5. Post-Training Evolution: DPO and Inference-Time Scaling</h3><p>The fundamental architecture discussed above is merely the “brain”; the alignment and post-training processes are what dictate how that brain operates, how it follows instructions, and how it interacts safely with users. The industry is rapidly moving away from the complex, multi-model pipelines that defined the early generative AI era.</p><h4>Direct Preference Optimization (DPO)</h4><p>The standard approach to alignment was Reinforcement Learning from Human Feedback (RLHF). This required a convoluted pipeline: first training a separate “Reward Model” to score how well an output matched human preferences, and then using complex reinforcement learning algorithms like PPO (Proximal Policy Optimization) to update the main LLM to maximize those scores. This process was notoriously difficult to tune, mathematically unstable, and required loading multiple massive models into memory simultaneously.</p><p><strong>DPO</strong> bypasses this entirely. By utilizing a clever mathematical shortcut derived from the RLHF objective, DPO proves that the optimal policy can be extracted directly without needing an explicit reward model. It treats the LLM <em>itself</em> as the implicit reward model. It directly updates the model’s weights based on pairs of human-preferred and rejected responses using a simple classification loss (cross-entropy). This direct approach is significantly faster, requires drastically less VRAM, is highly mathematically stable, and is far less prone to the “mode collapse” or “reward hacking” that plagued PPO.</p><h4>Inference-Time Scaling (The “o1” Paradigm)</h4><p>Perhaps the most defining trend of 2025 is the shift from “System 1” (instinctive, immediate) to “System 2” (deliberative, analytical) thinking via <strong>Inference-Time Scaling</strong>.</p><p>Previously, scaling laws only applied to training: more parameters and more training data yielded a smarter model. However, inference was always $O(1)$ in terms of “thought.” The model simply predicted the next token instantly based on surface-level patterns.</p><p>Models are now trained using sophisticated Reinforcement Learning techniques (often building on DPO or customized RL algorithms) to generate a hidden “Chain of Thought.” When presented with a complex math problem or coding task, they do not answer immediately. They are taught to search through different reasoning paths (often utilizing Monte Carlo Tree Search or similar search algorithms). They verify intermediate steps, critique their own logic, backtrack from dead ends, and self-correct mistakes <em>before</em> emitting the final answer to the user.</p><p>This paradigm proves a crucial hypothesis: scaling compute <em>during inference</em> (allowing the model to “think” longer and explore more paths) can yield massive leaps in reasoning capabilities that scaling parameters alone cannot achieve. A smaller model allowed to think for 30 seconds can often outperform a much larger model forced to answer instantly.</p><h3>6. The Edge Revolution: Quantization and Speculative Decoding</h3><p>While massive cloud models push the boundaries of reasoning, there is an equally strong push to deploy highly capable models efficiently in production environments and on local edge devices (laptops, phones, robotics). Two critical optimization techniques have become the industry standard for democratizing access to LLMs and dramatically lowering serving costs.</p><h4>1. Quantization (FP8/INT4 and Beyond)</h4><p>Models are typically trained using 16-bit or 32-bit floating-point precision (FP16/FP32), where every parameter is stored as a highly precise decimal number. <strong>Quantization</strong> is the process of reducing the precision of the model’s weights and/or activations to 8-bit (FP8/INT8) or even 4-bit (INT4) formats.</p><p>By mathematically compressing these values — akin to lowering the bit-depth of an audio file or reducing the color palette of an image — the model consumes dramatically less memory. A model that requires 32GB of VRAM in FP16 might run on just 8GB in INT4.</p><p>The challenge with quantization is that naive rounding introduces significant errors, degrading the model’s intelligence. Advanced quantization techniques (like AWQ, GPTQ, or GGUF) utilize calibration datasets to understand which weights are most important. They preserve the precision of critical “outlier” weights while aggressively quantizing the rest. This ensures that this massive compression introduces minimal degradation in output quality, making local deployment of powerful open-weights models viable on consumer hardware.</p><h4>2. Speculative Decoding: Breaking the Autoregressive Bottleneck</h4><p>LLM generation is fundamentally memory-bandwidth bound because it is autoregressive; it must generate one token at a time, reading the entire massive model weight matrix from memory into the compute cores for every single token generated.</p><p><strong>Speculative Decoding</strong> solves this inefficiency by pairing a massive, slow “Target” model (e.g., a 70B parameter model) with a tiny, lightning-fast “Draft” model (e.g., a 1B parameter model).</p><p>The draft model quickly guesses the next $K$ tokens (for example, the next 5 words). Because it is so small, it does this almost instantly. The large target model then takes those 5 guessed tokens and processes all of them in a <em>single</em> forward pass. Because modern GPUs are highly parallelized, evaluating 5 tokens simultaneously takes nearly the exact same amount of time as evaluating 1 token.</p><p>If the target model agrees with the draft model’s sequence (based on the probabilities it calculates), all 5 tokens are accepted instantly. If it disagrees at token 3, it accepts the first two, corrects the third, discards the rest, and the process repeats. This clever bypass of the autoregressive bottleneck often yields a 2x–3x speedup in generation time with absolutely zero mathematical degradation in output quality, as the final output is guaranteed to be drawn from the exact same distribution as the target model alone.</p><h3>Summary Checklist: What makes a 2025 LLM “Modern”?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*driL_3hq67Nh1P5C34yh9Q.jpeg" /></figure><p><strong>Conclusion:</strong></p><p>The era of simply stacking dense Transformer layers is over. The LLMs of 2025 are intricate, highly optimized engines. By leveraging sparse activation for parameter efficiency, hardware-aware mathematics for speed, hybrid state-space architectures for infinite context, and deliberate, deliberative reasoning paradigms, modern AI achieves unprecedented intelligence at a fraction of the expected computational cost. We have moved from brute force to elegant engineering, and the focus has shifted firmly from just building bigger brains to teaching them how to think.</p><h3>Appendix: <strong>How State Space Models (SSMs), Mamba works.</strong></h3><p>To understand State Space Models (SSMs) in detail, we have to look at how they bridge the gap between two older technologies: RNNs (which are fast but forgetful) and CNNs (which are good at patterns but fixed in size).</p><p>Here is the step-by-step breakdown of how a modern SSM (like Mamba) actually works.</p><p><strong>Step 1:</strong> The Origin (Control Theory)</p><p>SSMs come from classical engineering used to track rockets or planes. The core idea is that you have:</p><p>1. An Input (x): What is happening now (e.g., the current word in a sentence).<br>2. A Hidden State (h): The “memory” of everything that happened before.<br>3. An Output (y): The prediction of what comes next.</p><p>Mathematically, it looks like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/242/1*aEE0av3z_36UB6jlt1CxwA.png" /></figure><p>The Goal: Use Matrix ‘A’ to compress the past, and Matrix ‘B’ to decide how much of the new input to let in.</p><p><strong>Step 2:</strong> Discretization (Making it work for computers)</p><p>Computers don’t see “continuous flows” (like time); they see “discrete steps” (like words in a list). To make SSMs work for AI, we use a parameter called Delta.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/555/1*Y-fsLIIxCL7kYqbzyM23Ug.png" /></figure><p>This process converts continuous equations into a <strong>Recurrence</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/172/1*JMOa07X-Iige2ImxxTM0xA.png" /></figure><p>This looks exactly like an RNN. You take the old memory (h{t-1}), update it with the new word (x_t), and get a new memory (h_t).</p><p><strong>Step 3:</strong> The “Dual View” (The Secret Trick)</p><p>The reason Transformers replaced RNNs is that Transformers can be trained in parallel (all words at once), while RNNs have to wait for Word 1 to finish before starting Word 2.</p><p>SSMs solve this by having a Dual View:</p><p>1. <strong>Recurrent View:</strong> Used during Inference (chatting). It processes one token at a time, making it incredibly fast and memory-efficient.<br>2. <strong>Convolutional View:</strong> Used during Training. Through a mathematical trick, the entire sequence can be expanded into a “Convolution” (like a giant filter), allowing the GPU to train on all words simultaneously.</p><p>Summary so far: SSMs are fast like RNNs during chat, but fast like CNNs during training.</p><p><strong>Step 4:</strong> The “Selection” Breakthrough (The Mamba Innovation)</p><p>Traditional SSMs had a major flaw: The Matrices (A, B, C) were fixed. They<br>applied the same math to every word regardless of what the word was.</p><p>If you ask an old SSM to summarize a book, it treats the word “the” with the<br>same importance as the name of the “murderer.” It eventually runs out of “memory space” because it fills its hidden state with junk.</p><p>Mamba introduced “Selective SSMs” (S6):</p><ul><li>In Mamba, B, C, and Delta are functions of the input.</li><li>The model “sees” the word. If the word is important (e.g., “Murderer”), the model opens the gate (B) wide to store it in memory.</li><li>If the word is “the,” the model ignores it.</li></ul><p>This allows the model to compress information much more effectively than<br>previous versions.</p><p><strong>Step 5:</strong> Hardware-Aware Computation (Parallel Scan)</p><p>The biggest technical hurdle for Mamba was: If the math changes for every word (Selectivity), you can no longer use the Convolution trick to train in parallel.</p><p>To fix this, Mamba uses a technique called the Parallel Associative Scan.<br>Instead of processing words 1 to 2 to 3 to 4 (Linear), it does this:</p><p>1. Combines (1+2) and (3+4) in parallel.<br>2. Then combines those results.<br>3. This reduces the time complexity from O(N) to O(log N).</p><p>Mamba also uses Kernel Fusion, meaning it does all these complex calculations inside the GPU’s fastest memory (SRAM) without constantly writing back to the slower main memory (DRAM).</p><p><strong>Final Takeaway</strong></p><p>The State Space Model is essentially a “Smart Compressor.” While a Transformer keeps every single word in a “massive digital photo album” (KV Cache) to look at later, the SSM builds a “mental summary” (Hidden State) as it reads. Mamba is the first version of this technology that is smart enough to compete with Transformers in reasoning while being much faster at handling millions of words of context.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb01f111e2d7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Explainable AI (XAI)]]></title>
            <link>https://medium.com/@pratik.vyas_10544/explainable-ai-xai-3c3ebf51bc42?source=rss-d183e2545ed0------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c3ebf51bc42</guid>
            <category><![CDATA[xai]]></category>
            <category><![CDATA[explainable-ai]]></category>
            <category><![CDATA[shap]]></category>
            <dc:creator><![CDATA[Pratik]]></dc:creator>
            <pubDate>Tue, 15 Jul 2025 07:13:35 GMT</pubDate>
            <atom:updated>2025-07-15T07:22:51.348Z</atom:updated>
            <content:encoded><![CDATA[<h3>Abstract</h3><p>As artificial intelligence (AI) and machine learning (ML) models become increasingly integrated into critical decision-making processes, their inherent complexity often renders them “black boxes.” Explainable AI (XAI) has emerged as a critical discipline dedicated to demystifying these models, providing transparency, and fostering trust. This article provides a comprehensive overview of XAI, establishing its importance through the evolution of modeling. It introduces a formal taxonomy for classifying XAI methods based on their scope, methodology, and usage, details key techniques within this framework, and identifies the core stakeholders, principles, and challenges in the field.</p><p><strong>Index</strong></p><ol><li>Explainable AI (XAI)</li><li>XAI Methods</li><li>Explanation Technique</li><li>The Stakeholders and Principles of Explanation</li><li>The Significance and Imperative of XAI</li><li>Limitations and Future Challenges</li><li>Conclusion and Practical Resources</li></ol><h3>1. Explainable AI</h3><p>Explainable AI (XAI) is a collection of frameworks, tools, and techniques designed to help human users understand and interpret the outputs of machine learning models. It bridges the gap between a model’s predictive power and a human’s ability to comprehend its inner workings, enabling developers to debug models, improve performance, and build trust among stakeholders.</p><p>The need for XAI is a direct result of the evolution of predictive modeling. This journey can be viewed in three distinct phases:</p><p><strong>A) Classical Statistical Models</strong>: <br>In traditional modeling (e.g., linear regression), significant effort was dedicated to domain understanding and exploratory data analysis. The models were inherently interpretable, and validation relied on statistical measures like goodness-of-fit and hypothesis testing.</p><p><strong>B)</strong> <strong>Mainstream Machine Learning</strong>: <br>The rise of big data and computational power shifted the focus to developing highly flexible, complex models (e.g., Random Forest, Gradient Boosting). Here, the primary goal became predictive performance, validated through techniques like cross-validation. This led to the “black box” problem, where models delivered accurate predictions without clear reasoning.</p><p><strong>C) Human-Centered AI: </strong><br>We are now entering an era where model performance alone is insufficient. The emphasis is shifting towards model validation and justification, prioritizing XAI, fairness, ethics, and responsible AI. This ensures models are not only accurate but also transparent and equitable, a shift accelerated by regulations like the EU’s General Data Protection Regulation (GDPR) and its “right to explanation.”</p><h3>2. XAI Methods</h3><p>To navigate the landscape of XAI, it is essential to classify methods along three core dimensions: their Scope, Methodology, and Usage. This taxonomy provides a structured framework for selecting the appropriate technique for a given problem.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4E1GsfS_TVB78NceEVI_rw.png" /><figcaption><a href="https://arxiv.org/pdf/2006.11371.pdf">https://arxiv.org/pdf/2006.11371.pdf</a></figcaption></figure><h4>2.1 Scope: Where the Explanation is Focused</h4><p>This dimension defines the granularity of the explanation.</p><ul><li><strong>Local Scope</strong>: Focuses on explaining an individual prediction for a specific data instance. The goal is to understand why the model made a particular decision for that single case. For example, why was this specific customer’s loan application denied?</li><li><strong>Global Scope</strong>: Aims to understand the behavior of the model as a whole. It describes general trends and feature influences across the entire dataset, providing a holistic view of what the model has learned.</li></ul><h4>2.2 Usage: How the Explanation is Implemented</h4><p>This describes how the XAI method relates to the machine learning model itself.</p><ul><li><strong>Intrinsic:</strong> Explainability is built directly into the model’s architecture. These “white-box” models (e.g., Linear Regression, Decision Trees, Rule-based systems) are transparent by design. Their internal logic is simple enough for direct human inspection.</li><li><strong>Post-Hoc: </strong>The explanation technique is applied <em>after</em> a model has been trained. These methods are model-agnostic or designed for a specific class of models, allowing them to interpret complex, pre-existing “black boxes” like neural networks or ensemble models.</li></ul><h4>2.3 Methodology: The Algorithmic Approach</h4><p>This defines the core computational technique used to generate the explanation.</p><ul><li><strong>Backpropagation-based (Gradient-based)</strong>: Primarily used for differentiable models like neural networks, this approach relies on backpropagating gradients from the model’s output back to the input features. The magnitude of the gradient indicates how sensitive the output is to a change in each feature.</li><li><strong>Perturbation-based</strong>: This model-agnostic approach works by systematically altering or removing features from an input instance and observing how the model’s prediction changes. Features that cause the largest shifts in prediction are considered the most important.</li></ul><h3>3. Explanation Techniques</h3><p>With the taxonomy as our guide, we can now categorize the most common XAI techniques.</p><p><strong>1.Surrogate Models</strong>: <br>This post-hoc technique trains a simple, intrinsic model (like a decision tree) to mimic the predictions of a complex black-box model. By interpreting the surrogate, we gain global or local insights into the original model.</p><p><strong>2. Feature Attribution (e.g., SHAP, LIME, Integrated Gradients):</strong> <br>These are post-hoc methods that assign an importance score to each feature.</p><ul><li><strong>LIME (Local Interpretable Model-agnostic Explanations)</strong>: <br>A classic local, perturbation-based method that builds a simple linear model around a single prediction to explain it.</li><li><strong>SHAP (SHapley Additive exPlanations): </strong><br>A unified framework grounded in game theory. It offers both local and global explanations and typically uses a perturbation-based methodology (KernelSHAP) for model-agnostic use.</li><li><strong>Integrated Gradients</strong>: <br>A local technique for differentiable models that uses a backpropagation-based methodology to attribute predictions to input features.</li></ul><p><strong>3. Counterfactual and Contrastive Explanations:</strong> <br>These post-hoc, local methods explain a prediction by showing what needs to change.</p><ul><li>Counterfactuals identify the minimal change to an input that would alter the outcome (e.g., “Your loan would have been approved if your income was $5,000 higher”).</li><li>Contrastive explanations highlight the features that were critical for the given outcome (“pertinent positives”) and those whose absence was critical (“pertinent negatives”).</li></ul><p><strong>4. Example-Based Explanations:</strong> <br>These <strong>post-hoc</strong> methods explain a <strong>local </strong>prediction by finding similar examples in the training set that led to the same outcome, which is particularly intuitive in fields like medicine or law.</p><h3>4. The Stakeholders and Principles of Explanation</h3><p>To be effective, XAI must be tailored to its audience and purpose. This involves answering three core questions: Whom to explain, What to explain, and How to explain.</p><h4>4.1 Whom to Explain? (The Target Audience)</h4><ul><li><strong>Data Scientists &amp; Developers</strong>: Need detailed, local and global explanations to debug, validate, and improve models.</li><li><strong>Business Leaders &amp; Domain Experts</strong>: Require high-level, global explanations that connect model behavior to business outcomes.</li><li><strong>Regulators &amp; Auditors</strong>: Need evidence of fairness and compliance, often through global explanations of feature importance and bias checks.</li><li><strong>End-Users &amp; Affected Parties</strong>: Deserve clear, non-technical local explanations (like counterfactuals) for decisions that impact them.</li></ul><h4>4.2 What to Explain? (The Principles of Explanation)</h4><ol><li><strong>Explanation</strong>: The system must be able to provide reasoning for its outputs.</li><li><strong>Meaningful</strong>: The explanation must be understandable to its intended audience.</li><li><strong>Explanation Accuracy</strong>: The explanation must faithfully reflect the model’s true internal reasoning.</li><li><strong>Knowledge Limits</strong>: The system should declare when it is operating outside its area of expertise or on an input for which its confidence is low.</li></ol><h3>5. The Significance and Imperative of XAI</h3><p>XAI is a cornerstone of responsible AI and delivers tangible benefits:</p><ul><li><strong>Improves Trust and Transparency:</strong> Justifiable decisions foster user trust and facilitate adoption.</li><li><strong>Enhances Model Robustness</strong>: By revealing feature dependencies, XAI helps identify vulnerabilities.</li><li><strong>Promotes Fairness and Mitigates Bias</strong>: XAI can uncover reliance on sensitive or proxy variables, enabling teams to build more ethical systems.</li><li><strong>Ensures Accountability and Compliance</strong>: XAI provides the necessary audit trail for regulated industries.</li></ul><h3>6. Limitations and Future Challenges</h3><p>Despite its promise, XAI faces several significant challenges:</p><ul><li><strong>Fidelity-Interpretability Trade-off</strong>: Post-hoc explanations are approximations. A highly faithful explanation may be too complex to interpret, while a simple one may not be accurate.</li><li><strong>Vulnerability to Adversarial Attacks</strong>: Explanation methods themselves can be manipulated, potentially hiding a model’s unfair or malicious behavior.</li><li><strong>The Gap with Human Cognition</strong>: Human explanations are often contrastive, selective, and social. XAI methods typically provide exhaustive, quantitative lists of factors that may not align with human intuition.</li><li><strong>The AutoML Challenge</strong>: As AutoML creates increasingly complex models with minimal human intervention, developing XAI techniques that can keep pace is a critical frontier.</li></ul><h3>7. Conclusion and Practical Resources</h3><p>Explainable AI is transitioning from a “nice-to-have” feature to a fundamental requirement for the ethical and effective deployment of AI. It empowers organizations to build models that are not only powerful but also transparent, fair, and trustworthy.</p><p>For practitioners looking to implement XAI, several open-source libraries are invaluable:</p><ul><li><strong>SHAP</strong>: A powerful, model-agnostic library for both local and global feature attribution.</li><li><strong>LIME</strong>: An intuitive library for explaining individual predictions of any black-box model.</li><li><strong>ELI5 (“Explain Like I’m 5”):</strong> A library supporting multiple frameworks that provides intuitive model explanations.</li></ul><p>As AI continues to evolve, the principles and practices of XAI will be indispensable in ensuring that this powerful technology is developed and deployed for the benefit of all.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c3ebf51bc42" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“GenAI on Prod”:Observability for Generative AI on Kubernetess (Part-4)]]></title>
            <link>https://medium.com/@pratik.vyas_10544/genai-on-prod-observability-for-generative-ai-on-kubernetess-part-4-32d3e29cfc21?source=rss-d183e2545ed0------2</link>
            <guid isPermaLink="false">https://medium.com/p/32d3e29cfc21</guid>
            <category><![CDATA[kubernetes-observability]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[observability]]></category>
            <dc:creator><![CDATA[Pratik]]></dc:creator>
            <pubDate>Thu, 03 Jul 2025 17:26:47 GMT</pubDate>
            <atom:updated>2025-07-07T14:54:41.441Z</atom:updated>
            <content:encoded><![CDATA[<p>Read <a href="https://medium.com/p/e9b1469b95cb">Part-1 Deploying</a><br>Read <a href="https://medium.com/p/8678bfacfded">Part-2 Scaling</a><br>Read <a href="https://medium.com/@pratik.vyas_10544/genai-on-prod-mastering-gpu-efficiency-optimizing-genai-workloads-on-kubernetes-part-3-48b0c9e124b2">Part-3 Mastering GPU Efficiency</a></p><h3>Observability for Generative AI on Kubernetess</h3><p>Generative AI introduces a new frontier of applications, but running them in production on Kubernetes brings unique observability challenges. It’s not just about CPU and memory anymore. We need to understand token consumption, model latency, prompt/response quality, and specialized hardware like GPUs.</p><p>As shown below image, We’ll walk through the “what,” “why,” and “how” of setting up a robust observability stack, step-by-step, with code and commands you can use directly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OfLnf_Z1ynvLGU5trUINJQ.png" /></figure><h3>The Goal: A Unified View</h3><p>Our goal is to build a single pane of glass in Grafana where we can correlate:</p><ul><li><strong>Logs</strong>: What happened during a specific request? (e.g., the exact prompt, errors).</li><li><strong>Metrics</strong>: How is the system performing? (e.g., request latency, tokens per second, GPU utilization).</li><li><strong>Traces</strong>: Where did the time go? (e.g., how much time was spent in the model vs. pre-processing).</li></ul><h3>The Core Stack</h3><p>We will use the industry-standard, cloud-native observability stack:</p><ul><li><strong>Prometheus</strong>: For scraping and storing time-series metrics.</li><li><strong>Grafana</strong>: For visualizing everything in beautiful, interactive dashboards.</li><li><strong>Loki</strong>: For collecting and querying logs.</li><li><strong>OpenTelemetry</strong>: For generating and collecting traces.</li><li><strong>NVIDIA DCGM Exporter</strong>: For exposing detailed GPU metrics.</li></ul><h3>Step 1: The Sample GenAI Application</h3><p>First, we need an application to observe. Let’s create a simple Python Flask app that uses a pre-trained model to generate text.</p><p><strong>app.py</strong></p><p>This app will have a single endpoint /generate and will be instrumented for metrics and tracing.</p><pre>from flask import Flask, request, jsonify<br>from transformers import pipeline<br>import time<br>import logging<br>from prometheus_client import Counter, Histogram, generate_latest<br>from opentelemetry import trace<br>from opentelemetry.sdk.trace import TracerProvider<br>from opentelemetry.sdk.trace.export import ConsoleSpanExporter, BatchSpanProcessor<br><br># --- Basic Setup ---<br>app = Flask(__name__)<br>logging.basicConfig(level=logging.INFO)<br># --- Model Loading ---<br># Using a smaller model for demonstration purposes<br>generator = pipeline(&#39;text-generation&#39;, model=&#39;gpt2&#39;)<br># --- OpenTelemetry Tracing Setup ---<br>provider = TracerProvider()<br>provider.add_span_processor(BatchSpanProcessor(ConsoleSpanExporter())) # For now, just print traces to console<br>trace.set_tracer_provider(provider)<br>tracer = trace.get_tracer(__name__)<br># --- Prometheus Metrics Setup ---<br>REQUEST_LATENCY = Histogram(&#39;genai_request_latency_seconds&#39;, &#39;Latency of requests to the /generate endpoint&#39;)<br>REQUEST_COUNT = Counter(&#39;genai_requests_total&#39;, &#39;Total number of requests to the /generate endpoint&#39;, [&#39;model&#39;])<br>TOKENS_GENERATED = Counter(&#39;genai_tokens_generated_total&#39;, &#39;Total number of tokens generated&#39;, [&#39;model&#39;])<br># --- Flask Routes ---<br>@app.route(&#39;/generate&#39;, methods=[&#39;POST&#39;])<br>def generate():<br>    with tracer.start_as_current_span(&quot;generate-request&quot;) as parent_span:<br>        start_time = time.time()<br>        json_data = request.get_json()<br>        prompt = json_data.get(&#39;prompt&#39;)<br>        if not prompt:<br>            return jsonify({&quot;error&quot;: &quot;Prompt not provided&quot;}), 400<br>        # Log the incoming prompt<br>        logging.info(f&quot;Received prompt: {prompt}&quot;)<br>        parent_span.set_attribute(&quot;genai.prompt&quot;, prompt)<br>        # --- Model Inference ---<br>        with tracer.start_as_current_span(&quot;model-inference&quot;) as inference_span:<br>            try:<br>                outputs = generator(prompt, max_length=50, num_return_sequences=1)<br>                response_text = outputs[0][&#39;generated_text&#39;]<br>                num_tokens = len(response_text.split()) # Simple token counting<br>            except Exception as e:<br>                logging.error(f&quot;Model inference failed: {e}&quot;)<br>                return jsonify({&quot;error&quot;: &quot;Model inference failed&quot;}), 500<br>        # --- Record Metrics ---<br>        latency = time.time() - start_time<br>        REQUEST_LATENCY.observe(latency)<br>        REQUEST_COUNT.labels(model=&#39;gpt2&#39;).inc()<br>        TOKENS_GENERATED.labels(model=&#39;gpt2&#39;).inc(num_tokens)<br>        # --- Set Span Attributes ---<br>        parent_span.set_attribute(&quot;genai.response&quot;, response_text)<br>        parent_span.set_attribute(&quot;genai.tokens.generated&quot;, num_tokens)<br>        parent_span.set_attribute(&quot;genai.latency.seconds&quot;, latency)<br>        return jsonify({&quot;response&quot;: response_text})<br>@app.route(&#39;/metrics&#39;)<br>def metrics():<br>    return generate_latest()<br>if __name__ == &#39;__main__&#39;:<br>    app.run(host=&#39;0.0.0.0&#39;, port=5000)</pre><p><strong>requirements.txt</strong></p><pre>flask<br>transformers<br>torch<br>prometheus-client<br>opentelemetry-api<br>opentelemetry-sdk</pre><p><strong>Dockerfile</strong></p><pre>FROM python:3.9-slim<br>WORKDIR /app<br>COPY requirements.txt .<br>RUN pip install --no-cache-dir -r requirements.txt<br>COPY app.py .<br>CMD [&quot;python&quot;, &quot;app.py&quot;]</pre><h3>Step 2: Setting Up the Observability Stack in Kubernetes</h3><p>The easiest way to install Prometheus, Grafana, and Loki is with Helm.</p><pre># Add the necessary Helm repositories<br>helm repo add prometheus-community https://prometheus-community.github.io/helm-charts<br>helm repo add grafana https://grafana.github.io/helm-charts<br>helm repo update<br><br># Install the stack (kube-prometheus-stack includes Prometheus and Grafana)<br>helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace<br>helm install loki grafana/loki-stack --namespace monitoring --create-namespace</pre><p>This command sets up a powerful monitoring foundation in the monitoring namespace.</p><h3>Step 3: Deploying the Application and Exposing Metrics</h3><p>Now, let’s deploy our GenAI app to Kubernetes and tell Prometheus how to find its metrics.</p><p><strong>deployment.yaml</strong></p><p>This file defines our application’s Deployment and a Service to expose it internally. Note the annotations prometheus.io/scrape: &#39;true&#39; which is a simple way to tell Prometheus to scrape this pod.</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: genai-app<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: genai-app<br>  template:<br>    metadata:<br>      labels:<br>        app: genai-app<br>      annotations:<br>        # Tell Prometheus to scrape metrics from this pod<br>        prometheus.io/scrape: &#39;true&#39;<br>        prometheus.io/path:   /metrics<br>        prometheus.io/port:   &#39;5000&#39;<br>    spec:<br>      containers:<br>      - name: genai-app<br>        # Replace with your actual image registry/name<br>        image: your-repo/genai-app:latest<br>        ports:<br>        - containerPort: 5000<br>---<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: genai-app-service<br>spec:<br>  selector:<br>    app: genai-app<br>  ports:<br>    - protocol: TCP<br>      port: 5000<br>      targetPort: 5000</pre><p><strong>Deploy it:</strong></p><pre># First, build and push your Docker image<br># docker build -t your-repo/genai-app:latest .<br># docker push your-repo/genai-app:latest<br><br># Then, apply the Kubernetes configuration<br>kubectl apply -f deployment.yaml</pre><p>Your app is now running, and Prometheus will automatically start scraping the /metrics endpoint.</p><h3>Step 4: Visualizing Application Metrics in Grafana</h3><ol><li><strong>Access Grafana:</strong></li></ol><pre>kubectl port-forward svc/prometheus-grafana 3000:80 -n monitoring</pre><p>Open http://localhost:3000 in your browser. The default credentials are admin / prom-operator.</p><p>2. <strong>Create a Dashboard:</strong></p><ul><li>Go to Dashboards -&gt; New -&gt; New Dashboard.</li><li>Click Add visualization.</li><li>Select Prometheus as the data source.</li><li>In the “Metrics browser” query field, you can now find custom metrics:</li></ul><ol><li>To see the requests per second: <br>rate(genai_requests_total[5m])</li><li>To see the 95th percentile request latency :<br>histogram_quantile(0.95,sum(rate(genai_request_latency_seconds_bucket[ 5m])) by (le))</li><li>To see the rate of token generation:<br> sum(rate(genai_tokens_generated_total[5m])) by (model)</li></ol><p>Build panels for each of these queries to create your first GenAI application dashboard.</p><h3>Step 5: Capturing GPU Metrics (For GPU-enabled nodes)</h3><p>If your GenAI app runs on nodes with NVIDIA GPUs, you <em>must</em> monitor them. The dcgm-exporter is the standard tool for this.</p><p><strong>dcgm-exporter.yaml</strong></p><p>This deploys the exporter as a DaemonSet, meaning one instance will run on every node in your cluster.</p><pre>apiVersion: apps/v1<br>kind: DaemonSet<br>metadata:<br>  name: dcgm-exporter<br>  namespace: monitoring<br>spec:<br>  selector:<br>    matchLabels:<br>      name: dcgm-exporter<br>  template:<br>    metadata:<br>      labels:<br>        name: dcgm-exporter<br>    spec:<br>      containers:<br>      - image: nvcr.io/nvidia/k8s/dcgm-exporter:3.1.3-3.1.2-ubuntu20.04<br>        name: dcgm-exporter<br>        ports:<br>        - name: metrics<br>          containerPort: 9400<br>          hostPort: 9400</pre><p><strong>Deploy it:</strong></p><pre>kubectl apply -f dcgm-exporter.yaml</pre><p>Now, Prometheus will automatically discover these exporters. In Grafana, you can query for metrics like:</p><ul><li>DCGM_FI_DEV_GPU_UTIL: GPU Utilization (%).</li><li>DCGM_FI_DEV_FB_USED_PERCENT: GPU Framebuffer Memory Used (%).</li><li>DCGM_FI_DEV_POWER_USAGE: Power consumption in Watts.</li></ul><p>NVIDIA provides a pre-built Grafana dashboard for these metrics. You can import it by using the dashboard ID 12239.</p><h3>Step 6: Collecting and Querying Logs with Loki</h3><p>Our app already outputs logs. The loki-stack we installed earlier includes Promtail, an agent that automatically collects logs from all pods.</p><ol><li><strong>Explore Logs in Grafana:</strong></li></ol><ul><li>In Grafana, click the “Explore” icon on the left.</li><li>Select the Loki data source.</li><li>Use a LogQL query to find your app’s logs. The simplest is by application label:</li></ul><pre>{app=&quot;genai-app&quot;}</pre><ul><li>To find specific information, like all inference errors:</li></ul><pre>{app=&quot;genai-app&quot;} |= &quot;Model inference failed&quot;</pre><p><strong>The Power of Structured Logging</strong>: <br>For better querying, modify your Python logging to output JSON. This allows you to query specific fields directly in Loki, like json | line_format &quot;{{.message}}&quot; | prompt_length &gt; 100.</p><h3>Step 7: Tracing Requests with OpenTelemetry</h3><p>Traces show you the lifecycle of a request across different services or functions. Our current app just prints them to the console. Let’s send them to a real collector.</p><ol><li><strong>Install the OpenTelemetry Collector and Jaeger (for viewing traces):</strong></li></ol><pre>helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts<br>helm install otel-collector open-telemetry/opentelemetry-collector --namespace monitoring<br>helm install jaeger open-telemetry/jaeger --namespace monitoring</pre><p>2. <strong>Configure the Collector</strong>: <br>You’ll need to create a values.yaml to tell the collector to receive OTLP (OpenTelemetry Protocol) data and export it to Jaeger.</p><p>3. <strong>Update the Application</strong>: <br>Modify the app.py tracer setup to point to the OpenTelemetry Collector&#39;s service name (otel-collector.monitoring.svc.cluster.local).</p><p>This step is more advanced but provides invaluable insight into where latency is occurring in complex GenAI pipelines (e.g., pre-processing -&gt; model inference -&gt; post-processing).</p><h3>Conclusion</h3><p>By following these steps, you have built a comprehensive observability solution. You can now go to a single Grafana dashboard and see:</p><ul><li>High-level health: Request rate, latency, and error rate.</li><li>Cost/Performance: Token generation rate and GPU utilization.</li><li>Specific Problems: Drill down from a spike in latency on a graph directly to the logs and traces for the slow requests during that time.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=32d3e29cfc21" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“GenAI on Prod”:Mastering GPU Efficiency: Optimizing GenAI Workloads on Kubernetes (Part-3)]]></title>
            <link>https://medium.com/@pratik.vyas_10544/genai-on-prod-mastering-gpu-efficiency-optimizing-genai-workloads-on-kubernetes-part-3-48b0c9e124b2?source=rss-d183e2545ed0------2</link>
            <guid isPermaLink="false">https://medium.com/p/48b0c9e124b2</guid>
            <category><![CDATA[gpu-computing]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[gpu-optimization]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[gpu-utilization]]></category>
            <dc:creator><![CDATA[Pratik]]></dc:creator>
            <pubDate>Thu, 03 Jul 2025 04:48:52 GMT</pubDate>
            <atom:updated>2025-07-10T15:21:32.968Z</atom:updated>
            <content:encoded><![CDATA[<p>Read <a href="https://medium.com/p/e9b1469b95cb">Part-1 Deploying</a><br>Read <a href="https://medium.com/p/8678bfacfded">Part-2 Scaling</a><br>Read <a href="https://medium.com/@pratik.vyas_10544/genai-on-prod-mastering-gpu-efficiency-optimizing-genai-workloads-on-kubernetes-part-3-48b0c9e124b2">Part-3 Mastering GPU Efficiency</a><br>Read <a href="https://medium.com/@pratik.vyas_10544/genai-on-prod-observability-for-generative-ai-on-kubernetess-part-4-32d3e29cfc21">Part-4 Observability</a></p><p>Generative AI (GenAI) is transforming industries, but the underlying infrastructure can be a major challenge. Graphics Processing Units (GPUs) are the engines of GenAI, yet they are expensive and notoriously easy to underutilize. For developers building on Kubernetes, letting these powerful resources sit idle is like burning cash.</p><p>This guide is your practical, step-by-step roadmap to taming GPU costs and maximizing performance. We will walk through the entire process, from foundational setup to intelligent, hands-off automation, turning your expensive hardware from a cost center into a highly efficient powerhouse for your GenAI applications.</p><h3>The Goal:</h3><p>By the end of this guide, you will be able to:</p><ol><li>Correctly provision and schedule GPU workloads in Kubernetes.</li><li>Share a single physical GPU across multiple applications to dramatically increase utilization.</li><li>Monitor GPU performance with production-grade tools.</li><li>Automate pod and node scaling to dynamically match resource supply with demand.</li></ol><p>Let’s get started.</p><h3>Step 1: The Foundation — Correctly Allocating GPU Resources</h3><p>Before you can optimize, you must first establish control. Kubernetes doesn’t understand GPUs out of the box; we need to teach it how to see and manage this specialized hardware.</p><h4>1.1 Install the NVIDIA Device Plugin</h4><p>The first step is to make your Kubernetes cluster “GPU-aware.” The NVIDIA Device Plugin for Kubernetes is the standard tool for this. It’s a Kubernetes DaemonSet, meaning it runs on every node in your cluster, and is responsible for:</p><ul><li>Discovering GPUs: It detects the presence and number of NVIDIA GPUs on each node.</li><li>Exposing GPUs: It advertises the GPUs as a schedulable resource to the Kubernetes scheduler.</li><li>Monitoring Health: It performs basic health checks on the GPUs.</li></ul><p>You can deploy it easily using its Helm chart.</p><pre># Add the NVIDIA Helm repository<br>helm repo add nvdp https://nvidia.github.io/k8s-device-plugin<br>helm repo update<br><br># Install the device plugin into its own namespace<br>helm install nvidia-device-plugin nvdp/nvidia-device-plugin \<br>  --version=0.14.1 \<br>  --namespace nvidia-device-plugin \<br>  --create-namespace</pre><h4>1.2 Isolate GPU Nodes with Taints</h4><p>Your GPU nodes are expensive real estate. You don’t want a simple web server or a Redis cache consuming resources on a machine designed for heavy-duty AI workloads. We can reserve these nodes using taints. A taint on a node acts as a “repellent,” preventing pods from being scheduled on it unless they have a matching toleration.</p><p>You can apply taints declaratively when you define your node groups. Here’s how you would do it for an Amazon EKS managed node group using Terraform:</p><pre># Example: Terraform configuration for an EKS Managed Node Group with a GPU Taint<br><br>eks_managed_node_groups = {<br>  # A logical name for your GPU node group<br>  eks-gpu-mng = {<br>    # Choose an appropriate GPU-enabled instance type<br>    instance_types = [&quot;g5.2xlarge&quot;] # Or g6, p4, etc.<br><br>    # This taints every node created in this group.<br>    taints = [<br>      {<br>        # The key for the taint. Using a standard convention is best practice.<br>        key    = &quot;nvidia.com/gpu&quot;<br>        # The value associated with the key.<br>        value  = &quot;true&quot;<br>        # The effect of the taint. NO_SCHEDULE means the scheduler will not<br>        # place any new pod on this node unless it explicitly tolerates this taint.<br>        effect = &quot;NO_SCHEDULE&quot;<br>      }<br>    ]<br>  }<br>}</pre><h4>1.3 Request a GPU in Your Pod Specification</h4><p>Now that the plugin is running and our nodes are protected, you can request a GPU for your application. This is done in your Pod or Deployment manifest. Unlike CPU and memory, GPUs are an “extended resource” and must be specified under the resources.limits section.</p><p>This example manifest shows a pod that not only requests a GPU but also includes the necessary toleration to run on our tainted nodes and a nodeSelector for explicit targeting.</p><pre># Save this file as gpu-pod.yaml<br>apiVersion: v1<br>kind: Pod<br>metadata:<br>  name: gpu-demo-pod<br>spec:<br>  # --- Section 1: Permission to run on a Tainted Node ---<br>  # &#39;tolerations&#39; allow this Pod to be scheduled on nodes with matching taints.<br>  # This acts as the &quot;permission slip&quot; to enter our restricted GPU node area.<br>  tolerations:<br>  - key: &quot;nvidia.com/gpu&quot;<br>    operator: &quot;Exists&quot; # Matches any taint with this key, regardless of value.<br>    effect: &quot;NoSchedule&quot; # Must match the effect of the taint on the node.<br>  <br>  # --- Section 2: Explicitly Target a GPU Node ---<br>  # &#39;nodeSelector&#39; forces the pod to only run on nodes with specific labels.<br>  # This provides an extra layer of scheduling assurance.<br>  # An admin must first label the node: kubectl label node &lt;your-node-name&gt; hardware-type=gpu<br>  nodeSelector:<br>    hardware-type: gpu<br>    <br>  # --- Section 3: The Container and its GPU Request ---<br>  containers:<br>  - name: cuda-container<br>    # The nvidia/cuda image comes pre-loaded with CUDA drivers and libraries.<br>    image: nvidia/cuda:12.1.1-devel-ubuntu22.04<br>    # A command to keep the container running and use the GPU<br>    command: [&quot;/bin/sh&quot;, &quot;-c&quot;, &quot;while true; do nvidia-smi; sleep 30; done&quot;]<br>    resources:<br>      # For extended resources like GPUs, the request MUST be specified under &#39;limits&#39;.<br>      # Kubernetes treats GPUs as indivisible, non-compressible resources.<br>      limits:<br>        # This is the core resource request.<br>        # &#39;nvidia.com/gpu&#39; is the resource name exposed by the NVIDIA device plugin.<br>        # The value &#39;1&#39; requests exactly one full, physical NVIDIA GPU.<br>        nvidia.com/gpu: 1</pre><h3>Step 2: Beyond Allocation — Sharing GPUs to Maximize Utilization</h3><p>Allocating an entire GPU to a single inference pod is often wasteful. Inference workloads typically cause short bursts of GPU activity, leaving the hardware idle most of the time. Here are three powerful techniques to share a single physical GPU among multiple pods.</p><h4>Technique 2.1: NVIDIA Multi-Instance GPU (MIG)</h4><p>Available on NVIDIA Ampere (A100) and newer architectures, MIG is a hardware-level partitioning feature. It carves a single physical GPU into up to seven fully isolated “GPU Instances.” Each MIG instance has its own dedicated memory, cache, and compute cores, providing predictable performance and strong security isolation, as if they were physically separate GPUs.</p><p>You request a MIG instance by specifying its profile name in your pod spec.</p><pre># Example: Requesting a 1-slice, 10GB memory MIG instance (profile `1g.10gb`)<br># This is a snippet of a container spec.<br>resources:<br>  limits:<br>    # The resource name directly corresponds to the MIG profile.<br>    # The device plugin must be configured to expose these profiles.<br>    nvidia.com/mig-1g.10gb: 1</pre><h4>Technique 2.2: GPU Time-Slicing</h4><p>For GPUs that don’t support MIG (like the T4 or V100), time-slicing is an excellent software-based alternative. It allows multiple containers to share a GPU by giving each one a “slice” of execution time. The NVIDIA device plugin can be configured to advertise a single physical GPU as multiple virtual GPUs.</p><p>To enable this, you apply a ConfigMap to configure the device plugin.</p><pre># Save this file as time-slicing-config.yaml<br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: nvidia-device-plugin-config<br>  # IMPORTANT: The ConfigMap must be in the same namespace as the device plugin.<br>  namespace: nvidia-device-plugin<br>data:<br>  # The configuration is provided as a multi-line string under the &#39;any&#39; key.<br>  any: |-<br>    version: v1<br>    # This flag is critical. MIG and Time-Slicing are mutually exclusive.<br>    # You MUST disable MIG to use time-slicing.<br>    flags:<br>      migStrategy: none<br>    sharing:<br>      timeSlicing:<br>        resources:<br>        - name: nvidia.com/gpu<br>          # This tells the plugin: &quot;For every 1 physical GPU, advertise 10<br>          # virtual GPUs to the Kubernetes scheduler.&quot;<br>          replicas: 10</pre><p>After applying this ConfigMap and restarting the device plugin DaemonSet, a pod can request nvidia.com/gpu: 1 and receive one of the 10 available virtual slices.</p><h4>Technique 2.3: NVIDIA Multi-Process Service (MPS)</h4><p>MPS is another sharing technique where multiple CUDA processes can run <em>concurrently</em> on the same GPU. Unlike time-slicing which gives each process a turn, MPS allows them to share the GPU’s compute resources and memory simultaneously. This can reduce kernel launch latency and improve overall throughput for applications that issue many small CUDA kernels.</p><p>The configuration is similar to time-slicing but uses a different sharing strategy.</p><pre># Save this file as mps-config.yaml<br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: nvidia-device-plugin-config<br>  # IMPORTANT: The ConfigMap must be in the same namespace as the device plugin.<br>  namespace: nvidia-device-plugin<br>data:<br>  config.yaml: |<br>    version: v1<br>    sharing:<br>      mps:<br>        resources:<br>        - name: nvidia.com/gpu<br>          # This tells the plugin to advertise 10 schedulable resources<br>          # for every 1 physical GPU. Up to 10 pods can be scheduled,<br>          # and they will all share the GPU via the MPS daemon.<br>          replicas: 10</pre><h4>Comparison of Sharing Techniques</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g2JRaogHhOX7eo7Da8c3Iw.png" /></figure><h3>Step 3: Monitoring — You Can’t Optimize What You Can’t See</h3><p>Effective optimization requires data. The NVIDIA Data Center GPU Manager (DCGM) is the gold standard for collecting detailed GPU metrics like utilization, memory usage, temperature, and power draw. To get these metrics into a standard monitoring stack like Prometheus, we use the DCGM-Exporter.</p><h4>3.1 Deploy DCGM-Exporter</h4><p>The official Helm chart makes deployment simple. This Terraform resource automates it.</p><pre># Example: Deploying dcgm-exporter via its Helm chart using Terraform<br><br>resource &quot;helm_release&quot; &quot;dcgm_exporter&quot; {<br>  name             = &quot;dcgm-exporter&quot;<br>  repository       = &quot;https://nvidia.github.io/dcgm-exporter/&quot;<br>  chart            = &quot;dcgm-exporter&quot;<br>  version          = &quot;3.3.4&quot; # Pinning the chart version is a good practice<br>  namespace        = &quot;monitoring&quot;<br>  create_namespace = true<br><br>  # Enable the ServiceMonitor for Prometheus Operator to automatically scrape metrics<br>  set {<br>    name  = &quot;serviceMonitor.enabled&quot;<br>    value = &quot;true&quot;<br>  }<br>}</pre><h4>3.2 Verify Metrics are Flowing</h4><p>Once deployed, you can quickly check if metrics are being exposed using kubectl port-forward.</p><pre># Find the dcgm-exporter service in your monitoring namespace<br># Forward its port (9400) to your local machine<br>kubectl port-forward svc/dcgm-exporter -n monitoring 9400:9400<br><br># In a separate terminal, use curl to fetch the metrics<br>curl http://localhost:9400/metrics<br><br># You should see a large output of metrics, including GPU utilization:<br># HELP DCGM_FI_DEV_GPU_UTIL GPU utilization (in %).<br># TYPE DCGM_FI_DEV_GPU_UTIL gauge<br># DCGM_FI_DEV_GPU_UTIL{gpu=&quot;0&quot;,UUID=&quot;GPU-...&quot;,...} 0</pre><p>Now you can build Grafana dashboards to visualize real-time GPU utilization, memory, and more across your entire cluster, giving you the insights needed for optimization.</p><h3>Step 4: Automation — Implementing Intelligent Scaling</h3><p>With metrics in place, we can close the loop and automate scaling. The goal is to match resource supply with demand dynamically, at both the pod and node levels.</p><h4>4.1 Pod Scaling: Horizontal Pod Autoscaler (HPA)</h4><p>The HPA can automatically scale the number of pod replicas based on metrics. With our DCGM metrics in Prometheus, we can scale directly on GPU utilization.</p><p>This HPA will scale a Deployment named genai-inference up or down to keep the average GPU utilization across all its pods at around 75%.</p><pre># Save this file as gpu-hpa.yaml<br>apiVersion: autoscaling/v2<br>kind: HorizontalPodAutoscaler<br>metadata:<br>  name: genai-inference-hpa<br>spec:<br>  # --- What to scale ---<br>  scaleTargetRef:<br>    apiVersion: apps/v1<br>    kind: Deployment<br>    name: genai-inference # The name of the deployment to manage<br><br>  # --- Scaling limits ---<br>  minReplicas: 1<br>  maxReplicas: 10<br><br>  # --- When to scale ---<br>  metrics:<br>  - type: Pods # We are scaling based on the average of a metric across multiple pods<br>    pods:<br>      # The metric name must match what dcgm-exporter provides<br>      metric:<br>        name: &quot;DCGM_FI_DEV_GPU_UTIL&quot;<br>      # The HPA&#39;s goal: adjust replica count to maintain this average value<br>      target:<br>        type: AverageValue<br>        averageValue: &quot;75&quot; # Target 75% GPU utilization</pre><h4>4.2 Advanced Pod Scaling: KEDA</h4><p>For more advanced scenarios, like scaling based on queue length or other event sources, the Kubernetes Event-Driven Autoscaler (KEDA) is a perfect fit. It extends the HPA and, most importantly, can scale workloads down to zero when there is no demand, and back up again when an event occurs.</p><p>This ScaledObject uses a Prometheus query to trigger scaling. It will add pods if the average GPU utilization over the last minute exceeds 70%.</p><pre># Save this file as gpu-keda-scaler.yaml<br>apiVersion: keda.sh/v1alpha1<br>kind: ScaledObject<br>metadata:<br>  name: my-llama-deployment-scaler<br>spec:<br>  # --- What to scale ---<br>  scaleTargetRef:<br>    kind: Deployment<br>    name: my-llama-deployment<br><br>  # --- Scaling limits (KEDA can manage min/max itself) ---<br>  minReplicaCount: 0  # The magic of KEDA: scale to zero!<br>  maxReplicaCount: 8<br>  <br>  # --- When to scale ---<br>  triggers:<br>  - type: prometheus<br>    metadata:<br>      serverAddress: http://prometheus-server.monitoring.svc:9090 # Your Prometheus service address<br>      metricName: gpu_utilization_scaler # A friendly name for the scaler<br>      threshold: &quot;70&quot; # The target value for the query result<br>      <br>      # The heart of the logic: a PromQL query.<br>      # This query calculates the average GPU utilization across all pods of our<br>      # deployment over the last 60 seconds.<br>      query: |<br>        avg(<br>          DCGM_FI_DEV_GPU_UTIL{kubernetes_pod_name=~&quot;my-llama-deployment-.*&quot;}<br>        ) by (kubernetes_pod_name)</pre><h4>4.3 Node Scaling: Karpenter</h4><p>HPA and KEDA scale pods. But what happens when you need more pods than can fit on your existing GPU nodes? Karpenter is a high-performance node autoscaler that solves this problem. It watches for unschedulable pods and can provision a new, perfectly sized GPU node in seconds. Crucially, it also de-provisions idle nodes to save costs.</p><p>This Karpenter NodePool definition tells Karpenter how to provision GPU nodes.</p><pre># Save this file as gpu-nodepool.yaml<br>apiVersion: karpenter.sh/v1<br>kind: NodePool<br>metadata:<br>  name: gpu-nodepool<br>spec:<br>  # This template applies to all nodes created by this NodePool.<br>  # It includes the taint we defined in Step 1.<br>  template:<br>    metadata:<br>      labels:<br>        hardware-type: gpu<br>    spec:<br>      taints:<br>        - key: nvidia.com/gpu<br>          effect: NoSchedule<br>      requirements:<br>        # --- Define the &quot;shopping list&quot; for nodes ---<br>        - key: &quot;karpenter.k8s.aws/instance-category&quot;<br>          operator: In<br>          values: [&quot;g&quot;, &quot;p&quot;] # Allow GPU instance categories &#39;g&#39; and &#39;p&#39;<br>        - key: &quot;karpenter.sh/capacity-type&quot;<br>          operator: In<br>          values: [&quot;spot&quot;, &quot;on-demand&quot;] # Prioritize cheaper Spot instances<br>        - key: kubernetes.io/arch<br>          operator: In<br>          values: [&quot;amd64&quot;]<br><br>  # --- Define when to remove nodes ---<br>  disruption:<br>    # This policy is key for cost savings. Karpenter will terminate nodes<br>    # that are empty or can be consolidated.<br>    consolidationPolicy: WhenEmptyOrUnderutilized<br>    # Nodes will be considered for removal after being empty for 30 seconds.<br>    expireAfter: 30s</pre><h3>Conclusion: Putting It All Together</h3><p>Optimizing GPU resources in Kubernetes is not a single action but a layered strategy. By following this guide, you have built a robust, efficient, and scalable platform for your Generative AI applications:</p><ol><li>Allocate Correctly: You’ve laid a solid foundation with the NVIDIA device plugin and smart scheduling using taints, ensuring GPU resources are reserved for the workloads that need them.</li><li>Share and Maximize: You’ve unlocked massive utilization potential by implementing sharing strategies like MIG or Time-Slicing, eliminating the waste of idle, fully-allocated GPUs.</li><li>Monitor Everything: You’ve gained critical visibility into performance by integrating DCGM with Prometheus and Grafana, enabling data-driven decisions.</li><li>Automate Scaling: You’ve created a self-regulating system using HPA, KEDA, and Karpenter to dynamically match resources to demand at both the pod and node levels, ensuring performance while controlling costs.</li></ol><p>By embracing these strategies, you empower your team to innovate freely with GenAI, confident that your underlying infrastructure is as intelligent and efficient as the models you’re deploying.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=48b0c9e124b2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“GenAI on Prod”: Deploying Generative AI models on Kubernetes (Part-1)]]></title>
            <link>https://medium.com/@pratik.vyas_10544/genai-on-prod-deploying-generative-ai-models-on-kubernetes-part-1-e9b1469b95cb?source=rss-d183e2545ed0------2</link>
            <guid isPermaLink="false">https://medium.com/p/e9b1469b95cb</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[genai-deployment]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Pratik]]></dc:creator>
            <pubDate>Tue, 01 Jul 2025 07:57:27 GMT</pubDate>
            <atom:updated>2025-07-12T13:58:17.204Z</atom:updated>
            <content:encoded><![CDATA[<p>Read <a href="https://medium.com/p/8678bfacfded">Part-2 Scaling</a><br>Read <a href="https://medium.com/@pratik.vyas_10544/genai-on-prod-mastering-gpu-efficiency-optimizing-genai-workloads-on-kubernetes-part-3-48b0c9e124b2">Part-3 Mastering GPU Efficiency</a><br>Read <a href="https://medium.com/@pratik.vyas_10544/genai-on-prod-observability-for-generative-ai-on-kubernetess-part-4-32d3e29cfc21">Part-4 Observability</a></p><p><strong>Index</strong></p><ol><li>Need for Containers and Kubernetes for GenAI</li><li>Steps to Deploy a GenAI Model on Kubernetes</li><li>Sample Deployments</li></ol><p>Deploying Generative AI (GenAI) models on Kubernetes (K8s) is a powerful way to manage these resource-intensive applications with scalability, resilience, and efficiency. For a developer, understanding and implementing this process involves several key steps, from preparing your model to deploying and managing it in a cloud-native environment.</p><p>Here’s a breakdown of how to approach this, making it easy to understand and implement:</p><h3>1. Need for Containers and Kubernetes for GenAI</h3><p>Generative AI applications, especially those involving Large Language Models (LLMs), come with unique challenges:</p><ul><li><strong>Complex Dependencies:</strong> GenAI models rely on rapidly evolving open-source machine learning (ML) frameworks like PyTorch and TensorFlow, specific ML toolkits like Hugging Face Transformers, and continuously changing GPU hardware ecosystems.</li><li><strong>Computational Requirements:</strong> These models are large and complex, demanding substantial computational resources, including GPUs and custom accelerators for both training and inference.</li><li><strong>Scalability:</strong> As demand for AI/ML services grows, models need to scale seamlessly without sacrificing performance or cost-efficiency.</li><li><strong>Deployment Complexity:</strong> Packaging models with custom frameworks, plugin libraries, and other dependencies can lead to deployment issues.</li></ul><p><strong>Containers</strong> address these complexities by packaging your application code, runtime, libraries, and dependencies into a <strong>lightweight, standalone, executable unit</strong> called a container image. This ensures consistency and portability across different environments, from a developer’s local machine to production servers.</p><p><strong>Kubernetes (K8s)</strong>, an open-source container orchestration platform, then automates the deployment, scaling, and management of these containerized applications. It’s the “brain” that manages hundreds or thousands of containers across many virtual machines (VMs), ensuring high availability, load balancing, and efficient resource allocation.</p><h3>2. Steps to Deploy a GenAI Model on Kubernetes</h3><p>Let’s walk through the practical steps, using examples from the sources like deploying a Llama model:</p><h3>Step 1: Containerize Your GenAI Model</h3><p>The first crucial step is to package your GenAI model and its inference code into a Docker container image.</p><p><strong>1.1 Prepare your Model and Code:</strong> <br>You’ll typically have a pre-trained model (like Llama 2 or Llama 3) and an inference script (e.g., using Python Flask or FastAPI) that exposes your model as an API.</p><blockquote><strong>Tip for datasets:</strong> If your model needs specific datasets (e.g., for fine-tuning), it’s a best practice to store them in an <strong>external datastore</strong> like Amazon S3 rather than packaging them directly into the container image. This makes your image reusable for different datasets.</blockquote><p><strong>1.2 Create a </strong><strong>Dockerfile:</strong> <br>This text file contains instructions for building your container image.</p><ul><li><strong>Choose a Base Image:</strong> Start with a suitable base image. For GPU-accelerated GenAI models, <strong>nvidia/cuda</strong> images are common as they include necessary CUDA libraries. For CPU-only applications or smaller components, lighter images like python:slim might suffice.</li><li><strong>Install Dependencies:</strong> Your Dockerfile will include commands to install Python packages (e.g., torch, transformers, fastapi, uvicorn).</li><li><strong>Copy Your Code and Model Assets:</strong> Copy your inference script and any fine-tuned model assets (if not loaded from an external store at runtime) into the image.</li><li><strong>Define the Entrypoint:</strong> Specify the command to run your application when the container starts (e.g., CMD [&quot;uvicorn&quot;, &quot;main:app&quot;, &quot;--host&quot;, &quot;0.0.0.0&quot;, &quot;--port&quot;, &quot;80&quot;] for a FastAPI app).</li></ul><p><strong>1.3 Build the Image:<br></strong>Use the Docker CLI to build your image locally: <strong>docker build -t my-genai-model .</strong>.</p><h3>Step 2: Store Your Container Image</h3><p>Once built, push your container image to a <strong>container registry</strong> so your K8s cluster can access it.</p><p><strong>2.1 Use a Managed Registry:</strong> <br>Services like <strong>Amazon Elastic Container Registry (ECR)</strong> are fully managed and ideal for storing, sharing, and deploying container software.</p><p><strong>2.2 Push the Image:</strong> <br>You’ll tag your local image with the registry’s URL and then push it: docker tag my-genai-model &lt;registry_url&gt;/my-genai-model followed by docker push &lt;registry_url&gt;/my-genai-model.</p><p><strong>2.3 Ensure Image Immutability:</strong> <br>For production, set image tag mutability to IMMUTABLE in your registry. This prevents accidental or malicious overwrites of image tags, ensuring that a specific tag always refers to the same image, which is critical for consistent deployments and troubleshooting.</p><h3>Step 3: Set up a Kubernetes Cluster</h3><p>While you can set up K8s manually, for GenAI workloads, it’s highly recommended to use <strong>managed K8s services</strong> from cloud providers.</p><p><strong>3.1 Benefits of Managed Services:</strong> <br>These services (e.g., <strong>Amazon Elastic Kubernetes Service (EKS)</strong>, Google Kubernetes Engine (GKE), Azure Kubernetes Service (AKS)) handle the operational complexities of running K8s clusters, such as managing the control plane, upgrades, and patching, allowing you to focus on your applications.</p><p><strong>3.2 Infrastructure as Code (IaC):</strong> <br>Use tools like <strong>Terraform</strong> to automate the provisioning of your K8s cluster and its underlying cloud infrastructure (VPC, subnets, worker nodes). This ensures reproducibility and consistency.</p><h3>Step 4: Deploy Your Model on Kubernetes</h3><p>Now that your image is in a registry and your cluster is ready, deploy your model using K8s resources.</p><p><strong>4.1 Kubernetes Deployment:</strong> <br>This is the most common way to deploy stateless applications in K8s. It manages the lifecycle of your application Pods, ensuring a desired number of replicas are running, handling rolling updates, and automatically restarting failed Pods.</p><p><strong>4.2 Define a YAML Manifest:</strong> <br>Create a <strong>Deployment YAML file</strong> that specifies:</p><ul><li><strong>apiVersion</strong>, <strong>kind</strong> (Deployment), and <strong>metadata</strong> (name).</li><li><strong>spec.replicas</strong>: The desired number of Pod instances.</li><li><strong>spec.template.spec.containers</strong>: Your container image URL from the registry, port, and <strong>resource requests/limits</strong> (especially important for GPUs).</li><li><strong>GPU Allocation:</strong> When requesting GPU resources, always define them in the <strong>limits</strong> section (e.g., nvidia.com/gpu: 1). This ensures the GPU is reserved for your Pod.</li></ul><p><strong>4.3 Apply the Manifest:</strong> <br>Use the kubectl command-line tool to deploy your application: <strong>kubectl apply -f your-deployment.yaml</strong>.</p><h3>Step 5: Expose Your Model to Users</h3><p>For your GenAI model to be accessible, you need a K8s Service.</p><p><strong>5.1 Kubernetes Service:</strong> <br>A Service provides a stable network endpoint and load balancing for a set of Pods.</p><pre># genai-service-external.yaml<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: genai-model-public-service<br>  # --- Example Annotations for AWS NLB ---<br>  # These annotations instruct the AWS cloud provider to configure a specific type of Load Balancer.<br>  annotations:<br>    # Use an external-facing load balancer<br>    service.beta.kubernetes.io/aws-load-balancer-type: &quot;external&quot;<br>    # Create a high-performance Network Load Balancer (NLB) instead of a classic one<br>    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: &quot;ip&quot;<br>spec:<br>  # This type provisions an external IP/DNS from your cloud provider<br>  type: LoadBalancer<br>  # This selector finds Pods with the label &#39;app: genai-model&#39;<br>  selector:<br>    app: genai-model<br>  ports:<br>    - protocol: TCP<br>      # The public port that the Load Balancer will listen on.<br>      port: 80<br>      # The internal port on your Pods where the application is running (must match containerPort).<br>      targetPort: 8080</pre><p><strong>5.2 </strong><strong>LoadBalancer Service Type:</strong> <br>For external access (e.g., from your customers or other external applications), use the LoadBalancer service type. This automatically provisions a cloud provider&#39;s load balancer (like AWS Network Load Balancer) to distribute incoming traffic to your Pods.</p><ul><li><strong>Example Annotation for AWS NLB:</strong> Add annotations to your Service YAML to specify external-facing NLB type and target type.</li></ul><pre># genai-service-external.yaml<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: genai-model-public-service<br>  # --- Example Annotations for AWS NLB ---<br>  # These annotations instruct the AWS cloud provider to configure a specific type of Load Balancer.<br>  annotations:<br>    # Use an external-facing load balancer<br>    service.beta.kubernetes.io/aws-load-balancer-type: &quot;external&quot;<br>    # Create a high-performance Network Load Balancer (NLB) instead of a classic one<br>    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: &quot;ip&quot;<br>spec:<br>  # This type provisions an external IP/DNS from your cloud provider<br>  type: LoadBalancer<br>  # This selector finds Pods with the label &#39;app: genai-model&#39;<br>  selector:<br>    app: genai-model<br>  ports:<br>    - protocol: TCP<br>      # The public port that the Load Balancer will listen on.<br>      port: 80<br>      # The internal port on your Pods where the application is running (must match containerPort).<br>      targetPort: 8080</pre><p><strong>5.3 </strong><strong>ClusterIP Service Type:</strong> <br>If your GenAI model is only accessed by other services <em>within</em> the K8s cluster, use ClusterIP.</p><pre># genai-service-internal.yaml<br>apiVersion: v1<br>kind: Service<br>metadata:<br>  name: genai-model-internal-service<br>spec:<br>  # ClusterIP is the default type. It is only accessible within the cluster.<br>  type: ClusterIP<br>  # Find Pods with the label &#39;app: genai-model&#39;<br>  selector:<br>    app: genai-model<br>  ports:<br>    - protocol: TCP<br>      # The port that other services *inside the cluster* will use to connect.<br>      port: 8080<br>      # Forward traffic to the same port 8080 on the container.<br>      targetPort: 8080</pre><pre># command to implement <br>kubectl apply -f genai-service-internal.yaml</pre><pre><br>#Code (How other pods use it):<br>#Another application in the same cluster can now call your model using the service&#39;s name as a DNS host.<br><br># Example in a Python backend service<br>import requests<br><br># The service name becomes the hostname. Kubernetes DNS handles the rest.<br># The URL is http://&lt;service-name&gt;:&lt;service-port&gt;<br>internal_url = &quot;http://genai-model-internal-service:8080/predict&quot;<br><br>response = requests.post(internal_url, json={&quot;prompt&quot;: &quot;Explain ClusterIP&quot;})<br>print(response.text)</pre><p><strong>5.4 Get the External URL:</strong> <br>After applying the Service manifest, you can fetch the LoadBalancer URL to test your endpoint.</p><h3>Step 6: Optimize and Scale Your GenAI Applications</h3><p>GenAI workloads are dynamic, requiring careful resource management.</p><p><strong>6.1 Right-Sizing Resources:</strong> <br>Understand your application’s actual CPU, memory, and GPU needs to prevent over-provisioning (wasting money) or under-provisioning (poor performance). Tools like <strong>Kubecost</strong> and Goldilocks can provide recommendations.</p><p><strong>6.2 Horizontal Pod Autoscaler (HPA):</strong> <br>Use HPA to automatically adjust the number of Pod replicas based on metrics like CPU, memory, or <strong>GPU utilization</strong>. For GenAI, GPU utilization is a highly effective scaling metric.</p><p><strong>6.3 Kubernetes Event-Driven Autoscaler (KEDA):</strong> <br>For event-driven GenAI applications (e.g., processing messages from a queue), KEDA extends HPA to scale based on external event sources, even scaling to zero when there’s no demand.</p><p><strong>6.4 Karpenter:</strong> <br>A high-performance K8s cluster autoscaler that dynamically provisions right-sized compute resources (including GPU instances) based on Pod needs, optimizing for efficiency and performance. It can terminate underutilized nodes to save costs.</p><p><strong>6.5 GPU Partitioning and Sharing:</strong> <br>For expensive GPUs, techniques like <strong>NVIDIA MIG</strong> (Multi-Instance GPU), <strong>MPS</strong> (Multi-Process Service), and <strong>time-slicing</strong> allow multiple workloads to share a single physical GPU, improving utilization.</p><h3>Step 7: Ensure Observability for GenAI Workloads</h3><p>Monitoring your GenAI applications is crucial for identifying issues, optimizing performance, and ensuring reliability.</p><p><strong>7.1 The Three Pillars:</strong> <br>Observability relies on <strong>logs</strong> (detailed events), <strong>metrics</strong> (quantifiable performance data), and <strong>traces</strong> (end-to-end request flow).</p><p><strong>7.2 Logging:</strong> <br>Use agents like <strong>Fluentd</strong> or <strong>Fluent Bit</strong> deployed as DaemonSets to collect logs from your Pods and forward them to centralized storage solutions like Loki or Amazon CloudWatch Logs.</p><p><strong>7.3 Metrics:</strong> <br>Deploy <strong>Prometheus</strong> to collect time-series metrics from your K8s components and applications. For GPUs, use the <strong>NVIDIA DCGM Exporter</strong> to expose GPU utilization, memory usage, and temperature metrics to Prometheus.</p><p><strong>7.4 Visualization:</strong> <br>Use <strong>Grafana</strong> to create interactive dashboards to visualize your logs and metrics, providing real-time insights into system health and application performance.</p><p><strong>7.5 Debugging GenAI Applications:</strong> <br>Frameworks like <strong>LangChain</strong> offer built-in logging and tracing (verbose mode). Tools like <strong>LangFuse</strong> provide more advanced observability for LLM applications by tracking prompts, responses, and performance.</p><p>By following these steps and leveraging the powerful capabilities of containers and Kubernetes, developers can efficiently build, deploy, and manage scalable, secure, and cost-optimized GenAI solutions.</p><h3>Sample Deployments</h3><p>This is a comprehensive, real-world example for a GenAI application, broken down into multiple resource files for clarity and best practices.</p><p><strong>Core Concepts in this Production YAML</strong></p><ul><li><strong>Separation of Concerns</strong>: Configuration (ConfigMap), secrets (Secret), and the application definition (Deployment) are in separate files.</li><li><strong>Security</strong>: The Pod runs as a non-root user and uses a dedicated ServiceAccount.</li><li><strong>Resilience &amp; High Availability</strong>: Uses multiple replicas, a RollingUpdate strategy for zero-downtime deployments, and robust health checks (startup, readiness, liveness probes).</li><li><strong>Scheduling</strong>: Uses nodeAffinity and tolerations to ensure Pods are scheduled <em>only</em> on expensive GPU nodes that are properly configured.</li><li><strong>Configuration Management</strong>: Mounts configuration and secrets as environment variables, not hardcoded in the image.</li><li><strong>Performance</strong>: Includes a PersistentVolumeClaim for a model cache. This is a critical optimization for GenAI models, as downloading a 50GB model from Hugging Face on every Pod startup is incredibly slow and costly.</li></ul><p><strong>Prerequisites: Create the Config, Secret, and PVC</strong></p><p>Before deploying the main application, you need to create its dependencies in the cluster.</p><p>1. <strong>genai-configmap.yaml</strong></p><p>This holds non-sensitive configuration, like model names or logging levels.</p><pre># genai-configmap.yaml<br>apiVersion: v1<br>kind: ConfigMap<br>metadata:<br>  name: genai-inference-config<br>  namespace: prod-ml<br>data:<br>  # The name of the model to load from Hugging Face or a local cache<br>  MODEL_NAME: &quot;meta-llama/Llama-2-7b-chat-hf&quot;<br>  # Set the application log level<br>  LOG_LEVEL: &quot;INFO&quot;<br>  # Number of workers for your model server (e.g., Gunicorn, Uvicorn)<br>  WORKER_COUNT: &quot;4&quot;</pre><p>2. <strong>genai-secret.yaml</strong></p><p>This holds sensitive data like API keys. Never commit this to public Git. Use a tool like HashiCorp Vault, AWS Secrets Manager, or Sealed Secrets in a real-world scenario.</p><pre># genai-secret.yaml<br>apiVersion: v1<br>kind: Secret<br>metadata:<br>  name: genai-inference-secret<br>  namespace: prod-ml<br># Using &#39;stringData&#39; is convenient as it doesn&#39;t require base64 encoding.<br># Kubernetes will automatically encode it.<br>stringData:<br>  # Example: API key needed to download models from a private registry or Hugging Face<br>  HUGGING_FACE_TOKEN: &quot;hf_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx&quot;<br>  # Example: API key for a vector database your app connects to<br>  VECTOR_DB_API_KEY: &quot;super-secret-key-for-vectordb&quot;</pre><p>3. <strong>genai-pvc.yaml</strong></p><p>This creates a persistent volume for caching the large model files. This assumes you have a StorageClass (like gp2, ssd, or a custom one) available in your cluster that can provision persistent disks.</p><pre># genai-pvc.yaml<br>apiVersion: v1<br>kind: PersistentVolumeClaim<br>metadata:<br>  name: model-cache-pvc<br>  namespace: prod-ml<br>spec:<br>  # The storage class to use. This must exist in your cluster.<br>  storageClassName: standard-rwx # Use a ReadWriteMany class if multiple pods need to write<br>  accessModes:<br>    - ReadWriteOnce # Or ReadWriteMany if supported and needed<br>  resources:<br>    requests:<br>      # Allocate enough space for your largest models<br>      storage: 100Gi</pre><p>4. <strong>genai-deployment.yaml ( Main deployment )</strong></p><p>This is the core manifest that ties everything together.</p><pre># genai-deployment.yaml<br>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: genai-inference-api-deployment<br>  namespace: prod-ml<br>  labels:<br>    app: genai-inference-api<br>    tier: backend<br>spec:<br>  # Start with 3 replicas for high availability<br>  replicas: 3<br>  # Strategy for updating Pods to new versions with zero downtime<br>  strategy:<br>    type: RollingUpdate<br>    rollingUpdate:<br>      # Allow bursting up to 25% more Pods than &#39;replicas&#39; during an update<br>      maxSurge: 25%<br>      # Only allow 1 Pod to be unavailable during the update<br>      maxUnavailable: 1<br>  selector:<br>    matchLabels:<br>      app: genai-inference-api<br>  template:<br>    metadata:<br>      labels:<br>        app: genai-inference-api<br>        tier: backend<br>      # Annotations for monitoring systems like Prometheus to scrape metrics<br>      annotations:<br>        prometheus.io/scrape: &quot;true&quot;<br>        prometheus.io/port: &quot;8080&quot;<br>        prometheus.io/path: &quot;/metrics&quot;<br>    spec:<br>      # Use a dedicated, non-default service account for security<br>      serviceAccountName: genai-service-account<br>      # Security context to run the container as a non-root user<br>      securityContext:<br>        runAsUser: 1001<br>        runAsGroup: 1001<br>        fsGroup: 1001<br>      <br>      # --- Critical Scheduling for GPU Nodes ---<br>      affinity:<br>        nodeAffinity:<br>          # HARD REQUIREMENT: Only schedule on nodes with the specified label.<br>          # This prevents scheduling expensive GPU pods on general-purpose nodes.<br>          requiredDuringSchedulingIgnoredDuringExecution:<br>            nodeSelectorTerms:<br>            - matchExpressions:<br>              - key: nvidia.com/gpu.product<br>                operator: In<br>                values:<br>                - NVIDIA-A100-SXM4-80GB # Be specific about the GPU type you need<br>                - NVIDIA-A10G<br>      tolerations:<br>      # If your GPU nodes are &quot;tainted&quot; to repel normal pods, you must add a toleration.<br>      - key: &quot;nvidia.com/gpu&quot;<br>        operator: &quot;Exists&quot;<br>        effect: &quot;NoSchedule&quot;<br><br>      # Define volumes that will be mounted into the container<br>      volumes:<br>      - name: config-volume<br>        configMap:<br>          name: genai-inference-config<br>      - name: secret-volume<br>        secret:<br>          secretName: genai-inference-secret<br>      # The persistent volume for the model cache<br>      - name: model-cache-storage<br>        persistentVolumeClaim:<br>          claimName: model-cache-pvc<br><br>      containers:<br>      - name: genai-server-container<br>        # ALWAYS use a specific version tag in production, not &#39;latest&#39;.<br>        image: your-registry/your-genai-app:1.2.0<br>        imagePullPolicy: IfNotPresent<br>        <br>        # Mount the configuration and secrets as environment variables<br>        envFrom:<br>        - configMapRef:<br>            name: genai-inference-config<br>        - secretRef:<br>            name: genai-inference-secret<br>        <br>        ports:<br>        - containerPort: 8080<br>          name: http      # Naming ports is good practice<br>        - containerPort: 9090<br>          name: metrics   # Separate port for metrics if applicable<br><br>        # Mount the volumes into the container&#39;s filesystem<br>        volumeMounts:<br>        - name: model-cache-storage<br>          mountPath: /root/.cache/huggingface/hub # Standard Hugging Face cache location<br>          # Or /models if your app uses a custom directory<br><br>        # --- Health Probes: Essential for GenAI ---<br>        # GenAI models can take several minutes to load into memory.<br>        # A startup probe prevents Kubernetes from killing the pod before it&#39;s ready.<br>        startupProbe:<br>          httpGet:<br>            path: /healthz<br>            port: http<br>          # Check every 15 seconds, up to 40 times (10 minutes total)<br>          # Give it plenty of time to download and load the model.<br>          failureThreshold: 40<br>          periodSeconds: 15<br><br>        # Once started, check if the app is ready to serve traffic.<br>        readinessProbe:<br>          httpGet:<br>            path: /readyz<br>            port: http<br>          initialDelaySeconds: 5<br>          periodSeconds: 10<br>          failureThreshold: 3<br>        <br>        # Check if the application is still running and not deadlocked.<br>        livenessProbe:<br>          httpGet:<br>            path: /healthz<br>            port: http<br>          initialDelaySeconds: 60<br>          periodSeconds: 30<br><br>        # --- Resource Allocation for a Large Model ---<br>        resources:<br>          requests:<br>            cpu: &quot;4&quot;         # Request 4 CPU cores<br>            memory: &quot;32Gi&quot;   # Request 32 Gibibytes of RAM<br>          limits:<br>            cpu: &quot;8&quot;         # Limit to 8 CPU cores<br>            memory: &quot;64Gi&quot;   # Limit to 64 Gibibytes of RAM<br>            # Request 1 NVIDIA GPU. This must be in &#39;limits&#39;.<br>            nvidia.com/gpu: 1</pre><h3>How to Deploy</h3><ol><li>Create a Namespace: It’s best practice to deploy applications into their own namespace.</li></ol><pre>kubectl create namespace prod-ml</pre><p>2. Create Service Account (if needed):</p><pre>kubectl create serviceaccount genai-service-account --namespace prod-ml # In a real prod env, you&#39;d attach specific roles (RBAC) to this account.</pre><p>3. Apply the Manifests: Apply them in order of dependency.</p><pre>kubectl apply -f genai-configmap.yaml -n prod-ml <br>kubectl apply -f genai-secret.yaml -n prod-ml <br>kubectl apply -f genai-pvc.yaml -n prod-ml <br>kubectl apply -f genai-deployment.yaml -n prod-ml</pre><p>4. Verify the Deployment:</p><pre># Check the deployment status<br>kubectl get deployment -n prod-ml<br><br># Watch the Pods being created. They may be &#39;Pending&#39; while the scheduler finds a GPU node<br># and &#39;ContainerCreating&#39; while the large image is pulled.<br>kubectl get pods -n prod-ml -w<br><br># To debug a specific pod if it fails to start:<br>kubectl describe pod &lt;pod-name&gt; -n prod-ml<br>kubectl logs &lt;pod-name&gt; -n prod-ml</pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e9b1469b95cb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[“GenAI on Prod”: Scaling GenAI application on Kubernetes (Part-2)]]></title>
            <link>https://medium.com/@pratik.vyas_10544/genai-on-prod-scaling-genai-application-on-kubernetes-part-1-8678bfacfded?source=rss-d183e2545ed0------2</link>
            <guid isPermaLink="false">https://medium.com/p/8678bfacfded</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[genai-security]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[genai-deployment]]></category>
            <dc:creator><![CDATA[Pratik]]></dc:creator>
            <pubDate>Tue, 01 Jul 2025 05:47:13 GMT</pubDate>
            <atom:updated>2025-07-10T15:21:02.731Z</atom:updated>
            <content:encoded><![CDATA[<p>Read <a href="https://medium.com/p/e9b1469b95cb">Part-1 Deploying</a><br>Read <a href="https://medium.com/p/8678bfacfded">Part-2 Scaling</a><br>Read <a href="https://medium.com/@pratik.vyas_10544/genai-on-prod-mastering-gpu-efficiency-optimizing-genai-workloads-on-kubernetes-part-3-48b0c9e124b2">Part-3 Mastering GPU Efficiency</a><br>Read <a href="https://medium.com/@pratik.vyas_10544/genai-on-prod-observability-for-generative-ai-on-kubernetess-part-4-32d3e29cfc21">Part-4 Observability</a></p><p>Welcome to the second part of our series on running GenAI in production. In <a href="https://medium.com/p/e9b1469b95cb">Part 1</a>, we covered deployment strategies. Now, we’ll dive deep into scaling.</p><p>GenAI workloads are resource-hungry, especially when it comes to GPUs. Static resource allocation is inefficient — you either overprovision and waste money, or underprovision and suffer performance bottlenecks. The solution is dynamic, automated scaling. This guide will walk you through the essential Kubernetes autoscaling tools, providing step-by-step instructions, YAML configurations, and the technical details you need to implement them effectively.</p><h3>Index</h3><ol><li>Scaling Metrics: The Foundation of Autoscaling</li><li>Horizontal Pod Autoscaler (HPA): Scaling Out Your Pods</li><li>Vertical Pod Autoscaler (VPA): Scaling Up Your Pods</li><li>Kubernetes Event-Driven Autoscaler (KEDA): Scaling on Custom Events</li><li>Cluster Autoscaler (CA): Scaling Your Node Infrastructure</li><li>AWS Karpenter: The Next-Generation Node Autoscaler</li></ol><h3>1. Scaling Metrics: Know What to Measure</h3><p>Before you can autoscale, you must define <em>what</em> triggers a scaling event. The right metric ensures your application scales in response to actual demand.</p><p><strong>Conventional Metrics</strong></p><ul><li>CPU Usage: A classic metric. Good for CPU-bound preprocessing tasks.</li><li>Memory Usage: Can be used, but it’s often a poor scaling metric. Applications may not release memory efficiently, leading to scaling that doesn’t reflect the actual load.</li></ul><p><strong>Custom Metrics for GenAI</strong></p><p>For GenAI, conventional metrics are often insufficient. You need metrics that directly reflect the workload.</p><ul><li><strong>Queue Length</strong>: If you process inference jobs from a message queue (like RabbitMQ or AWS SQS), the number of messages in the queue is a perfect indicator of backlog.</li><li><strong>HTTP Requests Rate</strong>: Ideal for API-driven models (e.g., a RAG API). You can scale based on incoming traffic.</li><li><strong>GPU Utilization</strong>: This is the most critical metric for GenAI. It directly measures how busy your expensive GPU resources are.</li></ul><h4><strong>Step-by-Step: Exposing GPU Metrics with DCGM-Exporter and Prometheus</strong></h4><p>To scale on GPU utilization, you first need to expose those metrics. We’ll use NVIDIA’s Data Center GPU Manager (DCGM) Exporter and Prometheus.</p><p><strong>Step 1</strong>: <strong>Install the DCGM Exporter.</strong> This tool runs as a DaemonSet on your GPU-enabled nodes and exposes detailed GPU metrics.</p><pre># Add the GPU Operator Helm repository<br>helm repo add nvidia https://helm.ngc.nvidia.com/nvidia<br>helm repo update<br><br># Install the GPU Operator, which includes the dcgm-exporter<br>helm install --wait --generate-name \<br>     -n gpu-operator --create-namespace \<br>     nvidia/gpu-operator</pre><p><strong>Step 2:</strong> <strong>Install Prometheus.</strong> If you don’t have a monitoring stack, install Prometheus. It will scrape (collect) the metrics from the DCGM exporter.</p><pre># Add the Prometheus community repo<br>helm repo add prometheus-community https://prometheus-community.github.io/helm-charts<br>helm repo update<br><br># Install Prometheus<br>helm install prometheus prometheus-community/prometheus -n prometheus --create-namespace</pre><p><strong>Step 3</strong>: <strong>Configure Prometheus to Scrape GPU Metrics.</strong> Ensure your Prometheus configuration includes a scrape job that targets the dcgm-exporter endpoints. If you used the Helm charts, this is often configured automatically. If not, add a job like this to your prometheus.yml:</p><pre>- job_name: &#39;dcgm-exporter&#39;<br>  kubernetes_sd_configs:<br>  - role: endpoints<br>  relabel_configs:<br>  - source_labels: [__meta_kubernetes_service_label_app_kubernetes_io_name]<br>    action: keep<br>    regex: dcgm-exporter</pre><p>You can now query metrics like DCGM_FI_DEV_GPU_UTIL in Prometheus to see GPU utilization across your nodes.</p><h3>2. Horizontal Pod Autoscaler (HPA): Scaling Pod Replicas</h3><p>HPA automatically increases or decreases the number of pod replicas based on metrics like CPU utilization.</p><h4><strong>Step-by-Step: Implementing HPA for a UI Deployment</strong></h4><p>Let’s create an HPA for a simple RAG UI application.</p><p><strong>Step 1: Create a Deployment </strong>First, you need a workload to scale. Save this as rag-ui-deployment.yaml. Crucially, you must set resource requests for the metrics you want to scale on. HPA uses the request value to calculate utilization percentage.</p><pre>apiVersion: apps/v1<br>kind: Deployment<br>metadata:<br>  name: rag-ui-deployment<br>spec:<br>  replicas: 1<br>  selector:<br>    matchLabels:<br>      app: rag-ui<br>  template:<br>    metadata:<br>      labels:<br>        app: rag-ui<br>    spec:<br>      containers:<br>      - name: web-app<br>        image: your-rag-ui-image:latest<br>        ports:<br>        - containerPort: 80<br>        resources:<br>          requests:<br>            cpu: &quot;500m&quot; # Request 0.5 vCPU<br>          limits:<br>            cpu: &quot;1&quot;    # Limit to 1 vCPU</pre><p>Apply it: kubectl apply -f rag-ui-deployment.yaml</p><p><strong>Step 2: Create the HPA</strong> Now, define the HPA. Save this as rag-ui-hpa.yaml.</p><pre>apiVersion: autoscaling/v2<br>kind: HorizontalPodAutoscaler<br>metadata:<br>  name: rag-ui-hpa<br>spec:<br>  scaleTargetRef:<br>    apiVersion: apps/v1<br>    kind: Deployment<br>    name: rag-ui-deployment<br>  minReplicas: 1<br>  maxReplicas: 5<br>  metrics:<br>  - type: Resource<br>    resource:<br>      name: cpu<br>      target:<br>        type: Utilization<br>        # Scale up when the average CPU usage across all pods exceeds 70% of their requested CPU (i.e., &gt;350m)<br>        averageUtilization: 70</pre><p>Apply it: kubectl apply -f rag-ui-hpa.yaml</p><p>Kubernetes will now monitor the CPU usage of pods in rag-ui-deployment. If the average utilization exceeds 70%, it will add new pods, up to a maximum of 5.</p><p><strong>Fine-Tuning HPA Behavior</strong></p><p>To prevent the HPA from scaling up and down too aggressively (known as “flapping”), you can define scaling behavior.</p><pre># Add this &#39;behavior&#39; block to your HPA spec<br>behavior:<br>  scaleDown:<br>    # Wait 3 minutes before scaling down to ensure the traffic dip is stable<br>    stabilizationWindowSeconds: 180<br>    policies:<br>    # Rule: In any 15-second window, remove at most 50% of the current pods<br>    - type: Percent<br>      value: 50<br>      periodSeconds: 15<br>  scaleUp:<br>    # React immediately to load spikes<br>    stabilizationWindowSeconds: 0<br>    policies:<br>    # Rule: In any 15-second window, add at most 2 new pods<br>    - type: Pods<br>      value: 2<br>      periodSeconds: 15<br>    # If multiple policies were defined, choose the one that allows the most aggressive scale-up<br>    selectPolicy: Max</pre><h3>3. Vertical Pod Autoscaler (VPA): Right-Sizing Pod Resources</h3><p>VPA adjusts the CPU and memory requests of pods to match actual usage. This is great for optimizing resource allocation but can be disruptive because it often requires pod restarts.</p><h4>Step-by-Step: Implementing VPA</h4><p><strong>Step 1: Install VPA.</strong> VPA is not a standard Kubernetes component. You need to install it.</p><pre>git clone https://github.com/kubernetes/autoscaler.git<br><br>cd autoscaler/vertical-pod-autoscaler/<br>./hack/vpa-up.sh</pre><p><strong>Step 2: Create the VPA.</strong> Let’s create a VPA for our RAG UI deployment. The updateMode: &quot;Auto&quot; is powerful but use with caution—it will evict and recreate pods to apply new resource requests.</p><p>Save this as rag-ui-vpa.yaml.</p><pre>apiVersion: &quot;autoscaling.k8s.io/v1&quot;<br>kind: VerticalPodAutoscaler<br>metadata:<br>  name: rag-ui-vpa<br>spec:<br>  targetRef:<br>    apiVersion: &quot;apps/v1&quot;<br>    kind:       &quot;Deployment&quot;<br>    name:       &quot;rag-ui-deployment&quot;<br>  updatePolicy:<br>    # Automatically apply resource changes, which may involve restarting pods.<br>    # Other modes: &quot;Off&quot; (recommendations only), &quot;Initial&quot; (apply on creation only).<br>    updateMode: &quot;Auto&quot;<br>  resourcePolicy:<br>    containerPolicies:<br>      # Apply this policy to all containers in the pod<br>    - containerName: &#39;*&#39;<br>      # Set guardrails to prevent VPA from setting requests too low or too high<br>      minAllowed:<br>        cpu: &quot;1000m&quot;<br>        memory: &quot;2Gi&quot;<br>      maxAllowed:<br>        cpu: &quot;2000m&quot;<br>        memory: &quot;4Gi&quot;<br>      # VPA will manage both resource requests and limits<br>      controlledValues: &quot;RequestsAndLimits&quot;</pre><p>Apply it: kubectl apply -f rag-ui-vpa.yaml</p><blockquote><strong><em>Warning</em></strong><em>: Do not use HPA and VPA on the same metrics (e.g., CPU) at the same time in </em><em>Auto mode. They will conflict, causing scaling loops. VPA will change resource requests, which confuses HPA&#39;s utilization calculations.</em></blockquote><h3>4. Kubernetes Event-Driven Autoscaler (KEDA): Advanced, Event-Based Scaling</h3><p>KEDA is a game-changer for workloads that respond to external events, like jobs in a queue or custom Prometheus metrics. It extends HPA and can scale deployments down to zero replicas, offering huge cost savings.</p><h4>Step-by-Step: Scaling a Deployment with KEDA and AWS SQS</h4><p><strong>Step 1: Install KEDA.</strong> Use Helm for the easiest installation.</p><pre>helm repo add kedacore https://kedacore.github.io/charts<br>helm repo update<br>helm install keda kedacore/keda -n keda --create-namespace</pre><p><strong>Step 2: Configure Authentication</strong> <strong>(Example: AWS IRSA) </strong>KEDA needs credentials to access external services. The best practice on EKS is to use IAM Roles for Service Accounts (IRSA). You would create a service account that your deployment and KEDA can use, which has permissions to read from the SQS queue.</p><p><strong>Step 3: Create the ScaledObject.</strong> The ScaledObject is a custom resource that tells KEDA what to watch and how to scale. This example scales a demo-deployment based on the number of messages in an SQS queue.</p><p>Save as sqs-scaledobject.yaml.</p><pre>apiVersion: keda.sh/v1alpha1<br>kind: ScaledObject<br>metadata:<br>  name: amazonsqs-scaler<br>spec:<br>  scaleTargetRef:<br>    name: demo-deployment<br>  minReplicaCount: 0 # Key feature: scale to zero when the queue is empty<br>  maxReplicaCount: 10<br>  pollingInterval: 15 # Check the queue every 15 seconds<br>  cooldownPeriod: 180 # Wait 3 minutes after the last message before scaling to 0<br>  triggers:<br>  - type: aws-sqs-queue<br>    metadata:<br>      queueURL: &quot;YOUR_SQS_QUEUE_URL&quot;<br>      awsRegion: &quot;us-east-1&quot;<br>      # Target: aim for 1 pod for every 10 messages in the queue<br>      queueLength: &quot;10&quot;<br>    authenticationRef:<br>      # This references a TriggerAuthentication object or a service account with IRSA<br>      name: keda-service-account</pre><p>Apply it: kubectl apply -f sqs-scaledobject.yaml</p><h4><strong>Advanced KEDA Example: Scaling a GenAI Model on Multiple Metrics</strong></h4><p>You can combine triggers. KEDA will evaluate all triggers and scale to the replica count required by the <em>most demanding</em> one. This example scales a Llama model based on both HTTP traffic and GPU utilization.</p><pre>apiVersion: keda.sh/v1alpha1<br>kind: ScaledObject<br>metadata:<br>  name: my-llama-deployment-scaler<br>spec:<br>  scaleTargetRef:<br>    name: my-llama-deployment<br>  triggers:<br>    # Trigger 1: Scale on HTTP requests per second<br>    - type: prometheus<br>      metadata:<br>        serverAddress: http://prometheus-server.default.svc:9090<br>        metricName: http_requests_per_sec # A unique name for this metric<br>        # Target: Add a pod for every 10 requests/sec<br>        threshold: &quot;10&quot;<br>        # PromQL: Calculate the per-second rate of requests over 1 minute<br>        query: sum(rate(http_requests_total[1m]))<br>    # Trigger 2: Scale on average GPU utilization<br>    - type: prometheus<br>      metadata:<br>        serverAddress: http://prometheus-server.default.svc:9090<br>        metricName: gpu_utilization_percent # A unique name for this metric<br>        # Target: Keep average GPU utilization around 70%<br>        threshold: &quot;70&quot;<br>        # PromQL: Calculate the average GPU utilization across all llama pods<br>        query: avg(DCGM_FI_DEV_GPU_UTIL{pod=~&quot;my-llama-.*&quot;}[30s])</pre><h3>5. Kubernetes Cluster Autoscaler (CA): Adding and Removing Nodes</h3><p>When HPA or KEDA scales up your pods but there’s no room on existing nodes, the pods get stuck in a Pending state. The Cluster Autoscaler (CA) solves this by adding new nodes to the cluster. It also removes underutilized nodes to save costs.</p><p>CA is highly cloud-provider-specific. On AWS EKS, it operates on Auto Scaling Groups (ASGs). On GKE, it uses Managed Instance Groups (MIGs). You typically enable it through your cloud provider’s console or CLI. While effective, it’s slower because it works by polling and managing predefined node groups.</p><h3>6. AWS Karpenter: High-Performance Cluster Autoscaling</h3><p>Karpenter is a modern, high-performance replacement for the Cluster Autoscaler on AWS. It directly observes pending pods and launches perfectly-sized EC2 instances just-in-time, making it much faster and more efficient.</p><p><strong>Advantages Over CA</strong></p><ul><li>Speed: Launches nodes in near real-time, drastically reducing pod pending times.</li><li>Efficiency: Provisions nodes based on the aggregate resource requests of pending pods, avoiding wasted capacity.</li><li>Flexibility: Can select from a wide range of instance types and purchasing options (Spot/On-Demand) on the fly.</li><li>Consolidation: Actively works to drain and terminate underutilized nodes to reduce costs.</li></ul><h4>Step-by-Step: Implementing Karpenter for GenAI GPU Workloads</h4><p><strong>Step 1: Install Karpenter</strong> This involves setting up IAM roles, an SQS queue for eventing, and installing the Karpenter Helm chart. Follow the official <a href="https://karpenter.sh/docs/getting-started/getting-started-with-karpenter/">Karpenter Installation Guide</a> for the most up-to-date instructions.</p><p><strong>Step 2: Configure an </strong><strong>EC2NodeClass</strong> This resource defines the AWS-specific configuration for nodes that Karpenter creates.</p><p>Save as gpu-nodeclass.yaml.</p><pre>apiVersion: karpenter.k8s.aws/v1<br>kind: EC2NodeClass<br>metadata:<br>  name: default-gpu<br>spec:<br>  # Use Amazon Linux 2023. Karpenter finds the latest optimized AMI automatically.<br>  amiFamily: AL2023<br>  # The IAM role for the nodes, giving them permissions to connect to the EKS cluster.<br>  role: &quot;karpenter-node-role-eks-demo&quot;<br>  # Select subnets and security groups for the new nodes, usually via tags.<br>  subnetSelectorTerms:<br>    - tags:<br>        &quot;karpenter.sh/discovery&quot;: &quot;eks-cluster-name&quot;<br>  securityGroupSelectorTerms:<br>    - tags:<br>        &quot;karpenter.sh/discovery&quot;: &quot;eks-cluster-name&quot;</pre><p>Apply it: kubectl apply -f gpu-nodeclass.yaml</p><p><strong>Step 3: Configure a </strong><strong>NodePool</strong> The NodePool defines the rules for provisioning, like which instance types are allowed and how to handle node lifecycle.</p><p>Save as gpu-nodepool.yaml.</p><pre>apiVersion: karpenter.sh/v1<br>kind: NodePool<br>metadata:<br>  name: eks-gpu-np<br>spec:<br>  # Point to the NodeClass that defines AWS infrastructure settings<br>  nodeClassRef:<br>    name: default-gpu<br>  # Define the requirements for nodes in this pool<br>  requirements:<br>    # Only provision modern &#39;g6&#39; generation GPU instances<br>    - key: &quot;karpenter.k8s.aws/instance-generation&quot;<br>      operator: In<br>      values: [&quot;g6&quot;]<br>    # Prioritize cheaper Spot instances, fall back to On-Demand if Spot is unavailable<br>    - key: &quot;karpenter.sh/capacity-type&quot;<br>      operator: In<br>      values: [&quot;spot&quot;, &quot;on-demand&quot;]<br>  # Define how to deprovision nodes<br>  disruption:<br>    # Terminate nodes when they are empty OR if their pods can be moved to other nodes<br>    consolidationPolicy: WhenEmptyOrUnderutilized</pre><p>Apply it: kubectl apply -f gpu-nodepool.yaml</p><h4>Karpenter in Action: A GPU Workload Demo</h4><ol><li><strong>Submit a GPU Job</strong>: You submit a fine-tuning job that requires a GPU</li></ol><pre>apiVersion: batch/v1<br>kind: Job<br>metadata:<br>  name: my-llama-job<br>spec:<br>  template:<br>    metadata:<br>      labels:<br>        app: my-llama-job<br>    spec:<br>      restartPolicy: Never<br>      containers:<br>      - name: fine-tuning-container<br>        image: my-finetuning-image:latest<br>        resources:<br>          limits:<br>            nvidia.com/gpu: 1 # Request one GPU</pre><p>2. <strong>Pod is Pending</strong>: If no GPU nodes exist, the pod will be stuck.</p><pre>$ kubectl get pods -l app=my-llama-job<br>NAME                  READY   STATUS    RESTARTS   AGE<br>my-llama-job-plgb5    0/1     Pending   0          22s<br><br>$ kubectl describe pod -l app=my-llama-job | grep Scheduling<br>Warning  FailedScheduling  ... 0/2 nodes are available: 2 Insufficient nvidia.com/gpu.</pre><p>3. <strong>Karpenter Acts</strong>: Karpenter sees the pending pod, checks its NodePool requirements (g6 instance), and directly calls the EC2 API to provision a new g6.xlarge node.</p><p>4. <strong>Node Joins &amp; Pod Runs</strong>: The new GPU node joins the cluster within a minute or two. The Kubernetes scheduler immediately places the pending pod on the new node.</p><pre># A new GPU node appears<br>$ kubectl get nodes -l nvidia.com/gpu.present=true<br>NAME                                       STATUS   ROLES    AGE   VERSION<br>ip-10-0-32-176.us-west-2.compute.internal  Ready    &lt;none&gt;   2m    v1.28.5-eks<br><br># The pod is now running<br>$ kubectl get pods -l app=my-llama-job<br>NAME                  READY   STATUS    RESTARTS   AGE<br>my-llama-job-plgb5    1/1     Running   0          9m20s</pre><p>5. <strong>Job Finishes &amp; Node is Removed</strong>: Once the job completes, the node becomes empty. Karpenter’s consolidation policy detects this, and after a short period, terminates the expensive GPU node automatically. This just-in-time provisioning and cleanup cycle is what makes Karpenter incredibly cost-effective for bursty GenAI workloads.</p><h3>References</h3><ul><li><a href="https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/">https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/</a></li><li><a href="https://www.stacksimplify.com/aws-eks/aws-eks-kubernetes-autoscaling/learn-to-master-horizontal-pod-autoscaling-on-aws-eks/">https://www.stacksimplify.com/aws-eks/aws-eks-kubernetes-autoscaling/learn-to-master-horizontal-pod-autoscaling-on-aws-eks/</a></li><li><a href="https://cloud.google.com/kubernetes-engine/docs/concepts/horizontalpodautoscaler">https://cloud.google.com/kubernetes-engine/docs/concepts/horizontalpodautoscaler</a></li><li><a href="https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler">https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler</a></li><li><a href="https://kubernetes.io/docs/concepts/cluster-administration/cluster-autoscaling/">https://kubernetes.io/docs/concepts/cluster-administration/cluster-autoscaling/</a></li><li><a href="https://karpenter.sh/v1.4/">https://karpenter.sh/v1.4/</a></li><li><a href="https://github.com/kubernetes-sigs/prometheus-adapter">https://github.com/kubernetes-sigs/prometheus-adapter</a></li><li><a href="https://github.com/kubernetes-sigs/prometheus-adapter">https://github.com/kubernetes-sigs/prometheus-adapter</a></li></ul><h3>Read <a href="https://medium.com/@pratik.vyas_10544/genai-on-prod-deploying-generative-ai-models-on-kubernetes-part-1-e9b1469b95cb">Part-1</a></h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8678bfacfded" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Agentic AI is Transforming SaaS]]></title>
            <link>https://medium.com/@pratik.vyas_10544/how-agentic-ai-is-transforming-saas-6aee7ecb9520?source=rss-d183e2545ed0------2</link>
            <guid isPermaLink="false">https://medium.com/p/6aee7ecb9520</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[saas]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Pratik]]></dc:creator>
            <pubDate>Tue, 24 Jun 2025 16:23:20 GMT</pubDate>
            <atom:updated>2025-06-24T17:11:55.111Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/870/1*8Pxo56d_gLt2U4sxxp-gsg.png" /></figure><p>The integration of Agentic AI into SaaS platforms is not just an incremental upgrade but a paradigm shift, transforming software into autonomous agents capable of perceiving, planning, deciding, and acting independently to achieve complex business goals. This blog expands on the technical foundations, architectural components, and practical applications of Agentic AI in SaaS, highlighting how this synergy is revolutionizing enterprise software.</p><h3>Understanding Agentic AI Architecture in SaaS</h3><p>Agentic AI systems are designed as autonomous agents with modular components that work cohesively to deliver intelligent behavior. These components include:</p><ul><li>Perception Module: Gathers and interprets diverse data inputs from the environment, including textual, visual, and structured data sources. In SaaS, this could mean extracting insights from user interactions, system logs, or external APIs using advanced NLP and computer vision techniques</li><li>Cognitive Module: The “brain” where goals are represented, plans are formulated, and decisions are made. It uses goal-oriented architectures to define explicit or implicit objectives such as maximizing customer satisfaction or optimizing operational efficiency.</li><li>Planning Module: Generates sequences of actions or hierarchical sub-goals to meet objectives. For SaaS, this can involve orchestrating multi-step workflows like automated customer onboarding or dynamic resource allocation</li><li>Decision-Making Module: Evaluates options based on current context and goals, selecting optimal actions autonomously. This replaces manual rule-based systems with adaptive, context-aware decision processes.</li><li>Action Execution Module: Carries out decisions through virtual actions such as API calls, message dispatch, or database updates, effectively enabling the AI to interact with the SaaS environment and external systems.</li><li>Learning Module: Continuously improves performance via reinforcement learning, supervised or unsupervised learning, adapting to new data and feedback in real-time.</li></ul><p>This architecture enables SaaS platforms to evolve from static tools into self-optimizing, autonomous systems that can handle complex, multi-dimensional tasks with minimal human intervention.</p><h3>Technical Innovations Empowering Agentic AI in SaaS</h3><h4>Natural Language Interfaces (NLIs)</h4><p>Agentic AI leverages transformer-based models (e.g., GPT variants) to power NLIs that convert plain-language user queries into complex database commands or API calls. This eliminates traditional ORM layers and reduces dependency on technical expertise, enabling users to retrieve multi-dimensional insights by simply asking questions like “How much revenue was lost due to downtime this year?”</p><h4>Automated Query Generation and Data Retrieval</h4><p>Agentic AI automates the creation of sophisticated SQL or NoSQL queries by embedding structured and unstructured data representations. Models like Azure OpenAI GPT use vector embeddings to map data seamlessly, enabling real-time extraction and visualization of operational metrics, customer feedback, and more</p><h4>Intelligent Code Generation and Testing</h4><p>Agentic AI tools such as GitHub Copilot analyze millions of code repositories to auto-generate boilerplate code, macros, SQL queries, and unit tests, accelerating software development cycles and reducing human error</p><h4>Multi-Agent Collaboration</h4><p>Complex SaaS workflows are managed by multi-agent systems where specialized AI agents communicate and coordinate to accomplish tasks. For example, one agent might handle customer data analysis while another manages communication workflows, collectively delivering a cohesive service experience</p><h4>Real-Time Adaptation and Feedback Loops</h4><p>Agentic AI systems process real-time data streams to dynamically adjust their actions. Feedback loops monitor performance and user interactions, enabling continuous learning and refinement of strategies without manual reprogramming</p><h3>Key Metrics for Measuring the ROI of Agentic AI</h3><p>To measure the true return on investment of Agentic AI, businesses must look beyond simple automation and evaluate its impact across efficiency, revenue, and experience. Here are the essential metrics to track:</p><p><strong>1. Is It Saving Time and Money?</strong></p><ul><li><strong>Less Busywork:</strong> How much time do your employees get back now that the AI handles repetitive tasks? Look at the hours saved each week or month.</li><li><strong>Cheaper Operations:</strong> Is it costing you less to get things done? For example, calculate the cost of solving one IT support ticket before and after the AI.</li><li><strong>Focus on Important Work:</strong> Are your employees now able to focus on bigger goals instead of boring tasks? See if more time is being spent on creative or strategic work.</li><li><strong>Faster Fixes:</strong> How much quicker are problems getting solved for your employees or customers? Measure the time it takes from when someone asks for help to when the issue is closed.</li></ul><p><strong>2. Is It Helping You Make More Money?</strong></p><ul><li><strong>Happier, Paying Customers:</strong> Can you connect the AI to an increase in sales or see that customers are sticking around longer because they’re getting better service?</li><li><strong>New Ways to Earn:</strong> Has the AI allowed you to offer new services you couldn’t before? For example, providing 24/7 support or personalized recommendations that lead to new sales.</li></ul><p><strong>3. Are Your Customers Happier?</strong></p><ul><li><strong>Good Reviews:</strong> Are customer satisfaction scores or reviews getting better since the AI was introduced?</li><li><strong>Fewer Call-Backs:</strong> Is the AI solving customer problems on the first try, so they don’t have to keep contacting you?</li><li><strong>Always Available:</strong> Are customers getting instant answers and help around the clock, even at night or on weekends?</li></ul><p><strong>4. Are Your Employees Happier?</strong></p><ul><li><strong>Better Work Life:</strong> Do your employees feel more satisfied now that they can focus on more interesting work? You can find this out with simple surveys.</li><li><strong>Less Burnout:</strong> By taking over the most tedious tasks, is the AI helping to reduce employee stress and burnout?</li></ul><p><strong>5. Can It Grow with Your Business?</strong></p><ul><li><strong>Handles More Work:</strong> Can the AI take on a lot more work without you having to hire more people? It should be able to handle 100 requests as easily as it handles 10.</li><li><strong>Adapts to Change:</strong> How easy is it to teach the AI a new task or update it when your business changes? It should be much faster than retraining an entire team.</li></ul><h3>Practical SaaS Use Cases Enhanced by Agentic AI</h3><h4>1. Autonomous Sales and Account Management</h4><p>Agentic AI automates lead scoring by analyzing vast datasets of customer behavior and market signals, prioritizing prospects with the highest conversion potential. It crafts personalized outreach campaigns, monitors account health to predict churn, and manages multi-channel engagement with optimized timing and messaging. These capabilities significantly improve sales efficiency and pipeline accuracy</p><h4>2. Intelligent Customer Support</h4><p>Beyond traditional chatbots, agentic AI autonomously manages complex customer interactions, proactively guiding users through multi-step processes such as troubleshooting, billing inquiries, or product recommendations. It anticipates user needs, reducing friction and enhancing satisfaction</p><h4>3. Workflow Automation and Optimization</h4><p>Agentic AI orchestrates end-to-end workflows across departments — automating tasks like case creation, resource allocation, and compliance checks. For example, the SearchUnify Virtual Assistant (SUVA) integrates conversational AI with backend systems to streamline user queries and automate case management within a single interface</p><h4>4. Finance and Risk Management</h4><p>Agentic AI autonomously analyzes financial data, predicts cash flow trends, adjusts loan terms, and recommends best-fit financial products in real-time. This surpasses traditional robotic process automation by incorporating predictive analytics and adaptive decision-making</p><h4>5. Software Development Acceleration</h4><p>By generating code snippets, debugging assistance, and test cases automatically, agentic AI reduces developer workload and accelerates SaaS product iterations, enabling faster feature delivery and higher software quality</p><h3>Challenges and Technical Considerations</h3><ul><li>Transparency and Explainability: Autonomous decisions require clear audit trails and explainability to build trust and meet compliance requirements.</li><li>Modular and Scalable Design: Architectures must support modular agents that can be scaled horizontally to handle increasing complexity and workload.</li><li>Security and Privacy: Autonomous agents interacting with sensitive data must enforce strict security protocols and data governance.</li><li>Integration Complexity: Seamless integration with legacy enterprise systems demands robust APIs and middleware capable of supporting agentic workflows.</li><li>Continuous Monitoring: Real-time monitoring and feedback mechanisms are essential to detect anomalies and ensure reliable agent performance.</li></ul><h3>Conclusion</h3><p>Agentic AI is reshaping SaaS by embedding autonomous intelligence that perceives, plans, decides, acts, and learns continuously. This technical evolution enables SaaS platforms to deliver unprecedented efficiency, scalability, and personalization, driving competitive advantage in a fast-evolving digital landscape. As Agentic AI architectures mature, multi-agent collaboration, natural language interfaces, and real-time adaptation will become standard features, ushering in a new era of intelligent, autonomous SaaS solutions.</p><p>Businesses and developers embracing this shift today will lead the transformation toward software that not only supports but actively drives enterprise success.</p><p>This detailed exploration draws on recent technical insights and industry examples to provide a comprehensive understanding of Agentic AI’s role in advancing SaaS platforms</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6aee7ecb9520" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transformer : Mixture-of-expert and Switch Transformer ( Part 4)]]></title>
            <link>https://medium.com/@pratik.vyas_10544/transformer-mixture-of-expert-and-switch-transformer-part-4-8fea9f540073?source=rss-d183e2545ed0------2</link>
            <guid isPermaLink="false">https://medium.com/p/8fea9f540073</guid>
            <category><![CDATA[mixture-of-experts]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[switch-transformer]]></category>
            <dc:creator><![CDATA[Pratik]]></dc:creator>
            <pubDate>Sun, 20 Apr 2025 11:57:08 GMT</pubDate>
            <atom:updated>2025-04-22T07:33:14.276Z</atom:updated>
            <content:encoded><![CDATA[<h3>Transformer : Mixture-of-experts ( Part-4)</h3><p><strong>Read previous parts before proceeding to Part-4</strong></p><ul><li><strong>Read </strong><a href="https://medium.com/@pratik.vyas_10544/transformer-encoder-part-1-1d926707f74b"><strong>Part-1</strong></a><strong> Encoder</strong></li><li><strong>Read </strong><a href="https://medium.com/@pratik.vyas_10544/transformer-decoder-part-2-644f3a12ff60"><strong>Part-2</strong></a><strong> Encoder-Decoder</strong></li><li><strong>Read </strong><a href="https://medium.com/@pratik.vyas_10544/transformer-decoder-only-generative-model-part-3-0b38c10c2ae9"><strong>Part-3</strong></a><strong> Decoder only ( Generative Model )</strong></li><li>Part-5 Switch Transformer (<strong>Coming soon….</strong> )</li></ul><h3>Part-4 Mixture of Experts (MoE)</h3><ol><li>Overview</li><li>Architecture</li><li>Sparsely-gated MoE Layer ( How it works )</li><li>Why we need MoE (benefits)</li><li>Limitation of MoE and Switch Transformer</li></ol><h4>Overview</h4><p>Training and deploying massive, dense Transformer models — where every parameter is active for every input — quickly becomes prohibitively expensive. This is where <strong>Mixture-of-Experts (MoE)</strong> steps in as a game-changer.</p><p>Think of it like this:</p><ul><li><strong>Regular Transformer:</strong> A single, all-knowing individual trying to answer every question.</li><li><strong>MoE Transformer:</strong> A panel of specialized experts (e.g., a linguist, a scientist, a historian), each best suited to answer specific types of questions. A clever assistant (the router) directs each question to the most appropriate expert on the panel.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DTXy28_oGXUMeHndkjORxw.png" /></figure><h4>Architecture</h4><p>As shown below, MoE layers are typically integrated after Add/Norm(1) layers of a Transformer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*OuqtiSNAEoeVRimYfAaB4Q.png" /><figcaption>image source : <a href="https://huggingface.co/blog/moe">https://huggingface.co/blog/moe</a></figcaption></figure><p>An MoE is composed of two core components:</p><ul><li>Experts E1, …,En: n FFN that are each specialized at a given task.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/262/1*xpM9Vc6SW0oX3bb4qNQ9Lg.png" /></figure><ul><li>Gate G: A single network that decides the extent at which each of the n experts should be activated. Its output is an n-dimensional vector G(x), where the ith component G(x)i corresponds to the weight attributed to expert Ei.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/79/1*d72um9-0QDwnROc_E0A5Jg.png" /></figure><p>The gate G and experts Ei are sometimes called router and channels respectively.</p><p>There are mainly two types of MoEs</p><ul><li><strong>Densely-gated MoE: </strong><br>The gate provides weights that give information about how important <strong>each expert </strong>is.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/376/1*cI3zsubzaHv6e9WwP7PJ4g.png" /></figure><p>The final result is a weighted average of the output of <strong>each expert</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/276/1*xVx51pcFwef35xK_A4_Aug.png" /></figure><ul><li><strong>Sparsely-gated MoE: </strong><br><strong>Only k experts</strong> can be activated at any given time, where “k” is a hyperparameter chosen and fixed beforehand.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/559/1*O0FFFBZ50NK_Nvkh80MxOQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/275/1*xCzCKUj6j5rpmUiRTweCaw.png" /></figure><p>By noting “<em>Ik</em>” the set of indices corresponding to the “k” activated experts ( in this case , Exper 1 and Expert 3 ).</p><p>This setup is useful to bring down the number of activations in the forward pass. It only uses “k/n” of the activations used in the densely-gated setup.</p><h3><strong>Sparsely-gated </strong>MoE Layer ( How it works )</h3><p>In an MoE layer, instead of applying a single feed-forward network (FFN) to all tokens, we have multiple <strong>experts</strong> (each an FFN) and a <strong>gating mechanism</strong> that decides which experts to use for each token. The key components are:</p><ul><li><strong>Experts</strong>: Multiple FFNs, each parameterized by weight matrices.</li><li><strong>Gating Network</strong>: A linear layer that computes scores to route each token to one or more experts.</li><li><strong>Routing</strong>: The process of selecting experts based on gating scores, often using top-k routing.</li></ul><p>The output of the MoE layer is a weighted combination of the outputs from the selected experts, computed via matrix operations.</p><blockquote><strong>In continuation of </strong><a href="https://medium.com/@pratik.vyas_10544/transformer-encoder-part-1-1d926707f74b"><strong>Part-1</strong></a><strong> Section 5 Add/Norm(1)<br></strong>The output of the Add/Norm(1) operation, which we label as Norm1_Output(X), serves as the input to the Mixture-of-Experts (MoE) layer</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Us1jLnmiAI8DAAW6IIeEag.png" /></figure><p><strong>Let’s understand each steps as below</strong></p><p><strong>Gating Network</strong></p><p>The gating network takes the input x∈R<em>3×768</em> and computes a score for each expert. Suppose we have N experts (e.g., N=4 ). The gating network is a linear layer followed by a softmax:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/261/1*xfgtSsqb_GMUhqXtqhKDcg.png" /></figure><p>where:</p><ul><li>Wg (shape: 768 x N ) is the weight matrix of the gating network.</li><li>x * Wg (shape: 3 x N ) computes a score for each of the 3 tokens for each of the N experts.</li><li>The softmax is applied row-wise (over the N experts) to produce probabilities:</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/342/1*MMijip0yJKE24ZiP4YGK0A.png" /></figure><p>Thus, G(x) (shape: 3 x N ) , where each row sums to 1, representing the probability distribution over experts for each token.</p><p>For example, if N=4, G(x) might look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/1*gcijla3v2VueO20ceOKqtw.png" /></figure><p>Each row corresponds to one token, and each column corresponds to one expert.</p><p><strong>Routing: Top-k Selection</strong></p><p>In practice, to save computation, only the top-k experts (e.g., k=2 ) are selected for each token based on the gating scores. This is done by:</p><ol><li>For each row of G(x), select the indices of the k highest probabilities.</li><li>Create a sparse gating matrix S ( shape: 3 x N). <br>- where Sij=G(x)ij​ if expert j is in the top-k for token i, else Sij=0.<br>- For example, for token 1, if experts 1 and 2 have the highest scores (0.5 and 0.3), then: S1:=[0.5,0.3,0,0]<br>- The matrix S ensures that only the selected experts contribute to the output.</li></ol><p><strong>Experts: Feed-Forward Networks (FFNs)<br></strong>Each expert is a FFN, typically consisting of two linear layers with a non-linearity (e.g., ReLU or GeLU) in between. For expert i, the computation is:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/443/1*Fv7-x6QY2ZKZgjAdbh4YPA.png" /></figure><p>Where</p><ul><li>Wi,1​(shape: 768 x dg) (e.g. dg=3072) projects the input to a higher-dimensional space.</li><li>Wi,2​ (shape: dg x 768) projects back to the original dimension.</li><li>bi,1​,bi,2​ are biases.</li></ul><p>For a batch of tokens, we compute:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/407/1*kSKDMKYyRpLTui177D6NJQ.png" /></figure><p>where xWi,1 shape is ​3 x dg , and the output Ei(x) shape is 3 × 768</p><p>Each of the N experts produces an output matrix of shape (3,768) but only the selected experts (per the routing) are used.</p><p><strong>Combining (Weighted) Expert Outputs</strong></p><p>The final output of the MoE layer is a weighted sum of the expert outputs, weighted by the gating scores in S. For each token i:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/225/1*c89GfZ_atO1wiu5vvDgYvQ.png" /></figure><p>In matrix form, if we stack the expert outputs into a tensor E (shape: N x 3 x 768 ), the output y (shape: 3 x 768) is</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/202/1*WkSwU_8AM0lTTbG2FnrJSg.png" /></figure><p>where S:,j (shape: 3) is the jth column of S, and Ej(x) (shape: 3 x 768) is the output of expert j. This is a sparse operation, as S has at most k non-zero entries per row.</p><p>For efficiency, only the top-k experts are computed for each token, and their outputs are scaled by the corresponding gating probabilities.</p><h4>Why we need MoE (benefits)</h4><p>We need Mixture of Experts (MoE) architectures for several key reasons, all ultimately boiling down to the goal of creating larger, more capable, and more efficient models:</p><ol><li><strong>Scaling Model Capacity Beyond Computational Limits:</strong></li></ol><p><strong>The Problem:</strong> Training larger, dense models (models where every parameter is used for every input) requires exponentially more computational resources (memory, compute time, data). Eventually, you hit a wall. You can’t afford to train or even run the model on available hardware.</p><p><strong>MoE Solution:</strong> MoE introduces sparsity. Instead of having a single, giant network where every connection is active for every input, MoE uses multiple smaller “expert” networks and a routing mechanism. Only a <em>subset</em> of the experts are active for a given input. This allows the <em>total</em> number of parameters in the model to be very large, while the number of parameters used <em>per input</em> remains manageable.</p><p><strong>2. Specialization and Learning Complex Patterns:</strong></p><p><strong>The Problem:</strong>A single, dense neural network might struggle to effectively model complex, heterogeneous data distributions. Some parts of the data might require different representations or processing steps than others.</p><p><strong>MoE Solution:</strong> Experts can specialize in different aspects of the data. One expert might be good at handling grammatical structures, another at understanding semantic relationships, and yet another at dealing with specific topics. This allows the model to learn a richer and more nuanced representation of the data.</p><p><strong>3. Improved Generalization:</strong></p><p><strong>The Problem:</strong>Overfitting. Very large models can overfit to the training data, meaning they perform well on the training data but poorly on unseen data.</p><p><strong>MoE Solution:</strong> The sparsity introduced by MoE can act as a regularizer. Because only a subset of the parameters are used for each input, the model is forced to learn more robust and generalizable representations. It also forces the router to become more robust in its decision process because each expert is not “perfect” for every given input. This can lead to improved performance on unseen data.</p><p><strong>4. Efficiency in Inference (Using the Model):</strong></p><p><strong>The Problem: </strong>Even if you can train a large, dense model, using it for inference (making predictions on new data) can be computationally expensive.</p><p><strong>MoE Solution:</strong> Because only a small subset of the experts are active during inference, the computational cost of making a prediction is significantly lower than for a dense model with the same number of parameters. This makes it possible to deploy very large models in real-world applications.</p><h4>Limitation of MoE and Switch Transformer</h4><p>While Mixture-of-Experts (MoE) is a powerful concept, the Switch Transformer specifically addresses key limitations and challenges of the general MoE framework, leading to significant improvements in scalability, efficiency, and ease of implementation.</p><h4>1. Simplified Routing (Single Expert Selection)</h4><ul><li><strong>Traditional MoE</strong>: Typically selects multiple experts (e.g., top-k) per token and combines their outputs using weighted averages. This increases computation and memory usage, as multiple expert outputs must be computed and aggregated.</li><li><strong>Switch Transformer</strong>: Selects <strong>only one expert</strong> per token, reducing computational overhead. This “switching” mechanism simplifies the routing process and makes it more efficient, especially for large-scale models with thousands of experts.</li><li><strong>Benefit</strong>: Lower compute and memory requirements, enabling scaling to models with trillions of parameters.</li></ul><h4>2. Improved Scalability</h4><ul><li>Traditional MoE models often face challenges in scaling to extremely large sizes due to communication overheads (e.g., sending tokens to multiple experts) and load balancing issues.</li><li>The Switch Transformer’s single-expert routing and load balancing loss make it easier to scale across hundreds or thousands of devices. For example, it can handle models with 1.6 trillion parameters while maintaining computational efficiency.</li></ul><h4>3. Enhanced Load Balancing</h4><ul><li>Traditional MoE models may suffer from expert imbalance, where some experts are underutilized while others are overloaded, leading to inefficient resource use.</li><li>The Switch Transformer’s explicit load balancing loss ensures tokens are distributed evenly across experts, maximizing hardware utilization and reducing training/inference bottlenecks.</li></ul><h4>4. Practical Efficiency for Large Models</h4><ul><li>While MoE is theoretically efficient, implementing it at scale requires careful engineering. The Switch Transformer provides a practical framework for deploying MoE in real-world settings, with optimizations like:</li><li><strong>Sparsity</strong>: Only a fraction of parameters are used per token.</li><li><strong>Distributed Computing</strong>: Efficient sharding of experts across devices.</li><li><strong>Reduced Communication</strong>: Single-expert routing minimizes data transfer between devices.</li></ul><h4>5. Performance Gains</h4><ul><li>The Switch Transformer achieves comparable or better performance than dense transformers with significantly fewer computational resources. For example, a Switch Transformer with 1.6 trillion parameters can outperform a dense model with 100 billion parameters while using less compute.</li><li>This makes it ideal for applications where computational resources are a bottleneck, such as large-scale NLP tasks.</li></ul><h4>6. Addressing MoE Limitations</h4><ul><li><strong>Complexity</strong>: Traditional MoE models are complex to implement and tune due to multiple expert selections and weighted combinations.</li><li><strong>Training Instability</strong>: MoE models can suffer from training instability due to uneven expert utilization.</li><li>The Switch Transformer simplifies the architecture (single expert) and stabilizes training (load balancing loss), making MoE more accessible and robust.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8fea9f540073" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transformer : Decoder Only Generative Model( Part-3)]]></title>
            <link>https://medium.com/@pratik.vyas_10544/transformer-decoder-only-generative-model-part-3-0b38c10c2ae9?source=rss-d183e2545ed0------2</link>
            <guid isPermaLink="false">https://medium.com/p/0b38c10c2ae9</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[decoder-only-transformer]]></category>
            <category><![CDATA[decoder]]></category>
            <dc:creator><![CDATA[Pratik]]></dc:creator>
            <pubDate>Wed, 16 Apr 2025 08:34:28 GMT</pubDate>
            <atom:updated>2025-04-22T04:44:37.874Z</atom:updated>
            <content:encoded><![CDATA[<h3>Transformer : Decoder Only Generative Model( Part-3)</h3><ul><li>Read <a href="https://medium.com/@pratik.vyas_10544/transformer-encoder-part-1-1d926707f74b"><strong>Part-1</strong></a> <strong>Encoder</strong></li><li>Read <a href="https://medium.com/@pratik.vyas_10544/transformer-decoder-part-2-644f3a12ff60"><strong>Part-2</strong></a> <strong>Encoder-Decoder</strong></li><li>Read <a href="https://medium.com/@pratik.vyas_10544/transformer-mixture-of-expert-and-switch-transformer-part-4-8fea9f540073"><strong>Part-4</strong></a><strong> Mixture-of-Experts (MoEs)</strong></li><li>Part-5 Switch Transformer (Coming soon…. )</li></ul><h3>Part-3 : Decoder Only ( Generative Model )</h3><p>In this part, I will discuss following points.</p><ol><li>Decoder token generation</li><li>How to control decoder output</li><li>Parameters That Influence Text Generation</li></ol><h3>Decoder token generation</h3><p>When you type “I love pizza because” in ChatGPT, it generate below text.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KfCFpj0ZipJBy0QkhQYXog.png" /></figure><p>Let’s understand how it generate token by token.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vwEusi8WK1mEnYzzPmOfFw.png" /></figure><p>The image shows how a decoder-only GPT model generates tokens autoregressively over time. Each vertical stack of boxes represents one step in the generation process. The dashed blue lines represent the flow of information between steps.</p><blockquote>Decoder-only models, in contrast to “Encoder” or “Encoder-Decoder” architectures (<a href="https://medium.com/@pratik.vyas_10544/transformer-encoder-part-1-1d926707f74b"><strong>Part-1</strong></a> and <a href="https://medium.com/@pratik.vyas_10544/transformer-decoder-part-2-644f3a12ff60"><strong>Part-2</strong></a>), leverage solely the decoder module of the Transformer architecture.</blockquote><blockquote>A key characteristic is the implementation of masked self-attention (refer <a href="https://medium.com/@pratik.vyas_10544/transformer-decoder-part-2-644f3a12ff60"><strong>Part-2</strong></a><strong> </strong>for more details). <br>Crucially, decoder-only models generate text without employing encoder-decoder attention (cross-attention).</blockquote><p><strong>Key observation</strong></p><ul><li><strong>Autoregressive Nature:</strong> Each prediction depends on the previously generated tokens. The model is “conditioned” on its own past outputs.</li><li><strong>Context-Awareness:</strong> The Masked Self-Attention mechanism allows the model to consider the context provided by the preceding tokens when making its predictions. The longer the sequence, the more context the model has.</li></ul><blockquote>It causes K,V cache issue during inference. It is out of scope for this blog , however You can check my blog for more details <a href="https://medium.com/@pratik.vyas_10544/llm-inference-vllm-tgi-tensorrt-17872f7df1f5">https://medium.com/@pratik.vyas_10544/llm-inference-vllm-tgi-tensorrt-17872f7df1f5</a></blockquote><ul><li><strong>Sequential Generation:</strong> Tokens are generated one at a time, sequentially.</li><li><strong>Timestamp Representation:</strong> The timestamps clearly show the temporal flow of token generation. Each step adds a new token to the sequence based on the previous steps.</li></ul><p><strong>Timestamps and Token Generation:</strong></p><p><strong>t0:</strong></p><ul><li><strong>Input:</strong> The model receives the &lt;sos&gt; (start-of-sentence) token as input.</li><li><strong>Processing:</strong> The &lt;sos&gt; token is embedded and passed through the decoder blocks (Masked Self-Attention, FFNN, etc. as shown in <a href="https://medium.com/@pratik.vyas_10544/transformer-decoder-part-2-644f3a12ff60">Part-2</a>).</li><li><strong>Prediction:</strong> The softmax layer outputs a probability distribution over the vocabulary. The word with the highest probability is “I” (boxed in red above).</li><li><strong>Output:</strong> The model predicts the token “I”.</li></ul><p><strong>t1:</strong></p><ul><li><strong>Input:</strong> The input now is &lt;sos&gt; I. Note how “I” from the previous step is now part of the input sequence.</li><li><strong>Processing:</strong> Both &lt;sos&gt; and “I” are embedded and passed through the decoder blocks. Crucially, in the masked self-attention, the model can now attend to &lt;sos&gt; <em>and</em> “I” when predicting the next token.</li><li><strong>Prediction:</strong> The softmax layer produces a probability distribution. The word with the highest probability is “love” (boxed in red).</li><li><strong>Output:</strong> The model predicts the token “love”.</li></ul><p><strong>t2:</strong></p><ul><li><strong>Input:</strong> The input sequence is “&lt;sos&gt; I love”.</li><li><strong>Processing:</strong> &lt;sos&gt;, “I”, and “love” are processed. The masked self-attention allows the model to attend to these three tokens.</li><li><strong>Prediction:</strong> The softmax predicts “pizza” (boxed in red).</li><li><strong>Output:</strong> The model predicts “pizza”.</li></ul><p><strong>t3:</strong></p><ul><li><strong>Input:</strong> The input sequence is “&lt;sos&gt; I love pizza”.</li><li><strong>Processing:</strong> The tokens are processed, with the model able to attend to &lt;sos&gt;, “I”, “love”, and “pizza”.</li><li><strong>Prediction:</strong> The softmax predicts “because” (boxed in red).</li><li><strong>Output:</strong> The model predicts “because”.</li></ul><p><strong>t4:</strong></p><ul><li><strong>Input:</strong> The input sequence is “&lt;sos&gt; I love pizza because”.</li><li><strong>Processing:</strong> The tokens are processed, with the model able to attend to all tokens now.</li><li><strong>Prediction:</strong> The softmax predicts “irresistible” (boxed in red).</li><li><strong>Output:</strong> The model predicts “irresistible”.</li></ul><h3>Controlling decoder output</h3><p>LLMs use decoding methods for text generation, assigning scores to each word (token) based on probability. Choosing the most likely word right away might not be the best in the long run. Sometimes, picking a slightly less likely word opens up better options later on. The ideal approach (evaluating all possibilities) is too computationally expensive. <br>Decoding methods, therefore, strive to balance “greediness” (choosing the most probable token immediately) with exploration (considering multiple options simultaneously) to improve text quality and coherence.</p><p>Key decoding methods are</p><ol><li>Greedy Search</li><li>Sampling</li><li>Beam Search</li><li>Top-K Sampling</li><li>Top-p (Nucleus) Sampling</li></ol><p>let’s walk through each decoding method using the example “I love pizza because…” and seeing how the model might predict the next token.</p><p><strong>Assumed Probability Distribution (for demonstration):</strong></p><p>After processing “I love pizza because,” let’s say the model’s Softmax layer produces the following probabilities for the next word:</p><ul><li>“it’s”: 0.01</li><li>“irresistible”: 0.3</li><li>“delicious”: 0.25</li><li>“I”: 0.02</li><li>“of”: 0.005</li><li>“so”: 0.15</li><li>“it”: 0.015</li><li>“amazing”: 0.1</li><li>“tasty”: 0.05</li><li>… (and many other words with very low probabilities)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/819/1*4ma57j_fzdm0dZxgcclXkw.png" /></figure><p><strong>1. Greedy Search:</strong></p><ul><li><strong>Process:</strong> Greedy search simply selects the token with the highest probability.</li><li><strong>Result:</strong> The model would choose “irresistible” (probability 0.3).</li><li><strong>Generated Sequence:</strong> “I love pizza because irresistible”</li></ul><p><strong>2. Sampling:</strong></p><ul><li><strong>Process:</strong> Sampling treats the probabilities as weights in a random draw. So, “irresistible” has a 30% chance of being selected, “delicious” has a 25% chance, “so” has a 15% chance, etc.</li><li><strong>Result (Example):</strong> Let’s say, based on a random draw, the model selects “delicious” (even though “irresistible” has a higher probability).</li><li><strong>Generated Sequence:</strong> “I love pizza because delicious”</li></ul><p><strong>3. Beam Search (k=2):</strong></p><ul><li><strong>Process:</strong> We keep track of the two most likely sequences.</li><li><strong>Step 1:</strong> The two most likely tokens are “irresistible” (0.3) and “delicious” (0.25).<br><strong>Beam 1:</strong> “I love pizza because ”<br><strong>Beam 2:</strong> “I love pizza because ”</li><li><strong>Step 2:</strong> For each beam, we generate the next token, calculating their sequence probability.</li><li>Beam1<br>- Imagine, next word of “<em>I love pizza because </em>” is “<em>irresistible</em>” <br>- the probability of the sequence “I love pizza because” is 0.25<br>- probability of “<em>irresistible</em>” is 0.3<br>- probability of sequence is<em> 0.25 * </em>0.3=<strong>0.075</strong></li><li>Beam2<br>- Imagine, next word of “<em>I love pizza because </em>” is “delicious” <br>- the probability of the sequence “I love pizza because” is 0.25<br>- probability of “<em>delicious</em>” is 0.25<br>- probability of sequence is<em> 0.25 * </em>0.25=<strong>0.0625</strong></li><li><strong>Step 3:</strong> Select two beams with highest probability and repeat again for next tokens.</li><li><strong>Result:</strong> hightest probability among beams is 0.075 that is “<em>irresistible”</em></li><li><strong>Generated Sequence:</strong> “I love pizza because <em>irresistible</em>”</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/252/1*De7QpwZcPuTB-Jbz-kbeTQ.png" /></figure><p><strong>4. Top-K Sampling (k=4):</strong></p><ul><li><strong>Process:</strong> <br>We select the top 4 most likely tokens: “irresistible”, “delicious”, “so”, and “amazing”.<br>- “irresistible”: 0.3<br>- “delicious”: 0.25<br>- “so”: 0.15<br>- “amazing”: 0.1</li><li>Next, we normalize these probabilities so they add up to 1:<br>- “irresistible”: 0.3 / (0.3 + 0.25 + 0.15 + 0.1) = 0.375<br>- “delicious”: 0.25 / 0.8 = 0.3125<br>- “so”: 0.15 / 0.8 = 0.1875<br>- “amazing”: 0.1 / 0.8 = 0.125<br>Then, we sample from these 4 tokens based on their normalized probabilities.</li><li><strong>Result (Example):</strong> Let’s say “so” is randomly selected.</li><li><strong>Generated Sequence:</strong> “I love pizza because so”</li></ul><p><strong>5. Top-p (Nucleus) Sampling (p=0.7):</strong></p><ul><li><strong>Process:</strong> We order tokens by probability and add them until the cumulative probability reaches at least 0.7.<br>- “irresistible”: 0.3 (cumulative = 0.3)<br>- “delicious”: 0.25 (cumulative = 0.55)<br>- “so”: 0.15 (cumulative = 0.7)<br>- “amazing”: 0.1 (cumulative = 0.8)</li><li>So we select: “irresistible”, “delicious”, and “so”.</li><li>Then we normalize the probabilities of selected tokens:<br>- “irresistible”: 0.3/ (0.3+0.25+0.15) = 0.428<br>- “delicious”: 0.25/0.7 = 0.357<br>- “so”: 0.15/0.7 = 0.214</li><li>Then, we sample from these 3 tokens based on their normalized probabilities.</li><li><strong>Result (Example):</strong> Let’s say “delicious” is randomly selected.</li><li><strong>Generated Sequence:</strong> “I love pizza because delicious”</li></ul><p>In short,</p><ul><li><strong>Greedy</strong>: Choose “irresistible” (most predictable).</li><li><strong>Sampling</strong>: Random choice, e.g., “delicious.”</li><li><strong>Beam Search</strong>: Explores multiple options, might lead to “delicious” or “irresistible” depending on subsequent probabilities.</li><li><strong>Top-K</strong>: Limits the choice to top K (e.g., 4), e.g., “so”.</li><li><strong>Top-P</strong>: Dynamically selects a subset, e.g., “delicious” based on P.</li></ul><h3>Parameters That Influence Text Generation</h3><p>Three important parameters are</p><ol><li>Temperature</li><li>Stop Sequences</li><li>Frequency and Presence Penalties</li></ol><h4>1. Temperature:</h4><p>Temperature controls the randomness of the text generated. It affects the probability distribution of the possible next tokens.</p><ul><li><strong>Lower temperature:</strong> Makes the output more predictable and focused.</li><li><strong>Higher temperature:</strong> Makes the output more random and varied.</li></ul><p>Temperature works by tweaking the numbers the model uses <em>before</em> it decides which word to pick next (this process uses something called “softmax”). This helps balance how creative the output is with how good it is.</p><p>Let’s undestand in details</p><ul><li><strong>Logits:</strong> The language model starts by creating a list of scores, called “logits,” for each possible word. These scores show how likely the model <em>initially</em> thinks each word is.</li><li><strong>Softmax:</strong> This is a function that takes those “logits” and turns them into probabilities (chances). The important thing is that all the probabilities added together equal 1 (or 100%).</li><li><strong>Temperature’s role:</strong> The temperature setting controls how random the output is. It does this by dividing the “logits” by the temperature value <em>before</em> the “softmax” step.<br>- <strong>High temperature (like, higher than 1):</strong> <br>When the temperature goes up, the logits get smaller (<em>make it uniform across vocab. Like “irresistible”: 0.4 , “delicious”:0.35 , “tasty” : 0.30. All token are equally likely </em>). This makes all the probabilities from the “softmax” more similar. This means the model is more likely to pick a less common word, leading to more creative and surprising output. But it also means there’s a higher chance of making mistakes or saying something that doesn’t make sense. <br>- <strong>Low temperature (like, lower than 1):</strong> <br>When the temperature goes down, the logits get bigger <strong>(</strong> <em>make it peaked distribution. Like “irresistible”: 0.7 , “delicious”:0.2 , “tasty ”: 0.1. Here ‘irresistible’ is more likely </em><strong>)</strong>. This makes the probability of the most likely word much higher. The model is more likely to pick the “safest” word, leading to more accurate and predictable output. But the output will be less diverse and creative.<br>- <strong>Temperature = 1:</strong> <br>When the temperature is set to 1, the logits stay the same. This keeps the probabilities just as the model originally calculated them. It’s a good “middle ground” or “neutral” option.</li></ul><h4>2. Stop Sequences:</h4><p>Stop sequences define specific strings of text that, when encountered during generation, will cause the LLM to stop generating further text.</p><p>You provide a list of one or more stop sequences. As the LLM generates text, it constantly checks if the generated text ends with any of the stop sequences. If a match is found, the generation process is halted.</p><p>When to use them:</p><ul><li><strong>Controlling output length:</strong> If you want to limit the length of the generated text, you can set a stop sequence that’s likely to appear naturally after a reasonable amount of text.</li><li><strong>Preventing repetition:</strong> You can use stop sequences to prevent the LLM from repeating the same phrases or sentences over and over again.</li><li><strong>Defining clear boundaries:</strong> Useful in scenarios where the LLM is generating structured data, such as code or lists, and you need to clearly mark the end of a particular section.</li><li><strong>Chatbots:</strong> To define when the chatbot should stop talking and wait for the user’s next input. Common stop sequences might be things like “User:” or “Bot:”.</li></ul><h4>3. Frequency and Presence Penalties:</h4><p>These penalties discourage the LLM from repeating previously used tokens or phrases. They help generate more diverse and novel text.</p><ul><li><strong>Frequency Penalty:</strong> Penalizes tokens based on how frequently they have <em>already</em> appeared in the generated text. The more a token has been used, the lower its probability becomes for future selection.</li><li><strong>Presence Penalty:</strong> Penalizes tokens simply based on whether they have appeared <em>at all</em> in the generated text. This is a more aggressive form of repetition control. If a token has been used even once, it gets penalized.</li></ul><p>When to use them:</p><ul><li><strong>Overcoming repetition:</strong> If the LLM tends to get stuck in loops and repeats the same words or phrases, increasing the frequency and/or presence penalties can help.</li><li><strong>Encouraging diversity:</strong> When you want the LLM to explore different vocabulary and avoid predictable patterns, these penalties can be useful.</li><li><strong>Generating creative text:</strong> Similar to a higher temperature, these penalties can lead to more unexpected and novel text.</li></ul><h4>Next Read <a href="https://medium.com/@pratik.vyas_10544/transformer-mixture-of-expert-and-switch-transformer-part-4-8fea9f540073"><strong>Part-4</strong></a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0b38c10c2ae9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Transformer : Decoder ( Part 2)]]></title>
            <link>https://medium.com/@pratik.vyas_10544/transformer-decoder-part-2-644f3a12ff60?source=rss-d183e2545ed0------2</link>
            <guid isPermaLink="false">https://medium.com/p/644f3a12ff60</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[decoder-and-encoder]]></category>
            <category><![CDATA[decoder-only-transformer]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[decoder]]></category>
            <dc:creator><![CDATA[Pratik]]></dc:creator>
            <pubDate>Fri, 11 Apr 2025 11:24:19 GMT</pubDate>
            <atom:updated>2025-04-21T15:47:07.352Z</atom:updated>
            <content:encoded><![CDATA[<h3>Transformer : Encoder-Decoder ( Part-2)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*GbxSv1CaPeoHyzGtkwEuQg.png" /><figcaption>Image source: <a href="https://arxiv.org/pdf/1706.03762">https://arxiv.org/pdf/1706.03762</a></figcaption></figure><ul><li>Read <a href="https://medium.com/@pratik.vyas_10544/transformer-encoder-part-1-1d926707f74b"><strong>Part-1</strong></a><strong> Encoder</strong></li><li>Read <a href="https://medium.com/@pratik.vyas_10544/transformer-decoder-only-generative-model-part-3-0b38c10c2ae9"><strong>Part-3</strong></a><strong> Decoder only ( Generative Model )</strong></li><li>Read <a href="https://medium.com/@pratik.vyas_10544/transformer-mixture-of-expert-and-switch-transformer-part-4-8fea9f540073"><strong>Part-4</strong></a><strong> Mixture-of-Experts</strong></li><li>Part-5 Switch Transformer (Coming soon…. )</li></ul><h3>Part-2 : Encoder-Decoder</h3><p>In this Part-2 , I will explain Encoder-Decoder module.</p><ol><li>Decoder Overview</li><li>How Decoder works</li><li>Masking future words</li><li>Masked Multi-head attention</li><li>Encoder-decoder attention</li><li>Feed forward Neural network and Add/Norm</li><li>Linear and softmax layers</li><li>Summary</li></ol><h3>Decoder overview:</h3><p>Imagine you want to convert the English sentence (the original sentence) “I love pizza” into French: “J’adore la pizza” (the translated sentence). To accomplish this, we give the starting sentence, “I love pizza,” to a encoder. The encoder then figures out the meaning and structure of the sentence. In <a href="https://medium.com/@pratik.vyas_10544/transformer-encoder-part-1-1d926707f74b">Part-1</a> we explored how the encoder does this. Next, we use what the encoder produced (Representation) as input for the decoder. The decoder’s job is to take this information and generate the French sentence “J’adore la pizza,” as illustrated below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/445/1*BpH_6cx1hOjBZIxqb9LhlA.png" /><figcaption><strong>image 1</strong></figcaption></figure><p>In the <a href="https://medium.com/@pratik.vyas_10544/transformer-encoder-part-1-1d926707f74b">encoder part</a>, we saw that instead of just one encoder, we can actually use a bunch of them stacked on top of each other. It turns out we can do the same thing with decoders! We can stack them too. Like in the picture, the result from one decoder is passed on to the decoder above it. Something else to notice is that the encoder’s understanding of the original sentence (the encoder’s output) is given to <em>every</em> decoder in the stack. So, each decoder gets two things to work with: the output from the decoder below it, and the encoder’s overall understanding of the input.</p><h4>Let’s understand how decoder generate output</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4xAhXWEy5oktZXzRpu0Dtg.png" /><figcaption><strong>image 2</strong></figcaption></figure><p><strong>The Encoder (Left Side): Understanding the Input</strong></p><ul><li>In <a href="https://medium.com/@pratik.vyas_10544/transformer-encoder-part-1-1d926707f74b">Part-1</a> we explored how the encoder does this</li></ul><p><strong>The Decoder (Right Side): Generating the Output</strong></p><ul><li><strong>Decoder Stack:</strong> Just like the encoder, we have a stack of decoders (“Decoder 1”, “Decoder N-1”, …, “Decoder N”). Each decoder also processes the information in stages.</li><li><strong>Start of Sequence (&lt;SOS&gt;):</strong> At the bottom of the first decoder stack (timestamp t1), you see &lt;SOS&gt;. This is a special “start of sequence” token. It tells the decoder, “Okay, start generating the output sentence now!”</li><li><strong>Decoder Input:</strong> Each decoder in the stack gets <em>two</em> inputs:<br>- <strong>The encoder’s Representation</strong> (the orange arrows). This is the encoder’s “understanding” of the input sentence. The decoders refer to it at each step of the decoding process.<br>- <strong>Output from the previous decoder.</strong> Each decoder in the stack uses the output from the previous decoder to generate the next part of the sequence. The output from this decoder is also sent to the decoder stack at the next timestamp. This way the decoders remember what they predicted in the past.</li><li><strong>Step-by-Step Generation:<br>- t1:</strong> The first decoder stack (input &lt;SOS&gt;) looks at the encoder’s representation and generates the first word of the French sentence, “J’”.<strong><br>- t2:</strong> The second decoder stack (input &lt;SOS&gt; J’ ) looks at the encoder representation and generates the second word of the French sentence, “J’adore la”.<br>- <strong>t3:</strong> The third decoder stack (input &lt;SOS&gt; J’adore la) looks at the encoder representation and generates the final French sentence, “J’adore la pizza”.</li><li><strong>Output:</strong> <br>Each decoder generates the next word in the target sentence. So, the first decoder generates “J’”, the second generates “J’adore la”, and the final decoder generates the complete sentence “J’adore la pizza”.</li></ul><h3>How decoder works</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/422/1*wa43lsYDvpIBwWsg5t4y3g.png" /><figcaption><strong>image 3</strong></figcaption></figure><p>Looking closely at the decoder part of the system, we see it’s built in a similar way to the encoder. Inside each decoder, there are three main building blocks:</p><ul><li><strong>Masked Multi-Head Attention:</strong> This is like the decoder focusing on different parts of what it has already generated. It makes sure the decoder only considers words that came <em>before</em> the current word in the sequence.</li><li><strong>Multi-Head Attention:</strong> Here, the decoder pays attention to the encoded representation of the original sentence. This helps it understand the context from the source language.</li><li><strong>Feedforward Network:</strong> This is a standard neural network that helps further process the information from the attention layers, refining the output.</li></ul><h4><strong>Let’s understand how Decoder is trained</strong></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/442/1*ojFSYnV1tpLPpyLHljhMPA.png" /></figure><p>During training, we help the decoder learn by giving it the <em>correct</em> answer (the target sentence) directly. We add a special “start” signal (&lt;sos&gt;) to the beginning, so the decoder knows where to begin. The decoder then tries to predict the target sentence, and we compare its prediction to the <em>actual</em> target sentence to see how well it’s doing. This helps it learn to make good predictions later.</p><p>Imagine the decoder is learning to say “I love pizza” in French (“J’adore la pizza”).</p><ul><li><strong>We give the decoder this to start:</strong> “&lt;sos&gt; J’adore la pizza”. Remember, &lt;sos&gt; means “GO!”. So we’re saying, “GO! predict ‘J’adore la pizza’”.</li><li><strong>What we <em>want</em> the decoder to predict:</strong> “J’adore la pizza &lt;eos&gt;”. &lt;eos&gt; means “The End!”.</li><li><strong>Think of it like a game:</strong> We’re showing the decoder the <em>whole</em> French sentence, and we’re saying, “Okay, you need to learn to <em>say</em> this sentence, starting after the ‘GO!’ signal.”</li><li><strong>Why is this easier?</strong> Instead of making the decoder predict each word <em>one by one</em>, we give it the whole thing to look at. The decoder sees how all the French words go together to mean “I love pizza”. This is much faster than redicting one word at a time!</li></ul><p>This approach might seem a little odd at first. What’s the point of feeding the whole target sentence as input and then asking the decoder to predict a <em>shifted</em> version of it as output?</p><h3><strong>Masking future Words</strong></h3><p>Here, masked attention comes into picture. during <em>actual</em> translation (testing), the decoder generate the sentence word by word (ofcourse it uses self-attention for that ). It only sees the words it’s already generated. So, if it’s working on the second word, it only knows the &lt;sos&gt; and the first word, “J’”. It <em>doesn’t</em> know “adore”, “la”, or “pizza” yet!</p><p>To make sure the decoder learns to work this way, we have to train it the same way it will be used. That means when the decoder is processing “J’”, it should only pay attention to &lt;sos&gt; and “J’”, and <em>ignore</em> the words that come later in the sentence.</p><p>This is where the “mask” comes in.<strong> The mask prevents the decoder from “peeking” at future words</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/602/1*eLVk7emju5E2CoUuTjno6g.png" /><figcaption><strong>image 4</strong></figcaption></figure><p>As shown in Image 4 and 2. It follow below steps.</p><ol><li><strong>Timestamp 1 (Predicting “J’”):</strong> The decoder can only attend to &lt;sos&gt;. All other positions in the attention matrix for that row are -∞.</li><li><strong>Timestamp 2 (Predicting “adore”):</strong> The decoder can attend to &lt;sos&gt; and “J’”. The rest are -∞.</li><li><strong>Timestamp 3 (Predicting “la”):</strong> The decoder can attend to &lt;sos&gt;, “J’”, and “adore”. The rest are -∞.</li><li><strong>Timestamp 4 (Predicting “pizza”):</strong> The decoder can attend to &lt;sos&gt;, “J’”, “adore”, and “la”.</li></ol><h3>Masked Multi-head attention</h3><p>First we calculate self attention for all heads and concatenate all head output to calculate MHA (Refer <a href="https://medium.com/@pratik.vyas_10544/transformer-encoder-part-1-1d926707f74b">Part-1</a>)</p><p>Masking words like this help the self-attention mechanism to attend only to the words that would be available to the model during testing.<br>( Already discussed Self-attention in <a href="https://medium.com/@pratik.vyas_10544/transformer-encoder-part-1-1d926707f74b">Part-1</a> )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/302/1*HDUimT_widDAsvIrh4NnvQ.png" /></figure><p>Concatenate all attention head. ( Already discussed MHA in <a href="https://medium.com/@pratik.vyas_10544/transformer-encoder-part-1-1d926707f74b">Part-1</a> )</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/439/1*DtxywlBU6iuDzto1Ey2IRg.png" /></figure><h3>Encoder-decoder attention</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/487/1*r6ZEBCtP2TypiVgEqygYmw.png" /></figure><ul><li>Let encoder representation be ‘<strong>R</strong>’</li><li>Let attention matrix obtained as a result of the masked multi-head attention be ‘<strong>M</strong>’</li></ul><p>The first step in the multi-head attention mechanism is creating the query, key, and value matrices</p><ul><li>Query: Q = M * W<em>q</em></li><li>Key: K = R * W<em>k</em></li><li>Value: V = R * W<em>v</em></li></ul><p>This is the main difference compare to Encoder. (Refer <a href="https://medium.com/@pratik.vyas_10544/transformer-encoder-part-1-1d926707f74b">Part-1</a>)</p><ul><li>Query matrics are created from M ( instead of input X, which was the case in Encoder ).</li><li>Key and Value matrics are created from R ( instead of input X, which was the case in Encoder ).</li></ul><p>Since here we have an interaction between the encoder and decoder, this layer is also called an <strong>encoder-decoder attention layer</strong>.</p><h3>Feed forward NN and Add/Norm layer</h3><p>This is same as we discussed in <a href="https://medium.com/@pratik.vyas_10544/transformer-encoder-part-1-1d926707f74b">Part-1</a></p><h3>Linear and softmax layers</h3><p>The linear layer generates the logits whose size is equal to our vocabulary size. <br>Let’s say vocab size is 7 : [ J’ , adore, la, pizza , bonjour, commen, allez-vous].</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/374/1*4qPTDdXEjECVy70zmdIH8w.png" /></figure><p>In this case, Decoder Process would be:</p><p>Linear Layer</p><ul><li>Takes the output from the last decoder block.</li><li>Creates a “logits” vector. Logits are <em>raw scores</em> for each word; higher means “more likely”.<br>Because we have 7 words, the logits vector has 7 values: <br>Example: logits = [41, 58, 39, 38, 31, 20, 10].</li></ul><p>Softmax Layer</p><ul><li>Takes the logits vector as input.</li><li>Transforms it into a probability distribution (numbers between 0 and 1 that add up to 1): <br>Example: prob = [ 0.03937842, 0.1380208, 0.01602345, 0.00176855, 0.2524375, 0.03218583, 0.00886544]</li><li>Each probability represents the likelihood of that word being the next word.</li></ul><p>Prediction:</p><ul><li>The decoder selects the word with the <em>highest probability</em>.</li><li>In our example, ‘adore’ (index 1) has the highest probability (0.138).</li></ul><p>Output:</p><ul><li>The decoder predicts ‘adore’ as the next word.</li></ul><h3>Summary</h3><ol><li>First, we convert the input to the decoder into an embedding matrix and then add the position encoding to it and feed it as input to the bottom-most decoder (decoder 1).</li><li>The decoder takes the input and sends it to the masked multi-head attention layer, which returns the attention matrix, M, as output.</li><li>Now, we take the attention matrix, M, and also the encoder representation, R , and feed them as input to the multi-head attention layer (encoder-decoder attention layer), which again outputs the new attention matrix.</li><li>We take the attention matrix obtained from the encoder-decoder attention layer and feed it as input to the next sublayer, which is the feedforward network. The feedforward network takes the attention matrix as input and returns the decoder representation as output.</li><li>Next, we take the output obtained from decoder 1 and feed it as input to the decoder above it (decoder 2).</li><li>Decoder 2 carries the same process and returns the decoder representation of the target sentence as output.</li></ol><h4>Read <a href="https://medium.com/@pratik.vyas_10544/transformer-decoder-only-generative-model-part-3-0b38c10c2ae9">Part-3</a></h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=644f3a12ff60" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>