<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Luv Bansal on Medium]]></title>
        <description><![CDATA[Stories by Luv Bansal on Medium]]></description>
        <link>https://medium.com/@luv-bansal?source=rss-234f1d6be728------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*B4nSMMLKFedO31fn9xNGOw.jpeg</url>
            <title>Stories by Luv Bansal on Medium</title>
            <link>https://medium.com/@luv-bansal?source=rss-234f1d6be728------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:45:38 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@luv-bansal/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[DSpark Easily Explained: Confidence-Scheduled Speculative Decoding]]></title>
            <link>https://luv-bansal.medium.com/dspark-easily-explained-confidence-scheduled-speculative-decoding-20a9a158afc7?source=rss-234f1d6be728------2</link>
            <guid isPermaLink="false">https://medium.com/p/20a9a158afc7</guid>
            <category><![CDATA[speculative-decoding]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[llm-inference]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[inference-optimization]]></category>
            <dc:creator><![CDATA[Luv Bansal]]></dc:creator>
            <pubDate>Sat, 25 Jul 2026 08:34:28 GMT</pubDate>
            <atom:updated>2026-07-25T08:34:28.813Z</atom:updated>
            <content:encoded><![CDATA[<h4>Ultimate guide to DSpark, semi-autoregressive drafting, confidence scheduling, and why faster LLM inference is not only about proposing more tokens</h4><p>In this blog, we will understand <strong>DSpark</strong>, a speculative decoding framework from the paper <a href="https://arxiv.org/abs/2607.05147">DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation</a>.</p><p>This post continues the same speculative decoding thread from my earlier posts on <a href="https://luvbansall.substack.com/p/dflash-easily-explained-how-block">DFlash</a> and Domino.</p><p>DFlash made draft generation parallel.</p><p>Domino added causal correction on top of a parallel backbone.</p><p>DSpark goes one step further and asks a more production-oriented question:</p><pre>Even if the drafter proposes a long block,<br>how much of that block is actually worth verifying right now?</pre><p>That “<em>right now</em>” is important.</p><p>In a high-concurrency serving system, GPU capacity is not constant. Sometimes the target model has idle verification capacity. Sometimes it is saturated. Verifying a long suffix of low-confidence draft tokens can waste the exact batch capacity that other users are waiting for.</p><p>So DSpark is not only a drafter paper.</p><p>It is both:</p><ol><li>a <strong>modeling paper</strong>, because it improves draft quality with semi-autoregressive generation;</li><li>a <strong>serving-systems paper</strong>, because it dynamically schedules verification length based on confidence and hardware load.</li></ol><p>The short version:</p><blockquote><strong>DSpark keeps the DFlash-style parallel backbone, adds a lightweight sequential head to reduce suffix decay, then uses calibrated confidence scores to verify only the prefix that is worth the target-model cost.</strong></blockquote><p>The paper reports that DSpark improves macro-average accepted length over DFlash by 16.3%, 18.4%, and 18.3% on Qwen3-4B, Qwen3-8B, and Qwen3-14B respectively. In DeepSeek-V4 production serving, it reports 60% to 85% faster per-user generation for V4-Flash and 57% to 78% for V4-Pro at matched practical throughput levels.</p><p>Let’s build the intuition step by step.</p><h3>What is speculative decoding?</h3><p>LLM decoding is slow because normal generation is autoregressive.</p><p>The target model produces one token, appends it to the context, then produces the next token:</p><pre>prompt -&gt; token_1 -&gt; token_2 -&gt; token_3 -&gt; token_4 -&gt; ...</pre><p>For a small model, this is fine.</p><p>For a large model serving many users, this loop is expensive because every token requires another target-model step.</p><p><strong>Speculative decoding</strong> adds a cheaper draft model.</p><p>The draft model proposes several future tokens. The target model verifies those proposed tokens in one pass. If the draft tokens agree with the target distribution, multiple tokens can be accepted in a single verification round.</p><p>The loop looks like this:</p><pre>def speculative_decode(prompt, target_model, draft_model, max_rounds):<br>    context = prompt<br><br>    for _ in range(max_rounds):<br>        draft_tokens = draft_model.propose(context, num_tokens=8)<br><br>        accepted_prefix, bonus_token = target_model.verify(<br>            context=context,<br>            draft_tokens=draft_tokens,<br>        )<br><br>        context += accepted_prefix<br><br>        if bonus_token is not None:<br>            context += [bonus_token]<br><br>    return context</pre><p>The target model is still the source of truth. The draft model only proposes candidates.</p><p>The standard latency equation from the DSpark paper is:</p><pre>L = (T_draft + T_verify) / tau</pre><p>where:</p><ul><li>L is average latency per generated token.</li><li>T_draft is time spent generating draft tokens.</li><li>T_verify is time spent verifying with the target model.</li><li>tau is the average accepted length per speculative round.</li></ul><p>This equation is the whole game.</p><p>To make inference faster, we want:</p><ul><li>cheaper draft generation,</li><li>more accepted tokens per round,</li><li>less wasted verification compute.</li></ul><p>Most speculative decoding methods optimize the first two. DSpark cares deeply about the third.</p><h3>Why DFlash was already strong</h3><p>Before DSpark, DFlash was one of the cleanest directions for fast speculative decoding.</p><p>Traditional drafters often generate draft tokens autoregressively:</p><pre>draft_1 -&gt; draft_2 -&gt; draft_3 -&gt; draft_4 -&gt; ...</pre><p>This gives good dependency modeling, because each draft token can see the previous draft tokens.</p><p>But the drafting step itself becomes sequential.</p><p>DFlash changes that by using a <strong>block diffusion drafter</strong>. It predicts a whole block of future tokens in one forward pass:</p><pre>last_verified_token, [MASK], [MASK], [MASK], ...</pre><p>Then the target model verifies the proposed block.</p><p>DFlash also conditions the draft model on target-model hidden states through <strong>KV injection</strong>. This makes the draft model behave less like a tiny standalone language model and more like a fast target-conditioned adapter.</p><p>That is a powerful idea.</p><p>But DFlash has a natural weakness:</p><p><strong>all positions inside the draft block are mostly predicted without knowing the actual sampled previous draft tokens.</strong></p><p>This creates <strong>suffix decay</strong>.</p><p>The first draft token can be strong. The later draft tokens can become less coherent because they do not know which mode the earlier sampled token selected.</p><p>Example:</p><pre>Possible continuation 1: &quot;of course&quot;<br>Possible continuation 2: &quot;no problem&quot;<br>Bad mixed continuation: &quot;of problem&quot;<br>Bad mixed continuation: &quot;no course&quot;</pre><p>This matters because target verification is prefix-based. If token 2 is rejected, tokens 3, 4, and 5 are thrown away too.</p><p>So the question becomes:</p><p>Can we keep the parallel speed of DFlash, but give later draft positions just enough dependency on earlier sampled tokens?</p><p>That is the first half of DSpark.</p><h3>DSpark in one picture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/897/0*n2G50i0nTcPC_lDc.png" /><figcaption>Image: DSpark architecture and decoding cycle from the <a href="https://arxiv.org/abs/2607.05147">DSpark paper</a>.</figcaption></figure><p>The figure shows one DSpark decoding cycle.</p><p>At a high level:</p><ol><li>The target model processes the current context and produces an anchor token.</li><li>DSpark uses a parallel backbone to produce base logits for a draft block.</li><li>A lightweight sequential block samples draft tokens left to right.</li><li>A confidence head predicts how likely each token is to survive target verification.</li><li>A hardware-aware prefix scheduler decides how many prefix tokens to verify.</li><li>The target model verifies only that scheduled prefix.</li></ol><p>Notice the key difference from a fixed-block speculative decoder.</p><p>DSpark may draft E F G H, but choose to verify only E F G if H has low expected value.</p><p>That is the production mindset:</p><p>Do not just ask whether you can draft more tokens.</p><p>Ask whether verifying those tokens is worth the GPU cost.</p><h3>Semi-autoregressive drafting</h3><p>DSpark’s drafter has two parts:</p><ul><li>a <strong>parallel backbone</strong>;</li><li>a <strong>lightweight sequential head</strong>.</li></ul><p>The parallel backbone is DFlash-style. It does the heavy work in one pass and produces base logits:</p><pre>U_1, U_2, ..., U_gamma</pre><p>Here, gamma is the draft block length.</p><p>If DSpark stopped here, it would be close to DFlash-style parallel drafting.</p><p>The DSpark addition is a sequential correction term:</p><pre>p_k(v | x_0, x_&lt;k)<br>  = softmax(U_k(v) + B_k(x_0, x_&lt;k, v))</pre><p>where:</p><ul><li>x_0 is the anchor token;</li><li>x_&lt;k are the previously sampled draft tokens;</li><li>U_k(v) is the base logit from the parallel backbone;</li><li>B_k(...) is a sequential transition bias;</li><li>v is a candidate vocabulary token.</li></ul><p>The intuition is simple:</p><pre>parallel backbone:<br>  what token is likely at this future position?<br><br>sequential head:<br>  given the actual previous sampled token, what should we boost or suppress?</pre><p>The default DSpark head is a <strong>Markov head</strong>:</p><pre>B(x_{k-1}, .) = W_1[x_{k-1}] W_2</pre><p>This means the bias depends mainly on the immediately previous draft token.</p><p>It is like a learned low-rank transition table.</p><h3>How the low-rank Markov head is calculated</h3><p>The most direct way to model a previous-token transition would be a full vocabulary-by-vocabulary matrix:</p><pre>B in R^{V x V}</pre><p>where V is the vocabulary size.</p><p>If the previous token is x_{k-1}, we could look up the row:</p><pre>B[x_{k-1}, :]</pre><p>and use that row as a vocabulary-sized bias for the next draft position.</p><p>But this full matrix is too expensive when the vocabulary is large.</p><p>So DSpark factorizes it into two low-rank matrices:</p><pre>B = W_1 W_2</pre><p>where:</p><ul><li>W_1 in R^{V x r}</li><li>W_2 in R^{r x V}</li><li>r = 256 by default in the paper</li></ul><p>Now the transition bias is:</p><pre>B(x_{k-1}, .) = W_1[x_{k-1}] W_2</pre><p>W_1[x_{k-1}] is just an embedding lookup for the previous token. It gives a small r-dimensional vector.</p><p>Then W_2 projects that small vector back into vocabulary space.</p><p>So the full computation is:</p><pre>previous token id<br>  -&gt; lookup W_1 row<br>  -&gt; get r-dimensional transition embedding<br>  -&gt; multiply by W_2<br>  -&gt; get vocabulary-sized correction logits</pre><p>Important detail:</p><p>These matrices are <strong>learned during training</strong>. They are not calculated with an SVD of some pre-existing transition table. The full V x V transition table is only the conceptual object. DSpark directly trains the low-rank factors W_1 and W_2 with backpropagation as part of the draft model.</p><p>If the previous token is of, the head can boost course.</p><p>If the previous token is no, the head can boost problem.</p><p>This is why the Markov head is cheap.</p><p>It adds a local transition correction without running a full Transformer layer or a full autoregressive draft model for every token.</p><h3>How the RNN head works</h3><p>The Markov head only looks at the immediately previous token.</p><p>That is useful, but it is memoryless beyond one step.</p><p>The paper also studies an <strong>RNN head</strong> for a more expressive lightweight sequential module.</p><p>The RNN head keeps a recurrent state:</p><pre>s_k</pre><p>This state summarizes the sampled draft prefix inside the current block.</p><p>At each draft position k, DSpark builds:</p><pre>z_k = [s_{k-1}; W_1[x_{k-1}]; h_k]</pre><p>where:</p><ul><li>s_{k-1} is the previous recurrent state;</li><li>W_1[x_{k-1}] is the previous-token embedding from the Markov head;</li><li>h_k is the parallel backbone hidden state for position k;</li><li>[;] means concatenation.</li></ul><p>Then the RNN head applies a gated update:</p><pre>s_k =<br>  sigmoid(W_g z_k) * s_{k-1}<br>  + (1 - sigmoid(W_g z_k)) * tanh(W_c z_k)</pre><p>The transition bias is then:</p><pre>B_k(x_&lt;k, .) = W_2^T tanh(W_o z_k)</pre><p>In plain English:</p><p>The RNN head asks:</p><pre>Given the previous recurrent memory,<br>the previous sampled token,<br>and the current parallel hidden state,<br>what correction should I add to this position?</pre><p>This can capture more prefix information than the Markov head.</p><p>But it also adds more implementation complexity.</p><p>The paper finds that the RNN head gives only marginal additional gains over the Markov head, mainly at longer proposal lengths. Because of this, DSpark uses the Markov head as the default in the reported setup.</p><p>This is the nice part:</p><p>DSpark does not make the whole draft model autoregressive.</p><p>It keeps the heavy computation parallel and adds only a small sequential correction at the output.</p><h3>Why this reduces suffix decay</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/0*HHdEx5L35kww6vCf.png" /><figcaption>Image: Position-wise conditional acceptance from the <a href="https://arxiv.org/abs/2607.05147">DSpark paper</a>.</figcaption></figure><p>This figure is one of the most important figures in the paper.</p><p>It measures <strong>conditional acceptance</strong> by draft position.</p><p>For position k, the metric only includes cases where positions 1..k-1 were already accepted. This removes the penalty from earlier failures and asks:</p><pre>If the prefix survived, how good is this position?</pre><p>The pattern is:</p><ul><li>DFlash starts strong but decays later.</li><li>Eagle3 is more sequential, so later positions are more stable.</li><li>DSpark starts high and stays more stable.</li></ul><p>This explains why DSpark can beat both styles.</p><p>DFlash has the capacity advantage of parallel drafting, especially at the first position.</p><p>Eagle3 has stronger sequential dependency, especially later in the block.</p><p>DSpark tries to combine both:</p><pre>DFlash-like first-token strength<br>+ lightweight local dependency<br>= better accepted prefix length</pre><p>This is exactly why DSpark is better than DFlash in the paper’s setup.</p><p>DFlash attacks draft cost. DSpark attacks draft cost and suffix quality.</p><h3>How DSpark is trained</h3><p>Now let’s talk about the training setup, because this is where the method becomes clearer.</p><p>During training, DSpark samples multiple anchor positions from each target sequence.</p><p>Each anchor creates one gamma-token draft block.</p><p>Conceptually:</p><pre>target sequence:<br>  x_1, x_2, ..., x_t, x_{t+1}, ..., x_{t+gamma}<br><br>anchor:<br>  x_t<br><br>training target:<br>  x_{t+1}, ..., x_{t+gamma}</pre><p>The target model is frozen.</p><p>The draft model shares the target model’s embedding layer and LM head, and those are also frozen.</p><p>Only these parts are trained:</p><ul><li>the parallel draft backbone;</li><li>the sequential Markov or RNN block;</li><li>the confidence head.</li></ul><p>This matters because the draft model is trained to behave like a target-specific acceleration module, not as a standalone language model.</p><p>The paper uses three losses:</p><ol><li><strong>Cross-entropy loss</strong>, so the drafter predicts the ground-truth next tokens.</li><li><strong>Distribution-matching loss</strong>, so the draft distribution overlaps with the target distribution.</li><li><strong>Confidence loss</strong>, so the confidence head predicts whether draft tokens are likely to survive verification.</li></ol><p>All three losses are position-weighted:</p><pre>w_k = exp(-(k - 1) / gamma)</pre><p>Earlier positions get higher weight because speculative decoding is prefix-based.</p><p>If token 1 is rejected, the rest of the draft block does not matter.</p><h3>Cross-entropy loss</h3><p>The cross-entropy loss trains DSpark to predict the ground-truth token:</p><pre>L_ce = - sum_{k=1}^{gamma} w_k log p_k^d(x_k*)</pre><p>where:</p><ul><li>x_k* is the ground-truth token at draft position k;</li><li>p_k^d is the draft distribution;</li><li>w_k is the position weight.</li></ul><p>This is the normal “predict the correct next token” part.</p><p>But speculative decoding needs more than next-token accuracy.</p><p>It needs the draft distribution to match the target distribution.</p><h3>Distribution-matching loss</h3><p>The distribution-matching loss is:</p><pre>L_tv = sum_{k=1}^{gamma} w_k ||p_k^d - p_k^t||_1</pre><p>where:</p><ul><li>p_k^d is the draft distribution;</li><li>p_k^t is the target distribution;</li><li>||.||_1 is L1 distance over the vocabulary.</li></ul><p>This loss is important because speculative acceptance depends on distribution overlap.</p><p>The expected per-step acceptance probability is:</p><pre>1 - (1 / 2) * ||p_k^d - p_k^t||_1</pre><p>So minimizing the distribution gap directly helps maximize expected acceptance.</p><p>This is the part that makes the loss aligned with speculative decoding, not just language modeling.</p><h3>Confidence loss</h3><p>The confidence head predicts:</p><pre>c_k</pre><p>which should estimate the conditional probability that token k survives, assuming previous tokens survived.</p><p>The paper trains it with a soft analytical label:</p><pre>c_k* = 1 - (1 / 2) * ||p_k^d - p_k^t||_1</pre><p>Then it uses binary cross-entropy:</p><pre>L_conf =<br>  - sum_{k=1}^{gamma} w_k [<br>      c_k* log c_k<br>      + (1 - c_k*) log(1 - c_k)<br>    ]</pre><p>The total objective is:</p><pre>L = alpha_ce L_ce + alpha_tv L_tv + alpha_conf L_conf</pre><p>The default weights from the paper are:</p><ul><li>alpha_ce = 0.1</li><li>alpha_tv = 0.9</li><li>alpha_conf = 1.0</li></ul><p>So DSpark leans heavily toward distribution matching, which makes sense.</p><p>For speculative decoding, being close to the target distribution is more important than only predicting the single ground-truth token.</p><h3>Results: accepted length</h3><p>The paper evaluates DSpark on Qwen3–4B, Qwen3–8B, Qwen3–14B, and Gemma4–12B across math, code, and chat benchmarks.</p><p>The important metric here is accepted length tau per decoding round. Higher is better.</p><p>Here are the macro-average accepted lengths from Table 1:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qqMzC0f_SD0HZwyG.png" /><figcaption>report accepted length (𝜏) per decoding round (higher is better) for different target models and domains from <a href="https://arxiv.org/pdf/2607.05147">DSpark paper</a></figcaption></figure><p>Two things stand out.</p><p>First, DSpark beats DFlash in every shown Qwen3–8B benchmark column.</p><p>Second, chat has lower accepted length than math and code. This matters because open-ended chat is exactly where long low-confidence suffixes can waste verification compute.</p><p>This is the bridge from the modeling idea to the scheduling idea.</p><h3>A little autoregression goes a long way</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/0*_TInUjbyuA-Cuuru.png" /></figure><p>Image: Effect of proposal length and latency overhead from the <a href="https://arxiv.org/abs/2607.05147">DSpark paper</a>.</p><p>This figure compares DSpark and DFlash as draft length grows.</p><p>The main pattern:</p><ul><li>DSpark beats DFlash at draft lengths 4, 8, 12, and 16.</li><li>The gap grows as draft length increases.</li><li>The RNN head gives only marginal gains over the Markov head.</li><li>The sequential head adds only about 0.2% to 1.3% full-round latency overhead over DFlash in the reported setup.</li></ul><p>The paper reports:</p><ul><li>at gamma = 7, DSpark improves accepted length over DFlash by 16% on math, 15% on code, and 18% on chat;</li><li>at gamma = 15, those gains expand to 30% on math, 26% on code, and 22% on chat.</li></ul><p>This is a very important practical result.</p><p>If the sequential head were expensive, the idea would be less compelling. But the paper’s argument is that the target verification pass dominates the round latency, so the small sequential output head is cheap enough.</p><p>The mental model:</p><pre>DFlash:<br>  big win from parallel block drafting,<br>  but suffix decay limits the value of longer blocks.<br><br>DSpark:<br>  keeps the parallel backbone,<br>  adds cheap local dependency,<br>  so longer blocks remain useful.</pre><h3>Confidence scheduling</h3><p>Now we get to the second major DSpark idea.</p><p>Even if the drafter proposes a long block, should the target model verify the whole thing?</p><p>The answer depends on confidence and load.</p><p>For each request r and draft position k, DSpark predicts a confidence score:</p><pre>c_{r,k}</pre><p>This is a conditional probability:</p><pre>probability token k survives,<br>assuming tokens 1..k-1 already survived</pre><h3>How the confidence head is calculated</h3><p>The confidence head is intentionally lightweight.</p><p>For each draft position k, it takes:</p><ul><li>the parallel backbone hidden state h_k;</li><li>the previous-token Markov embedding W_1[x_{k-1}].</li></ul><p>Then it concatenates them and applies a linear projection followed by a sigmoid:</p><pre>c_k = sigmoid(w^T [h_k; W_1[x_{k-1}]])</pre><p>where:</p><ul><li>h_k tells the confidence head what the parallel backbone believes about position k;</li><li>W_1[x_{k-1}] tells it what token was sampled just before position k;</li><li>w is a learned projection vector;</li><li>sigmoid maps the score into (0, 1).</li></ul><p>In plain English:</p><pre>Given the hidden state at this draft position<br>and the previous sampled token,<br>how likely is this draft token to survive target verification?</pre><p>This confidence is conditional. It assumes previous draft tokens have already survived.</p><p>The scheduler wants prefix survival probability:</p><pre>a_{r,j} = product_{i &lt;= j} c_{r,i}</pre><p>This is the probability that the whole prefix through position j survives.</p><p>Here is a small Python example:</p><pre>from itertools import accumulate<br>from operator import mul<br><br><br>def prefix_survival(confidences: list[float]) -&gt; list[float]:<br>    &quot;&quot;&quot;Compute cumulative prefix survival probabilities.&quot;&quot;&quot;<br>    return list(accumulate(confidences, mul))<br><br><br>confidences = [0.92, 0.85, 0.70, 0.40]<br>print(prefix_survival(confidences))<br># [0.92, 0.782, 0.5474, 0.21896]</pre><p>The fourth token has conditional confidence 0.40, but the probability that the whole four-token prefix survives is only around 0.219.</p><p>Under low load, maybe verifying it is fine.</p><p>Under high load, maybe that verification slot should go to another request with a higher expected return.</p><p>That is the scheduling problem.</p><h3>Verify smarter, not longer</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/947/0*_ljEmDOvy0TkU5Hw.png" /><figcaption>Image: Confidence threshold sweep from the <a href="https://arxiv.org/abs/2607.05147">DSpark paper</a>.</figcaption></figure><p>The paper first shows a simple diagnostic: prune tokens below a confidence threshold.</p><p>As the threshold increases, DSpark verifies fewer low-confidence suffix tokens. The fraction of verified tokens that are actually accepted goes up.</p><p>The reported acceptance-rate changes are:</p><ul><li>Math: 76.9% to 92.5%.</li><li>Code: 67.6% to 92.0%.</li><li>Chat: 45.7% to 95.7%.</li></ul><p>The chat result is the most intuitive one.</p><p>Open-ended chat has much more uncertainty in later draft positions, so fixed-length verification wastes more target capacity.</p><p>But production DSpark does not use one fixed threshold.</p><p>It uses a hardware-aware objective:</p><pre>Theta = tau * SPS(B)</pre><p>where:</p><ul><li>tau is expected accepted tokens;</li><li>B is the target verification batch size;</li><li>SPS(B) is the profiled target-engine steps per second at batch size B;</li><li>Theta is expected system-wide token throughput.</li></ul><p>The scheduler sorts candidate prefix tokens by cumulative survival probability and admits them while expected throughput improves.</p><p>Here is a compact version of the idea:</p><pre>from collections.abc import Callable<br><br><br>def schedule_prefix_lengths(<br>    batch_confidences: list[list[float]],<br>    sps: Callable[[int], float],<br>) -&gt; list[int]:<br>    &quot;&quot;&quot;Greedy DSpark-style prefix scheduler.<br><br>    batch_confidences[r][k] is calibrated conditional confidence<br>    for request r and draft position k.<br>    &quot;&quot;&quot;<br>    candidates: list[tuple[float, int, int]] = []<br><br>    for request_id, confidences in enumerate(batch_confidences):<br>        survival = 1.0<br>        for prefix_length, confidence in enumerate(confidences, start=1):<br>            survival *= confidence<br>            candidates.append((survival, request_id, prefix_length))<br><br>    candidates.sort(reverse=True)<br><br>    num_requests = len(batch_confidences)<br>    prefix_lengths = [0] * num_requests<br>    best_prefix_lengths = prefix_lengths.copy()<br><br>    batch_size = num_requests<br>    expected_tokens = float(num_requests)<br>    best_throughput = expected_tokens * sps(batch_size)<br><br>    for survival, request_id, prefix_length in candidates:<br>        if prefix_length &lt;= prefix_lengths[request_id]:<br>            continue<br><br>        prefix_lengths[request_id] = prefix_length<br>        batch_size += 1<br>        expected_tokens += survival<br><br>        throughput = expected_tokens * sps(batch_size)<br>        if throughput &gt; best_throughput:<br>            best_throughput = throughput<br>            best_prefix_lengths = prefix_lengths.copy()<br>        else:<br>            break<br><br>    return best_prefix_lengths<br><br><br>def toy_sps(batch_size: int) -&gt; float:<br>    &quot;&quot;&quot;Toy throughput curve. Real systems should profile this.&quot;&quot;&quot;<br>    return 1.0 / (1.0 + 0.08 * max(batch_size - 1, 0))<br><br><br>confidences = [<br>    [0.95, 0.88, 0.60, 0.30],<br>    [0.90, 0.70, 0.35, 0.20],<br>    [0.80, 0.75, 0.72, 0.68],<br>]<br><br>print(schedule_prefix_lengths(confidences, toy_sps))</pre><p>The exact output depends on the real SPS(B) curve.</p><p>And that is the point.</p><p>DSpark is not only using model confidence. It is combining confidence with measured hardware behavior.</p><h3>Calibration is not optional</h3><p>If the scheduler uses probabilities, then the confidence scores need to behave like probabilities.</p><p>Ranking is not enough.</p><p>A score of 0.80 should mean roughly an 80% chance of survival, not just &quot;higher than a score of 0.60.&quot;</p><p>The DSpark confidence head is trained with a soft target based on expected acceptance:</p><pre>c_k* = 1 - (1 / 2) * ||p_k^d - p_k^t||_1</pre><p>where:</p><ul><li>p_k^d is the draft distribution;</li><li>p_k^t is the target distribution;</li><li>||.||_1 is L1 distance over the vocabulary.</li></ul><p>This is connected to total variation distance. If draft and target distributions overlap strongly, expected acceptance is high. If they disagree, expected acceptance is low.</p><p>The paper then applies <strong>Sequential Temperature Scaling</strong> on a held-out set because raw neural confidence estimates are often overconfident.</p><p>In the Alpaca reliability analysis, raw confidence has strong discrimination, with ROC-AUC around 0.81 to 0.90, but expected calibration error is around 3% to 8%. After calibration, average ECE drops to about 1%.</p><p>For a scheduler, this matters a lot.</p><p>Bad calibration can make the system verify too many bad suffix tokens or prune too aggressively.</p><h3>Production results</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/997/0*iC_oxTFvoZg3kYuC.png" /><figcaption>Image: Production throughput versus per-user generation speed from the <a href="https://arxiv.org/abs/2607.05147">DSpark paper</a>.</figcaption></figure><p>The production section is where DSpark becomes more than an offline benchmark paper.</p><p>The paper says DSpark is deployed with preview versions of:</p><ul><li>DeepSeek-V4-Flash;</li><li>DeepSeek-V4-Pro.</li></ul><p>The production drafter uses:</p><ul><li>a parallel backbone with 3 MoE layers</li><li>mHC</li><li>sliding-window attention of 128</li><li>maximum block size gamma = 5</li><li>the Markov head</li><li>a confidence head trained end to end and calibrated with STS.</li></ul><p>The reported production gains are:</p><ul><li>V4-Flash: 60% to 85% faster per-user generation at matched practical throughput.</li><li>V4-Pro: 57% to 78% faster per-user generation at matched practical throughput.</li><li>At moderate SLA points, aggregate throughput improves by about 51% for Flash and 52% for Pro.</li></ul><p>The paper also reports very large nominal gains at strict SLA points, such as 661% for Flash and 406% for Pro.</p><p>I would read those carefully.</p><p>Those huge numbers mainly show that DSpark extends the serving frontier when the baseline enters a low-concurrency regime. They are not a universal “everything is 6x faster” claim.</p><p>That careful reading matters because speculative decoding speedups are workload and system dependent.</p><h3>DSpark vs DFlash: why DSpark is better</h3><p>I would not say DSpark makes DFlash irrelevant.</p><p>DFlash is the foundation DSpark builds on.</p><p>DFlash solves a major bottleneck:</p><pre>drafting should not be token-by-token if we can draft a block in parallel</pre><p>DSpark agrees with this.</p><p>But DSpark adds two missing pieces:</p><ol><li><strong>Semi-autoregressive correction</strong> for suffix decay.</li><li><strong>Confidence-scheduled verification</strong> for production batch efficiency.</li></ol><h4>Why is DSpark better in the paper’s setup?</h4><p>First, it improves accepted length.</p><p>Across Qwen3–4B, Qwen3–8B, and Qwen3–14B, DSpark improves macro-average accepted length over DFlash by 16.3%, 18.4%, and 18.3%.</p><p>Second, the gap grows with proposal length.</p><p>At gamma = 15, DSpark improves accepted length over DFlash by 30% on math, 26% on code, and 22% on chat.</p><p>Third, the added sequential head is cheap in the reported setup.</p><p>The paper reports only 0.2% to 1.3% full-round latency overhead over DFlash while delivering much larger accepted-length gains.</p><p>Fourth, DSpark handles the system-level waste that DFlash alone does not solve.</p><p>DFlash can propose a long block. But if the suffix is low-confidence and the target engine is saturated, verifying that suffix can be harmful. DSpark uses confidence and the profiled throughput curve to decide how much of the block is worth verifying.</p><p>So my summary is:</p><pre>DFlash answers:<br>  How do we make drafting parallel?<br><br>DSpark answers:<br>  How do we keep parallel drafting useful at longer suffixes,<br>  and how do we spend target verification compute under load?</pre><p>That is why DSpark feels more production-complete.</p><h3>How would you use the open-source artifacts?</h3><p>DeepSeek released <a href="https://github.com/deepseek-ai/DeepSpec">DeepSpec</a>, which is the training and evaluation codebase for speculative decoding draft models.</p><p>The repo includes DSpark, DFlash, and Eagle3 implementations.</p><p>The workflow is:</p><ol><li>prepare data</li><li>regenerate target-model answers</li><li>build the target cache</li><li>train the draft model</li><li>evaluate accepted length on benchmarks.</li></ol><h3>How I think about DSpark</h3><p>The easiest way to remember DSpark is:</p><pre>Draft long blocks cheaply.<br>Repair local dependency with a tiny sequential head.<br>Verify only the prefix that is worth the GPU cost.</pre><p>That last line is the most important one.</p><p>Many speculative decoding methods focus on the drafter.</p><p>DSpark focuses on the full serving loop.</p><p>In an offline benchmark, a long accepted prefix looks great.</p><p>In production, a long verified suffix that mostly gets rejected is a batch-capacity tax.</p><p>DSpark’s confidence scheduler is interesting because it turns speculative decoding into a resource allocation problem:</p><pre>which draft tokens deserve target-model verification under current load?</pre><p>That is a very practical framing.</p><h3>Final thoughts</h3><p>DSpark is one of the more interesting speculative decoding papers because it combines model design and serving design cleanly.</p><p>DFlash showed that parallel block drafting can remove a major sequential bottleneck.</p><p>DSpark keeps that idea, then fixes two downstream problems:</p><ul><li>suffix tokens need some dependency on earlier sampled draft tokens;</li><li>target verification should be allocated based on confidence and hardware load.</li></ul><p>For me, the main takeaway is:</p><p><strong>The best speculative decoding system is not the one that drafts the most tokens. It is the one that produces the most accepted tokens per unit of target-model compute.</strong></p><p>DSpark improves that trade-off by combining semi-autoregressive drafting with confidence-scheduled verification.</p><p>That is why it is worth paying attention to.</p><h3>References</h3><ul><li><a href="https://arxiv.org/abs/2607.05147">DSpark paper on arXiv</a></li><li><a href="https://arxiv.org/pdf/2607.05147">DSpark PDF</a></li><li><a href="https://github.com/deepseek-ai/DeepSpec">DeepSpec GitHub repository</a></li><li><a href="https://huggingface.co/collections/deepseek-ai/deepspec">DeepSpec checkpoint collection</a></li><li><a href="https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-DSpark">DeepSeek-V4-Pro-DSpark checkpoint</a></li><li><a href="https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-DSpark">DeepSeek-V4-Flash-DSpark checkpoint</a></li><li><a href="https://arxiv.org/abs/2602.06036">DFlash paper on arXiv</a></li><li><a href="https://github.com/z-lab/dflash">DFlash GitHub repository</a></li><li><a href="https://luvbansall.substack.com/p/dflash-easily-explained-how-block">My DFlash blog post</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=20a9a158afc7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Domino Easily Explained: Causal Correction for Faster Speculative Decoding]]></title>
            <link>https://luv-bansal.medium.com/domino-easily-explained-causal-correction-for-faster-speculative-decoding-09e6ee93370d?source=rss-234f1d6be728------2</link>
            <guid isPermaLink="false">https://medium.com/p/09e6ee93370d</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[inference-optimization]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[speculative-decoding]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Luv Bansal]]></dc:creator>
            <pubDate>Sat, 18 Jul 2026 17:00:58 GMT</pubDate>
            <atom:updated>2026-07-18T17:00:58.472Z</atom:updated>
            <content:encoded><![CDATA[<h4>Ultimate guide to Domino, DFlash, block-parallel drafting, and why causal correction improves LLM inference speed</h4><p>In this blog, we will understand <strong>Domino</strong>, a speculative decoding method from the paper <a href="https://arxiv.org/abs/2605.29707">Domino: Decoupling Causal Modeling from Autoregressive Drafting in Speculative Decoding</a>.</p><p>This post is a continuation of the same story I wrote about in my <a href="https://luvbansall.substack.com/p/dflash-easily-explained-how-block">DFlash blog</a>.</p><p><strong>DFlash made speculative decoding much more interesting by asking:</strong></p><blockquote><em>Can the draft model predict a whole block of future tokens in parallel?</em></blockquote><p><strong>Domino asks the next question:</strong></p><blockquote><em>Can we keep that block-parallel speed, but bring back some of the causal modeling accuracy that parallel drafting loses?</em></blockquote><p>That question is the whole point of Domino.</p><p>The short version:</p><p><strong>DFlash makes drafting parallel. Domino keeps drafting parallel, then adds a lightweight causal correction head so later draft tokens can depend on earlier draft tokens.</strong></p><p>The paper reports that, compared with DFlash on Qwen3 models, Domino improves average acceptance length and end-to-end speedup while adding only a small amount of draft overhead. In the Qwen3–8B greedy-decoding setting, the average speedup goes from 4.66x with DFlash to 5.49x with Domino.</p><p>Let’s build the intuition from the beginning.</p><h3>What is speculative decoding?</h3><p>LLM generation is slow because normal decoding is autoregressive.</p><p>The model predicts one token, appends it to the context, predicts the next token, and keeps going:</p><pre>token_1 -&gt; token_2 -&gt; token_3 -&gt; token_4 -&gt; ...</pre><p>This sequential loop is painful for long outputs, reasoning traces, code generation, agents, and anything where the user waits for hundreds or thousands of tokens.</p><p><strong>Speculative decoding</strong> tries to speed this up with two models:</p><ol><li>A large <strong>target model</strong>, which is the model we actually trust.</li><li>A faster <strong>draft model</strong>, which proposes several future tokens.</li></ol><p>The loop is:</p><pre>def speculative_decode(prompt, target_model, draft_model):<br>    context = prompt<br><br>    while not done(context):<br>        draft_tokens = draft_model.propose(context, num_tokens=16)<br>        accepted_tokens, bonus_token = target_model.verify(context, draft_tokens)<br>        context += accepted_tokens<br><br>        if bonus_token is not None:<br>            context += [bonus_token]<br><br>    return context</pre><p>The important trick is that the target model can verify a block of draft tokens in one pass.</p><p>If the draft model guesses well, we get multiple accepted tokens per target-model call. If the draft model guesses badly, most of the draft block is rejected and we do not get much speedup.</p><p>A useful way to think about the latency is:</p><pre>speed depends on accepted_tokens_per_cycle / draft_plus_verify_cost</pre><p>So the entire game becomes:</p><p>How do we get many accepted tokens while keeping the draft step cheap?</p><h3>Why DFlash was already a big step</h3><p>Traditional speculative decoding methods often make the draft model generate tokens autoregressively:</p><pre>draft_1 -&gt; draft_2 -&gt; draft_3 -&gt; draft_4 -&gt; ...</pre><p>That gives the drafter causal dependency between its own draft tokens, but it also makes draft generation sequential.</p><p>DFlash changed this by using a <strong>block diffusion drafter</strong>.</p><p>Instead of producing future draft tokens one at a time, DFlash creates a masked block:</p><pre>last_verified_token, [MASK], [MASK], [MASK], ...</pre><p>Then it predicts the future block in parallel.</p><p>This is powerful because the draft cost no longer scales linearly with the number of draft tokens. It becomes much closer to one parallel drafter pass.</p><p>DFlash also conditions the drafter on target-model context features through <strong>KV injection</strong>, so the small draft model behaves more like an efficient adapter attached to the target model.</p><p>That is why DFlash is strong:</p><p>It attacks the draft latency directly.</p><p>But there is a trade-off.</p><p>When all future draft positions are predicted in parallel, token draft_7 does not naturally know what was sampled at draft_3. That weakens <strong>intra-block causal modeling</strong>.</p><p>This is exactly the gap Domino targets.</p><h3>The bottleneck Domino tries to fix</h3><p>Imagine the target model would naturally continue with:</p><pre>The answer is 42 because ...</pre><p>If the drafter predicts each future position independently inside a block, then later positions may not be properly conditioned on earlier sampled draft tokens.</p><p>That matters because language is causal.</p><p>The token after The answer is should depend on the exact earlier tokens. The token after 42 should depend on 42. Code tokens depend on indentation and previous symbols. Math reasoning tokens depend on the previous step.</p><p>Autoregressive drafters model this dependency naturally, but they pay for it with sequential draft execution.</p><p>Block-parallel drafters avoid the sequential loop, but they lose some of that causal dependency.</p><p>Domino is trying to keep the good parts of both sides:</p><ul><li>Keep the <strong>parallel backbone</strong> from DFlash.</li><li>Add a cheap <strong>causal correction</strong> branch.</li><li>Avoid running a full draft model autoregressively.</li><li>Avoid repeatedly applying an expensive full LM head.</li></ul><p>This is why the title says Domino decouples causal modeling from autoregressive drafting.</p><p>The causal information comes back, but the expensive autoregressive draft loop does not.</p><h3>Domino in one picture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*pIZ1vBXHMoMuFY6P.png" /></figure><p>Image: Domino pipeline from the <a href="https://huggingface.co/Huang2020/Qwen3-8B-Domino-b16">Domino Hugging Face model card</a>.</p><p>Domino has two main parts:</p><ol><li>A <strong>parallel draft backbone</strong>.</li><li>A <strong>Domino head</strong>.</li></ol><p>The parallel backbone is instantiated using the DFlash architecture. It takes target-model context features plus a masked draft block and produces hidden states for the whole draft block in one forward pass.</p><p>Then the Domino head refines the draft logits.</p><p>The Domino head is the important new part. It has:</p><ul><li>A lightweight <strong>causal encoder</strong>, implemented as a GRU in the paper.</li><li>A <strong>low-rank correction head</strong>, which produces residual correction logits.</li></ul><p>In plain English:</p><p>The backbone says, “Here is my best parallel guess for every draft position.”</p><p>The Domino head says, “Before sampling this position, let me cheaply summarize the previous sampled draft tokens and adjust the logits.”</p><p>So later draft tokens are no longer completely blind to earlier draft tokens.</p><h3>Step by step: how Domino works</h3><p>Let’s walk through one Domino decoding cycle.</p><h3>1. Start from the verified prefix</h3><p>We begin with the text that the target model has already verified.</p><pre>x_1, x_2, ..., x_t</pre><p>The last verified token becomes the anchor for the next draft block.</p><p>Domino constructs a masked draft block:</p><pre>x_t, [MASK], [MASK], [MASK], ..., [MASK]</pre><p>In the released Qwen3 Domino checkpoints, the block size is 16.</p><h3>2. Run the parallel draft backbone</h3><p>The DFlash-style backbone consumes:</p><ul><li>target-model context features from the verified prefix</li><li>embeddings of the masked draft block</li></ul><p>It produces hidden states for all positions in the block in one parallel pass:</p><pre>H_t, H_{t+1}, ..., H_{t+B-1}</pre><p>Then the frozen target LM head is applied in parallel to produce base logits:</p><pre>base_logits_i = LMHead(H_i)</pre><p>At this point, Domino has a parallel draft distribution for every position.</p><p>This is basically the DFlash-style part.</p><h3>3. Use the causal encoder to summarize previous draft tokens</h3><p>Now Domino adds a small sequential correction branch.</p><p>For each draft position, the causal encoder summarizes the previous sampled draft tokens:</p><pre>causal_state_i = GRU(previous_draft_token_embeddings)</pre><p>The paper uses a GRU with hidden dimension 1024.</p><p>This branch is sequential, but it is tiny compared with running a full draft transformer autoregressively.</p><p>That is the design move.</p><p>Domino is willing to pay a small sequential cost, but only for a lightweight correction path.</p><h3>4. Produce low-rank correction logits</h3><p>For each position, Domino combines:</p><ul><li>the parallel backbone representation</li><li>the causal state from previous draft tokens</li></ul><p>Then it passes them through a low-rank correction head:</p><pre>correction_logits_i = LowRankMLP(backbone_hidden_i, causal_state_i)<br>final_logits_i = base_logits_i + correction_logits_i</pre><p>The correction hidden size in the paper is 256.</p><p>This is much cheaper than rerunning a full LM head after every causal update.</p><p><strong>The correction happens in logit space, not hidden space</strong>. That detail matters because a hidden-space correction would need another expensive LM-head projection.</p><p>Domino keeps the expensive base LM-head computation parallel, then adds a small residual correction.</p><h3>5. Verify with the target model</h3><p>After Domino samples a draft block, the target model verifies the proposed tokens just like standard speculative decoding.</p><p>Only tokens accepted by the target model are committed.</p><p>So the final generation remains target-model-correct. Domino is only changing how candidate draft tokens are proposed.</p><h3>Why the training setup matters</h3><p>The Domino architecture is not the whole story. The training choices matter a lot.</p><p>The paper calls out two choices:</p><ol><li><strong>Teacher-forced causal encoding</strong>.</li><li><strong>Base-anchored curriculum</strong>.</li></ol><p>Let’s unpack both.</p><h3>Teacher-forced causal encoding</h3><p>The causal encoder consumes previous draft tokens.</p><p>One option is to train it on self-generated draft prefixes. This would mimic test time, but those self-generated prefixes can be noisy, especially early in training.</p><blockquote><em>Domino instead uses teacher forcing: it feeds the encoder ground-truth previous tokens during training.</em></blockquote><p>This is a good fit for speculative decoding because position i only matters if all previous draft tokens were accepted. And if they were accepted, those previous tokens match the target sequence.</p><p>So the most useful regime to train on is the accepted-prefix regime.</p><h3>Base-anchored curriculum</h3><p>Teacher forcing introduces another problem.</p><p>If the correction branch gets clean previous tokens, it can become too strong and shortcut the parallel backbone. Then the base logits become weak, and the whole system leans too much on the correction branch.</p><p>Domino avoids this with a curriculum:</p><pre>loss = (1 - lambda) * final_logit_loss + lambda * base_logit_loss</pre><p>At the beginning, lambda is high, so training forces the parallel backbone to learn a strong base distribution.</p><p>Over time, lambda is annealed toward zero, so the final corrected logits become the main objective.</p><p>This is a very practical idea:</p><p>First make the parallel drafter good. Then let the causal correction head improve it.</p><blockquote><em>The ablation shows why this matters. On Qwen3–8B, the reported average acceptance length improves from </em><em>3.80 with training-time testing, to </em><em>3.96 with teacher forcing, to </em><em>4.19 with teacher forcing plus the base-anchored curriculum.</em></blockquote><h3>Results: what does Domino report?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dSHLi4zu5aGbXG5B.png" /></figure><p>Image: Domino speedup comparison from the <a href="https://huggingface.co/Huang2020/Qwen3-8B-Domino-b16">Domino Hugging Face model card</a>.</p><p>The paper evaluates Domino on Qwen3–4B and Qwen3–8B across math, code, and chat/instruction benchmarks.</p><p>The most direct comparison for me is Domino vs DFlash because Domino uses a DFlash-style parallel backbone and adds causal correction on top.</p><p>On Qwen3–8B with greedy decoding, the paper reports:</p><ul><li>GSM8K: DFlash 5.21x, Domino 7.92x</li><li>MATH-500: DFlash 6.18x, Domino 7.38x</li><li>HumanEval: DFlash 5.21x, Domino 5.89x</li><li>MBPP: DFlash 4.71x, Domino 5.53x</li><li>MT-Bench: DFlash 2.73x, Domino 3.29x</li><li>Alpaca: DFlash 2.23x, Domino 2.78x</li></ul><blockquote><em>The paper also reports that, compared with DFlash, Domino adds only </em><em>56M parameters, about </em><em>5.3%, and increases total draft-then-verify latency by only </em><em>2.8%.</em></blockquote><p>That is the key trade-off.</p><p>Domino pays a tiny overhead for causal correction, but gets more accepted tokens per cycle.</p><h3>Domino vs DFlash: why Domino is better</h3><p>I would not describe Domino as “DFlash is wrong.”</p><p>That misses the point.</p><p>DFlash is the foundation Domino builds on.</p><p>DFlash showed that block-parallel drafting is a very strong direction. Domino keeps that idea and asks how to recover the causal dependency that block-parallel drafting gives up.</p><p>DFlash is excellent when you want a simple block-parallel drafter.</p><p>Domino is better in the reported Qwen3 experiments because it improves the thing that matters most after draft latency is low:</p><p><strong>average acceptance length</strong>.</p><p>Speculative decoding speedup is not just about proposing tokens quickly. It is about proposing tokens that the target model accepts.</p><p>DFlash reduces draft cost aggressively. Domino keeps that low draft cost, then nudges the draft distribution closer to the target model by adding prefix-dependent correction.</p><p>The intuition is:</p><pre>DFlash: &quot;Predict the block in parallel.&quot;<br>Domino: &quot;Predict the block in parallel, but let each position see a cheap summary of previous sampled draft tokens.&quot;</pre><p>That causal summary is exactly what helps later draft tokens become more coherent with earlier draft tokens.</p><p>And because the correction head is lightweight, Domino does not fall back into the expensive autoregressive draft loop.</p><p>So the reason Domino is better than DFlash in this setup is not one single magic component. It is the combination:</p><ul><li>DFlash-style block-parallel backbone</li><li>causal encoder over previous draft tokens</li><li>low-rank residual correction in logit space</li><li>teacher-forced training</li><li>base-anchored curriculum</li><li>optimized runtime with fused kernels and CUDA Graphs</li></ul><p>This combination shifts the speed-quality trade-off.</p><h3>How would you run Domino?</h3><p>The released Hugging Face checkpoint is not a standalone language model. It is a draft model that must be paired with the target model.</p><p>For example, the Qwen3–8B Domino draft model is paired with Qwen/Qwen3-8B:</p><pre>from transformers import AutoModel, AutoModelForCausalLM, AutoTokenizer<br><br>draft_model = AutoModel.from_pretrained(<br>    &quot;Huang2020/Qwen3-8B-Domino-b16&quot;,<br>    trust_remote_code=True,<br>    dtype=&quot;auto&quot;,<br>    device_map=&quot;cuda:0&quot;,<br>).eval()<br><br>target_model = AutoModelForCausalLM.from_pretrained(<br>    &quot;Qwen/Qwen3-8B&quot;,<br>    dtype=&quot;auto&quot;,<br>    device_map=&quot;cuda:0&quot;,<br>).eval()<br><br>tokenizer = AutoTokenizer.from_pretrained(&quot;Qwen/Qwen3-8B&quot;)<br>messages = [<br>    {<br>        &quot;role&quot;: &quot;user&quot;,<br>        &quot;content&quot;: &quot;How many positive whole-number divisors does 196 have?&quot;,<br>    }<br>]<br><br>text = tokenizer.apply_chat_template(<br>    messages,<br>    tokenize=False,<br>    add_generation_prompt=True,<br>    enable_thinking=False,<br>)<br><br>model_inputs = tokenizer([text], return_tensors=&quot;pt&quot;).to(draft_model.device)<br><br>output_ids = draft_model.spec_generate(<br>    input_ids=model_inputs[&quot;input_ids&quot;],<br>    target=target_model,<br>    max_new_tokens=2048,<br>    temperature=0.0,<br>    stop_token_ids=[tokenizer.eos_token_id],<br>)<br><br>generated_ids = output_ids[:, model_inputs[&quot;input_ids&quot;].shape[1]:]<br><br>print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))</pre><p>The repo also includes benchmark scripts for Hugging Face and SGLang.</p><p>For SGLang, the official repo currently points to a Domino-compatible branch:</p><pre>git clone --branch sglang-feat/dflash-domino https://github.com/jianuo-huang/Domino.git sglang-domino<br>cd sglang-domino<br>python -m pip install -e ./python</pre><p>The practical point is simple:</p><p>Domino is an inference-system technique, not just a model checkpoint. The checkpoint, target model, serving runtime, CUDA kernels, and benchmark configuration all matter.</p><h3>How I think about Domino</h3><p>The best way to understand Domino is:</p><p><strong>Domino is a correction layer on top of parallel speculative decoding.</strong></p><p>DFlash made the drafter fast by predicting a block in parallel.</p><p>Domino says:</p><blockquote>What if the parallel block was almost right, but each position needed a cheap causal adjustment?</blockquote><p>That is a very ML-systems-like idea.</p><p>Do not throw away the fast path. Keep it.</p><p>Then add the smallest possible correction mechanism that fixes the biggest weakness.</p><p>The Domino head does not try to become a full autoregressive language model. It only tries to provide enough prefix-dependent information to improve draft acceptance.</p><p>This is why the method feels elegant to me.</p><p>It respects the actual bottleneck.</p><h3>Final thoughts</h3><p>Speculative decoding is becoming less about “use a small model to guess tokens” and more about designing specialized draft systems that match the target model and hardware.</p><p>DFlash pushed the field toward block-parallel drafting.</p><p>Domino pushes it one step further by showing that we can reintroduce causal dependency without returning to expensive autoregressive draft execution.</p><p>For me, the main lesson is:</p><p><strong>The best drafter is not necessarily the most language-model-like drafter. It is the drafter that gives the target model the most acceptable tokens per unit of latency.</strong></p><p>Domino improves that trade-off by combining a DFlash-style parallel backbone with lightweight causal correction.</p><p>That is why it is worth paying attention to.</p><h3>References</h3><ul><li><a href="https://arxiv.org/abs/2605.29707">Domino paper on arXiv</a></li><li><a href="https://arxiv.org/pdf/2605.29707">Domino PDF</a></li><li><a href="https://github.com/jianuo-huang/Domino">Domino GitHub repository</a></li><li><a href="https://huggingface.co/Huang2020/Qwen3-8B-Domino-b16">Qwen3–8B Domino draft model on Hugging Face</a></li><li><a href="https://huggingface.co/collections/Huang2020/domino">Domino model collection on Hugging Face</a></li><li><a href="https://arxiv.org/abs/2602.06036">DFlash paper on arXiv</a></li><li><a href="https://github.com/z-lab/dflash">DFlash GitHub repository</a></li><li><a href="https://luvbansall.substack.com/p/dflash-easily-explained-how-block">My DFlash blog post</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=09e6ee93370d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DFlash Easily Explained: How Block Diffusion Makes Speculative Decoding Faster]]></title>
            <link>https://luv-bansal.medium.com/dflash-easily-explained-how-block-diffusion-makes-speculative-decoding-faster-9c742c0fed9e?source=rss-234f1d6be728------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c742c0fed9e</guid>
            <category><![CDATA[speculative-decoding]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[inference]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Luv Bansal]]></dc:creator>
            <pubDate>Sat, 04 Jul 2026 19:23:19 GMT</pubDate>
            <atom:updated>2026-07-04T19:23:19.747Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>Ultimate guide to DFlash, KV injection, block diffusion drafting, and why this paper matters for faster LLM inference.</em></h4><p>In this blog, we will understand <strong>DFlash</strong>, a new speculative decoding method from the paper <a href="https://arxiv.org/abs/2602.06036">DFlash: Block Diffusion for Flash Speculative Decoding</a> by Jian Chen, Yesheng Liang, and Zhijian Liu. The paper was accepted at ICML 2026, and the authors released code and model checkpoints through the <a href="https://github.com/z-lab/dflash">z-lab/dflash GitHub repository</a> and <a href="https://huggingface.co/z-lab/Qwen3-8B-DFlash-b16">Hugging Face models</a>.</p><p>The problem DFlash tries to solve is simple to state but very hard to fix:</p><blockquote>LLMs are slow during generation because they produce text one token at a time.</blockquote><p>Even if your GPU is extremely powerful, autoregressive decoding forces the model into a sequential loop:</p><pre>token_1 -&gt; token_2 -&gt; token_3 -&gt; token_4 -&gt; ...</pre><p>This is especially painful for long reasoning traces, code generation, agent loops, and any application where the user waits for hundreds or thousands of output tokens.</p><p>DFlash is interesting because it asks a different question:</p><blockquote><em>What if the draft model in speculative decoding did not generate future tokens one by one?</em></blockquote><p>Instead, DFlash uses a small <strong>block diffusion drafter</strong> to predict a full block of future tokens in a single forward pass. Then the target LLM verifies those tokens, preserving the output distribution of the target model.</p><p>The paper reports over 6x lossless acceleration across models and tasks, and up to 2.5x higher speedup than EAGLE-3, which is one of the strongest speculative decoding baselines.</p><p>Let’s start from the basics and build up the full idea.</p><h4>What is speculative decoding?</h4><p>Speculative decoding is an inference trick for making autoregressive LLMs faster without changing the final model distribution.</p><p>The setup has two models:</p><ol><li>A <strong>target model</strong>, which is the high-quality model we actually want to sample from.</li><li>A <strong>draft model</strong>, which is faster and proposes several future tokens.</li></ol><p>The loop looks like this:</p><pre>def speculative_decode(prompt, target_model, draft_model, max_steps):<br>    context = prompt<br><br>    for _ in range(max_steps):<br>        # 1. Draft several future tokens quickly<br>        draft_tokens = draft_model.propose(context, num_tokens=8)<br><br>        # 2. Verify them with the target model in one pass<br>        accepted_prefix, bonus_token = target_model.verify(context, draft_tokens)<br><br>        # 3. Commit only the tokens accepted by the target model<br>        context += accepted_prefix<br><br>        # 4. Add the target model&#39;s bonus token if verification stops<br>        if bonus_token is not None:<br>            context += [bonus_token]<br><br>    return context</pre><p>The key idea is that verification is parallel. Instead of running the large target model separately for every future token, we ask it to score a candidate block in one pass.</p><p>If the draft model guesses well, we accept multiple tokens per target-model call.</p><p>The paper summarizes this trade-off with a very useful latency equation:</p><pre>L = (T_draft + T_verify) / tau</pre><p>where:</p><ul><li>T_draft is the time spent generating draft tokens.</li><li>T_verify is the time spent verifying them with the target model.</li><li>tau is the average number of accepted tokens per decoding cycle, including the target bonus token.</li></ul><p>This equation is the whole game.</p><p>To make decoding faster, we need to reduce draft time, reduce verification time, increase accepted tokens, or find a better trade-off among all three.</p><h4>The hidden bottleneck in existing speculative decoding</h4><p>Many speculative decoding systems already verify tokens in parallel.</p><p>But many strong draft methods still generate those draft tokens autoregressively.</p><p>That means the draft side still looks like this:</p><pre>draft_1 -&gt; draft_2 -&gt; draft_3 -&gt; draft_4 -&gt; ...</pre><p>So if the drafter proposes gamma tokens, its draft cost is roughly:</p><pre>T_draft = gamma * t_step</pre><p>This is the limitation DFlash focuses on.</p><p>EAGLE-style methods are very strong because they use target-model features to produce better drafts. But the draft process is still sequential. If we make the drafter deeper, it may become more accurate, but each token becomes more expensive. If we make it shallow, it is fast, but acceptance length saturates.</p><p>DFlash changes the draft process itself.</p><p>Instead of drafting future tokens one at a time, it creates a block of masked positions and predicts all of them together:</p><pre>[MASK, MASK, MASK, MASK] -&gt; [draft_1, draft_2, draft_3, draft_4]</pre><p>Now the draft cost becomes closer to:</p><pre>T_draft = t_parallel</pre><p>For a moderate block size, the cost is much less sensitive to the number of speculative tokens.</p><p>This is why DFlash can afford a more expressive drafter. It is not paying a full sequential forward pass for each draft token.</p><h4>DFlash in one picture</h4><figure><img alt="DFlash architecture showing target context features, KV injection, block diffusion drafting, and target verification" src="https://cdn-images-1.medium.com/max/1024/0*eynkuWpZ4eYmTByN.png" /></figure><p><em>Image: DFlash system architecture from the </em><a href="https://github.com/z-lab/dflash"><em>DFlash GitHub repository</em></a><em>.</em></p><p>At a high level, DFlash has three core ingredients:</p><ol><li>A small <strong>block diffusion draft model</strong> that predicts multiple future tokens in parallel.</li><li><strong>Target-model hidden features</strong> that tell the drafter what the large model already knows about the context.</li><li><strong>KV injection</strong>, where target features are injected into the key-value cache of every draft layer.</li></ol><p>The most important intuition is:</p><p>The diffusion drafter should not behave like a tiny standalone model.</p><p>It should behave like a lightweight adapter attached to the target model.</p><p>This is a very practical framing. Diffusion LLMs do not need to beat autoregressive LLMs at full generation quality. They only need to be good enough at proposing candidate blocks that the target model accepts.</p><h4>Step by step: how DFlash works during inference</h4><p>Let’s walk through a single DFlash decoding cycle.</p><h4>1. Run the target model and extract hidden features</h4><p>The target LLM first runs on the current context. During this pass, DFlash extracts hidden states from several target layers.</p><p>In the paper setup, hidden states are sampled from shallow to deep layers and then fused through a projection layer:</p><pre>target_context = projection(concat(hidden_layer_1, hidden_layer_2, ..., hidden_layer_k))</pre><p>Why not just use the target model’s logits?</p><p>Because logits only describe the next-token distribution. Hidden states carry richer information about long-range context, semantics, reasoning state, and likely continuations.</p><p>The paper’s key observation is that a strong autoregressive target model already contains useful information about multiple future tokens inside its hidden features. DFlash turns those hidden features into conditioning context for the drafter.</p><h4>2. Inject target features into every draft layer</h4><p>Many feature-based drafters feed target features into the drafter only at the input.</p><p>DFlash does something stronger: it injects the target context into the key and value projections of every draft layer.</p><p>Conceptually, each draft layer does this:</p><pre>Q_i = Wq_i * draft_hidden<br>K_i = concat(Wk_i * target_context, Wk_i * draft_hidden)<br>V_i = concat(Wv_i * target_context, Wv_i * draft_hidden)</pre><p>The draft tokens produce queries. The target context and draft tokens together provide keys and values.</p><p>This matters because the target signal remains available throughout the full depth of the drafter. It does not get washed out after the first layer.</p><p>The paper’s ablation shows that KV injection beats simple input fusion. This is one of the most useful lessons from DFlash: if the target model has the knowledge, treat it as persistent context, not just as an initial hint.</p><h4>3. Predict a block of tokens in parallel</h4><p>DFlash then creates a block of masked future positions:</p><pre>last_verified_token, [MASK, MASK, MASK, MASK, ...]</pre><p>The block diffusion drafter predicts the masked positions in one forward pass.</p><p>This is different from a normal autoregressive drafter:</p><pre># Autoregressive drafter<br>for i in range(num_draft_tokens):<br>    next_token = draft_model(context)<br>    context.append(next_token)</pre><p>DFlash is closer to:</p><pre>def dflash_draft(last_verified_token, target_context, block_size):<br>    masked_block = [last_verified_token] + [&quot;[MASK]&quot;] * (block_size - 1)<br><br>    logits = diffusion_drafter(<br>        tokens=masked_block,<br>        injected_target_context=target_context,<br>    )<br><br>    draft_tokens = sample_or_argmax(logits)<br>    return draft_tokens</pre><p>All draft positions are predicted together.</p><h4>4. Verify with the target model</h4><p>Finally, the target model verifies the drafted block.</p><p>Only the longest valid prefix is accepted. If a draft token diverges from the target model’s distribution, verification stops and the target model supplies the next valid token.</p><p>This is why DFlash is called a lossless acceleration method in the speculative decoding sense. The draft model proposes candidates, but the target model remains the final authority.</p><h4>Training design: why DFlash is not just a small diffusion model</h4><p>DFlash is trained as a draft adapter for a frozen target model.</p><p>It is not trained as a standalone diffusion LLM.</p><p>That distinction is important.</p><h4>Shared embedding and LM head</h4><p>DFlash shares the target model’s token embedding layer and LM head. These parts stay frozen.</p><p>This has two benefits:</p><ol><li>The drafter stays aligned with the target vocabulary and output space.</li><li>The number of trainable parameters stays small.</li></ol><p>Only the intermediate draft Transformer layers are trained.</p><h4>Random anchor sampling</h4><p>Standard block diffusion training divides text into blocks and masks tokens inside each block.</p><p>DFlash adapts this to speculative decoding.</p><p>During training, it samples random anchor tokens from the response. Each anchor is kept clean, and the following positions are masked:</p><pre>anchor, [MASK, MASK, MASK, ...]</pre><p>The draft model learns to predict the continuation from that clean anchor while conditioned on target hidden features.</p><p>This mirrors inference, where the drafter starts from a token that survived target verification.</p><p>The paper reports that random anchor sampling improves both acceptance length and speedup compared with more standard block construction.</p><h4>Sparse attention</h4><p>Multiple sampled blocks can be packed into one training sequence.</p><p>But we must avoid information leakage across unrelated blocks.</p><p>DFlash uses sparse attention so that tokens can attend inside the right block and to the right target features, but cannot cheat by seeing unrelated clean future tokens.</p><p>This is where Flex Attention becomes useful in the training implementation.</p><h4>Position-weighted loss</h4><p>In speculative decoding, early draft tokens matter more than later ones.</p><p>If the first token in a draft block is wrong, the rest of the block will be rejected anyway.</p><p>DFlash therefore weights early positions more heavily:</p><pre>w_k = exp(-(k - 1) / gamma)</pre><p>This is a very practical loss design. It optimizes for what actually matters during verification: getting the earliest tokens right so the accepted prefix grows.</p><p>The appendix reports training details such as AdamW, 6 epochs, learning rate 6e-4, gradient clipping 1.0, warmup ratio 0.04, max sequence length 3072 tokens or 4096 for Qwen3-Coder, 512 sampled anchors per sequence, and different loss-decay values depending on block size.</p><h4>What results does DFlash report?</h4><p>The headline result is strong:</p><ul><li>The arXiv abstract reports over 6x lossless acceleration across models and tasks.</li><li>The camera-ready page states DFlash is accepted at ICML 2026.</li><li>The Hugging Face Qwen3–8B DFlash model card reports up to 6.17x lossless acceleration for Qwen3-8B.</li><li>The project page reports that DFlash is roughly 2.5x faster than EAGLE-3 on many Qwen3-8B tasks.</li></ul><figure><img alt="DFlash speedup chart for Qwen3-8B compared with EAGLE-3 and autoregressive decoding" src="https://cdn-images-1.medium.com/max/1024/0*ef16_WQz6VN8zJCb.png" /></figure><p><em>Image: DFlash speedup chart from the </em><a href="https://huggingface.co/z-lab/Qwen3-8B-DFlash-b16"><em>Qwen3–8B-DFlash-b16 model card</em></a><em>.</em></p><p>The paper evaluates DFlash across math, code, chat, and reasoning tasks. Your biggest practical takeaway should be this:</p><p>DFlash helps most when decoding is the bottleneck and when the draft model can maintain high acceptance length without adding too much overhead.</p><p>This is especially relevant for:</p><ul><li>reasoning models with long traces;</li><li>code generation;</li><li>math problem solving;</li><li>agentic workflows with long outputs;</li><li>serving stacks where decode throughput is memory-bound and sequential.</li></ul><p>The paper also evaluates realistic serving with SGLang on a single B200 GPU using FlashAttention-4 and Spec-v2 overlap. It reports that speedups remain useful across concurrency levels from 1 to 32, although speedup naturally changes with batch size and workload.</p><p>The DFlash ecosystem is also moving into serving frameworks. The GitHub repository includes vLLM, SGLang, Transformers, and MLX examples. The vLLM speculators documentation describes DFlash as a small diffusion-LLM draft model that predicts an entire block in one pass, conditioned on target hidden states.</p><h4>How would you run DFlash?</h4><p>The model card gives a simple SGLang launch path for Qwen3–8B:</p><pre>python -m sglang.launch_server \<br>    --model-path Qwen/Qwen3-8B \<br>    --speculative-algorithm DFLASH \<br>    --speculative-draft-model-path z-lab/Qwen3-8B-DFlash-b16 \<br>    --tp-size 1 \<br>    --dtype bfloat16 \<br>    --attention-backend fa3 \<br>    --mem-fraction-static 0.75 \<br>    --trust-remote-code</pre><p>For vLLM, the setup is a config change around speculative decoding:</p><pre>vllm serve Qwen/Qwen3-8B \<br>  --speculative-config &#39;{&quot;method&quot;: &quot;dflash&quot;, &quot;model&quot;: &quot;z-lab/Qwen3-8B-DFlash-b16&quot;, &quot;num_speculative_tokens&quot;: 15}&#39; \<br>  --attention-backend flash_attn \<br>  --max-num-batched-tokens 32768</pre><p>And if you want to understand it at the Transformers level, the draft model exposes a custom spec_generate method:</p><pre>from transformers import AutoModel, AutoModelForCausalLM, AutoTokenizer<br><br>draft = AutoModel.from_pretrained(<br>    &quot;z-lab/Qwen3-8B-DFlash-b16&quot;,<br>    trust_remote_code=True,<br>    dtype=&quot;auto&quot;,<br>    device_map=&quot;cuda:0&quot;,<br>).eval()<br><br>target = AutoModelForCausalLM.from_pretrained(<br>    &quot;Qwen/Qwen3-8B&quot;,<br>    dtype=&quot;auto&quot;,<br>    device_map=&quot;cuda:0&quot;,<br>).eval()<br><br>tokenizer = AutoTokenizer.from_pretrained(&quot;Qwen/Qwen3-8B&quot;)<br><br>messages = [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;Write a quicksort in Python.&quot;}]<br>input_ids = tokenizer.apply_chat_template(<br>    messages,<br>    return_tensors=&quot;pt&quot;,<br>    add_generation_prompt=True,<br>    enable_thinking=False,<br>).to(draft.device)<br><br>output = draft.spec_generate(<br>    input_ids=input_ids,<br>    max_new_tokens=2048,<br>    temperature=0.0,<br>    target=target,<br>    stop_token_ids=[tokenizer.eos_token_id],<br>)<br><br>print(tokenizer.decode(output[0], skip_special_tokens=True))</pre><p>Note: the drafter is target-specific. The Qwen3–8B DFlash drafter must be paired with Qwen/Qwen3-8B. This is not a generic draft model you can attach to any random LLM.</p><h4>Why KV injection is the most important design idea</h4><p>For me, the most interesting part of DFlash is not just “<em>diffusion is parallel</em>.”</p><p>The more important part is how DFlash makes diffusion useful.</p><p>A tiny diffusion drafter without target information has a hard job. It must predict future tokens from scratch, while being much smaller than the target model.</p><p>DFlash avoids this by giving the drafter a rich target context.</p><p>This changes the role of the draft model. It no longer needs to understand everything independently. It needs to translate the target model’s hidden state into a good speculative block.</p><p>That is why KV injection matters.</p><blockquote>If the target context is only added at the input, the signal may fade as the draft network gets deeper. But if each draft layer can attend to target features through its key-value cache, then the target signal remains alive throughout the draft computation.</blockquote><p>The paper’s ablation supports this:</p><ul><li>KV injection improves acceptance length compared with input fusion.</li><li>A five-layer drafter gives the best speed-quality trade-off in the reported setup.</li><li>More target hidden features help, but they also increase storage or feature extraction cost.</li><li>Training with a larger block size can generalize down to smaller inference block sizes better than the reverse.</li></ul><p>That last point is useful for production. You can imagine training a draft model with a larger block size and then using smaller block sizes when concurrency is high or verification becomes more expensive.</p><h4>Practical gotchas before using DFlash</h4><p>DFlash is promising, but there are a few practical details to keep in mind.</p><h4>1. Speedup depends on workload and concurrency</h4><p>Speculative decoding speedups are not constants.</p><p>They depend on prompt length, output length, batch size, sampling temperature, target model, hardware, and serving framework.</p><p>At low concurrency, decoding latency is often the main bottleneck, so DFlash can help a lot. At higher concurrency, verification and batching behavior can change the trade-off.</p><h4>2. Acceptance length is everything</h4><p>If the drafter proposes many tokens but the target model rejects them early, speedup collapses.</p><p>That is why target conditioning, KV injection, and position-weighted training are central to the method.</p><h4>3. The drafter adds memory and engineering complexity</h4><p>Even a lightweight draft model is still another model. You need to load it, schedule it, and integrate it with target hidden states.</p><p>For production systems, this means you should benchmark end-to-end throughput, not just isolated token generation.</p><h4>4. Long context may need adaptation</h4><p>The paper notes that base DFlash models trained on shorter contexts can degrade at longer context lengths, and light long-context fine-tuning can help.</p><p>This matters for RAG, coding agents, and long reasoning traces where context length grows quickly.</p><h4>5. Framework support is still evolving</h4><p>The repository already includes examples for SGLang, vLLM, Transformers, and MLX, but some paths require specific versions or branches.</p><p>So before treating DFlash as drop-in infrastructure, test it on your actual serving stack.</p><h4>How I think about DFlash</h4><p>DFlash reframes diffusion models in a very practical way.</p><p>Most discussions about diffusion LLMs ask whether they can replace autoregressive LLMs.</p><p>DFlash asks something more useful:</p><p>Can diffusion be used as a fast draft mechanism while the autoregressive model remains the source of truth?</p><p>That is a much easier and more production-friendly target.</p><p>The drafter does not need to be perfect. It only needs to be good enough that the verifier accepts more tokens per cycle. The target model protects quality, while the diffusion drafter exposes more parallelism.</p><p>This is the reason I like the paper. It is not just another speculative decoding variant. It changes the draft model design space.</p><p>In EAGLE-style drafting, deeper drafters become expensive because drafting is sequential. In DFlash, deeper drafters can make sense because a block is drafted in one pass.</p><p>That is the core shift.</p><h4>Final thoughts</h4><p>DFlash is one of the most interesting recent papers in LLM inference optimization because it combines three ideas cleanly:</p><ol><li>Use speculative decoding to preserve target-model quality.</li><li>Use block diffusion to draft many tokens in parallel.</li><li>Use target hidden states and KV injection to make the small drafter accurate enough.</li></ol><p>If you are building LLM serving systems, the lesson is not simply “try DFlash.”</p><p>The broader lesson is:</p><p>Drafting quality and drafting latency must be optimized together.</p><p>A draft model that is too weak will not be accepted. A draft model that is too expensive will erase the speedup. DFlash finds a better point on that trade-off by making the draft process parallel and target-conditioned.</p><p>For workloads with long generations, especially reasoning and coding, this direction is worth watching closely.</p><h4>References</h4><ul><li><a href="https://arxiv.org/abs/2602.06036">DFlash: Block Diffusion for Flash Speculative Decoding</a></li><li><a href="https://z-lab.ai/projects/dflash/">DFlash project page</a></li><li><a href="https://github.com/z-lab/dflash">z-lab/dflash GitHub repository</a></li><li><a href="https://huggingface.co/z-lab/Qwen3-8B-DFlash-b16">Qwen3–8B-DFlash-b16 model card</a></li><li><a href="https://docs.vllm.ai/projects/speculators/en/latest/user_guide/algorithms/dflash/">vLLM Speculators documentation for DFlash</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c742c0fed9e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Build an AI Agent from Scratch — It’s Simpler Than You Think]]></title>
            <link>https://blog.stackademic.com/how-to-build-an-ai-agent-from-scratch-its-simpler-than-you-think-85273f071fdb?source=rss-234f1d6be728------2</link>
            <guid isPermaLink="false">https://medium.com/p/85273f071fdb</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Luv Bansal]]></dc:creator>
            <pubDate>Sat, 21 Feb 2026 23:44:28 GMT</pubDate>
            <atom:updated>2026-02-21T23:44:28.429Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*F2hH2liWaTY0wENWPM5HlQ.png" /><figcaption>Image illustration created with AI</figcaption></figure><h4>A step-by-step guide to building a fully functional code-editing AI agent with Python — from a basic chatbot to an agent that reads, lists, and edits files using LLM tool use</h4><p>In this blog, we’ll build a fully functional AI agent from scratch, one that can read files, list directories, and even edit code all in less than 400 lines of Python code.</p><p>If you’ve ever watched an AI agent editing files, running commands, recovering from errors, and retrying different strategies, it might seem like there’s some deep magic behind it. There isn’t. It’s an LLM, a loop, and enough tokens. That’s it.</p><p>I was inspired by <a href="https://ampcode.com/notes/how-to-build-an-agent">Thorsten Ball’s excellent post</a> where he builds a code-editing agent in Go. In this blog, I’ll walk you through building the same thing in Python step by step using the Anthropic SDK (for LLM inference). By the end, you’ll have a working agent that can autonomously navigate your file system and edit code.</p><p>But before diving in, let’s understand what an AI agent actually is and how tool use works.</p><h4>What is an AI Agent?</h4><p>An <strong>AI Agent</strong> is simply an LLM with access to tools, giving it the ability to modify something outside its context window.</p><p>That’s the whole definition. Let’s break it down:</p><ul><li><strong>LLM</strong> — The brain that reasons and makes decisions</li><li><strong>Tools</strong> — Functions the agent can call to interact with the outside world (read files, search the web, execute code, etc.)</li><li><strong>Loop</strong> — The agent keeps running until the task is done, calling tools as needed</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-VTthVpBHpFH9J0ivYSFEA.png" /><figcaption>Image illustration created with AI</figcaption></figure><p>In simple terms, think of it like this: you (the user) give a task to the agent. The agent thinks about it, decides it needs more information, calls a tool to get that information, processes the result, decides if it needs to do more, and repeats until the task is done. That’s the entire loop.</p><p>Now let’s build one.</p><h4>Prerequisites</h4><p>Here’s what you’ll need to follow along:</p><ul><li><strong>Python 3.10+</strong> installed</li><li><strong>Anthropic API key</strong> — Get one from the <a href="https://console.anthropic.com/settings/keys">Anthropic Console</a></li><li>A terminal and a text editor</li></ul><p>Install the required library:</p><pre>pip install anthropic</pre><p>Set your API key as an environment variable:</p><pre>export ANTHROPIC_API_KEY=&quot;your-api-key-here&quot;</pre><h4>Step 1: Build a Basic Chatbot</h4><p>Let’s start simple. Before we add any agent capabilities, let’s build a basic terminal chatbot that talks to Claude.</p><p>Create a new file called agent.py:</p><pre>import os<br>import json<br>from anthropic import Anthropic<br><br>client = Anthropic()<br><br>def chat():<br>    &quot;&quot;&quot;Basic chatbot loop - talk to Claude in the terminal.&quot;&quot;&quot;<br>    conversation = []<br>    <br>    print(&quot;Chat with Claude (use &#39;ctrl-c&#39; to quit)&quot;)<br>    <br>    while True:<br>        user_input = input(&quot;\033[94mYou\033[0m: &quot;)<br>        <br>        # Add user message to conversation<br>        conversation.append({<br>            &quot;role&quot;: &quot;user&quot;,<br>            &quot;content&quot;: user_input<br>        })<br>        <br>        # Send to Claude<br>        response = client.messages.create(<br>            model=&quot;claude-sonnet-4-20250514&quot;,<br>            max_tokens=1024,<br>            messages=conversation<br>        )<br>        <br>        # Extract and print response<br>        assistant_message = response.content[0].text<br>        print(f&quot;\033[93mClaude\033[0m: {assistant_message}&quot;)<br>        <br>        # Add assistant response to conversation<br>        conversation.append({<br>            &quot;role&quot;: &quot;assistant&quot;,<br>            &quot;content&quot;: response.content<br>        })<br><br>if __name__ == &quot;__main__&quot;:<br>    chat()</pre><pre>Run it:</pre><pre>python agent.py</pre><pre>Chat with Claude (use &#39;ctrl-c&#39; to quit)<br>You: Hey! I&#39;m Luv! How are you?<br>Claude: Hi Luv! I&#39;m doing well, thanks for asking. How are you doing today?<br>You: What&#39;s my name?<br>Claude: Your name is Luv!</pre><p><strong>What’s happening here?</strong></p><ol><li>We print a prompt and wait for user input</li><li>We add the user’s message to the conversation list</li><li>We send the entire conversation to Claude</li><li>We print Claude’s response and add it to the conversation</li><li>Repeat</li></ol><p>Notice how Claude remembers your name from the first message. That’s because we send the <strong>entire conversation</strong> every time. The server (Anthropic’s API) is <strong>stateless</strong> — it only sees what’s in the conversation list we send. It’s up to us to maintain that context.</p><p>This is essentially every AI chat application you’ve ever used, just in the terminal. But it’s not an agent yet. Let’s change that.</p><h4>What is Tool Use (Function Calling)?</h4><p>Before we add tools to our agent, let’s understand the concept. The basic idea is this:</p><p><strong>Step 1:</strong> You tell the model what tools are available (name, description, input parameters)</p><p><strong>Step 2:</strong> When the model wants to use a tool, it responds with a special tool_use message instead of regular text</p><p><strong>Step 3:</strong> You execute the tool locally and send the result back to the model</p><p><strong>Step 4:</strong> The model uses that result to continue its response</p><p>In simple terms, imagine you’re talking to a friend and you tell them: <em>“In the following conversation, wink if you want me to raise my arm.”</em> That’s essentially what tool use is: we define signals (tools) for the model, and when it “winks” (requests a tool), we “raise our arm” (execute the function) and tell it what happened.</p><p>We can already demonstrate this concept without any code changes. Try this conversation in our basic chatbot:</p><pre>You: You are a weather expert. When I ask about weather, reply with <br>     get_weather(&lt;location&gt;). I will tell you the actual weather.<br>Claude: Understood! I&#39;ll use get_weather(&lt;location&gt;) when you ask about <br>        the weather. Ready when you are.<br>You: What&#39;s the weather in Mumbai?<br>Claude: get_weather(Mumbai)<br>You: Hot and humid, 32 degrees celsius<br>Claude: The current weather in Mumbai is hot and humid at 32°C (about 90°F).<br>        That&#39;s fairly typical for Mumbai. Stay hydrated!</pre><p>See what happened? We told Claude to “wink” with get_weather when it wants weather data. It did, we replied with the result, and Claude processed it naturally.</p><p>These models are trained and fine-tuned to use tools they’re very eager to do so. By now, in 2026, they essentially “know” that they don’t know everything and can use tools to get more information.</p><p>Now, instead of doing this manually, let’s formalize it with proper API support.</p><h4>Step 2: Add the Tool Use Infrastructure</h4><p>Let’s restructure our code to support tools. We’ll create an Agent class with the core agent loop:</p><pre>import os<br>import json<br>from anthropic import Anthropic<br><br>client = Anthropic()<br><br>class Agent:<br>    def __init__(self, tools=None):<br>        self.tools = tools or []<br>        self.tool_functions = {}<br>        <br>    def register_tool(self, name, description, input_schema, function):<br>        &quot;&quot;&quot;Register a tool that Claude can use.&quot;&quot;&quot;<br>        self.tools.append({<br>            &quot;name&quot;: name,<br>            &quot;description&quot;: description,<br>            &quot;input_schema&quot;: input_schema<br>        })<br>        self.tool_functions[name] = function<br>    <br>    def execute_tool(self, tool_name, tool_input):<br>        &quot;&quot;&quot;Execute a tool and return the result.&quot;&quot;&quot;<br>        if tool_name not in self.tool_functions:<br>            return f&quot;Error: Tool &#39;{tool_name}&#39; not found&quot;<br>        <br>        try:<br>            result = self.tool_functions[tool_name](tool_input)<br>            return result<br>        except Exception as e:<br>            return f&quot;Error: {str(e)}&quot;<br>    <br>    def run(self):<br>        &quot;&quot;&quot;Main agent loop.&quot;&quot;&quot;<br>        conversation = []<br>        <br>        print(&quot;Chat with Claude (use &#39;ctrl-c&#39; to quit)&quot;)<br>        <br>        while True:<br>            # Get user input<br>            user_input = input(&quot;\033[94mYou\033[0m: &quot;)<br>            conversation.append({<br>                &quot;role&quot;: &quot;user&quot;,<br>                &quot;content&quot;: user_input<br>            })<br>            <br>            # Agent loop - keeps running until no more tool calls<br>            while True:<br>                response = client.messages.create(<br>                    model=&quot;claude-sonnet-4-20250514&quot;,<br>                    max_tokens=4096,<br>                    messages=conversation,<br>                    tools=self.tools<br>                )<br>                <br>                # Add assistant response to conversation<br>                conversation.append({<br>                    &quot;role&quot;: &quot;assistant&quot;,<br>                    &quot;content&quot;: response.content<br>                })<br>                <br>                # Process response - check for tool use<br>                tool_results = []<br>                for block in response.content:<br>                    if block.type == &quot;text&quot;:<br>                        print(f&quot;\033[93mClaude\033[0m: {block.text}&quot;)<br>                    elif block.type == &quot;tool_use&quot;:<br>                        print(f&quot;\033[92mtool\033[0m: {block.name}({json.dumps(block.input)})&quot;)<br>                        result = self.execute_tool(block.name, block.input)<br>                        tool_results.append({<br>                            &quot;type&quot;: &quot;tool_result&quot;,<br>                            &quot;tool_use_id&quot;: block.id,<br>                            &quot;content&quot;: str(result)<br>                        })<br>                <br>                # If no tool calls, we&#39;re done - wait for next user input<br>                if not tool_results:<br>                    break<br>                <br>                # Send tool results back to Claude<br>                conversation.append({<br>                    &quot;role&quot;: &quot;user&quot;,<br>                    &quot;content&quot;: tool_results<br>                })</pre><p><strong>Let’s break down the key part — the inner </strong><strong>while True loop:</strong></p><p>This is the heartbeat of every AI agent. Here’s what happens:</p><ol><li>Send the conversation to Claude (including tool definitions)</li><li>Get the response back</li><li>Check each content block in the response:</li></ol><ul><li>If it’s text → print it</li><li>If it’s tool_use → execute the tool and collect the result and append it to the conversation list</li></ul><p>4. Then the same loop If there were tool calls → send results back to Claude and loop again</p><p>5. If no tool calls → Claude is done, break and wait for next user input</p><p>That’s it. This inner loop is what transforms a chatbot into an agent. The agent keeps calling tools and processing results until it decides it has enough information to give you a final answer.</p><h4>Step 3: The read_file Tool</h4><p>Now let’s build our first tool: read_file. This tool allows Claude to read the contents of any file in the working directory.</p><p>Each tool needs four things:</p><ul><li><strong>Name</strong> — What to call it</li><li><strong>Description</strong> — Tells the model what the tool does, when to use it, when NOT to use it</li><li><strong>Input schema</strong> — JSON schema describing what inputs the tool expects</li><li><strong>Function</strong> — The actual implementation that runs when Claude calls it</li></ul><pre>def read_file(input_data):<br>    &quot;&quot;&quot;Read the contents of a file.&quot;&quot;&quot;<br>    path = input_data[&quot;path&quot;]<br>    <br>    try:<br>        with open(path, &quot;r&quot;) as f:<br>            return f.read()<br>    except FileNotFoundError:<br>        return f&quot;Error: File &#39;{path}&#39; not found&quot;<br>    except Exception as e:<br>        return f&quot;Error reading file: {str(e)}&quot;</pre><p>Register it with our agent:</p><pre>agent = Agent()<br><br>agent.register_tool(<br>    name=&quot;read_file&quot;,<br>    description=&quot;Read the contents of a given relative file path. Use this when you want to see what&#39;s inside a file. Do not use this with directory names.&quot;,<br>    input_schema={<br>        &quot;type&quot;: &quot;object&quot;,<br>        &quot;properties&quot;: {<br>            &quot;path&quot;: {<br>                &quot;type&quot;: &quot;string&quot;,<br>                &quot;description&quot;: &quot;The relative path of a file in the working directory.&quot;<br>            }<br>        },<br>        &quot;required&quot;: [&quot;path&quot;]<br>    },<br>    function=read_file<br>)</pre><p>Let’s test it! First, create a test file:</p><pre>echo &#39;What animal is the most disagreeable because it always says neigh?&#39; &gt; secret-file.txt</pre><p>Now run the agent:</p><pre>You: Help me solve the riddle in secret-file.txt<br>tool: read_file({&quot;path&quot;: &quot;secret-file.txt&quot;})<br>Claude: The answer to the riddle is: A horse! 🐴<br><br>The riddle plays on the word &quot;neigh&quot; which is the sound a horse <br>makes — sounding like &quot;nay&quot;, meaning to disagree. So horses are <br>&quot;disagreeable&quot; because they always say &quot;neigh/nay&quot;!</pre><p>Notice something powerful here: <strong>we never told Claude</strong> “if a user asks about a file, read the file.” We also didn’t say “if something looks like a filename, figure out how to read it.” We simply said “help me solve the thing in this file” and Claude <strong>figured out on its own</strong> that it should read the file to answer the question.</p><p>This is the magic of tool use, you define capabilities, and the LLM decides when and how to use them.</p><h4>Step 4: The list_files Tool</h4><p>Every good agent needs to orient itself. Just like the first thing you do on a new server is run ls, let&#39;s give Claude the same ability:</p><pre>import os<br><br>def list_files(input_data):<br>    &quot;&quot;&quot;List files and directories at a given path.&quot;&quot;&quot;<br>    dir_path = input_data.get(&quot;path&quot;, &quot;.&quot;)<br>    <br>    try:<br>        entries = []<br>        for item in os.listdir(dir_path):<br>            full_path = os.path.join(dir_path, item)<br>            if os.path.isdir(full_path):<br>                entries.append(f&quot;{item}/&quot;)<br>            else:<br>                entries.append(item)<br>        return json.dumps(sorted(entries))<br>    except Exception as e:<br>        return f&quot;Error listing files: {str(e)}&quot;<br><br>agent.register_tool(<br>    name=&quot;list_files&quot;,<br>    description=&quot;List files and directories at a given path. If no path is provided, lists files in the current directory.&quot;,<br>    input_schema={<br>        &quot;type&quot;: &quot;object&quot;,<br>        &quot;properties&quot;: {<br>            &quot;path&quot;: {<br>                &quot;type&quot;: &quot;string&quot;,<br>                &quot;description&quot;: &quot;Optional relative path to list files from. Defaults to current directory if not provided.&quot;<br>            }<br>        },<br>        &quot;required&quot;: []<br>    },<br>    function=list_files<br>)</pre><pre>Now Claude can combine tools — and it does this automatically:</pre><pre>You: Tell me about all the Python files in here. Be brief!<br>tool: list_files({})<br>tool: read_file({&quot;path&quot;: &quot;agent.py&quot;})<br>Claude: Here&#39;s a quick overview:<br><br>**agent.py**: Implements an AI agent that can interact with the local <br>filesystem. It uses the Anthropic SDK to communicate with Claude, <br>provides tools for reading files and listing directories, and manages <br>the conversation loop with tool execution.<br><br>This is a simple terminal-based chatbot with file system access.</pre><p>First it listed files to find the Python files, then it read each one. Just like you would — ls first, then cat the files you care about.</p><h4>Step 5: The edit_file Tool — Where It Gets Exciting</h4><p>Here’s the tool that transforms our agent from a reader into an editor. The implementation might surprise you with its simplicity:</p><pre>def edit_file(input_data):<br>    &quot;&quot;&quot;Edit a file by replacing old_str with new_str, or create a new file.&quot;&quot;&quot;<br>    path = input_data[&quot;path&quot;]<br>    old_str = input_data.get(&quot;old_str&quot;, &quot;&quot;)<br>    new_str = input_data.get(&quot;new_str&quot;, &quot;&quot;)<br>    <br>    if old_str == new_str:<br>        return &quot;Error: old_str and new_str must be different&quot;<br>    <br>    # If file doesn&#39;t exist and old_str is empty, create new file<br>    if not os.path.exists(path) and old_str == &quot;&quot;:<br>        # Create directories if needed<br>        dir_name = os.path.dirname(path)<br>        if dir_name:<br>            os.makedirs(dir_name, exist_ok=True)<br>        <br>        with open(path, &quot;w&quot;) as f:<br>            f.write(new_str)<br>        return f&quot;Successfully created file {path}&quot;<br>    <br>    # Read existing file<br>    try:<br>        with open(path, &quot;r&quot;) as f:<br>            content = f.read()<br>    except FileNotFoundError:<br>        return f&quot;Error: File &#39;{path}&#39; not found&quot;<br>    <br>    # Replace old_str with new_str<br>    if old_str not in content:<br>        return &quot;Error: old_str not found in file&quot;<br>    <br>    new_content = content.replace(old_str, new_str, 1)  # Replace first occurrence only<br>    <br>    with open(path, &quot;w&quot;) as f:<br>        f.write(new_content)<br>    <br>    return &quot;OK&quot;<br><br>agent.register_tool(<br>    name=&quot;edit_file&quot;,<br>    description=&quot;&quot;&quot;Make edits to a text file.<br><br>Replaces &#39;old_str&#39; with &#39;new_str&#39; in the given file. &#39;old_str&#39; and &#39;new_str&#39; MUST be different from each other.<br><br>If the file specified with path doesn&#39;t exist, it will be created.&quot;&quot;&quot;,<br>    input_schema={<br>        &quot;type&quot;: &quot;object&quot;,<br>        &quot;properties&quot;: {<br>            &quot;path&quot;: {<br>                &quot;type&quot;: &quot;string&quot;,<br>                &quot;description&quot;: &quot;The path to the file&quot;<br>            },<br>            &quot;old_str&quot;: {<br>                &quot;type&quot;: &quot;string&quot;,<br>                &quot;description&quot;: &quot;Text to search for - must match exactly and must only have one match exactly&quot;<br>            },<br>            &quot;new_str&quot;: {<br>                &quot;type&quot;: &quot;string&quot;,<br>                &quot;description&quot;: &quot;Text to replace old_str with&quot;<br>            }<br>        },<br>        &quot;required&quot;: [&quot;path&quot;, &quot;new_str&quot;]<br>    },<br>    function=edit_file<br>)</pre><p>Yes, <strong>string replacement</strong> to edit files. That’s it. Modern LLMs like Claude are extremely good at generating precise string replacements. They read the file, identify exactly what needs to change, and send the old and new strings.</p><p>Now let’s see it in action:</p><pre>You: Create a fizzbuzz.py that prints FizzBuzz from 1 to 100<br>tool: edit_file({&quot;path&quot;: &quot;fizzbuzz.py&quot;, &quot;old_str&quot;: &quot;&quot;, &quot;new_str&quot;: &quot;def fizzbuzz(n):\n    for i in range(1, n + 1):\n        output = &#39;&#39;\n        if i % 3 == 0:\n            output += &#39;Fizz&#39;\n        if i % 5 == 0:\n            output += &#39;Buzz&#39;\n        print(output or i)\n\nif __name__ == &#39;__main__&#39;:\n    print(&#39;Running FizzBuzz:&#39;)\n    fizzbuzz(100)&quot;})<br>Claude: I&#39;ve created fizzbuzz.py! You can run it with:<br>  python fizzbuzz.py</pre><p>Let’s verify it works:</p><pre>$ python fizzbuzz.py<br>Running FizzBuzz:<br>1<br>2<br>Fizz<br>4<br>Buzz<br>Fizz<br>...<br>14<br>FizzBuzz</pre><p>Now let’s ask it to <strong>edit</strong> the file:</p><pre>You: Edit fizzbuzz.py to only print until 15<br>tool: read_file({&quot;path&quot;: &quot;fizzbuzz.py&quot;})<br>tool: edit_file({&quot;path&quot;: &quot;fizzbuzz.py&quot;, &quot;old_str&quot;: &quot;fizzbuzz(100)&quot;, &quot;new_str&quot;: &quot;fizzbuzz(15)&quot;})<br>Claude: Done! I&#39;ve updated fizzbuzz.py to only print until 15.</pre><p>Notice the workflow: Claude first <strong>reads</strong> the file to understand its contents, then makes a <strong>precise edit</strong> replacing only the part that needs to change. It didn’t rewrite the entire file — it surgically replaced fizzbuzz(100) with fizzbuzz(15).</p><h4>Putting It All Together</h4><p>Here’s the complete agent.py — less than 200 lines of actual logic:</p><pre>import os<br>import json<br>from anthropic import Anthropic<br><br>client = Anthropic()<br><br># ---- Tool Implementations ----<br><br>def read_file(input_data):<br>    &quot;&quot;&quot;Read the contents of a file.&quot;&quot;&quot;<br>    path = input_data[&quot;path&quot;]<br>    try:<br>        with open(path, &quot;r&quot;) as f:<br>            return f.read()<br>    except FileNotFoundError:<br>        return f&quot;Error: File &#39;{path}&#39; not found&quot;<br>    except Exception as e:<br>        return f&quot;Error reading file: {str(e)}&quot;<br><br>def list_files(input_data):<br>    &quot;&quot;&quot;List files and directories at a given path.&quot;&quot;&quot;<br>    dir_path = input_data.get(&quot;path&quot;, &quot;.&quot;)<br>    try:<br>        entries = []<br>        for item in os.listdir(dir_path):<br>            full_path = os.path.join(dir_path, item)<br>            if os.path.isdir(full_path):<br>                entries.append(f&quot;{item}/&quot;)<br>            else:<br>                entries.append(item)<br>        return json.dumps(sorted(entries))<br>    except Exception as e:<br>        return f&quot;Error listing files: {str(e)}&quot;<br><br>def edit_file(input_data):<br>    &quot;&quot;&quot;Edit a file by replacing old_str with new_str, or create a new file.&quot;&quot;&quot;<br>    path = input_data[&quot;path&quot;]<br>    old_str = input_data.get(&quot;old_str&quot;, &quot;&quot;)<br>    new_str = input_data.get(&quot;new_str&quot;, &quot;&quot;)<br>    <br>    if old_str == new_str:<br>        return &quot;Error: old_str and new_str must be different&quot;<br>    <br>    if not os.path.exists(path) and old_str == &quot;&quot;:<br>        dir_name = os.path.dirname(path)<br>        if dir_name:<br>            os.makedirs(dir_name, exist_ok=True)<br>        with open(path, &quot;w&quot;) as f:<br>            f.write(new_str)<br>        return f&quot;Successfully created file {path}&quot;<br>    <br>    try:<br>        with open(path, &quot;r&quot;) as f:<br>            content = f.read()<br>    except FileNotFoundError:<br>        return f&quot;Error: File &#39;{path}&#39; not found&quot;<br>    <br>    if old_str and old_str not in content:<br>        return &quot;Error: old_str not found in file&quot;<br>    <br>    new_content = content.replace(old_str, new_str, 1)<br>    with open(path, &quot;w&quot;) as f:<br>        f.write(new_content)<br>    return &quot;OK&quot;<br><br># ---- Tool Definitions ----<br><br>TOOLS = [<br>    {<br>        &quot;name&quot;: &quot;read_file&quot;,<br>        &quot;description&quot;: &quot;Read the contents of a given relative file path. Use this when you want to see what&#39;s inside a file. Do not use this with directory names.&quot;,<br>        &quot;input_schema&quot;: {<br>            &quot;type&quot;: &quot;object&quot;,<br>            &quot;properties&quot;: {<br>                &quot;path&quot;: {<br>                    &quot;type&quot;: &quot;string&quot;,<br>                    &quot;description&quot;: &quot;The relative path of a file in the working directory.&quot;<br>                }<br>            },<br>            &quot;required&quot;: [&quot;path&quot;]<br>        }<br>    },<br>    {<br>        &quot;name&quot;: &quot;list_files&quot;,<br>        &quot;description&quot;: &quot;List files and directories at a given path. If no path is provided, lists files in the current directory.&quot;,<br>        &quot;input_schema&quot;: {<br>            &quot;type&quot;: &quot;object&quot;,<br>            &quot;properties&quot;: {<br>                &quot;path&quot;: {<br>                    &quot;type&quot;: &quot;string&quot;,<br>                    &quot;description&quot;: &quot;Optional relative path to list files from. Defaults to current directory.&quot;<br>                }<br>            },<br>            &quot;required&quot;: []<br>        }<br>    },<br>    {<br>        &quot;name&quot;: &quot;edit_file&quot;,<br>        &quot;description&quot;: &quot;Make edits to a text file. Replaces &#39;old_str&#39; with &#39;new_str&#39; in the given file. If the file doesn&#39;t exist and old_str is empty, creates a new file with new_str as content.&quot;,<br>        &quot;input_schema&quot;: {<br>            &quot;type&quot;: &quot;object&quot;,<br>            &quot;properties&quot;: {<br>                &quot;path&quot;: {<br>                    &quot;type&quot;: &quot;string&quot;,<br>                    &quot;description&quot;: &quot;The path to the file&quot;<br>                },<br>                &quot;old_str&quot;: {<br>                    &quot;type&quot;: &quot;string&quot;,<br>                    &quot;description&quot;: &quot;Text to search for - must match exactly&quot;<br>                },<br>                &quot;new_str&quot;: {<br>                    &quot;type&quot;: &quot;string&quot;,<br>                    &quot;description&quot;: &quot;Text to replace old_str with&quot;<br>                }<br>            },<br>            &quot;required&quot;: [&quot;path&quot;, &quot;new_str&quot;]<br>        }<br>    }<br>]<br><br>TOOL_FUNCTIONS = {<br>    &quot;read_file&quot;: read_file,<br>    &quot;list_files&quot;: list_files,<br>    &quot;edit_file&quot;: edit_file,<br>}<br><br># ---- Agent Loop ----<br><br>def run_agent():<br>    &quot;&quot;&quot;Main agent loop with tool use.&quot;&quot;&quot;<br>    conversation = []<br>    <br>    print(&quot;🤖 Code-Editing Agent (use &#39;ctrl-c&#39; to quit)&quot;)<br>    print(&quot;I can read, list, and edit files in this directory.\n&quot;)<br>    <br>    while True:<br>        user_input = input(&quot;\033[94mYou\033[0m: &quot;)<br>        conversation.append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: user_input})<br>        <br>        # Inner loop: keep running until no more tool calls<br>        while True:<br>            response = client.messages.create(<br>                model=&quot;claude-sonnet-4-20250514&quot;,<br>                max_tokens=4096,<br>                messages=conversation,<br>                tools=TOOLS<br>            )<br>            <br>            # Add response to conversation<br>            conversation.append({<br>                &quot;role&quot;: &quot;assistant&quot;,<br>                &quot;content&quot;: response.content<br>            })<br>            <br>            # Process response blocks<br>            tool_results = []<br>            for block in response.content:<br>                if block.type == &quot;text&quot;:<br>                    print(f&quot;\033[93mClaude\033[0m: {block.text}&quot;)<br>                elif block.type == &quot;tool_use&quot;:<br>                    print(f&quot;\033[92mtool\033[0m: {block.name}({json.dumps(block.input)})&quot;)<br>                    <br>                    func = TOOL_FUNCTIONS.get(block.name)<br>                    if func:<br>                        result = func(block.input)<br>                    else:<br>                        result = f&quot;Error: Unknown tool &#39;{block.name}&#39;&quot;<br>                    <br>                    tool_results.append({<br>                        &quot;type&quot;: &quot;tool_result&quot;,<br>                        &quot;tool_use_id&quot;: block.id,<br>                        &quot;content&quot;: str(result)<br>                    })<br>            <br>            # If no tool calls, break inner loop<br>            if not tool_results:<br>                break<br>            <br>            # Send tool results back and continue<br>            conversation.append({&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: tool_results})<br><br>if __name__ == &quot;__main__&quot;:<br>    run_agent()</pre><h4>How Does This Compare to Production Agents?</h4><p>You might be wondering if building an agent is this simple, what makes production agents like <a href="https://ampcode.com/">Amp</a>, <a href="https://cursor.sh/">Cursor</a>, or <a href="https://github.com/features/copilot">GitHub Copilot</a> so impressive?</p><p>The core loop is exactly what we built. The difference is engineering around that loop:</p><h4>What We Built vs Production Agent</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ctqy49wPDootY6qxVZ8IA.png" /><figcaption>Image generated with AI</figcaption></figure><p>As <a href="https://openai.com/business/guides-and-resources/a-practical-guide-to-building-ai-agents/">OpenAI’s agent guide</a> explains, production agents also use orchestration patterns like:</p><ul><li><strong>Single agent with many tools</strong> — Start here, expand incrementally</li><li><strong>Manager pattern</strong> — One central agent coordinates specialized sub-agents</li><li><strong>Decentralized handoff</strong> — Agents pass tasks to each other based on specialization</li></ul><p>The key insight from their guide: <em>“maximize a single agent’s capabilities first”</em> before jumping to multi-agent architectures.</p><h4>Key Lessons from Building This Agent</h4><p>Here are the important takeaways:</p><p><strong>1. Tool descriptions matter more than you think</strong></p><p>The description you write for each tool is essentially a prompt. Claude uses it to decide <em>when</em> and <em>how</em> to use the tool. Write clear, specific descriptions, include what the tool does, what input it expects, and when NOT to use it.</p><p><strong>2. The model decides when to use tools</strong></p><p>We never wrote any conditional logic like “if the user mentions a file, call read_file.” The LLM figures that out on its own based on the tool descriptions and the user’s intent. This is the most powerful aspect of tool use.</p><p><strong>3. Keep tool outputs simple</strong></p><p>Return plain strings or JSON. The simpler the format, the easier it is for the model to process. We return “OK” for successful edits and error messages for failures, that’s all Claude needs.</p><p><strong>4. The loop is everything</strong></p><p>The inner while True loop that keeps processing tool calls until there are none left that&#39;s the entire difference between a chatbot and an agent. A chatbot does one inference call per turn. An agent does as many as it needs.</p><p><strong>5. String replacement works surprisingly well</strong></p><p>You might expect file editing to require AST parsing, tree-sitter, or some complex diff algorithm. But Claude is trained to generate precise string replacements, and it works remarkably well. This is how many production code editors implement their edit tools.</p><h4>What’s Next? Extending Your Agent</h4><p>Now that you have a working agent, here are some ideas to extend it:</p><ul><li><strong>Add a </strong><strong>run_command tool</strong> — Let Claude execute shell commands and see the output</li><li><strong>Add a </strong><strong>search_files tool</strong> — Grep through files to find relevant code</li><li><strong>Add a system prompt</strong> — Tell Claude about the project, coding standards, and guidelines</li><li><strong>Add streaming</strong> — Use the Anthropic streaming API for real-time output</li><li><strong>Add safety guardrails</strong> — Confirm before destructive operations (deletes, overwrites)</li><li><strong>Add memory</strong> — Save conversation summaries between sessions</li></ul><p>Each of these is just another tool definition + function. The agent loop stays the same.</p><h4>Final Thoughts</h4><p>Building an AI agent is not rocket science. It’s an LLM, a loop, and tools. The three tools we built read_file, list_files, and edit_file are enough to create an agent that can autonomously navigate your codebase and make meaningful edits.</p><p>The real insight is how little code it takes. Less than 200 lines of Python logic, and you have something that genuinely feels like an intelligent assistant editing your code. The models are incredibly powerful now all they need is the right tools and the right loop.</p><p>If you’re building AI applications and haven’t explored tool use yet, I highly recommend starting with this simple agent. Once you see it read a file, think about it, and make a precise edit — all on its own, you’ll understand why everyone in the industry is talking about agents.</p><p>The emperor has no clothes. And that’s actually the exciting part — it means <em>you</em> can build this too.</p><p><em>Inspired by Thorsten Ball’s </em><a href="https://ampcode.com/notes/how-to-build-an-agent"><em>How to Build an Agent</em></a><em>. Original implementation in Go, adapted to Python for this blog.</em></p><p><em>All code from this blog is available and ready to copy-paste. Try building your own agent and see how far you can take it!</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=85273f071fdb" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/how-to-build-an-ai-agent-from-scratch-its-simpler-than-you-think-85273f071fdb">How to Build an AI Agent from Scratch — It’s Simpler Than You Think</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[KV Cache Explained: The Complete Guide to KV Cache in LLM Inference]]></title>
            <link>https://luv-bansal.medium.com/the-evolution-of-kv-cache-from-simple-buffers-to-distributed-memory-systems-df51cb8ce26f?source=rss-234f1d6be728------2</link>
            <guid isPermaLink="false">https://medium.com/p/df51cb8ce26f</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[kv-cache]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[mlops]]></category>
            <dc:creator><![CDATA[Luv Bansal]]></dc:creator>
            <pubDate>Sat, 21 Feb 2026 11:32:18 GMT</pubDate>
            <atom:updated>2026-02-21T12:15:03.504Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*s5tXsrpdVqSUJ0Tmt-itYw.png" /><figcaption>AI-generated illustration</figcaption></figure><h4>From PagedAttention to Distributed Caching — How KV Cache Evolved Across 5 Eras to Power Modern LLM Inference</h4><p>If you have ever deployed an LLM in production, you know that the model weights are only half the story. The other half is <strong>KV cache</strong>, the runtime memory that stores attention states so your model can generate tokens without recomputing everything from scratch. Managing this memory efficiently is what separates a sluggish demo from a production-grade inference system.</p><p>In this blog, we will walk through the six eras of KV cache management, from the days when it did not exist at all to the cutting-edge unified memory systems being built today. Along the way, I will share practical insights from deploying 250+ models and compare how vLLM, SGLang, and TensorRT-LLM handle each challenge. By the end, you will have a clear mental model for choosing the right approach for your use case.</p><p>Let’s start by understanding what KV cache actually is.</p><h4>Background: Prefill, Decode, and the KV Cache</h4><p>LLM inference happens in two phases:</p><ol><li><strong>Prefill phase</strong>: The model processes all input tokens in parallel and computes the Key and Value vectors for each token at every attention layer. This is compute-bound and benefits from GPU parallelism.</li><li><strong>Decode phase</strong>: The model generates tokens one at a time, autoregressively. Each new token attends to all previous Key-Value pairs. This phase is memory-bound because the GPU spends most of its time reading KV cache from HBM rather than doing math.</li></ol><p>The <strong>KV cache</strong> stores those Key and Value vectors so we do not recompute them at every decode step. Without it, generating each token would require re-running attention over the entire sequence, making inference impossibly slow.</p><p>Here is a quick sense of scale. For <strong>Llama-3–70B</strong> with a 8K context:</p><pre>KV cache per token = 2 (K+V) x 80 layers x 8 KV heads x 128 head_dim x 2 bytes (FP16)<br>                   = 2 x 80 x 8 x 128 x 2 = 327,680 bytes ≈ 320 KB per token<br><br>For 8K tokens: 320 KB x 8,192 = 2.56 GB per request<br>For 32 concurrent requests: 2.56 GB x 32 = 81.9 GB</pre><p>That is more than an entire A100 80GB GPU just for KV cache, leaving zero room for model weights. This is why KV cache management matters so much.</p><p><em>Autoregressive LLM inference: the prefill phase processes all input tokens, while the decode phase generates one token at a time using cached Key-Value pairs. (Source: </em><a href="https://www.modular.com/blog/the-five-eras-of-kvcache"><em>Modular Blog</em></a><em>)</em></p><h4>Era 0: Pre-GenAI (Before 2017)</h4><p>Before transformers took over, deep learning was dominated by stateless, feed-forward architectures like <strong>ResNet</strong>, <strong>YOLO</strong>, <strong>VGG</strong>, and <strong>Inception</strong>. These models processed each input independently with no persistent state across inference steps. The concept of a KV cache simply did not exist.</p><p>Inference frameworks like ONNX Runtime and TensorRT were designed for these stateless workloads: load model, run forward pass, return result. Simple.</p><p><strong>Key takeaway</strong>: If you are still serving traditional vision or tabular models, none of this complexity applies to you. Enjoy the simplicity.</p><h4>Era 1: Continuous KV Cache (2017)</h4><p>The original <a href="https://arxiv.org/pdf/1706.03762">Transformer paper (2017)</a> introduced the architecture that now dominates ML. With the self-attention mechanism came the need to cache Key and Value tensors across decode steps.</p><p>Early inference engines like HuggingFace Transformers implemented KV caches in the simplest way possible:</p><ul><li><strong>Pre-allocate</strong> a contiguous tensor with max_seq_len tokens for each request</li><li>Storage per request: 2 x num_layers x num_heads x head_dim x max_seq_len</li></ul><p><em>Era 1: Continuous KV cache allocates a fixed contiguous block per request. Shorter requests waste the unused portion. (Source: </em><a href="https://www.modular.com/blog/the-five-eras-of-kvcache"><em>Modular Blog</em></a><em>)</em></p><p><strong>Pros:</strong></p><ul><li>Simple to implement</li><li>Huge speedup over recomputing attention at every step</li></ul><p><strong>Cons:</strong></p><ul><li>Memory scales with max_seq_len x batch_size, not actual sequence lengths</li><li>Massive internal fragmentation (most requests are far shorter than max length)</li><li>Severely limits concurrent batch size</li><li>No memory sharing between requests</li></ul><p>In practice, profiling showed that <strong>only 20–38% of allocated KV cache memory actually stored useful token states</strong> in these systems. The rest was wasted on padding and fragmentation.</p><h4>Era 2: PagedAttention (2023)</h4><p>This is the era that changed everything. <a href="https://arxiv.org/abs/2309.06180">PagedAttention</a>, introduced by the vLLM team at UC Berkeley, borrowed a fundamental idea from operating systems: <strong>virtual memory with paging</strong>.</p><p>Instead of allocating one big contiguous block per request, PagedAttention divides the KV cache into fixed-size <strong>pages</strong> (blocks) that can be allocated dynamically as sequences grow. A block table maps logical pages to physical memory, just like an OS page table maps virtual addresses to physical RAM.</p><p><em>Era 2: PagedAttention allocates KV cache in fixed-size pages, dramatically reducing fragmentation. Pages can be non-contiguous in physical memory. (Source: </em><a href="https://www.modular.com/blog/the-five-eras-of-kvcache"><em>Modular Blog</em></a><em>)</em></p><p>The results were dramatic. The <a href="https://arxiv.org/abs/2309.06180">vLLM paper</a> showed:</p><ul><li><strong>2–4x throughput improvement</strong> over FasterTransformer and Orca</li><li><strong>Near-zero memory waste</strong> (under 4% fragmentation vs 60–80% in previous systems)</li><li>Enabled hundreds to thousands of concurrent requests</li></ul><p>PagedAttention also unlocked <a href="https://arxiv.org/abs/2312.07104"><strong>Prefix Caching</strong></a> via SGLang’s RadixAttention. If multiple requests share a common prefix (like a system prompt or a shared document), their KV cache pages can be shared instead of recomputed. This is a massive throughput multiplier for multi-turn chat and RAG workloads.</p><p><strong>Pros:</strong></p><ul><li>Near-zero memory waste</li><li>Enables much larger batch sizes</li><li>Supports prefix caching for KV reuse</li><li>Enables complex decoding (beam search, parallel sampling) with shared pages</li></ul><p><strong>Cons:</strong></p><ul><li>Adds complexity to attention kernels (non-contiguous memory access)</li><li>Block size selection requires tuning</li><li>Assumes homogeneous KV cache (same size at every layer)</li></ul><p>PagedAttention became the de-facto standard. Today, <strong>vLLM, SGLang, and TensorRT-LLM all use it</strong> as their foundation.</p><h4>Practical comparison: vLLM vs SGLang prefix caching</h4><p>Both frameworks support prefix caching, but their implementations differ:</p><ul><li><strong>vLLM</strong> uses hash-based prefix matching at the block level</li><li><strong>SGLang</strong> uses a <a href="https://arxiv.org/abs/2312.07104">RadixAttention tree</a> that maintains an LRU cache of KV blocks in a radix tree structure, enabling automatic reuse across multiple generation calls</li></ul><p>In my experience, SGLang’s approach tends to achieve higher cache hit rates for complex multi-call workloads (agents, tree-of-thought), while vLLM’s approach is simpler and works well for standard chat.</p><h4>Era 3: Heterogeneous KV Caches (2024)</h4><p>The world of ML became far more complex in 2024. New model architectures and optimization techniques require <strong>multiple different kinds of cached state</strong>, each with different shapes, lifetimes, and access patterns. The term “KV cache” is being stretched far beyond its original meaning.</p><p>Here is what changed:</p><p><strong>Speculative decoding</strong> uses a small draft model to propose multiple tokens, then a larger target model verifies them in one pass. This requires maintaining <strong>separate KV caches for draft and target models</strong>.</p><p><strong>Vision-Language Models (VLMs)</strong> like QwenVL and InternVL have vision encoders that generate large image embeddings. These embeddings can be cached and reused across requests, but they have different sizes from text KV cache.</p><p><strong>Quantized KV Cache</strong> uses FP8 or other low-precision formats to reduce KV storage. This requires managing additional scaling factors alongside the cache data.</p><p><strong>Sliding Window Attention (SWA)</strong> only attends to the last window_size tokens, so KV cache management needs to track which tokens are inside vs outside the window for eviction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z2pI5UkgkmAtyd-_.png" /><figcaption><em>Sliding Window Attention limits each token to attending only a fixed window of previous tokens, changing KV cache lifetime requirements. (Source: </em><a href="https://arxiv.org/abs/2503.18292"><em>Jenga Paper, arxiv:2503.18292</em></a><em>)</em></figcaption></figure><p><a href="https://arxiv.org/abs/2312.00752"><strong>Mamba / State Space Models</strong></a> replace attention with a recurrent state that updates a fixed-size vector for each new token. This state is fundamentally different from KV cache, it cannot be easily rolled back or shared at the token level.</p><p><strong>Hybrid Models</strong> combine multiple layer types within a single model:</p><ul><li>Sliding Window + Full Attention (Gemma 2/3, Ministral)</li><li>Mamba + Full Attention (Jamba, Bamba)</li><li>Local Chunked + Full Attention (Llama 4)</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JZbYkQbYhnvkjGkW.png" /><figcaption><em>Modern LLMs combine heterogeneous layer types, each requiring different KV cache management strategies. (Source: </em><a href="https://arxiv.org/abs/2503.18292"><em>Jenga Paper, arxiv:2503.18292</em></a><em>)</em></figcaption></figure><p>The <a href="https://arxiv.org/abs/2503.18292">Jenga paper</a> quantified this problem: for Llama 3.2 11B Vision, the memory waste from treating all layers uniformly reaches <strong>79.6%</strong>. For Gemma-2, it is up to 25%. For Ministral, 56.25%.</p><p><strong>Challenges with heterogeneous caches:</strong></p><ul><li>Memory fragmentation from multiple disjoint cache managers</li><li>Hard to predict memory allocation at server startup</li><li>Prefix caching implementations are separate per cache type, reducing hit rates</li><li>Feature composition becomes extremely complex</li></ul><p>In practice, frameworks like vLLM ended up with separate managers: a normal KV cache, a Vision Encoding Cache, a Mamba Cache, etc. This works but is fragile and hard to extend.</p><h4>Era 4: Distributed KV Cache (2025+)</h4><p>As models grow larger and workloads increase, a single GPU or even a single node is no longer sufficient. KV cache management is now becoming a <strong>multi-node, datacenter-scale problem</strong>.</p><h4>Disaggregated Inference</h4><p><a href="https://arxiv.org/abs/2401.09670">DistServe</a> proposed separating the prefill and decode phases onto different GPU instances. The key insight: prefill is compute-bound while decode is memory-bound, so they benefit from different hardware configurations and parallelism strategies.</p><p>DistServe showed <strong>4.48x more requests served</strong> (or 10.2x tighter SLO) compared to colocated systems. The critical challenge is efficiently transferring KV cache from prefill nodes to decode nodes.</p><p>This pattern has evolved further. <a href="https://blog.vllm.ai/2025/12/15/vllm-epd.html">vLLM’s Encoder Disaggregation</a> separates the vision encoder into its own scalable service for multimodal models, achieving <strong>2–2.5x better goodput</strong> by eliminating encoder-decode interference.</p><h4>KV Cache-aware Load Balancing</h4><p>NVIDIA Dynamo introduced <a href="https://docs.nvidia.com/dynamo/archive/0.4.0/architecture/kv_cache_routing.html">KV cache-aware routing</a>, where the request router prioritizes instances that already hold the relevant KV cache. This maximizes prefix cache hits across a cluster, requiring a cluster-wide view of cache state on every instance.</p><h4>Hierarchical KV Cache</h4><p><a href="https://arxiv.org/abs/2407.00079">Mooncake</a> from Moonshot AI implements a KV cache-centric disaggregated architecture where cold KV pages can be spilled from GPU HBM to CPU DRAM or SSD. This extends effective cache capacity while keeping hot pages on GPU. The latency of loading/storing KV for one layer from a lower tier can be hidden by overlapping it with GPU execution for the prior layer.</p><p>Mooncake achieved up to <strong>525% throughput increase</strong> in long-context scenarios while meeting SLOs. Under real workloads (serving Kimi), it handles <strong>75% more requests</strong>.</p><p><strong>The distributed era is hard:</strong></p><ul><li>Many optimizations (speculative decoding, VLMs) are still incompatible with distributed inference</li><li>Requires expert knowledge and patience to deploy</li><li>Inter-node networking (InfiniBand, RoCE) is challenging; libraries like <a href="https://github.com/ai-dynamo/nixl">NIXL</a> are nascent</li><li>Failover, stragglers, hardware defects, and auto-scaling add real-world complexity</li></ul><p>New Kubernetes-native solutions like <strong>NVIDIA Dynamo</strong>, <strong>vLLM Production Stack</strong>, <strong>llm-d</strong>, and <strong>AIBrix</strong> are emerging to tame this complexity, but we are still early.</p><h4>Era 5: Unified Hybrid KV Caches (2025+)</h4><p>The current frontier is building <strong>unified memory systems</strong> where heterogeneous KV types share a common memory pool rather than isolated allocators. The overarching theme is composability: every optimization should work with every other optimization.</p><h4>Jenga: Huge Pages + LCM Sizing</h4><p><a href="https://arxiv.org/abs/2503.18292">Jenga</a> introduces a two-level memory allocator. The key idea: use the <strong>Least Common Multiple (LCM)</strong> of different embedding sizes as the “huge page” size, so different KV shapes can co-exist efficiently within the same memory pool.</p><p>For example, if image token KV is 256 bytes and text token KV is 384 bytes, Jenga uses LCM(256, 384) = 768 bytes as the large page size. Each large page is subdivided into small pages matching the specific layer type.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/0*hR_QPhyzLpfXD8BK.png" /><figcaption><em>Jenga’s two-level memory allocation: large LCM-sized pages are subdivided into type-specific small pages, enabling unified memory management across heterogeneous KV caches. (Source: </em><a href="https://arxiv.org/abs/2503.18292"><em>Jenga Paper</em></a><em>)</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/0*j910tize3HeWGmZB.png" /><figcaption><em>Jenga’s unified prefix caching considers all KV cache types simultaneously, with balanced and aligned eviction policies. (Source: </em><a href="https://arxiv.org/abs/2503.18292"><em>Jenga Paper</em></a><em>)</em></figcaption></figure><p>Jenga achieves up to <strong>79.6% better GPU memory utilization</strong> and <strong>4.92x higher throughput</strong> (1.80x on average) compared to vanilla vLLM.</p><h4>SGLang: CUDA Virtual Memory</h4><p><a href="https://pytorch.org/blog/hybrid-models-meet-sglang-more-than-full-attention/">SGLang’s approach</a> uses <strong>CUDA Virtual Memory APIs</strong> to dynamically remap device memory. This enables virtually contiguous but physically scattered KV pages, with an elastic memory pool that dynamically adjusts allocation ratios between different pool types (e.g., Mamba pool vs KV cache pool) at runtime.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3CeA-_6irZ8nQQb_.png" /><figcaption><em>SGLang uses CUDA Virtual Memory APIs to create elastic, resizable memory pools that dynamically adjust to workload demands. (Source: </em><a href="https://pytorch.org/blog/hybrid-models-meet-sglang-more-than-full-attention/"><em>PyTorch Blog</em></a><em>)</em></figcaption></figure><p>SGLang’s <a href="https://github.com/sgl-project/sglang/issues/12780">2026 Q1 roadmap</a> emphasizes feature composability as a critical goal: running a VLM with speculative decoding across multiple nodes in a disaggregated setup. This requires long-term re-architecting of core engine components.</p><h4>Comparison Table: Eras at a Glance</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fCzHMKVcCUgX-CAXP8BcYA.png" /><figcaption>Comparison Table for different KV Caching Eras</figcaption></figure><h4>Which Era Matters Most for Your Use Case?</h4><p>Based on my experience deploying models in production:</p><p><strong>Standard text LLM serving (chatbot, completion)</strong>: Era 2 (PagedAttention) is your bread and butter. Pick vLLM or SGLang and you are in good shape. Enable prefix caching if you have shared system prompts.</p><p><strong>Multimodal models (VLMs)</strong>: You are in Era 3 territory. Pay attention to how your framework handles vision embeddings. vLLM’s encoder disaggregation (Era 4) is worth evaluating if image-heavy workloads dominate.</p><p><strong>Hybrid architectures (Gemma 3, Jamba, Llama 4)</strong>: Era 5 is directly relevant. SGLang’s CUDA virtual memory approach and Jenga’s LCM allocator are designed for this. Watch these closely.</p><p><strong>High-throughput production at scale</strong>: Era 4 is where you need to be. Disaggregated prefill/decode with KV-aware routing can dramatically improve your cost efficiency. NVIDIA Dynamo and Mooncake are the reference architectures.</p><p><strong>Long-context workloads (100K+ tokens)</strong>: Hierarchical KV cache (Era 4) with GPU-to-CPU spillover is essential. Without it, you simply run out of GPU memory.</p><h4>Key Takeaways</h4><ol><li><strong>KV cache is the bottleneck</strong>, not model weights. For Llama-3–70B, KV cache for 32 concurrent 8K-token requests exceeds 80GB, more than an entire A100.</li><li><strong>PagedAttention was the biggest single leap.</strong> Going from 20–38% memory utilization to 96%+ enabled the modern LLM serving era. Every major framework uses it.</li><li><strong>Model heterogeneity is the current challenge.</strong> Hybrid models (SWA + full attention + Mamba) break the assumption that all layers have the same KV shape. This is where active research is happening.</li><li><strong>Distributed KV management is hard but necessary.</strong> Disaggregated inference, KV-aware routing, and hierarchical caching are production requirements at scale, not nice-to-haves.</li><li><strong>Composability is the next frontier.</strong> The ability to run speculative decoding on a hybrid VLM across disaggregated nodes with unified prefix caching is the holy grail. We are not there yet, but SGLang and vLLM/Jenga are racing toward it.</li><li><strong>For practitioners</strong>: Start with vLLM or SGLang with PagedAttention enabled. Enable prefix caching for multi-turn workloads. Evaluate disaggregated inference when you need to scale. Watch the Era 5 developments for hybrid model support.</li></ol><p>The evolution of KV cache management mirrors the evolution of operating system memory management, from simple contiguous allocation to virtual memory, paging, and distributed shared memory. The difference is that this evolution happened in just 8 years instead of 40, driven by the explosive growth of LLM workloads. If you are building LLM infrastructure, understanding these eras is not optional. It is the foundation everything else is built on.</p><p><em>This post is inspired by Brian Zhang’s excellent </em><a href="https://www.modular.com/blog/the-five-eras-of-kvcache"><em>Five Eras of KVCache</em></a><em> article, enriched with additional context from the referenced papers and practical deployment experience.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df51cb8ce26f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understand your GPU for GenAI and LLMs — Part 1 (Difference between GPUs and CPUs)]]></title>
            <link>https://luv-bansal.medium.com/understand-your-gpu-for-genai-and-llms-part-1-difference-between-gpus-and-cpus-af72b69672b6?source=rss-234f1d6be728------2</link>
            <guid isPermaLink="false">https://medium.com/p/af72b69672b6</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[hardware-accelerator]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[gpu]]></category>
            <dc:creator><![CDATA[Luv Bansal]]></dc:creator>
            <pubDate>Tue, 17 Jun 2025 05:00:46 GMT</pubDate>
            <atom:updated>2025-06-17T05:00:46.351Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lPjecWXFu4T1D3Oo6AX5vA.jpeg" /><figcaption>Image created by author using Flux (Pro) Image Generation Model</figcaption></figure><h3>Understand your GPU for GenAI and LLMs — Part 1 (Difference between GPUs and CPUs)</h3><h4>The Role of GPUs and CPUs in GenAI &amp; LLM Workloads</h4><p>I’m starting a new series of articles to explore GPUs in detail— from their architecture to how they handle an incredible number of operations. We’ll start from all the way from basics and step by step take it to the more advance level. I believe understanding GPUs is essential if you really want to take their full advantage for Generative AI (GenAI) and Large Language Models (LLMs) Application.</p><p>In the first article of this series, we’ll learn about GPU Basics and how it’s different from CPU and what’s really makes it so powerful, if you are curious about these questions too then you’re really going to enjoy reading further.</p><h3>Difference between GPUs and CPUs</h3><p>Let’s start by understanding the fundamental differences between GPUs and CPUs.</p><p>The Graphics Processing Unit (GPU), housed inside a graphics card, typically contains over 10,000 cores. In contrast, the Central Processing Unit (CPU), mounted on the motherboard, has an integrated chip with only about 4 to 64 cores (modern CPUs may go up to 128 cores in some cases). At first glance, it might seem like the GPU is far more powerful than the CPU. After all, 10,000 cores versus 64 sounds like an easy win, right? However, it’s more complicated than that.</p><p>CPUs are highly versatile and can handle a wide range of programs and instructions. On the other hand, GPUs are less flexible, optimized for executing simple, repetitive instructions like basic arithmetic. They excel at handling specific workloads but are not capable of running operating systems or interfacing directly with input devices and networks.</p><h4>CPU vs. GPU Breakdown</h4><p>Every computer has a CPU, which typically consists of a few powerful cores and is connected to RAM (Random Access Memory). The CPU communicates with RAM to perform operations, a process we are all familiar with.</p><p>GPUs, however, differ from CPUs in architecture and memory design:</p><ul><li><strong>Cores:</strong> GPUs have many simpler cores compared to the few powerful cores in a CPU.</li><li><strong>Memory:</strong> GPUs use <strong>High Bandwidth Memory (HBM)</strong>, which is faster than traditional DRAM but limited in capacity</li></ul><h4>CPU</h4><ul><li><strong>Cores:</strong> Modern CPUs have 4–64 cores, designed for sequential task execution. Each core is highly optimized for a wide range of instructions.</li><li><strong>Clock Speed:</strong> Typically operates at a high clock speed (3–5 GHz), enabling quick execution of individual tasks.</li><li><strong>Cache Memory:</strong> CPUs have larger caches (L1, L2, L3) for low-latency access to frequently used data.</li><li><strong>Instruction Set:</strong> Optimized for complex instructions with advanced branch prediction and out-of-order execution.</li></ul><h4>GPU</h4><ul><li><strong>Cores:</strong> A GPU contains thousands of smaller, simpler cores (e.g., CUDA cores for NVIDIA, Stream processors for AMD) designed for massive parallelism.</li><li><strong>Clock Speed:</strong> Lower clock speeds (1–2 GHz) compared to CPUs.</li><li><strong>Cache Memory:</strong> Smaller caches compared to CPUs but highly optimized for graphics workloads.</li><li><strong>Instruction Set:</strong> Specialized for floating-point arithmetic and parallel operations.</li></ul><h3>Final Breakdown</h3><h4>The CPU: A Latency-Optimized Machine</h4><p>CPUs are designed to minimize latency for a single thread of execution. They have <strong>fewer threads</strong> and focus on reducing the time it takes to switch between tasks. This makes CPUs ideal for applications where a single thread needs to run as fast as possible.</p><h4>The GPU: A Throughput-Optimized Machine</h4><p>GPUs, on the other hand, are designed for <strong>throughput</strong>. They have <strong>thousands of threads</strong> and are optimized for running many tasks simultaneously. This allows GPUs to hide latency by keeping the computational units busy with other work while waiting for memory access. GPUs are <strong>oversubscribed</strong>, meaning they have more threads than they can run at any given time, ensuring that there is always work to do.</p><h4>The Key Difference: Threads vs. Latency</h4><p>The fundamental difference between CPUs and GPUs lies in how they handle latency. CPUs focus on <strong>reducing latency</strong> for a single thread, while GPUs focus on <strong>adding more threads</strong> to hide latency. This makes GPUs ideal for applications that can be parallelized across many threads, such as graphics rendering, Data processing and machine learning.</p><h4>Which is Faster?</h4><p>So, which is faster: the CPU or the GPU? It depends on the type of task.</p><ul><li><strong>For Parallel Tasks</strong>: If you’re working on tasks requiring simultaneous processing of large datasets — such as training a large language model (LLM) — a GPU is much faster. Its ability to execute thousands of operations simultaneously makes it ideal for such workloads.</li><li><strong>For Serial Tasks</strong>: When working with smaller datasets or tasks requiring low-latency processing, a CPU is faster due to its higher clock speeds, sophisticated caching mechanisms, and advanced architectures.</li><li><strong>For Versatility</strong>: If your workload includes running an operating system, managing network connections, or handling a variety of applications and hardware, a CPU is indispensable.</li></ul><h4>Conclusions</h4><ul><li>CPUs excel at <strong>serial tasks</strong> due to their high clock speed and low latency.</li><li>GPUs shine at <strong>parallel tasks</strong> thanks to their massive core count and high memory bandwidth.</li><li>GPUs and CPUs are intended for fundamentally different types of workloads. CPUs are typically designed for multitasking and fast serial processing, while GPUs are designed to produce high computational throughput using their massively parallel architectures.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af72b69672b6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[torch.compile(): How It Makes PyTorch Models So Fast?]]></title>
            <link>https://aiadvances.org/torch-compile-how-it-makes-pytorch-models-so-fast-d8362488911f?source=rss-234f1d6be728------2</link>
            <guid isPermaLink="false">https://medium.com/p/d8362488911f</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[torch]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <dc:creator><![CDATA[Luv Bansal]]></dc:creator>
            <pubDate>Wed, 15 Jan 2025 13:19:01 GMT</pubDate>
            <atom:updated>2025-01-15T13:19:01.701Z</atom:updated>
            <content:encoded><![CDATA[<h4>torch.compile(): A Game-Changer for PyTorch Model Performance</h4><p>torch.compile() is an incredible innovation from the PyTorch team. Think of it as a compiler for neural networks—similar to how GCC works for C and C++ code, but specifically designed for optimizing deep learning models. This breakthrough makes running PyTorch models faster and more efficient than ever before.</p><p>The best part? Using torch.compile() is remarkably simple. You can apply it to any PyTorch model or Hugging Face model with just a single line of code:</p><pre>import torch<br>model =toch.compile(model)</pre><p>While this one-liner introduces a slight overhead during the compilation process, the benefits far outweigh the cost. The compiled model runs significantly faster, making it ideal for high-performance inference and training scenarios.</p><h3>How Does torch.compile() Speed Up Your Model?</h3><p>We now know that running a model with torch.compile() results in significant speedups. This improvement primarily comes from two key factors:</p><ul><li><strong>Reducing Python Overhead</strong></li><li><strong>Optimizing GPU Read/Writes</strong></li></ul><p>Let’s dive into each of these in detail.</p><h3>Reducing Python Overhead</h3><p>Without torch.compile(), PyTorch operates in eager mode, where the Python interpreter executes code line by line. In this mode, the interpreter doesn’t know what operations will be executed in subsequent lines. This lack of foresight creates inefficiencies, as the code is executed layer by layer, with no opportunity for global optimization.</p><p>With torch.compile(), however, the entire model is analyzed at once. The compiler inspects all the operations in the model, determining what will happen and when. This global view allows it to optimize the execution process. Unlike eager mode, where execution happens step-by-step with limited foresight, torch.compile() sees the entire model as a whole and eliminates unnecessary steps and that’s how it optimises that process .</p><p>The first thing torch.compile() does is completely remove the reliance on the Python interpreter during execution. Instead, it compiles the entire neural network into a single, optimized object. By doing so, it avoids the inefficiencies of interpreting Python code and executes the operations in a highly efficient, streamlined manner.</p><p>This optimization ensures faster execution by reducing the computational overhead associated with Python’s dynamic interpretation, allowing the model to run much more efficiently.</p><h3>Optimizing GPU read/writes</h3><p>To understand how torch.compile() optimizes GPU read/writes, we first need to delve into how GPUs operate and handle data for computations.</p><h4>How GPUs Work</h4><p>Let’s break this down with a simple explanation:</p><p>Every computer has a CPU, which typically consists of a few powerful cores and is connected to RAM (Random Access Memory). The CPU communicates with RAM to perform operations, a process we are all familiar with.</p><p>GPUs, however, differ from CPUs in architecture and memory design:</p><ul><li><strong>Cores:</strong> GPUs have many simpler cores compared to the few powerful cores in a CPU.</li><li><strong>Memory:</strong> GPUs use <strong>High Bandwidth Memory (HBM)</strong>, which is faster than traditional DRAM but limited in capacity.</li></ul><h4>Here’s how computations generally occur on a GPU:</h4><ol><li>All the model parameters, activations, and other variables are stored in the GPU’s HBM memory.</li><li>For any operation to be performed, the required input data must travel from HBM memory to the GPU cores.</li><li>Inside the GPU, the data moves through caches and registers on the GPU chip for computations.</li><li>After the operation is completed, the result is written back to HBM memory.</li></ol><p>This constant data transfer between the HBM and GPU cores is time-consuming and significantly impacts performance. Even though operations within the GPU cores are extremely fast, the memory bandwidth between the HBM and GPU cores often becomes a bottleneck.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y5Yvmpv38e64MK7qH_yFBg.png" /><figcaption>CPU and GPU Architecture (Image sourced from <a href="https://developer.nvidia.com/blog/improving-gpu-memory-oversubscription-performance/">https://developer.nvidia.com/blog/improving-gpu-memory-oversubscription-performance/</a>)</figcaption></figure><h4>The Challenge: GPU Memory Bandwidth</h4><p>Each GPU has a fixed <strong>Memory Bandwidth</strong>, which defines the rate at which data can move between its memory (HBM) and its compute cores. This becomes a critical bottleneck for deep learning tasks, which often require frequent memory access for:</p><ol><li><strong>Loading inputs into the GPU cores</strong></li><li><strong>Performing calculations on the GPU</strong></li><li><strong>Writing the outputs back to HBM memory</strong></li></ol><p>Without torch.compile(), PyTorch processes operations sequentially in eager mode, and this constant back-and-forth creates inefficiencies, significantly increasing the time it takes to complete operations.</p><h3>How torch.compile() Solves This</h3><p>torch.compile() optimizes GPU read/write operations in two key ways:</p><ol><li><strong>Reducing Memory Transfers:</strong> The compiler analyzes the entire computation graph upfront and minimizes unnecessary data transfers between HBM and GPU cores.</li><li><strong>Kernel Fusion:</strong> Multiple operations are fused into a single GPU kernel, reducing the number of trips between memory and compute cores.</li></ol><h4>Kernel Fusion: A Key Optimization</h4><p>Kernel fusion combines multiple operations into a single kernel. Instead of performing multiple back-and-forth memory operations, fused kernels execute all operations in one go. This reduces overhead and improves execution efficiency.</p><h4>Why Is This Important?</h4><p>To better understand the impact, consider the GPU memory hierarchy:</p><ul><li><strong>HBM:</strong> Fast but limited in capacity (~40–80GB).</li><li><strong>On-Chip Memory:</strong> Extremely fast but small (~tens of MBs).</li></ul><p>Frequent data movement between these memory tiers introduces inefficiencies. torch.compile() addresses this by streamlining data transfers and reducing memory bandwidth bottlenecks.</p><h3>Conclusion</h3><p>torch.compile() is a game-changing feature that allows PyTorch models to run faster by:</p><ul><li><strong>Reducing Python Overhead:</strong> Compiling the entire model into an optimized object eliminates inefficiencies in dynamic interpretation.</li><li><strong>Optimizing GPU Read/Writes:</strong> Techniques like kernel fusion minimize memory transfers and maximize GPU performance.</li></ul><p>With just one line of code, you can achieve impressive speedups and unlock the full potential of your hardware for deep learning workloads. Whether you’re training massive models or running inference at scale, torch.compile() is a powerful tool you should add to your deep learning arsenal.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d8362488911f" width="1" height="1" alt=""><hr><p><a href="https://aiadvances.org/torch-compile-how-it-makes-pytorch-models-so-fast-d8362488911f">torch.compile(): How It Makes PyTorch Models So Fast?</a> was originally published in <a href="https://aiadvances.org/">AI Advances</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Build And Train GPT From Scratch]]></title>
            <link>https://pub.towardsai.net/build-and-train-gpt-from-scratch-8ae985a8fc25?source=rss-234f1d6be728------2</link>
            <guid isPermaLink="false">https://medium.com/p/8ae985a8fc25</guid>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[openai]]></category>
            <category><![CDATA[gpt]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Luv Bansal]]></dc:creator>
            <pubDate>Tue, 26 Nov 2024 04:37:14 GMT</pubDate>
            <atom:updated>2024-12-12T12:26:17.604Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GkAVwrnbw8MmkmNSMgO7qA.jpeg" /><figcaption>Image created by author using Dalle-3 via Bing Chat</figcaption></figure><h4>From Zero to GPT: Understanding the Core Components of GPT</h4><p>In this blog, we’ll build a GPT model from scratch using only Python and PyTorch, without relying on any external libraries. We’ll create and train our own GPT model to match the performance of the original GPT-2 and even fine-tune it on custom data.</p><p>By the end of this blog, you will have designed and built a smaller version of a GPT architecture using Python and PyTorch. You will also have trained it entirely from scratch on your own dataset, following the same principles used to train OpenAI’s GPT-2 and GPT-3 language models.</p><h4>What You can expect by the End of This Blog</h4><ul><li><strong>Understand GPT Internals:</strong> You’ll learn what happens under the hood of GPT and the key mechanisms that make it work.</li><li><strong>Build a GPT Model Step-by-Step: </strong>You’ll define and construct the GPT architecture piece by piece and train it on your own dataset.</li><li><strong>Train a Custom GPT Model:</strong> By the end, you’ll have a smaller GPT model built using only Python and PyTorch that you can train and fine-tune on your own data.</li><li><strong>Simplify Training and Fine-Tuning:</strong> You’ll discover the simplest way to train or fine-tune a GPT-like transformer model without using external libraries</li></ul><p>Before starting, it would be good to introduce GPT and how it’s been built</p><h4><strong>Introducing GPT</strong></h4><p>At its core, GPT is a language model designed to predict the next token in a sequence based on the given context. In simpler terms, when you provide a question or some context, GPT generates words sequentially from left to right to complete the sequence.</p><p>If you’ve used ChatGPT, you might have noticed that it can generate multiple responses for a single prompt. This is because GPT is a probabilistic system, meaning it can produce different answers to the same prompt depending on its internal state and sampling strategy.</p><p>It has been build using the technique introduced in the paper <a href="https://arxiv.org/abs/1706.03762">Attention is all you Need</a>, If you’re unfamiliar with the workings of transformers and attention mechanisms, which form the backbone of GPT models, I highly recommend checking out my <a href="https://luv-bansal.medium.com/transformer-attention-is-all-you-need-easily-explained-with-illustrations-d38fdb06d7db">blog</a> that simplifies these concepts and explains how attention works.</p><p><a href="https://luv-bansal.medium.com/transformer-attention-is-all-you-need-easily-explained-with-illustrations-d38fdb06d7db">Transformer — Attention Is All You Need Easily Explained With Illustrations</a></p><p>GPT is short form of Generative Pretrained Transformer, so Transformer is a Neural Network which actually does all the heavy lifting under the hood.</p><h3><strong>Let’s start building the GPT Language Model</strong></h3><p>We will start with Downloading and inspecting the Dataset.</p><h4><strong>Downloading the Dataset</strong></h4><p>To train our GPT model, we need a dataset. And we will use the Shakespeare dataset to train our GPT model</p><pre>!wget https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt</pre><h4><strong>Load and inspect dataset</strong></h4><p>Let’s load and inspect the dataset to get an understanding of its structure.</p><pre># read it in to inspect it<br>with open(&#39;input.txt&#39;, &#39;r&#39;, encoding=&#39;utf-8&#39;) as f:<br>    text = f.read()<br><br># let&#39;s look at the first 1000 characters<br>print(text[:1000])</pre><p>For simplicity and to accommodate limited compute resources, we’ll build a character-level language model. This means we’ll tokenize and train the model based on characters instead of word tokens. Training a token-level language model would require significantly more computational power.</p><h4><strong>Identifying unique characters in the dataset</strong></h4><pre>chars = sorted(list(set(text)))<br>vocab_size = len(chars)<br>print(&#39;&#39;.join(chars))<br>print(&#39;Total number of unique characters in the dataset:&#39;, vocab_size)</pre><blockquote>!$&amp;’,-.3:;?ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz Total number of unique characters in the dataset: 65</blockquote><p>As shown above, the dataset contains 65 unique characters, which include all alphabetical characters, spaces, and some special symbols.</p><h4><strong>Tokenize the Input text based on characters</strong></h4><p>We’ll tokenize the text by mapping each character to an integer</p><pre># create a mapping from characters to integers<br>stoi = { ch:i for i,ch in enumerate(chars) }<br>itos = { i:ch for i,ch in enumerate(chars) }<br>encode = lambda s: [stoi[c] for c in s] # encoder: take a string, output a list of integers<br>decode = lambda l: &#39;&#39;.join([itos[i] for i in l]) # decoder: take a list of integers, output a string<br><br>print(encode(&quot;hii there&quot;))<br>print(decode(encode(&quot;hii there&quot;)))</pre><p>This simple character-level tokenization method works for our purposes, but there are more advanced techniques like SentencePiece by Google and tiktoken by OpenAI, which tokenize based on words and subwords.</p><p>We’ll look at all these tokenization techniques latter on but for simplicity and understanding the GPT model in the better way, we will stick to our character level tokenization.</p><h4><strong>Let’s tokentize the entire dataset</strong></h4><p>Next, we encode the entire text dataset into integers and store it in a PyTorch tensor.</p><pre># let&#39;s now encode the entire text dataset and store it into a torch.Tensor<br>import torch <br>data = torch.tensor(encode(text), dtype=torch.long)</pre><h4><strong>Splitting data into train and validation sets</strong></h4><p>Splitting data into 90% training split and 10% validation split</p><pre># Let&#39;s now split up the data into train and validation sets<br>n = int(0.9*len(data)) # first 90% will be train, rest val<br>train_data = data[:n]<br>val_data = data[n:]</pre><h4><strong>Preparing the Dataset for Training</strong></h4><p>It’s important to note that we don’t feed the entire training dataset into the transformer model at once. Doing so would be highly inefficient and computationally expensive. Instead, we work with smaller chunks of the dataset.</p><p>So when we actually train Transformers on a lot of these datasets we work with chunks on the dataset therefore before training the Transformer we sample random little chunks out of the training set and train on just chunks at a time. These chunks are limited by a <strong><em>maximum length</em></strong>, referred to as the <em>context length</em> or <em>block size</em>. This approach significantly reduces memory requirements while maintaining efficiency.</p><pre>block_size = 8 # the context window length<br><br>x = train_data[:block_size]   # input sequence to the transformer <br>y = train_data[1:block_size+1]  # output sequence to the transformer (shifted one position) and it&#39;s the target for each position in the input<br>for t in range(block_size):<br>    context = x[:t+1]<br>    target = y[t]<br>    print(f&quot;when input is {context} the target: {target}&quot;)</pre><p>We train on all the eight(block_size) examples here with context size between 1 and context of block_size because we want the transformer model to seeing context all the way from as little as 1 to all the way to block size. And this is really going to useful later during infernece because we can generate text of any length by just giving it a context of any length.</p><blockquote><strong>Note:</strong> Model can take maximum of block_size context length as input, so during inference when generating text more than block_size length, it will start truncating the context from the beginning. i.e. it take last block_size tokens as input and generate next token and append it to the input and generate next token and so on.</blockquote><h4><strong>Batch Processing</strong></h4><p>We will add another dimension to out dataset which is batch size because GPUs are very good at processing data parallely therefor all the inputs in the batch will be process parallelly, that will make the training more efficient.</p><pre>torch.manual_seed(1337)<br>batch_size = 4 # how many independent sequences will we process in parallel?<br>block_size = 8 # what is the maximum context length for predictions?<br><br>def get_batch(split, batch_size, block_size):<br>    # generate a small batch of data of inputs x and targets y<br>    data = train_data if split == &#39;train&#39; else val_data<br>    ix = torch.randint(len(data) - block_size, (batch_size,))<br>    x = torch.stack([data[i:i+block_size] for i in ix])<br>    y = torch.stack([data[i+1:i+block_size+1] for i in ix])<br>    return x, y<br><br>xb, yb = get_batch(&#39;train&#39;, batch_size, block_size)</pre><h3><strong>Self-attention: The basic building block of the Transformer</strong></h3><p>Now that we have the basic setup for our model, it’s time to dive into one of the key components of the Transformer architecture — <strong>self-attention</strong>.</p><h4><strong>Intro: Self-Attention</strong></h4><p>Self-attention is a mechanism that relates each word in a sentence to every other word, this allows each word to understand its context within the sentence. This process generates final `out_dim` dimention output vector for each word, which captures its relationship with every other word in the sentence.</p><h4>Key Components: Query, Key, and Value</h4><p>To compute self-attention, we need three vectors: query, key, and value. These vectors are derived from the word embeddings of each word in the sentence. They are typically smaller in size than the original embeddings. In our case, these vectors are 64-dimensional.</p><blockquote><strong>Note:</strong> These new vectors are smaller in dimension than the embedding vector.</blockquote><h4><strong>Query, Key and Values can be interpreted as follows:</strong></h4><ul><li><strong>query:</strong> what I are looking for (what I are interested in)</li><li><strong>key:</strong> what I have</li><li><strong>value:</strong> If the key matches what I are looking for, this is the information I want to retrieve</li></ul><p>These <em>query</em>, <em>key</em>, and <em>values</em> vectors helps to captures the relationship for each word in a sentence to every other word.</p><pre>import torch.nn as nn<br>import torch.nn.functional as F<br><br>torch.manual_seed(1337)<br>batch_size = B = 4 # how many independent sequences will we process in parallel?<br>block_size = T = 8 # what is the maximum context length for predictions?<br>embed_dim = C = 32 # embedding dimension of the input tokens<br><br>x  = torch.randn(B, T, C) # input tokens<br><br># let&#39;s create a single head perform self-attention<br>head_size = H = 16 # how big is the head (size of the key, query, value projections)<br>key = nn.Linear(C, head_size) # linear layer for key<br>query = nn.Linear(C, head_size) # linear layer for query<br>value = nn.Linear(C, head_size) # linear layer for value<br><br># apply the linear projections to the input tokens x to get the keys, queries and values for all tokens in the batch<br>k = key(x)   # shape: (B, T, H) (batch size, time steps, head size)<br>q = query(x)  # shape: (B, T, H) <br>v = value(x)  # shape: (B, T, H)</pre><h4><strong>Calculating scores in Self-Attention</strong></h4><p>To calculate self-attention, we compute a score for each word in the sequence. This score is determined by the dot product of the query (q) for a word and the key (k) for every word in the sequence. This score tells us how much attention a word should pay to each token in the sequence, including itself.</p><blockquote>Scaling: We scale the dot products by dividing by the square root of the head size. This ensures that the variance of the attention scores remains consistent, preventing saturation during the softmax operation.</blockquote><pre># let&#39;s compute the raw self-attention scores (a simple dot product between the queries and the keys for all tokens in the batch)<br><br>scores_matrix = q @ k.transpose(-2, -1) # (B, T, H) @ (B, H, T) -&gt; (B, T, T) (batch size, time steps, time steps)<br><br># we also divide the scores by the square root of the head_size <br># This is because we are scaling the dot products by the head_size, so that the result is not too large and the softmax is not too peaked. <br>scores_matrix = scores_matrix / torch.sqrt(torch.tensor(H).float()) # shape: (B, T, T)</pre><h4><strong>Masking Scores: Attenting tokens that come before the current token</strong></h4><p>In language modeling, each token should only attend to the tokens before it in the sequence, not the ones that come after. This prevents the model from peeking at future tokens when predicting the current token.</p><p>This is a specific case of language modeling, where we want to mask out the upper half of the matrix, so that future tokens cannot communicate with the past tokens.</p><p>This is done by masking the upper half of the scores matrix, ensuring that the model cannot attend to future tokens. After masking, we apply the softmax function to the scores matrix.</p><blockquote>Also note that this is not a constraint for general case of self-attention mechanism there are many other usecase where we don’t want to mask the matrix like in case of translation task we want to allow the future tokens to communicate with the past tokens and other use cases like sentence classification etc.</blockquote><pre>mask = torch.tril(torch.ones(T, T))<br>mask_score_matrix = scores_matrix.masked_fill(mask == 0, float(&#39;-inf&#39;)) # masking out the upper half of the matrix #shape: (B, T, T)<br><br>print(f&quot;mask_score_matrix shape: {mask_score_matrix.shape}&quot;)<br><br>print(mask_score_matrix[0])</pre><h4><strong>Calculating attention weights using Softmax</strong></h4><p>Now we apply the softmax function to the scores matrix to get the attention weights, with the softmax applied on the last dimension (time dimension) of the scores matrix we will get the attention weights for each token in the sequence to all other tokens in the sequence.</p><pre>attention_weights = F.softmax(mask_score_matrix, dim=-1) # shape: (B, T, T)<br><br>print(f&quot;shape of the attention weights: {attention_weights.shape}&quot;)<br><br>print(f&quot;Sum of the attention weights for each token in the batch: {attention_weights.sum(dim=-1)}&quot;)</pre><h4><strong>Output of the self-attention mechanism in single head setting</strong></h4><p>Output of the self-attention mechanism is computed by a weighted sum of the values for all tokens in the batch, which are weighted by the attention weights.</p><p>This can effectively be computed by a matrix multiplication of the attention weights with the values.</p><pre># now we compute the output of the self-attention layer by multiplying the attention weights with the values<br>attention_output = attention_weights @ v # shape: (B, T, T) @ (B, T, H) -&gt; (B, T, H) (batch size, time steps, head size)<br><br>print(f&quot;shape of the attention output: {attention_output.shape}&quot;)</pre><h3><strong>Let’s put everything together in a single nn.Module</strong></h3><h4>Self-Attention Head</h4><pre>class SelfAttentionHead(nn.Module):<br>    &quot;&quot;&quot; one head of self-attention &quot;&quot;&quot;<br><br>    def __init__(self, embed_dim, block_size, head_size, dropout=0.2):<br>        super().__init__()<br>        self.key = nn.Linear(embed_dim, head_size, bias=False)<br>        self.query = nn.Linear(embed_dim, head_size, bias=False)<br>        self.value = nn.Linear(embed_dim, head_size, bias=False)<br>        self.register_buffer(&#39;tril&#39;, torch.tril(torch.ones(block_size, block_size)))<br><br>        self.dropout = nn.Dropout(dropout)<br><br>    def forward(self, x):<br>        # input of size (batch, time-step, channels)<br>        # output of size (batch, time-step, head size)<br>        B,T,C = x.shape<br>        k = self.key(x)   # (B,T,hs)<br>        q = self.query(x) # (B,T,hs)<br>        # compute attention scores (&quot;affinities&quot;)<br>        wei = q @ k.transpose(-2,-1) * k.shape[-1]**-0.5 # (B, T, hs) @ (B, hs, T) -&gt; (B, T, T)<br>        wei = wei.masked_fill(self.tril[:T, :T] == 0, float(&#39;-inf&#39;)) # (B, T, T)<br>        wei = F.softmax(wei, dim=-1) # (B, T, T)<br>        wei = self.dropout(wei)<br>        # perform the weighted aggregation of the values<br>        v = self.value(x) # (B,T,hs)<br>        out = wei @ v # (B, T, T) @ (B, T, hs) -&gt; (B, T, hs)<br>        return out</pre><h4><strong>Additional Informantion about Attentions</strong></h4><ul><li>Attention is a <strong>**communication mechanism**</strong>. Can be seen as nodes in a directed graph looking at each other and aggregating information with a weighted sum from all nodes that point to them, with data-dependent weights.</li><li>There is no notion of space. Attention simply acts over a set of vectors. This is why we need to positionally encode tokens.</li><li>Each example across batch dimension is of course processed completely independently and never “talk” to each other</li><li>“self-attention” just means that the keys and values are produced from the same source as queries. In “cross-attention”, the queries still get produced from x, but the keys and values come from some other, external source (e.g. an encoder module)</li></ul><h4><strong>Multi-Head Attention</strong></h4><p>Multi-head attention is just calculating multiple self-attention heads in parallel and then concatenating their results. And this help the model to learn different representation of the input data.</p><pre>class MultiHeadSelfAttention(nn.Module):<br>    &quot;&quot;&quot; a stack of self-attention layers &quot;&quot;&quot;<br><br>    def __init__(self, embed_dim, block_size, num_heads, head_size, dropout=0.2):<br>        super().__init__()<br>        self.heads = nn.ModuleList([SelfAttentionHead(embed_dim, block_size, head_size, dropout) for _ in range(num_heads)])<br>        self.projection = nn.Linear(num_heads*head_size, embed_dim)<br>        self.dropout = nn.Dropout(dropout)<br><br><br>    def forward(self, x):<br>        # input of size (batch, time-step, channels)<br>        # output of size (batch, time-step, channels)<br>        head_out = [head(x) for head in self.heads] # a list of tensors of shape (B, T, hs)<br>        out = torch.cat(head_out, dim=-1) # concatenate along the feature dimension<br>        out = self.projection(out) # project back to the original feature dimension<br>        out = self.dropout(out)<br>        return out</pre><h4><strong>Positional Encoding</strong></h4><p>Irrespective of RNNs where word tokens are input one at a time into the model, whereas in the attention model, all the words are input at the same time i.e. all the words taken as input parallelly into the encoder model. And the position and order of words are essential parts of any language since they define the grammar and thus the actual semantics of a sentence. So, we need to do something to maintain the order of sequence therefore, to provide the model with positional information we add positional encoding to the input embeddings.</p><p>Positional encoding is a way to inject some information about the relative or absolute position of the tokens in the sequence. The positional encoding vector is added to the embedding vector. Embeddings represent a token in a d-dimensional space where tokens with similar meaning will be closer to each other. But the embeddings do not encode the relative position of words in a sentence. So after adding positional encoding, words will be closer to each other based on the similarity of their meaning and their position in the sentence, in the d-dimensional space.</p><pre>import math<br><br># Position embeddings<br>class PositionalEncoding(nn.Module):<br>    def __init__(self, embed_dim, max_len=5000):<br>        super().__init__()<br>        # Compute the positional encodings once in log space.<br>        pe = torch.zeros(max_len, embed_dim)<br>        position = torch.arange(0, max_len).unsqueeze(1).float()<br>        div_term = torch.exp(torch.arange(0, embed_dim, 2).float() * -(math.log(10000.0) / embed_dim))<br>        pe[:, 0::2] = torch.sin(position * div_term)<br>        pe[:, 1::2] = torch.cos(position * div_term)<br>        pe = pe.unsqueeze(0)<br>        self.register_buffer(&#39;pe&#39;, pe)<br><br>    def forward(self, x):<br>        return x + self.pe[:, :x.size(1)]</pre><h4><strong>Feed forward network</strong></h4><p>This is a simple feed-forward network with a simple liner layer followed by non-linear activation functions that is applied to each positional token separately and identically.</p><p>This helps the model learn more complex functions than just the linear ones that self-attention provides.</p><pre># Feed forward network<br># hidden_dim is 4 times the embedding dimension as per the paper &quot;Attention is all you need&quot;<br><br>class FeedForward(nn.Module):<br>    def __init__(self, embed_dim, hidden_dim, dropout=0.2):<br>        super().__init__()<br>        self.net = nn.Sequential(<br>            nn.Linear(embed_dim, hidden_dim),<br>            nn.ReLU(),<br>            nn.Linear(hidden_dim, embed_dim),<br>            nn.Dropout(dropout)<br>        )<br><br>    def forward(self, x):<br>        return self.net(x)</pre><h3><strong>Putting it all together in a Transformer Block</strong></h3><h4><strong>Transformer Block</strong></h4><p>The transformer block is composed of a multi-head self-attention layer, followed by a feed-forward network.</p><pre># Putting it all together in a transformer block<br># The transformer block is composed of a multi-head self-attention layer, followed by a feed-forward network.<br><br>class TransformerBlock(nn.Module):<br>    def __init__(self, embed_dim, block_size, num_heads, head_size, ff_hidden_dim, dropout=0.2):<br>        super().__init__()<br>        self.attention = MultiHeadSelfAttention(embed_dim, block_size, num_heads, head_size, dropout)<br>        self.ffn = FeedForward(embed_dim, ff_hidden_dim)<br><br>    def forward(self, x):<br>        x = self.pos_emb(x)<br>        x = self.attention(x)<br>        x = self.ffn(x)<br>        return x</pre><h4><strong>Skip connections in Transformer Block</strong></h4><p>Adding skip connections around the multi-head self-attention and feed-forward network is crucial for training a very deep model.</p><p>Without skip connections, the gradients tend to either vanish (in very deep feed-forward networks) or explode (in very deep networks with attention). Therefore, skip connections are crucial for training very deep models.</p><p>We add a residual connection around the multi-head self-attention and feed-forward network, followed by layer normalization. We first look at the skip connection and then at the layer normalization.</p><pre># Skip connections<br><br>class TransformerBlock(nn.Module):<br>    def __init__(self, embed_dim, block_size, num_heads, head_size, ff_hidden_dim, dropout=0.2):<br>        super().__init__()<br>        self.attention = MultiHeadSelfAttention(embed_dim, block_size, num_heads, head_size, dropout=dropout)<br>        self.ffn = FeedForward(embed_dim, ff_hidden_dim)<br><br><br>    def forward(self, x):<br>        x = self.pos_emb(x)  # shape (B, T, C)<br>        # apply the self-attention mechanism<br>        att = self.attention(x) # shape (B, T, C)<br>        # add skip connection <br>        x = x + att  <br>        # apply the feed-forward network<br>        ffn = self.ffn(x)  # shape (B, T, C)<br>        # add skip connection<br>        x = x + ffn  # shape (B, T, C)<br>        return x</pre><p><strong>Transformer Block with Layer Normalization</strong></p><p>This helps stabilize the training of deep networks and speeds up convergence by normalizing the pre-activation outputs of each layer.</p><p>In the orignal paper, Layer normalization is applied to the output of the self-attention and the feed-forward network but now generally it is applied to the input of the self-attention and the feed-forward network and not the output.</p><pre>class TransformerBlock(nn.Module):<br>    def __init__(self, embed_dim, block_size, num_heads, head_size, ff_hidden_dim, dropout=0.2):<br>        super().__init__()<br>        self.attention = MultiHeadSelfAttention(embed_dim, block_size, num_heads, head_size, dropout=dropout)<br>        self.norm1 = nn.LayerNorm(embed_dim)<br>        self.ffn = FeedForward(embed_dim, ff_hidden_dim)<br>        self.norm2 = nn.LayerNorm(embed_dim)<br><br>    def forward(self, x):<br>        # apply the self-attention mechanism<br>        att = self.attention(x) # shape (B, T, C)<br>        # add skip connection and apply layer normalization<br>        x = x + self.norm1(att)  <br>        # apply the feed-forward network<br>        ffn = self.ffn(x)  # shape (B, T, C)<br>        # add skip connection and apply layer normalization<br>        x = x + self.norm2(ffn)<br>        return x</pre><h3><strong>GPT Model Architecture</strong></h3><p>Let’s now create a decoder based GPT transformer model composed of multiple transformer blocks</p><pre>class GPTLanguageModel(nn.Module):<br>    def __init__(self, vocab_size,embed_dim, block_size, num_heads, head_size, ff_hidden_dim, num_layers, dropout=0.2):<br>        super().__init__()<br>        self.emd = nn.Embedding(vocab_size, embed_dim)<br>        self.pos_emb = PositionalEncoding(embed_dim)<br>        self.layers = nn.ModuleList([<br>            TransformerBlock(embed_dim, block_size, num_heads, head_size, ff_hidden_dim, dropout=dropout)<br>            for _ in range(num_layers)<br>        ])<br>        self.norm = nn.LayerNorm(embed_dim)<br>        self.head = nn.Linear(embed_dim, vocab_size)<br><br>    def forward(self, x):<br>        x = self.emd(x)<br>        x = self.pos_emb(x)<br>        for layer in self.layers:<br>            x = layer(x)<br>        x = self.norm(x)<br>        x = self.head(x)<br>        return x</pre><h4><strong>Model Initialization</strong></h4><p>We will now intiatilze our model with different model settings and calculate number of training parameters at the end</p><pre># Model Initialization and Number of Parameters <br><br>torch.manual_seed(1337)<br><br># model hyperparameters<br>chars = sorted(list(set(text)))<br>vocab_size = len(chars)<br>embed_dim = 384<br>num_heads = 6<br>head_size = 64<br>ff_hidden_dim = 4*384<br>num_layers = 6<br>block_size = 256<br>dropout = 0.3<br><br>model = GPTLanguageModel(vocab_size, embed_dim, block_size, num_heads, head_size, ff_hidden_dim, num_layers, dropout=dropout)<br><br>device = torch.device(&quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;)<br><br>model = model.to(device)<br><br># Number of parameters in the model<br>num_params = sum(p.numel() for p in model.parameters())<br><br>print(f&quot;Number of parameters in the model: {num_params}&quot;)</pre><h3><strong>Let’s Train Our GPT Transformer Model</strong></h3><pre># Let&#39;s now train our transformer model<br># we will use the cross-entropy loss and the Adam optimizer to train the model.<br><br>device = torch.device(&#39;cuda&#39; if torch.cuda.is_available() else &#39;cpu&#39;)<br>print(f&quot;Training on device {device}.&quot;)<br><br># training parameters<br>lr = 3e-4<br>batch_size = 64<br>num_iterations = 3000<br>eval_iters = 200<br><br># intiatizing loss and optimizer<br>criterion = nn.CrossEntropyLoss()<br>optimizer = torch.optim.Adam(model.parameters(), lr=lr)<br><br>@torch.no_grad()<br>def estimate_loss(model, criterion, eval_iters, batch_size, block_size):<br>    model.eval()<br>    out = []<br>    losses = torch.zeros(2, eval_iters)<br><br>    @torch.no_grad()<br>    def evaluate(model, criterion, batch_size, block_size):<br>        model.eval()<br>        out = []<br>        with torch.no_grad():<br>            for split in [&#39;train&#39;, &#39;val&#39;]:<br>                x, y = get_batch(split, batch_size, block_size)<br>                x, y = x.to(device), y.to(device)<br>                y_pred = model(x)<br>                loss = criterion(y_pred.view(-1, vocab_size), y.view(-1))<br>                out.append(loss.item())<br>        return out<br><br>    for i in range(eval_iters):<br>        losses[0, i], losses[1, i] = evaluate(model, criterion, batch_size, block_size)<br>    return losses.mean(dim=1).tolist()<br><br><br><br># training function<br>def train(model, criterion, optimizer, num_iterations, batch_size, block_size):<br>    model.train()<br>    losses = []<br><br>    output_losses = []<br>    for i in range(num_iterations):<br>        x, y = get_batch(&#39;train&#39;, batch_size, block_size)<br>        x, y = x.to(device), y.to(device)<br>        optimizer.zero_grad()<br>        y_pred = model(x)  # shape (B, T, vocab_size)<br>        loss = criterion(y_pred.view(-1, vocab_size), y.view(-1))<br>        loss.backward()<br>        optimizer.step()<br>        losses.append(loss.item())<br>        if i % eval_iters == 0:<br>            # mean training loss over the last 100 batches<br>            mean_train_loss = sum(losses[-eval_iters:]) / float(eval_iters)<br><br>            print(f&#39;iteration {i}, mean training loss: {mean_train_loss:.2f}&#39;)<br><br>            train_loss, val_loss = estimate_loss(model, criterion, eval_iters, batch_size, block_size)<br>            output_losses.append((train_loss, val_loss))<br>            print(f&#39;train loss: {train_loss:.2f}, val loss: {val_loss:.2f}&#39;)<br><br>    return output_losses<br><br>losses = train(model, criterion, optimizer, num_iterations, batch_size, block_size)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lAljFCPej5DVUwLTIEkgcQ.png" /><figcaption>Training and validation losses (Image by author)</figcaption></figure><h4><strong>Virsualizing the Training and Validation losses</strong></h4><p>Seeing the training and validation loss over time we can see that the model is learning and the validation loss is decreasing over time for a certain point but after that validation loss starts to increase again. This is a sign of overfitting.</p><pre>import matplotlib.pyplot as plt<br><br>train_loss, val_loss = zip(*losses)<br><br>plt.figure(figsize=(10, 5))<br>plt.plot(train_loss, label=&#39;train loss&#39;)<br>plt.plot(val_loss, label=&#39;val loss&#39;)<br>plt.xlabel(&#39;iterations&#39;)<br>plt.ylabel(&#39;loss&#39;)<br>plt.legend()<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*mjTcizKBNlC0iIyFPcYm4A.png" /><figcaption>training and validation loss plot (Image by author)</figcaption></figure><h4><strong>Generation from Trained Model</strong></h4><p>Let’s now generate some text using the trained transformer mode</p><pre>@torch.no_grad()<br>def generate_text(model, input, max_new_tokens = 100, temperature = 1.0):<br>    model.eval()<br><br>    out = &#39;&#39;<br>    if isinstance(input, str):<br>        input = encode(input)<br>    input = torch.tensor(input, dtype=torch.long, device=device).unsqueeze(0)<br>    for _ in range(max_new_tokens):<br>        with torch.no_grad():<br>            input = input[:, -block_size:]<br>            y = model(input)  # shape (B, T, vocab_size)<br>            # take the logits at the final time-step and scale by the temperature<br>            y = y[0, -1, :] / temperature  # shape (vocab_size,)<br>            y = torch.softmax(y, dim=-1)<br>            next_token = torch.multinomial(y, 1)<br>            # if next_token == 0:<br>            #     break<br>            print(decode([next_token.item()]), end=&#39;&#39;, flush=True)<br>            out += decode([next_token.item()])<br>            input = torch.cat((input, next_token.unsqueeze(0)), dim=1)<br>    return out<br><br>generate_text(model, &quot;The&quot;, max_new_tokens=1000, temperature=0.8)</pre><h4>Output</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XdeYfJyWrssbJYROLJS5pw.png" /><figcaption>Model’s output (Image by author)</figcaption></figure><h3>Conclusions</h3><p>Building and training a GPT model from scratch is a rewarding yet challenging journey. By delving deep into the mechanics of transformers, self-attention, and multi-head attention, we’ve gained a solid understanding of what powers large language models like GPT-2 and GPT-3. Piece by piece, we constructed the GPT architecture, trained it on custom data, and observed its capabilities in text generation.</p><h4>Key Takeaways from Building GPT from Scratch</h4><ul><li><strong>Deep Understanding of GPT Internals:</strong> You now have a clear understanding of the internal mechanics of GPT models, including how self-attention, multi-head attention, and transformer blocks work.</li><li><strong>Hands-on Experience:</strong> Successfully built a smaller version of a GPT model using only Python and PyTorch, starting from basic components.</li><li><strong>Practical Knowledge of Training:</strong> Learned how to train and fine-tune a GPT model on your own dataset using simplified methodologies.</li></ul><h4>Future Work</h4><ul><li>Improve the overall performance of the model using hyperpamaters, AdamW, gradient clipping, etc</li><li>Improve the validation loss and reduce overfitting using hyperpamaters, learning rate scheduler, etc</li><li>Optimize the inference speed using different optimizations like <strong><em>torch.compile, </em></strong>kernel fusion<strong><em>, </em></strong>flash attention, float16 conversion, etc.</li><li>Optimize training speed — distributed data parallel (DDP), LORA, etc.</li></ul><h4>References</h4><p><a href="https://www.youtube.com/playlist?list=PLAqhIrjkxbuWI23v9cThsA9GvCAUhRvKZ">Neural Networks: Zero to Hero</a> by Andrej Karpathy — YT playlist</p><p><a href="https://arxiv.org/pdf/1706.03762">Attention Is All You Need</a> — Research Paper</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8ae985a8fc25" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/build-and-train-gpt-from-scratch-8ae985a8fc25">Build And Train GPT From Scratch</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Speculative Decoding — Make LLM Inference Faster]]></title>
            <link>https://medium.com/ai-science/speculative-decoding-make-llm-inference-faster-c004501af120?source=rss-234f1d6be728------2</link>
            <guid isPermaLink="false">https://medium.com/p/c004501af120</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Luv Bansal]]></dc:creator>
            <pubDate>Mon, 08 Apr 2024 05:26:36 GMT</pubDate>
            <atom:updated>2024-04-13T09:51:50.399Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*843SrnbC9zEBZfsVEkweaA.jpeg" /><figcaption>Image created by author using Dalle-3 via Bing Chat</figcaption></figure><h3>Speculative Decoding — Make LLM Inference Faster</h3><h4>Improve LLM inference speed by 2–3X without degrading any accuracy</h4><p>In this blog, we’ll discuss about Speculative Decoding in detail which is a method to improve LLM inference speed by around 2–3X without degrading any accuracy. We’ll also look into implementing Speculative Decoding and and see how fast it is compared naive transformer implementation.</p><p>Using speculative decoding can speed up the process of generating text <strong>without changing the final result</strong>. Speculative decoding involves <strong>running two models parallel</strong> , which has shown to promise a speed increase of 2–3 times for language model inference.</p><h3>Naive Autoregressive Sampling</h3><h4>Autoregressive Sampling</h4><p>The standard way of generating text from a language model is with autoregressive sampling where decoding <em>K</em> tokens takes <em>K</em> serial runs of the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*wf_Ne5qzR9hlwiGKIZE6nQ.png" /><figcaption>Autoregressive Sampling</figcaption></figure><p>Inference from large autoregressive models like Transformers is slow — decoding K tokens takes K serial runs of the model.</p><p>In code:</p><pre>def generate(prompt: str, tokens_to_generate: int) -&gt; str:<br>    tokens = tokenize(prompt)<br>    for i in range(tokens_to_generate):<br>        next_token = model(tokens)<br>        tokens.append(next_token)<br>    return detokenize(tokens)</pre><h3>Speculative Decoding</h3><p>Using a new method called Speculative Decoding could make our language model (LLM) work much faster without changing its results. Speculative Decoding that promising 2–3X speedups of LLM inference by <strong><em>running two models in parallel.</em></strong></p><ul><li><strong>Target Model:</strong> The main LLM we want to use for our task.</li><li><strong>Small Draft Model:</strong> A smaller, lightweight LLM that runs alongside to help speed up the main LLM’s inference process.</li></ul><h4><strong>Note</strong></h4><p>The <em>target model</em> and the <em>draft model</em> must both use the same tokenizer.</p><h4>There are two key idea behind Speculative Decoding:</h4><ol><li>In the below following image, predicting a token ‘<strong><em>of</em></strong> ‘ is really easy and it probably get easily predicted by much smaller model, therefore using a smaller model to predict the easy tokens and the use the big model only for predicting more difficult tokens.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cKEACKReWvmo3KL3giHcRA.png" /><figcaption>predicting a token ‘<strong><em>of</em></strong> ‘ is really easy and can easily predicted by much smaller model while prediction of token ‘<strong>Edinburg</strong>’ is difficult comparatively which smaller model might not able to predict</figcaption></figure><p>2. The second idea takes advantage of how Transformer models work. Even though these models typically generate one word at a time, they can process multiple tokens at once. While generating the next token, they can check all tokens in the sequence at once. It does this by calculating the probability for each token in the sequence. In the example below, smaller model predicts “Toronto” but the correct word is “Edinburgh” the bigger model can see that the probability of “Toronto” is low and correct it to “Edinburgh”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PXL_AnLPazawSM1Dm2o7iQ.png" /><figcaption>smaller model predicts “<strong>Toronto</strong>” but the correct word is “<strong>Edinburgh</strong>” the bigger model can see that the probability of “Toronto” is low and correct it to “Edinburgh”.</figcaption></figure><h4>How Speculative Decoding Works?</h4><p>At the heart of Speculative Decoding approach lie the observations that:</p><ul><li>Hard language-modeling tasks often include easier subtasks that can be solved well by more efficient lightweight models which take very less time to execute.</li><li>When performing inference on an LLM, speculative decoding uses a smaller draft model which generates speculative tokens and then target LLM verify those draft output tokens generated by smaller draft model.</li><li>With speculative execution, exact decoding from the large model can be generated faster. This works by running the larger model at the same time on the rough guesses from smaller model. This means we can generate several tokens in one forward pass of larger model, without changing the output distribution.</li></ul><p>The speedup provided by speculative decoding heavily depends on the choice of the draft model.</p><h4>Core Idea of Speculative Decoding</h4><p>The core idea is:</p><p>(1) use the more efficient small model <em>Mq</em> to generate γ completions.</p><p>(2) Use the target model <em>Mp</em> to evaluate all of the guesses and their respective probabilities from <em>Mq</em> in parallel, accepting all those that can lead to an identical distribution.</p><p>(3) Sampling an additional token from an adjusted distribution to fix the first one that was rejected, or to add an additional one if they are all accepted. That way, each parallel run of the target model Mp will produce at least one new token (so the number of serial runs of the target model can never, even in the worst case, be larger than the simple autoregressive method), but it can potentially generate many new tokens, up to γ + 1, depending on how well <em>Mq</em> approximates <em>Mp</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UQLuD2szod77-xTkro0wnA.png" /><figcaption>Image Source: <a href="https://arxiv.org/pdf/2302.01318.pdf">Accelerating Large Language Model Decoding with Speculative Sampling</a></figcaption></figure><h3>Experiments</h3><p>Evaluation setup of Speculative Decoding</p><blockquote>I’ve used <strong>NVIDIA RTX 6000 Ada Generation GPU</strong> for running and evaluating LLM.</blockquote><h4>Code Implementation</h4><p>Code for implementing Speculative Decoding and Benchmark with autoregressive sampling.</p><p>I’ve used <a href="https://github.com/shreyansh26/Speculative-Sampling/tree/main">Speculative-Sampling: main GitHub Repository</a> for implementing and benchmarking the results. I found this repository to be very good wrt Speculative Decoding implementation.</p><h4>Speculative Decoding with Huggingface Transformers</h4><p>Speculative Decoding could be easily implemented with Huggingface transformer library. Checkout the following code snippet:</p><pre>from transformers import AutoModelForCausalLM, AutoTokenizer<br>import torch<br><br>prompt = &quot;Alice and Bob&quot;<br>checkpoint = &quot;EleutherAI/pythia-1.4b-deduped&quot;<br>assistant_checkpoint = &quot;EleutherAI/pythia-160m-deduped&quot;<br>device = &quot;cuda&quot; if torch.cuda.is_available() else &quot;cpu&quot;<br><br>tokenizer = AutoTokenizer.from_pretrained(checkpoint)<br>inputs = tokenizer(prompt, return_tensors=&quot;pt&quot;).to(device)<br><br>model = AutoModelForCausalLM.from_pretrained(checkpoint).to(device)<br>assistant_model = AutoModelForCausalLM.from_pretrained(assistant_checkpoint).to(device)<br>outputs = model.generate(**inputs, assistant_model=assistant_model)<br>print(tokenizer.batch_decode(outputs, skip_special_tokens=True))<br># [&#39;Alice and Bob are sitting in a bar. Alice is drinking a beer and Bob is drinking a&#39;]<br></pre><h4>Parameters:</h4><p><strong>num_assistant_tokens</strong>: Defines the number of _speculative tokens_ that shall be generated by the assistant or draft model before being checked by the target model at each iteration. Higher values for `num_assistant_tokens` make the generation more _speculative_ : If the assistant model is performant larger speed-ups can be reached, if the assistant model requires lots of corrections, lower speed-ups are reached.</p><h4>Benchmark Results</h4><p><strong>Note</strong><br>The target model and the draft model must both use the same tokenizer.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/70ac41981f29569dbcddc17dd99743d1/href">https://medium.com/media/70ac41981f29569dbcddc17dd99743d1/href</a></iframe><p>Results showing the speedup (as ratio) of <strong>speculative sampling</strong> over naive <strong>autoregressive sampling</strong>. These results are from different benchmarking runs.</p><p>Above benchmark results shows that Speculating Decoding gives around 2X speed increase over naive autoregressive sampling<strong>.</strong></p><h3>Conclusions</h3><ul><li>Speed increase is around 2X in most of the cases based on our results.</li><li>The draft model must be <strong>significantly</strong> smaller(10–50X smaller) to achieve inference acceleration</li><li>The speedup ratio seems to increase as the target model size increases (and when the draft model is also relatively big enough). So, the speedup ratio of 2–2.5x mentioned in the Deepmind paper, could also be true for a 70B target model and a 7B draft model.</li><li>However, there are some instances where the speed actually slows down. We need to look into these cases more closely. It could be because different models require specific prompt formats?</li></ul><h3>Future Work</h3><h4>1. Benchmark with different lookheads</h4><h4>2. vLLM AND Speculative decoding Integration</h4><p>Integrating Speculative decoding in vLLM is in progress (I believe it’s in final stage) , I believe vLLM and Speculative decoding will give huge speed-up.</p><p>PR: <a href="https://github.com/vllm-project/vllm/pull/2188">vllm-public: [WIP] Speculative decoding using a draft model<strong>OPEN</strong></a></p><h4>3. Medusa</h4><p>It’s also a <strong>speculative sampling </strong>but instead of using draft model, it trains few new parameters to <em>predict multiple future tokens</em>.</p><p><strong>GitHub:</strong> <a href="https://github.com/FasterDecoding/Medusa/tree/main">Medusa: main</a></p><p><a href="https://arxiv.org/abs/2401.10774"><strong>arXiv</strong></a></p><p><em>Medusa adds extra “heads” to LLMs to predict multiple future tokens simultaneously. When augmenting a model with Medusa, the original model stays untouched, and only the new heads are fine-tuned during training. During generation, these heads each produce multiple likely words for the corresponding position</em>.</p><p>It solve the challenges associated with speculative decoding by implementing the following ideas:</p><ul><li>Instead of introducing a new model, we train multiple decoding heads on the <em>same</em> model.</li><li>The training is parameter-efficient so that even the “GPU-Poor” can do it. And since there is no additional model, there is no need to adjust the distributed computing setup.</li><li>Relaxing the requirement of matching the distribution of the original model makes the non-greedy generation even faster than greedy decoding</li></ul><h3>References</h3><p><a href="https://arxiv.org/pdf/2211.17192.pdf"><strong>Fast Inference from Transformers via Speculative Decoding</strong></a><strong> </strong>by Google. — Research paper</p><p><a href="https://arxiv.org/pdf/2302.01318.pdf"><strong>Accelerating Large Language Model Decoding with Speculative Sampling</strong></a> by Deepmind — Research paper</p><p><a href="https://arxiv.org/abs/2402.01528"><strong>Decoding Speculative Decoding</strong></a><strong> — </strong>Research paper</p><p><a href="https://www.youtube.com/watch?v=hm7VEgxhOvk"><strong>Speculative Decoding Explained</strong></a><strong> </strong>— YT video</p><p><a href="https://www.youtube.com/watch?v=S-8yr_RibJ4&amp;t=229s"><strong>Speculative Decoding: When Two LLMs are Faster than One</strong></a> — YT video</p><p><a href="https://jaykmody.com/blog/speculative-sampling/"><strong>Speculative Sampling</strong></a><strong> </strong>— blog</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c004501af120" width="1" height="1" alt=""><hr><p><a href="https://medium.com/ai-science/speculative-decoding-make-llm-inference-faster-c004501af120">Speculative Decoding — Make LLM Inference Faster</a> was originally published in <a href="https://medium.com/ai-science">AI Science</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM Quantisation: Quantise Hugging face Model with GPTQ, AWQ and Bitsandbytes]]></title>
            <link>https://pub.towardsai.net/llm-quantisation-quantise-hugging-face-model-with-gptq-awq-and-bitsandbytes-a4ad45cd8b48?source=rss-234f1d6be728------2</link>
            <guid isPermaLink="false">https://medium.com/p/a4ad45cd8b48</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[model-quantization]]></category>
            <dc:creator><![CDATA[Luv Bansal]]></dc:creator>
            <pubDate>Mon, 18 Mar 2024 18:02:16 GMT</pubDate>
            <atom:updated>2024-03-18T18:02:16.317Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f4nmYJ_NgIk_9lW0cvfQhg.jpeg" /><figcaption>Image created by author using Dalle-3 via Bing Chat</figcaption></figure><h3>LLM Quantization: Quantize Model with GPTQ, AWQ, and Bitsandbytes</h3><h4>The ultimate guide to Quantizing LLM — How to Quantize a model with AWQ, GPTQ, and Bitsandbytes, push a quantized model on the 🤗 Hub, load an already quantized model from the Hub</h4><p>This blog will be ultimate guide for Quantization of models, We’ll talk about various ways to quantizing models like GPTQ, AWQ and Bitsandbytes. We’ll discuss the pros and cons of each method (GPTQ vs AWQ vs Bitsandbytes), in the end, use quantized weights for efficient language model inference.</p><p>But before diving in, let’s begin by understanding what quantization is and why it’s necessary.</p><h3>Quantization</h3><p>A Large Language Model is represented by a bunch of weights and activations. These values are generally represented by the usual 32-bit floating point (float32) datatype.</p><blockquote>The number of bits tells us something about how many values it can represent. <strong>Float32</strong> can represent values between 1.18e-38 and 3.4e38, quite a number of values! The lower the number of bits, the fewer values it can represent.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*KLPKaFB9_f6jKcyl.png" /><figcaption>FLoat representation | <a href="https://www.cerebras.net/machine-learning/to-bfloat-or-not-to-bfloat-that-is-the-question/">source</a></figcaption></figure><p>As you might expect, if we choose a lower bit size, then the model becomes less accurate but it also needs to represent fewer values, thereby decreasing its size and memory requirements.</p><p>One of the most effective methods to reduce the model size in memory is <strong>quantization</strong>. You can see quantization as a compression technique for LLMs. In practice, the main goal of quantization is to lower the precision of the LLM’s weights, typically from 16-bit to 8-bit, 4-bit, or even 3-bit. However, we do not simply want to use a smaller bit variant but map a larger bit representation to a smaller bit without losing too much information.</p><h4>Quantisation Techniques</h4><p>There are three popular quantization methods for LLMs, we will discuss briefly about each of them, then we will look into how to use each of the techniques in code.</p><ul><li>GPTQ</li><li>AWQ</li><li>Bitsandbytes NF4</li></ul><h3>GPTQ Quantization</h3><p><a href="https://arxiv.org/abs/2210.17323">GPTQ</a> is a <strong>p</strong>ost-<strong>t</strong>raining <strong>q</strong>uantization (PTQ) method for quantization that focuses primarily on <strong>GPU</strong> inference and performance.</p><blockquote>Post-<strong>t</strong>raining <strong>q</strong>uantization (PTQ) is the sort of quantization, wherein the weights of an already skilled model are transformed to a lower precision <strong>without any retraining</strong>. This is a trustworthy and clean-to-put-into-effect method</blockquote><p>It is based on one-shot weight quantization method based on approximate second-order information, that is both <strong>highly- accurate</strong> and <strong>highly-efficient.</strong></p><p>The GPTQ algorithm makes it possible to quantize models up to <strong>2 bits</strong>! However, this might come with severe quality degradation. The recommended number of bits is 4, which seems to be a great tradeoff for GPTQ at this time.</p><p>The idea behind the method is that it will try to compress all weights to a 4-bit quantization by minimizing the mean squared error to that weight.</p><h4>Quantize Model to GPTQ</h4><p>GTPQ requires a lot of GPU VRAM. For this experiment, we are using Nvidia A10 GPU which has 24 GB of memory and we are quantizing Llama 2–7B model and the VRAM consumption peaked at around 16 GB.</p><p>Quantization with GPTQ is also slow. It took 35 min with one A10, The quantization speed and VRAM/RAM consumption are the same for the 4-bit, 3-bit, and 2-bit precisions.</p><p>While you can’t quantize Llama 2 with GPTQ on the Google Colab free tier. Smaller models (&lt;4B parameters) can be quantized with a colab-free tier.</p><p>I used the same snippet that <a href="https://huggingface.co/TheBlokeAI">TheBloke</a> uses to quantify HF models to GPTQ. You can find the script <a href="https://github.com/TheBlokeAI/AIScripts/blob/main/quant_autogptq.py">here</a>.</p><p>You can use the same TheBloke script easily with the below command.</p><blockquote>Note: You need to install transformers optimum accelerate auto-gptq first</blockquote><pre>python ./quant_autogptq.py meta-llama/Llama-2-7b-chat-hf gptq_checkpoints c4 --bits 4 --group_size 128 --desc_act 1 --damp 0.1 --seqlen 4096</pre><p>I’m simplifying the script above to make it easier for you to understand what’s in it.</p><p>To quantize with GPTQ, I installed the following libraries:</p><pre>pip install transformers optimum accelerate auto-gptq</pre><h4><strong>Importing libraries and loading dataset</strong></h4><pre>from datasets import load_dataset<br>import os<br>import random<br>import torch<br>import numpy as np<br>from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig<br><br>model_id = &quot;meta-llama/Llama-2-7b-hf&quot;<br><br>tokenizer = AutoTokenizer.from_pretrained(model_id, <br>                                               use_fast=True,)<br>def get_c4():<br> <br>    traindata = load_dataset(<br>        &#39;allenai/c4&#39;, data_files={&#39;train&#39;: &#39;en/c4-train.00000-of-01024.json.gz&#39;}, split=&#39;train&#39;, use_auth_token=False<br>    )<br>    seqlen  = 4096<br>    trainloader = []<br>    for _ in range(self.num_samples):<br>        while True:<br>            i = random.randint(0, len(traindata) - 1)<br>            trainenc = tokenizer(traindata[i][&#39;text&#39;], return_tensors=&#39;pt&#39;)<br>            if trainenc.input_ids.shape[1] &gt;= self.seqlen:<br>                break<br>        i = random.randint(0, trainenc.input_ids.shape[1] - self.seqlen - 1)<br>        j = i + seqlen<br>        inp = trainenc.input_ids[:, i:j]<br>        attention_mask = torch.ones_like(inp)<br>        trainloader.append({&#39;input_ids&#39;:inp,&#39;attention_mask&#39;: attention_mask})<br>    return trainloader</pre><ol><li>It specifies the pre-trained model ID (“meta-llama/Llama-2–7b-hf”) to load the tokenizer.</li><li><em>get_c4() </em>function is to load the dataset which is used for quantization.</li></ol><blockquote>Note that TheBloke didn’t use the entire ‘c4’ dataset which is quite large and it won’t fit in 24GB A10 GPU (I tried it and was giving OOM or Out Of Memory Error while quantising). Instead, only a small part of the ‘c4’ dataset was used.</blockquote><h4><strong>The quantization and serialization with Transformers</strong></h4><pre>quantize_config = BaseQuantizeConfig(<br>    bits=bits,<br>    group_size=group_size,<br>    desc_act=desc_act,<br>    damp_percent=damp<br>)<br><br>torch_dtype = torch.float16<br><br>traindataset = get_c4()<br><br>model = AutoGPTQForCausalLM.from_pretrained(model_id, quantize_config=quantize_config,<br>                                            low_cpu_mem_usage=True, torch_dtype=torch_dtype)<br><br><br>model.quantize(traindataset, use_triton=self.use_triton, batch_size=self.batch_size, cache_examples_on_gpu=self.cache_examples)</pre><p>3. It specifies the pre-trained model ID (“meta-llama/Llama-2–7b-hf”) that will be loaded and quantized.</p><p>4. It creates a BaseQuantizeConfig object, which configures the GPTQ quantization process:</p><ul><li><strong>bits</strong>: Precision of the quantization. You can set 4, 3, and 2.</li><li><strong>group_size: </strong>The group size to use for quantization. The recommended value is 128 and -1 uses per-column quantization</li><li><strong>desc_act: </strong>Whether to quantize columns in order of decreasing activation size. Setting it to False can significantly speed up inference but the perplexity may become slightly worse. Also known as act-order.</li></ul><h4><strong>Save quantized Model</strong></h4><pre>output_dir = &#39;Llama-2-7b-hf-gptq&#39;<br><br>model.save_quantized(output_dir, use_safetensors=True)<br>tokenizer.save_pretrained(output_dir)</pre><p>5. Finally mention the path where you want to save quantize model weights, and from where quantized model will be loaded for inference.</p><h4>Push Quantise Model to Hub</h4><p>You can upload (push) the quantized language model to the Hugging Face Hub, Then, anyone can use it right away for their work. It’s also great for collaborative contributions in the open-source community.</p><pre>from huggingface_hub import HfApi<br><br>api = HfApi()<br>api.upload_folder(<br>    folder_path=&quot;Llama-2-7b-hf-gptq&quot;,<br>    repo_id=&quot;luv2261/Llama-2-7b-hf-gptq&quot;,<br>    repo_type=&quot;model&quot;,<br>)</pre><p>Check out the models I quantized using the method mentioned above <a href="https://huggingface.co/luv2261">here</a> and feel free to use them for your own work.</p><h4>Load and Inference quantized models from the 🤗 Hub</h4><pre>from transformers import AutoTokenizer, AutoModelForCausalLM<br><br>model_id = &quot;luv2261/Llama-2-7b-hf-gptq&quot;<br><br>model = AutoModelForCausalLM.from_pretrained(model_id, device_map=&quot;auto&quot;)<br>tokenizer = AutoTokenizer.from_pretrained(model_id)<br><br>prompt = &quot;&lt;s&gt;[INST] Write a tweet on future of AI [/INST]&quot;<br>inputs = tokenizer(prompt, return_tensors=&quot;pt&quot;).to(0)<br><br>out = model.generate(**inputs, max_new_tokens=250, temperature = 0.6, top_p=0.95, tok_k=40)<br><br>print(tokenizer.decode(out[0], skip_special_tokens=True))</pre><h3>AWQ Quantization</h3><p>AWQ (<a href="https://arxiv.org/abs/2306.00978">Activation-aware Weight Quantization</a>) which is a quantization method similar to GPTQ. There are several differences between AWQ and GPTQ as methods but the most important one is that AWQ assumes that not all weights are equally important for an LLM’s performance.</p><p>In other words, a small fraction of weights will be skipped during quantization, which helps with the quantization loss.</p><p>As a result, their paper mentions a significant speed-up compared to GPTQ whilst keeping similar, and sometimes even better, performance.</p><h4>Quantize Model to AWQ</h4><p>AWQ performs zero point quantization down to a precision of 4-bit integers.</p><h4>AutoAWQ integration with Transformers</h4><p>Note:</p><ul><li>Some models like Falcon is only compatible with group size 64.</li><li>To use Marlin, you must specify zero point as False and version as Marlin.</li></ul><pre>from awq import AutoAWQForCausalLM<br>from transformers import AutoTokenizer<br><br>model_path = &#39;meta-llama/Llama-2-7b-hf&#39;<br><br>quant_config = { &quot;zero_point&quot;: True, &quot;q_group_size&quot;: 128, &quot;w_bit&quot;: 4, &quot;version&quot;: &quot;GEMM&quot; }<br># Load model<br>model = AutoAWQForCausalLM.from_pretrained(<br>    model_path, **{&quot;low_cpu_mem_usage&quot;: True, &quot;use_cache&quot;: False}<br>)<br>tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)<br># Quantize<br>model.quantize(tokenizer, quant_config=quant_config)</pre><h4>Make compatible with Transformers and Save quantized Model</h4><p>In order to make it compatible with transformers, we need to modify the config file and then save the quantized model.</p><pre>from transformers import AwqConfig, AutoConfig<br>from huggingface_hub import HfApi<br><br># modify the config file so that it is compatible with transformers integration<br>quantization_config = AwqConfig(<br>    bits=quant_config[&quot;w_bit&quot;],<br>    group_size=quant_config[&quot;q_group_size&quot;],<br>    zero_point=quant_config[&quot;zero_point&quot;],<br>    version=quant_config[&quot;version&quot;].lower(),<br>).to_dict()<br><br># the pretrained transformers model is stored in the model attribute + we need to pass a dict<br>model.model.config.quantization_config = quantization_config<br># a second solution would be to use Autoconfig and push to hub (what we do at llm-awq)<br><br># Save quantized model<br>quant_path = &#39;Llama-2-7b-hf-awq&#39;<br><br>model.save_quantized(quant_path)<br>tokenizer.save_pretrained(quant_path)<br>print(f&#39;Model is quantized and saved at &quot;{quant_path}&quot;&#39;)</pre><p>Uploading the quantized model to the hugging face hub and loading the model from the hub is pretty much like what I explained for GPTQ.</p><h3>Bitsandbytes NF4</h3><p><a href="https://github.com/TimDettmers/bitsandbytes">bitsandbytes</a> is the easiest way for quantizing a model to 8 and 4-bit</p><p>It works in three steps:</p><ol><li><strong>Normalization</strong>: The weights of the model are normalized so that we expect the weights to fall within a certain range. This allows for more efficient representation of more common values.</li><li><strong>Quantization</strong>: The weights are quantized to 4-bit. In NF4, the quantization levels are evenly spaced with respect to the normalized weights, thereby efficiently representing the original 32-bit weights.</li><li><strong>Dequantization</strong>: Although the weights are stored in 4-bit, they are dequantized during computation which gives a performance boost during inference.</li></ol><p>It represents the weights with 4-bit quantization but does the inference in 16-bit.</p><p>If you want to know more about how this quantisation works, I highly recommend to go though <a href="https://mlabonne.github.io/blog/posts/Introduction_to_Weight_Quantization.html">this blog</a></p><p>This is a wonderful technique but it seems rather wasteful to have to apply them every time we have to load the model.</p><h4>Quantise model with Bitsandbytes</h4><p>Using this quantisation is straightforward with HuggingFace. we just need to define a configuration for the quantization with <a href="https://github.com/TimDettmers/bitsandbytes">Bitsandbytes</a>:</p><pre>from transformers import BitsAndBytesConfig<br>from torch import bfloat16<br><br># Our 4-bit configuration to load the LLM with less GPU memory<br>bnb_config = BitsAndBytesConfig(<br>    load_in_4bit=True,  # 4-bit quantization<br>    bnb_4bit_quant_type=&#39;nf4&#39;,  # Normalized float 4<br>    bnb_4bit_use_double_quant=True,  # Second quantization after the first<br>    bnb_4bit_compute_dtype=bfloat16  # Computation type<br>)</pre><p>This configuration allows us to specify which quantization levels we are going for. Generally, we want to represent the weights with 4-bit quantization but do the inference in 16-bit.</p><p>Loading the model in a pipeline is then straightforward:</p><pre>from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline<br><br># Zephyr with BitsAndBytes Configuration<br>tokenizer = AutoTokenizer.from_pretrained(&quot;HuggingFaceH4/zephyr-7b-alpha&quot;)<br><br>model = AutoModelForCausalLM.from_pretrained(<br>    &quot;HuggingFaceH4/zephyr-7b-alpha&quot;,<br>    quantization_config=bnb_config,<br>    device_map=&#39;auto&#39;,<br>)<br><br># Create a pipeline<br>pipe = pipeline(model=model, tokenizer=tokenizer, task=&#39;text-generation&#39;)<br><br>prompt = &#39;&#39;&#39;<br>&lt;|system|&gt; \<br>You are a friendly chatbot who always responds in the style of a pirate.\<br>&lt;/s&gt;\<br>&lt;|user|&gt;\<br>Write a tweet on future of AI. &lt;/s&gt;\<br>&lt;|assistant|&gt;<br>&#39;&#39;&#39;<br><br>outputs = pipe(<br>    prompt, <br>    max_new_tokens=256, <br>    do_sample=True, <br>    temperature=0.7, <br>    top_p=0.95<br>)<br>print(outputs[0][&quot;generated_text&quot;])</pre><h3>Bitsandbytes vs GPTQ vs AWQ</h3><p>A quick camparition between Bitsandbytes, GPTQ and AWQ quantization, so you can choose which methods to use according to your use case.</p><h4><strong>Bitandbytes</strong></h4><ul><li>This method quantise the model using HF weights, so <strong>very easy to implement</strong></li><li>Slower than other quantisation methods as well as 16-bit LLM model.</li><li>This is a wonderful technique but it seems rather wasteful to have to apply them every time we have to load the model.</li><li>Take longer time to load the models weights</li></ul><h4><strong>GPTQ</strong></h4><ul><li>Much Faster as compared to Bitandbytes</li><li>New model architectures are promptly supported in <a href="https://github.com/AutoGPTQ/AutoGPTQ">AutoGPTQ</a></li></ul><p><strong>Challenges</strong></p><ul><li>Need to quantise the model weights to GPTQ weights beforehand to use it in production.</li><li>High computation to quantise the model</li><li>Around <strong>~ <em>16GB</em></strong><em> </em>GPU memory required to quantise 7B parameter model</li></ul><h4><strong>AWQ</strong></h4><ul><li>Paper mentions a significant speed-up compared to GPTQ whilst keeping similar, and sometimes even better, performance</li></ul><p><strong>Limitations</strong></p><ul><li>Newer architectures like Gemma or DeciLM doesn’t support in AWQ quantization yet.</li></ul><h4>Final thoughts</h4><p>This article discusses various techniques to quantize models like GPTQ, AWQ and Bitsandbytes. It looks at the pros and cons of each method (GPTQ vs AWQ vs bitsandbytes), explains quantizing hugging-face model weights using these methods and finally use quantize weights for LLM inference. These techniques can help you create and use Large Language Models more effectively in real-world applications.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a4ad45cd8b48" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/llm-quantisation-quantise-hugging-face-model-with-gptq-awq-and-bitsandbytes-a4ad45cd8b48">LLM Quantisation: Quantise Hugging face Model with GPTQ, AWQ and Bitsandbytes</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>