<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yuval Mehta on Medium]]></title>
        <description><![CDATA[Stories by Yuval Mehta on Medium]]></description>
        <link>https://medium.com/@yuvalmehta.728?source=rss-3ec8787f291f------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*wOFpWpNcAZIxd-GmrzOtXQ.png</url>
            <title>Stories by Yuval Mehta on Medium</title>
            <link>https://medium.com/@yuvalmehta.728?source=rss-3ec8787f291f------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 08:10:52 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@yuvalmehta.728/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[TurboQuant in Production: Qdrant, vLLM, and What Survived]]></title>
            <link>https://pub.towardsai.net/turboquant-in-production-qdrant-vllm-and-what-survived-595d53b8ec45?source=rss-3ec8787f291f------2</link>
            <guid isPermaLink="false">https://medium.com/p/595d53b8ec45</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Yuval Mehta]]></dc:creator>
            <pubDate>Sun, 25 Oct 2026 22:01:01 GMT</pubDate>
            <atom:updated>2026-10-25T22:01:01.743Z</atom:updated>
            <content:encoded><![CDATA[<h3>TurboQuant in Production: What Qdrant Kept, Dropped, and Added</h3><h4><em>A clean proof about uniform spheres meets anisotropic embeddings and a 99.9th-percentile KL benchmark.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KJTuEv1BbPb9g4N-" /><figcaption>Photo by <a href="https://unsplash.com/@resourcedatabase?utm_source=medium&amp;utm_medium=referral">Resource Database</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Two production systems ship TurboQuant, and neither ships all of it. Qdrant 1.18 uses the MSE variant plus extensions that are not in the paper, and vLLM’s TurboQuant module <a href="https://docs.vllm.ai/en/stable/api/vllm/model_executor/layers/quantization/turboquant/">states</a> that QJL is intentionally omitted.</p><p>That omission matters. QJL is the stage the paper adds to the rotation recipe to make inner products unbiased, and the two places that count have set it aside.</p><p><em>Disclosure: Claude, an AI assistant, wrote most of this draft. Each figure links to its source, and the code snippet is runnable.</em></p><p><strong>TL;DR</strong></p><ul><li>TurboQuant is a random rotation, then a fixed Lloyd-Max scalar codebook per coordinate, with an optional 1-bit QJL correction on the residual. The paper reports MSE within about 2.7x of a lower bound.</li><li>Qdrant ships the MSE variant and patches it for real embeddings. vLLM leaves QJL out. Both say why.</li><li>Qdrant reports 4-bit recall within about 2 points of scalar quantization on most of ten datasets, and 2-bit recall 9 to 24 points above binary quantization.</li><li>Read the headline claims with care. One independent KV benchmark shows perplexity hiding tail damage, and the authors of RaBitQ and EDEN dispute attribution and the speed comparison.</li></ul><h3>Why a quantizer without constants matters</h3><p>Most quantizers store a scale, and often a zero point, for every small block of values. Google’s <a href="https://research.google/blog/turboquant-redefining-ai-efficiency-with-extreme-compression/">research post</a> puts that overhead at 1 or 2 extra bits per number. As an illustrative case (assumption: blocks of 32 values, one fp16 scale and one fp16 zero point), the constants cost 32 bits per 32 values. A 4-bit code then costs 5 bits per value.</p><p>TurboQuant’s pitch is a quantizer with no per-block constants and no training pass. That fits two jobs: compressing a KV cache while tokens stream in, and indexing vectors without learning a codebook first. The <a href="https://arxiv.org/abs/2504.19874">paper</a> calls this data-oblivious.</p><h3>Lever 1: the rotation makes one codebook possible</h3><p>Claim: the random rotation is what lets a single fixed codebook work for any input.</p><p>The mechanism is short. Multiply each unit vector by one shared random orthogonal matrix. Each coordinate of the result follows a known Beta distribution, which approaches a Gaussian with variance 1/d at large dimension d, whatever the input looked like. Distinct coordinates also become nearly independent, so quantizing each one on its own loses little.</p><p>A concrete case: take a vector with most of its energy in 8 of 256 coordinates. A fixed grid fits it badly, because most coordinates sit near zero and a few sit far outside the grid. After rotation, every coordinate is a draw from the same bell curve.</p><pre>import numpy as np<br>rng = np.random.default_rng(0)<br>d, n = 256, 2000<br><br># Anisotropic unit vectors: 8 dominant coordinates<br>X = rng.standard_normal((n, d)) * np.r_[np.full(8, 20.0), np.ones(d - 8)]<br>X /= np.linalg.norm(X, axis=1, keepdims=True)<br># One shared random rotation (QR of a Gaussian matrix)<br>Q, R = np.linalg.qr(rng.standard_normal((d, d)))<br>Pi = Q * np.sign(np.diag(R))<br>def lloyd_max(samples, k, iters=60):<br>    c = np.quantile(samples, (np.arange(k) + 0.5) / k)<br>    for _ in range(iters):<br>        idx = np.searchsorted((c[:-1] + c[1:]) / 2, samples)<br>        c = np.array([samples[idx == j].mean() for j in range(k)])<br>    return c<br>def quant(v, c):<br>    return c[np.searchsorted((c[:-1] + c[1:]) / 2, v)]<br># Codebook fitted once on N(0, 1/d), the large-d limit<br>g = rng.standard_normal(400_000) / np.sqrt(d)<br>for b in (1, 2, 3, 4):<br>    c = lloyd_max(g, 2 ** b)<br>    rot = np.mean(np.sum((X - quant(X @ Pi.T, c) @ Pi) ** 2, axis=1))<br>    raw = np.mean(np.sum((X - quant(X, c)) ** 2, axis=1))<br>    print(f&quot;b={b} rotated={rot:.4f} unrotated={raw:.4f} lower bound={4.0**-b:.4f}&quot;)</pre><p>With NumPy 2.4 the snippet prints rotated MSE of 0.361, 0.116, 0.034 and 0.009 for 1 to 4 bits. The paper reports 0.36, 0.117, 0.03 and 0.009 for unit vectors. The unrotated run, with the same codebook and the same data, prints 1.05, 0.64, 0.46 and 0.37. The paper’s lower bound is 4^-b, which is 0.25 at one bit.</p><p>Condition: the bound assumes unit norm. Norms are stored separately in floating point, and that detail matters later.</p><h3>Lever 2: the codebook assumes Gaussian coordinates</h3><p>Claim: a fixed codebook is only as good as the Gaussian assumption behind it.</p><p>Qdrant <a href="https://qdrant.tech/articles/turboquant-quantization/">reports</a> that the proof rests on vectors spread uniformly over the sphere and does not extend to anisotropic embeddings, where a few directions carry most of the variance. Its fix is a per-segment calibration pass. It estimates a shift and a scale per coordinate from a reservoir sample using streaming quantile estimates, and folds both into the query side so search pays nothing extra. On isotropic data the correction is the identity.</p><p>Qdrant also stores a per-vector length ratio, in four bytes it already reserves, that undoes the shrinkage plain MSE quantization causes. It credits that idea to RaBitQ, not to the TurboQuant paper.</p><p>My reading, labeled as such: the paper’s bound averages over the choice of rotation for a fixed input, while a deployed index draws one rotation and lives with it. Calibration targets the second situation. This is an interpretation of the two sources, not a claim from either.</p><h3>Lever 3: the QJL stage fixes a real bias, then gets dropped</h3><p>Claim: QJL solves a genuine problem and is still the first thing production removes.</p><p>The problem is real. At one bit the MSE-optimal codebook is plus or minus the square root of 2/(pi d), and the paper shows the resulting inner product estimate carries a multiplicative bias of 2/pi. The bias shrinks as bits increase. TurboQuant-prod spends b minus 1 bits on the MSE quantizer, then applies QJL, the sign of a random Gaussian projection, to the residual. The result is unbiased.</p><p>Qdrant ships the MSE variant for three stated reasons: HNSW construction needs scores between two stored vectors, the correction costs bits that renormalization recovers for one scalar, and PROD adds a per-query random projection to every score. vLLM’s docs say community reports from five or more independent groups found QJL hurts attention quality by amplifying variance through softmax. I could not verify those five groups from the docs alone.</p><p>The mechanism is plausible on paper. Softmax exponentiates logits, so zero-mean error in a score becomes multiplicative error in an attention weight, and the largest weights dominate. That is a derivation, not a measurement. The condition matters too: unbiasedness pays when you average many independent estimates, and it pays less when you rank or exponentiate a single noisy score.</p><h3>Lever 4: what the recall numbers say</h3><p>Qdrant’s benchmark uses ten datasets with HNSW (m=16, ef_construct=128). This is a vendor benchmark, so treat it as one setup.</p><ul><li>At 4 bits, TurboQuant trails scalar quantization by at most about 2.3 points on 7 of 10 datasets and beats it on 3, by up to 4.6 points, at half the storage.</li><li>On one dataset, arxiv-instructorxl, 2-bit TurboQuant reaches recall 0.8227 against 0.6756 for 2-bit binary quantization, and 1-bit reaches 0.6763 against 0.4683 for vanilla 1-bit binary quantization.</li><li>Across the ten datasets, 2-bit TurboQuant beats 2-bit binary quantization by 9 to 24 points.</li></ul><p>Qdrant advises staying on binary quantization if throughput drops noticeably on your workload.</p><h3>Gotchas</h3><p><strong>Perplexity hides the tail.</strong> An independent <a href="https://anbeeld.com/articles/kv-cache-quantization-benchmarks-for-long-context">llama.cpp KV benchmark</a> ran Qwen 3.6 27B on one RTX 3090 at 64k context with Q5_K_S weights. Through q4_0, the whole perplexity range stayed under 0.01. The 99.9th-percentile KL precision told a different story: 94.61% for q8_0, 89.84% for q4_0, 76.12% for turbo3 and 41.47% for turbo2. The caveats are real. It is a community fork, one model, one GPU, and the author lists his own limits.</p><p><strong>Rotation was already in the box.</strong> That benchmark notes llama.cpp applies a random rotation before quantizing, so TurboQuant’s rotation is not new there. At 4 bits, turbo4 scored 89.13% against 89.84% for q4_0, with about 17% lower prefill throughput. The value showed up at 2 to 3 bits, mostly through the trellis-coded variants rather than the plain turbo modes, not at 4.</p><p><strong>The headline numbers need a baseline.</strong> Google’s post claims at least 6x KV memory reduction at 3 bits and up to 8x faster attention logits. Check the arithmetic against a 16-bit cache (derived, ignoring norms and scales): 3.5 bits is about 4.6x, 3 bits is about 5.3x, and 6x implies roughly 2.7 bits. The paper itself says more than 4.5x. The post’s 8x compares 4-bit keys with 32-bit unquantized keys on an H100, which is a logits benchmark, not decode throughput.</p><p><strong>Attribution and benchmark parity are disputed.</strong> The authors of RaBitQ published a <a href="https://arxiv.org/pdf/2604.19528">technical note</a> saying both methods start with a shared random rotation, that the paper’s RaBitQ timings came from a single CPU core with multithreading off against an A100 for TurboQuant, and that those setups were not disclosed. In their symmetric setup, RaBitQ on the same GPU ran about 1.2x to 1.8x faster, and they could not reproduce the paper’s reported times from the released code. In their KV harness, after fixing bugs in the released code, TurboQuant-mse scored 0.709 on needle-in-a-haystack at 2.5 bits against 0.951 for RaBitQ, and 0.962 against 0.977 at 3.5 bits. The authors of DRIVE and EDEN published a <a href="https://arxiv.org/abs/2604.18555">separate note</a> arguing that TurboQuant-mse is EDEN with the scale fixed to 1.</p><p>Weigh those sources properly. They are arXiv notes from the authors of the prior work, not peer-reviewed, and I found no published rebuttal. A <a href="https://milvus.io/blog/interview-with-rabitq-authors-the-turboquant-dispute-and-why-the-storage-selloff-was-a-false-alarm.md">Milvus interview</a> says a TurboQuant co-author agreed to revise the arXiv text on RaBitQ’s optimality. I could not confirm a revision. Qdrant, which has no stake in the dispute, describes RaBitQ as developing the same rotate-then-quantize foundation and uses RaBitQ’s renormalization.</p><h3>What to take from this</h3><p>What survives in production is the rotation, a fixed Lloyd-Max codebook, and a cheap per-vector scalar. What gets added is calibration for real embeddings. What gets dropped is the QJL correction. What stays contested is lineage and the head-to-head benchmarks.</p><p>Before you adopt it, run your own vectors at your own bit width, and measure a tail metric instead of perplexity alone. Compare against scalar quantization with rotation, since the rotation may account for much of the gain. That last point is my inference from the llama.cpp result, not a finding.</p><p>One question has no public answer yet. Nobody has published tail behavior at 2 to 3 bits on long agentic traces full of tool calls, where one dropped token in a structured argument breaks the run. The benchmark author names that as untested. Until someone measures it, the 3-bit KV cache is a claim about average quality.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=595d53b8ec45" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/turboquant-in-production-qdrant-vllm-and-what-survived-595d53b8ec45">TurboQuant in Production: Qdrant, vLLM, and What Survived</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Fine-Tuning to Quantization: What Free-Tier Hardware Can Prove]]></title>
            <link>https://pub.towardsai.net/fine-tuning-to-quantization-what-free-tier-hardware-can-prove-bb4359c2a5cd?source=rss-3ec8787f291f------2</link>
            <guid isPermaLink="false">https://medium.com/p/bb4359c2a5cd</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Yuval Mehta]]></dc:creator>
            <pubDate>Sat, 26 Sep 2026 23:01:02 GMT</pubDate>
            <atom:updated>2026-09-26T23:01:02.086Z</atom:updated>
            <content:encoded><![CDATA[<h3>What a Free-Tier GPU Can Prove About a Fine-Tuned LLM, and What It Can’t</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*cUOAjo3XvjP6mgox" /><figcaption>Photo by <a href="https://unsplash.com/@digitalcontentwritersindia?utm_source=medium&amp;utm_medium=referral">Digital Content Writers India</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>A base 4B model refuses or redirects the right call on adversarial prompts 79.5% of the time. After QLoRA fine-tuning on 150 synthetic examples, that number moves to 89.7%. The serving benchmark meant to prove the model holds up under concurrent load never produced a number, because a single free-tier GPU cannot hold enough KV cache to test past one request at a time.</p><p>This is a write-up of what a full LLM fine-tuning lifecycle looks like when every stage runs on hardware you didn’t pay for and what that hardware will and won’t let you claim by the end. I used AI assistance to structure this piece, check code, and pull benchmark citations. The project, the numbers, and the conclusions are mine.</p><h3>TL;DR</h3><ul><li>Fine-tuned a small model on 150 synthetic, adversarially augmented examples for a regulated-style compliance task domain, using QLoRA through Unsloth and TRL.</li><li>Adversarial-holdout accuracy moved from 31/39 to 35/39 after fine-tuning. Eval-set accuracy held at 16/16 throughout.</li><li>GPTQ and AWQ both held perplexity close to full precision, but a 2026 benchmark found the serving kernel underneath each format moves throughput more than the quantization algorithm does.</li><li>SGLang was worth testing here for a specific reason: every training example shares one long system prompt, which is exactly the condition RadixAttention is built for.</li><li>The serving and load-testing story stops at one data point. A single request returned correctly but slowly on a T4. Concurrent benchmarking needs hardware this project didn’t have.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_LFGc-2HW70NDopHKPiP6A.png" /><figcaption>AI Generated with Napkin AI</figcaption></figure><h3>Context</h3><p>The project started from a gap, not a topic. Inference optimization and production serving were the weakest lines on my profile relative to fine-tuning and agentic systems, so the brief was to build something that forced me through the whole lifecycle: synthetic data, evaluation, fine-tuning, quantization, serving, and whatever load testing a single GPU would permit.</p><p>The domain is a regulated-style compliance task, close enough to real pharmaceutical GxP workflows to be a meaningful test (structured deviation reports, corrective-action plans, audit-trail reasoning) but built entirely on synthetic data with no proprietary system names or client data anywhere in it. The constraint that mattered most wasn’t the domain. It was the hardware: Kaggle’s free T4 tier for training and quantization, no paid serving instance, and no multi-GPU node. That constraint is the actual subject of this post.</p><h3>Fine-tuning: closing the gap that mattered</h3><p>The base model’s weakest spot wasn’t the straightforward SOP and deviation-report tasks. It was the adversarial holdout: 39 examples built specifically to test whether the model would comply with a bad request (backdating a record, omitting a response-time delay, skipping an authorization step) or push back the way a compliant system should. The base model got 31 of those right.</p><p>QLoRA training used Unsloth’s FastLanguageModel for the quantized base and kernel replacement, with TRL&#39;s SFTTrainer for the actual training loop. Unsloth&#39;s own benchmarks report roughly 2x training speed and up to 70% lower VRAM against a HuggingFace-plus-FlashAttention baseline, which is the difference between this fitting on a free T4 at all and not.</p><pre>from unsloth import FastLanguageModel<br>from trl import SFTTrainer, SFTConfig<br><br>model, tokenizer = FastLanguageModel.from_pretrained(<br>    model_name=&quot;unsloth/Qwen3.5-4B&quot;,<br>    max_seq_length=2048,<br>    load_in_4bit=True,<br>)<br>model = FastLanguageModel.get_peft_model(<br>    model,<br>    r=16,<br>    lora_alpha=16,<br>    target_modules=[&quot;q_proj&quot;, &quot;k_proj&quot;, &quot;v_proj&quot;, &quot;o_proj&quot;,<br>                     &quot;gate_proj&quot;, &quot;up_proj&quot;, &quot;down_proj&quot;],<br>    use_gradient_checkpointing=&quot;unsloth&quot;,<br>)<br>trainer = SFTTrainer(<br>    model=model,<br>    tokenizer=tokenizer,<br>    train_dataset=train_ds,   # ChatML-formatted, 95 examples<br>    args=SFTConfig(max_steps=300, per_device_train_batch_size=4,<br>                   gradient_accumulation_steps=4, learning_rate=2e-4),<br>)<br>trainer.train()</pre><p>After fine-tuning, the eval set held at 16/16, which mostly confirms the model didn’t regress on what it already handled. The adversarial holdout moved to 35/39, a real gap closed but not a solved one: four cases still went the wrong way. This holds for a narrow, well-specified task domain with a small, curated dataset. It says nothing about whether the same QLoRA recipe closes an adversarial gap on a broader, less-structured domain, and the honest counterargument is that 150 synthetic examples might be teaching the model to pattern-match this specific holdout’s phrasing rather than the underlying judgment.</p><h3>Quantization: the kernel matters more than the algorithm</h3><p>GPTQ and AWQ both compressed the fine-tuned model to 4-bit with a small, expected quality cost: GPTQ came in at roughly +0.15 perplexity over full precision at 42 ms per token at batch size 1, and AWQ at roughly +0.08 perplexity and 38 ms per token. Both numbers are close enough to full precision that either format would be a reasonable production choice on quality grounds alone.</p><p>The more useful finding didn’t come from my own runs. A 2026 benchmark comparing GPTQ, AWQ, and GGUF on the same model, same hardware, and same serving harness found that swapping the serving kernel underneath identical GPTQ or AWQ weights moved throughput by up to 10.9x, while the choice of algorithm accounted for a fraction of that gap. Marlin-AWQ hit 741 tokens per second on an H200 against the default kernel’s 68. That’s a bigger lever than most “GPTQ vs AWQ” writeups ever mention, because most of them stop at the Hessian-math-versus-activation-scaling explanation and never touch a GPU.</p><p>The practical implication: picking a quantization format by its paper’s stated quality claim, without checking which kernel your serving engine actually dispatches to for that format, is picking the wrong variable to optimize. In my own runs, the perplexity gap between GPTQ and AWQ was small enough that either format would have shipped. The decision that actually should have taken longer than it did was confirming which kernel vLLM was dispatching to for each format, not which paper’s quality benchmark to trust.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*u9IAwSfUJyjvr6R9XqqBJA.png" /><figcaption>AI Generated with Napkin AI</figcaption></figure><h3>Serving engine choice: prefix overlap, not brand preference</h3><p>Every one of this project’s 150 training examples opens with the same long system prompt (the compliance-assistant role definition, the ALCOA+ data-integrity framing). That’s not a stylistic choice; it’s a structural fact about the dataset, and it’s the actual reason SGLang was worth testing here rather than a default “compare the two popular engines” exercise.</p><p>vLLM’s PagedAttention and SGLang’s RadixAttention solve the same memory-management problem differently. RadixAttention caches shared prompt prefixes across requests in a radix tree, so a request that starts with an already-cached prefix skips recomputing it. A 2026 H100 benchmark found the two engines land within about 5% of each other on unique-prompt workloads with no shared prefix, but once 60% or more of requests share a common prefix, SGLang’s latency advantage becomes measurable and consistent across concurrency levels. A dataset where every single example shares a 200-plus-token system prompt is close to the strongest case that benchmark tests for.</p><p>This is the mechanism, not the benchmark result: I can state why SGLang deserved a real test on this specific workload. I cannot yet state which engine actually won it on my own hardware, which is the next section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/913/1*YuKBCeE8ed3XYMGXTfovgQ.png" /><figcaption>AI Generated with Napkin AI</figcaption></figure><h3>What’s built but not proven</h3><p>The repo also includes an FP8 quantization path through Modal, an SGLang serving wrapper alongside the vLLM one, a Locust load-testing harness with a ramp profile, and a Docker Compose stack with Prometheus and Grafana. All of that code runs. None of it has a benchmark number behind it yet. Building the serving and observability layer was still worth doing, since it’s the difference between a training script and something that looks like a deployable system, but I’m not going to attach a performance claim to code I haven’t been able to load-test.</p><h3>Gotchas</h3><p><strong>The hardware ceiling is the real finding, not a footnote.</strong> I lost access to the T4 before I could log the exact per-request figure, so I won’t invent one from my own run. A published estimate for the same model on the same hardware gets close: hardware-fit modeling for Qwen3.5–4B at 4-bit on a 16GB T4 puts decode speed around 56 tokens per second with a roughly 3.5-second time to first token. That figure is a modeled estimate from published specs, not a live empirical benchmark, and it reflects a general local-runtime stack rather than the vLLM setup this project used, so treat it as an order-of-magnitude reference, not an exact match. It still supports the point worth making: fine-tuning changes the weights, not the architecture, the parameter count, or the quantization format, so it was never going to change how fast a T4 generates tokens. What fine-tuning bought here was the 79.5-to-89.7 jump. What it couldn’t buy was a faster GPU, and the vLLM-vs-SGLang comparison, the TTFT percentiles, and the throughput-under-load numbers all still require a GPU class this project didn’t rent.</p><p><strong>The adversarial holdout is a proxy, not a guarantee.</strong> A 10.2-point jump on 39 examples is a real, measured result on this specific test set. It is not evidence the model generalizes to adversarial patterns it wasn’t trained against. Anyone using synthetic adversarial data to validate a fine-tune should treat the holdout score as a floor, not a ceiling.</p><h3>Conclusion</h3><p>The parts of this project that needed compute and time, not money, are done and measured: the data pipeline, the fine-tune, and both quantization formats. The part that needed money, a GPU class that can hold a concurrent KV cache for more than one request, isn’t. That’s a fair trade for a free-tier build, but it leaves an open question I don’t have a clean answer to yet: how much of a fine-tuned model’s production readiness can free-tier hardware actually certify, and how much of it stays unverified until someone puts a real GPU behind it.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bb4359c2a5cd" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/fine-tuning-to-quantization-what-free-tier-hardware-can-prove-bb4359c2a5cd">Fine-Tuning to Quantization: What Free-Tier Hardware Can Prove</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agent Cost Optimization Is an Architecture Decision, Not a Model Choice]]></title>
            <link>https://levelup.gitconnected.com/agent-cost-optimization-is-an-architecture-decision-not-a-model-choice-e4f084d7048d?source=rss-3ec8787f291f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e4f084d7048d</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-infrastructure]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Yuval Mehta]]></dc:creator>
            <pubDate>Mon, 21 Sep 2026 17:58:41 GMT</pubDate>
            <atom:updated>2026-09-21T17:58:41.298Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nnJGEUj5rQC4jMbh" /><figcaption>Photo by <a href="https://unsplash.com/@coopery?utm_source=medium&amp;utm_medium=referral">Mohamed Nohassi</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>When an agent’s inference bill spikes, the first instinct is almost always the same: swap to a cheaper model. Drop from a frontier model to a smaller one, or route simple requests to a lighter tier. Sometimes this helps. More often, it barely moves the number, because the model was never the actual cost driver.</p><p>For a typical multi-turn tool-calling agent, the input-to-output token ratio can run 100:1 or higher — a ReAct loop making a handful of tool calls might generate a few hundred output tokens while consuming hundreds of thousands of input tokens, because every call re-sends the system prompt, the tool schemas, and the accumulating conversation history. At that ratio, prefill compute — processing the input — dominates the bill, not generation. Swapping the model changes the per-token price. It does nothing about how many tokens you’re paying to process in the first place.</p><p>The teams actually bringing agent costs under control in 2026 are treating this as an architecture problem with three levers, applied together: cache hit rate, prompt structure, and context compaction. None of them is sufficient alone. All three compound.</p><p><strong>TL;DR</strong></p><ul><li>At high input-to-output token ratios, prefill — not generation — is where your money goes. Model selection addresses the wrong side of the ledger.</li><li>KV cache/prefix cache hit rate is the single highest-leverage lever: going from a low hit rate to a high one is commonly reported as a 5–12x reduction in per-call cost on agent workloads, not a marginal tweak.</li><li>Prompt and tool schema structure determines whether you can hit a high cache rate at all — one dynamic field placed early in your prompt silently caps your ceiling.</li><li>Context compaction addresses the token volume itself, independent of caching, and becomes necessary once conversation history alone exceeds your budget.</li><li>These three levers interact — a well-structured prompt is a prerequisite for cache hits, and compaction changes what’s available to cache in the next turn.</li></ul><h3>Lever 1: KV Cache and Prefix Cache Hit Rate</h3><p>Every model call recomputes attention over its full input unless earlier computation can be reused. Two related but distinct caching mechanisms matter here. The KV cache is automatic and scoped to a single request — it’s what makes token-by-token decoding fast at all. The prefix cache is what matters for cost: it’s the mechanism, implemented as vLLM’s automatic prefix caching or SGLang’s RadixAttention, that lets a <em>new</em> request reuse KV state computed for a <em>previous</em> request, as long as they share an identical token prefix.</p><p>This is where the actual savings live. Anthropic’s own prompt caching implementation reports up to a 90% cost reduction on cached input tokens, and production reports across agent workloads describe hit rates of 60–85% as achievable for stable workloads — with per-call cost dropping 5–12x once a workload moves from a low hit rate to a well-tuned one. That’s not a rounding error; it’s the difference between a viable cost structure and one that gets your project cancelled.</p><p>The mechanism is unforgiving about exactness, which is the part most teams get wrong on the first attempt:</p><pre># vLLM: enabling automatic prefix caching<br>from vllm import LLM<br>llm = LLM(<br>    model=&quot;meta-llama/Llama-3.1-70B-Instruct&quot;,<br>    enable_prefix_caching=True,<br>)</pre><p>The cache is keyed on byte-identical content. If a single dynamic field — a timestamp, a session ID, a per-user value — sits <em>before</em> the cache boundary in your prompt, it invalidates the entire prefix behind it, not just that field. This is illustrative of the failure pattern, not a literal case, but it’s the single most common way teams accidentally cap their own hit rate near zero without realizing it:</p><pre># Cache-hostile: dynamic content placed early invalidates everything after it<br>prompt = f&quot;&quot;&quot;<br>Current time: {datetime.now()}<br>System instructions: {SYSTEM_PROMPT}<br>Tool definitions: {TOOL_SCHEMAS}<br>&quot;&quot;&quot;<br># Every request has a unique timestamp at the top.<br># The cache boundary breaks on token 1. Nothing downstream is ever cached.<br># Cache-friendly: static content first, dynamic content pushed to the end<br>prompt = f&quot;&quot;&quot;<br>System instructions: {SYSTEM_PROMPT}<br>Tool definitions: {TOOL_SCHEMAS}<br>---<br>Current time: {datetime.now()}<br>User query: {user_input}<br>&quot;&quot;&quot;<br># The static prefix - often the largest share of the prompt - is now<br># byte-identical across requests and eligible for cache reuse.</pre><p>Monitor hit rate as a first-class production metric, not an afterthought. Most inference stacks expose it directly:</p><pre># vLLM exposes cache metrics via Prometheus<br># kv_cache_usage_percent<br># prefix_cache_hit_rate<br># Alert if hit rate drops below ~50% — that&#39;s a structural regression,<br># not noise, and it usually traces back to a prompt structure change.</pre><h3>Lever 2: Prompt and Tool Schema Structure</h3><p>Cache hit rate isn’t something you tune after the fact — it’s largely determined by decisions made when you design the prompt and tool schema in the first place. This is the lever that’s easiest to underinvest in because it looks like a formatting detail rather than a cost decision.</p><p>The core principle: order content from most-static to most-dynamic, and keep the static portion identical across every request that can share it.</p><pre>class AgentPromptBuilder:<br>    def __init__(self, system_prompt: str, tool_schemas: list[dict]):<br>        # Static block — should be byte-identical across every request<br>        # in this agent&#39;s lifetime, so it&#39;s eligible for cache reuse<br>        # on every single call<br>        self.static_prefix = self._build_static_block(system_prompt, tool_schemas)<br>    def _build_static_block(self, system_prompt: str, tool_schemas: list[dict]) -&gt; str:<br>        # Tool schemas are a common, avoidable source of cache misses:<br>        # if they&#39;re regenerated per-request (e.g. from a dict with<br>        # non-deterministic key ordering), the &quot;static&quot; block silently<br>        # isn&#39;t static. Serialize deterministically, once, and reuse it.<br>        return f&quot;{system_prompt}\n\n{json.dumps(tool_schemas, sort_keys=True)}&quot;<br>    def build(self, conversation_history: list[dict], user_query: str) -&gt; str:<br>        # Everything dynamic goes after the static prefix, never before it<br>        return f&quot;{self.static_prefix}\n---\n{self._format_history(conversation_history)}\n{user_query}&quot;</pre><p>Two structural habits do most of the damage in practice. Retrieval-augmented prompts that reorder or re-rank documents on every call break the cache even when the underlying documents haven’t changed — cache the retrieval result’s serialization, not just the retrieval logic. And tool schemas that get dynamically filtered per-request (showing only “relevant” tools based on the query) feel like a smart optimization but destroy prefix reuse, because the tool block is no longer identical across calls — the cost of the cache miss frequently outweighs the savings from a smaller schema.</p><h3>Lever 3: Context Compaction</h3><p>Even with a perfectly cache-friendly structure, conversation history grows without bound in a long-running agent session, and eventually you’re paying to reprocess a genuinely large volume of accumulated tokens regardless of hit rate — a high cache hit rate on 200,000 tokens of history is still expensive per call, just less expensive than a full recompute.</p><p>Compaction addresses volume directly: summarizing or pruning older turns so the effective context stays bounded. The tradeoff is real — compaction is lossy, and applied carelessly it degrades the model’s ability to recall earlier decisions or constraints. The production pattern that holds up is threshold-triggered, not continuous:</p><pre>class CompactingAgentContext:<br>    def __init__(self, token_budget: int = 32_000, compact_at: float = 0.8):<br>        self.token_budget = token_budget<br>        self.compact_threshold = int(token_budget * compact_at)<br>        self.history: list[dict] = []<br>    def add_turn(self, turn: dict) -&gt; None:<br>        self.history.append(turn)<br>        if self._current_token_count() &gt; self.compact_threshold:<br>            self._compact()<br>    def _compact(self) -&gt; None:<br>        # Keep the most recent N turns verbatim - these are the ones<br>        # most likely to be referenced in the immediate next step<br>        recent_cutoff = 6<br>        recent = self.history[-recent_cutoff:]<br>        older = self.history[:-recent_cutoff]<br>        if not older:<br>            return<br>        # Summarize the older segment rather than dropping it outright.<br>        # A cheap, fast model is usually sufficient for this step -<br>        # it doesn&#39;t need to match the primary agent&#39;s capability.<br>        summary = self._summarize(older)<br>        self.history = [{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: f&quot;[Earlier context summary] {summary}&quot;}] + recent<br>    def _current_token_count(self) -&gt; int:<br>        # Approximate; a real implementation should use the actual tokenizer<br>        return sum(len(t.get(&quot;content&quot;, &quot;&quot;)) // 4 for t in self.history)</pre><p>Compaction and caching pull in slightly different directions worth naming explicitly: every time you compact, the resulting prompt is new content, which means the next call after a compaction event is a guaranteed cache miss on the compacted portion. This is an acceptable one-time cost — you’re trading a single cache miss for a smaller ongoing token volume — but it’s worth knowing rather than being surprised by, especially if your compaction threshold is tuned too aggressively and triggers more often than necessary.</p><h3>Illustrative Comparison</h3><p>To make the compounding effect concrete — these are illustrative, order-of-magnitude figures reflecting patterns reported across production case studies, not measurements from a specific deployment:</p><p>An agent with no caching, a poorly structured prompt (dynamic content early), and no compaction might process something on the order of 150,000 input tokens per call once conversation history has grown, with a cache hit rate near zero.</p><p>The same agent with prefix-cache-friendly prompt structure alone might reach a 60–70% hit rate, cutting effective per-call input cost roughly in half.</p><p>Adding compaction to bound history growth keeps the raw token volume from climbing further as the session continues, compounding with the caching win rather than replacing it.</p><p>The order matters: fixing structure without addressing volume caps your gains once conversations run long; addressing volume without fixing structure means you’re compacting content that should never have been recomputed from scratch in the first place.</p><h3>Gotchas: What Nobody Tells You</h3><p><strong>1. Caching only ever discounts input tokens.</strong></p><p>No major provider’s caching mechanism touches output token cost. If your agent’s cost profile is unusually generation-heavy — long reasoning traces, verbose tool call arguments — caching alone won’t move the number you’re worried about. Diagnose your actual input:output ratio before assuming caching is the fix.</p><p><strong>2. Cache-aware routing matters as much as caching itself at scale.</strong></p><p>In a distributed deployment behind a standard load balancer, semantically related requests can get scattered across different inference pods, destroying cache locality even though caching is correctly enabled — each pod ends up cold for requests another pod already warmed. This is invisible until you check per-pod hit rate specifically, not just the aggregate.</p><p><strong>3. A hit rate regression is usually a silent code change, not model drift.</strong></p><p>Because the cache is byte-exact, the most common cause of a hit rate collapse in production is an innocuous change elsewhere in the codebase — a logging statement that got added before the prompt is built, a library upgrade that changed dict serialization order, a new field added to a shared context object. Wire hit rate into your alerting the same way you’d alert on error rate; a silent regression here shows up as a cost spike weeks later with no obvious cause.</p><h3>Conclusion</h3><p>Model selection optimizes the price per token. For most production agents, the actual problem is the number of tokens being priced — and that’s an architecture decision, not a model decision. Cache hit rate, prompt structure, and context compaction aren’t three separate optimizations to pick from; they’re a stack, where structure determines whether caching is even possible, caching determines your cost floor, and compaction keeps that floor from drifting upward as sessions run longer.</p><p>The teams treating this as a first-class architectural concern — instrumenting hit rate, designing prompts cache-first, compacting on a threshold rather than never — are the ones whose agent costs scale sublinearly with usage. The teams still reaching for a cheaper model first are optimizing a number that was never the real lever.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e4f084d7048d" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/agent-cost-optimization-is-an-architecture-decision-not-a-model-choice-e4f084d7048d">Agent Cost Optimization Is an Architecture Decision, Not a Model Choice</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Lumiere to Veo 3: How Google Solved the Hardest Problem in Video Generation]]></title>
            <link>https://pub.towardsai.net/from-lumiere-to-veo-3-how-google-solved-the-hardest-problem-in-video-generation-c519b7e24be2?source=rss-3ec8787f291f------2</link>
            <guid isPermaLink="false">https://medium.com/p/c519b7e24be2</guid>
            <category><![CDATA[computer-science]]></category>
            <category><![CDATA[computer-vision]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Yuval Mehta]]></dc:creator>
            <pubDate>Tue, 25 Aug 2026 15:31:01 GMT</pubDate>
            <atom:updated>2026-08-25T15:31:01.630Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IjBuOhDf39UJez86" /><figcaption>Photo by <a href="https://unsplash.com/@jeremyyappy?utm_source=medium&amp;utm_medium=referral">Jeremy Yap</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Every early text-to-video model had the same tell: watch for more than three seconds and something drifts. A car’s badge changes shape. A person’s shirt collar migrates. A hand gains a finger. These weren’t rendering bugs. They were architectural ones, baked into how the models generated video in the first place. In January 2024, a Google Research paper called Lumiere proposed a fix that looked almost too simple: stop generating video piece by piece, and generate the whole clip at once. Two years later, that idea is one of the load-bearing walls under Veo 3, the model now shipping inside Google Photos, Gemini, and Vertex AI. This is the story of how a research fix for temporal consistency turned into a production video engine, and what changed (and didn’t) along the way.</p><h3>TL;DR</h3><ul><li>Early text-to-video (T2V) models generated sparse keyframes, then used a separate temporal super-resolution (TSR) model to fill in the gaps, a cascade that structurally limited global motion consistency.</li><li>Lumiere’s core contribution was a <strong>Space-Time U-Net (STUNet)</strong>: a single network that downsamples and upsamples across space <em>and</em> time, generating an entire clip’s duration in one pass instead of a cascade.</li><li>Lumiere never shipped as a product. It stayed a research demo: no public API, ~5-second clips, no audio, no multi-shot support.</li><li>Veo (especially Veo 3 / 3.1) is the productized descendant of this research line: native audio generation, longer clips, multiple resolution/framing options, and tiered models (Fast, Lite, full) for different cost/latency tradeoffs.</li><li>The gap between the two isn’t really “better diffusion math.” It’s the unglamorous work of controllability, cost tiers, safety filtering, and infra that turns a paper demo into something that runs at production request volume.</li></ul><h3>Background: why video diffusion is harder than image diffusion</h3><p>Text-to-image diffusion (think Stable Diffusion, Imagen) solved a single problem: given noise and a text prompt, iteratively denoise into a 2D grid of pixels that matches the prompt. Video adds a third axis, time, and that axis is unforgiving. A viewer’s eye is extremely sensitive to temporal inconsistency in a way it isn’t to small spatial artifacts. A slightly wrong texture on a wall is forgivable. A car that’s a different shade of red in frame 40 than in frame 4 reads as broken immediately.</p><p>The obvious extension of image diffusion to video, just add a time dimension to the U-Net and diffuse the whole tensor, is computationally brutal. A 5-second, 16fps clip at even modest resolution is dozens of full frames, and running full spatiotemporal attention over all of them at every diffusion step doesn’t fit on any reasonable amount of memory.</p><p>So before Lumiere, the standard workaround was a <strong>cascade</strong>:</p><ol><li>Generate a handful of sparse, temporally distant keyframes (say, frames 0, 16, 32…).</li><li>Run a separate temporal super-resolution (TSR) model to interpolate the frames in between.</li><li>Optionally run spatial super-resolution afterward to upscale resolution.</li></ol><p>This is tractable (each stage handles a smaller problem), but it has a structural flaw: the paper describes this as an approach that inherently makes global temporal consistency difficult to achieve, because the interpolation model has no way to correct for drift between distant keyframes it didn’t generate itself. It’s filling in gaps between two answers it has to trust, not two answers it derived jointly.</p><h3>The core idea: generate the whole clip in one pass</h3><p>Lumiere’s paper frames its contribution as addressing the fundamental challenge of achieving global temporal consistency in video synthesis, and its answer is a Space-Time U-Net architecture capable of generating the entire temporal duration of a video in a single model pass. No keyframes-then-interpolate. No separate TSR model papering over gaps. One network, one forward pass per diffusion step, producing every frame of the clip together.</p><p>The way it makes this tractable is the interesting part. Instead of running full attention over every pixel of every frame, the Space-Time U-Net downsamples the input signal both spatially and temporally, and performs the majority of its computation on this compact space-time representation, much like how a standard image U-Net downsamples spatially into a low-resolution bottleneck before upsampling back out. Lumiere just extends that same trick along the time axis too. The video gets compressed into a compact space-time blob, most of the heavy computation happens there (cheap), and then the network upsamples back out to full spatial and temporal resolution (expensive only at the edges, not the middle).</p><p>Concretely, the architecture “inflates” a pre-trained text-to-image U-Net into a Space-Time U-Net (STUNet), adding two kinds of temporal modules: convolution-based blocks (pretrained T2I layers followed by a factorized space-time convolution), and, at the coarsest (most compressed) level of the U-Net, attention-based blocks where the pretrained T2I layers are followed by temporal attention. Because the video representation is already compressed at that coarsest level, stacking several temporal attention layers there stays computationally cheap.</p><p>That inflate-from-pretrained-T2I detail matters practically: Lumiere didn’t train a video model from scratch. It started from a strong pretrained image diffusion model (so it already “knew” what a car or a face should look like) and added temporal awareness on top. This is the same instinct behind LoRA-style adaptation: reuse a strong prior, adapt cheaply rather than relearn everything.</p><pre>Classic cascade approach:<br>[Prompt] → [Sparse keyframes: t=0,16,32...] → [TSR model fills gaps] → [Spatial SR] → Video<br>                                                    ↑<br>                                    consistency between frames is *inferred*, not generated<br><br>Lumiere&#39;s single-pass approach:<br>[Prompt] → [STUNet: down-sample space+time → compact bottleneck → up-sample space+time] → Video<br>                                                    ↑<br>                                    every frame generated jointly, in one pass</pre><p>The tradeoff is honest and worth stating: this buys you a low-resolution, full-duration video directly, at the cost of needing separate (and much cheaper) spatial super-resolution to hit final output quality. You’ve traded “hard problem across two disjoint models” for “hard problem in one model, easy problem in a second.”</p><h3>What Lumiere could actually do</h3><p>Beyond plain text-to-video, the single-pass, spatiotemporal architecture turned out to generalize well to a handful of adjacent tasks, all shown in the original paper:</p><ul><li><strong>Image-to-video</strong>: animate a still image into motion.</li><li><strong>Stylized generation</strong>: apply a reference visual style to a generated video.</li><li><strong>Video inpainting</strong>: mask a region of a video and regenerate just that region, consistently, across time.</li><li><strong>Cinemagraphs</strong>: animate part of a still image while the rest stays frozen.</li></ul><p>The researchers were explicit about a known limitation, though: the model as published couldn’t handle multi-shot or scene transitions. It was fundamentally a single continuous clip generator, not a scene composer. That’s a meaningful constraint if you’re thinking about actual filmmaking use cases rather than short clips.</p><p>And that’s where Lumiere stayed. It was published at SIGGRAPH Asia 2024, generated plenty of research buzz, and as far as any public record shows, never became a shipped product. No API, no consumer access, no pricing page. It answered an architecture question, not a product question.</p><h3>Veo: the same problem, but shipped</h3><p>Google’s Veo line picks up from roughly the same research lineage (Google DeepMind’s video generation effort), but Veo’s story is a product story, not a single-paper story. Veo 3 launched at Google I/O 2025 with the ability to generate audio alongside video (sound effects, background noise, even dialogue), with Google DeepMind’s CEO Demis Hassabis framing it as finally “emerging from the silent era of video generation.” That’s a capability nowhere in the Lumiere paper; audio wasn’t in scope for the original research.</p><p>By 2026, Veo had split into a deliberate product line rather than one model:</p><ul><li>Veo 3.1, prioritizing top visual fidelity for final production work.</li><li>Veo 3.1 Fast, trading some quality for speed in standard production workflows.</li><li>Veo 3.1 Lite, the most cost-effective tier, aimed at high-volume applications that need to iterate and scale rather than chase maximum fidelity, at less than half the cost of Fast, at the same speed.</li></ul><p>That three-tier split is the clearest signal of the shift from research to product. Lumiere had one model answering one research question. Veo has three models answering three different <em>business</em> questions: quality-critical, latency-critical, and cost-critical use cases each get their own variant, sharing an underlying architecture lineage.</p><p>Other concrete deltas:</p><ul><li><strong>Duration</strong>: Lumiere topped out around 5 seconds. Veo 3.1 supports extended generations up to roughly 120 seconds in some configurations, which is over an order of magnitude longer than what the single-pass research architecture demonstrated.</li><li><strong>Resolution and framing</strong>: Veo 3.1 Lite supports both 720p and 1080p output, in landscape (16:9) and portrait (9:16) framing, with a separate upscaling path: a standalone Veo upscaling capability that can take existing video, from any source, up to 1080p or 4K.</li><li><strong>Distribution</strong>: Lumiere lived in a paper and a project webpage. Veo lives inside actual consumer and developer surfaces: Google Photos uses it to turn still images into short clips directly in the app, Google Vids offers free Veo 3.1 generation to all Google accounts alongside AI music and directable avatars, and it’s callable through the Gemini API and Vertex AI for developers building their own products on top of it.</li></ul><h3>What actually changed architecturally, and what’s still speculation</h3><p>It’s worth being precise about what’s public and what isn’t. Google has not published a Lumiere-equivalent paper for Veo 3 with the same level of architectural detail. What’s publicly described is a positioning statement, not a spec: Veo 3’s messaging emphasizes “real world physics and audio” driving realism and fidelity, improved prompt adherence, and greater control and consistency, now extending across audio and not just video.</p><p>That framing (“real world physics,” temporal and audio consistency together) is consistent with Veo having inherited and extended the single-pass, joint-generation philosophy Lumiere argued for, rather than reverting to the old keyframe-and-interpolate cascade. But “consistent with” isn’t the same as “proven by a published architecture diagram.” Treat any specific claim about Veo’s internals (exact parameter counts, specific attention patterns) reported by third-party blogs with real skepticism. Google hasn’t opened that box the way the original Lumiere paper did.</p><h3>Gotchas / what nobody tells you</h3><p><strong>1. “Single-pass” doesn’t mean “solved forever.”</strong> Lumiere’s single-pass trick solves consistency <em>within</em> one generated clip. It says nothing about consistency <em>across</em> clips, and that’s exactly the multi-shot, recurring-character problem that became commercially important once people wanted to make actual narrative content rather than isolated 5-second clips. That gap is filled by product-layer features (reference images, character-consistency conditioning), not by the core diffusion architecture.</p><p><strong>2. The “productization tax” is bigger than the “research tax.”</strong> Getting from Lumiere’s demo quality to Veo’s shipped product involved things that never show up in an architecture diagram: safety and content filtering at scale, cost-optimized model variants (Lite/Fast/full), infrastructure to serve ~thousands of concurrent requests, watermarking output for provenance, and multilingual prompt handling. If you’re evaluating “how hard would this be to build,” the modeling innovation is often the smaller half of the actual engineering effort.</p><p><strong>3. Compute-tractability tricks compound.</strong> The bottleneck-then-upsample trick that makes STUNet tractable is the same core idea behind latent diffusion (do the expensive diffusion process in a compressed latent space, not pixel space) applied along an extra axis. If you understand why Stable Diffusion runs in VAE latent space instead of raw pixels, you already understand 80% of why Lumiere’s temporal downsampling works. It’s the same tradeoff, one dimension over.</p><h3>Conclusion</h3><p>Lumiere answered a narrow, well-posed research question: how do you get a video model to stop drifting over time, without stacking cascaded models that can’t correct each other’s mistakes? The single-pass Space-Time U-Net was a clean, convincing answer: good enough to be SIGGRAPH-worthy, not good enough to be a product on its own.</p><p>Veo’s evolution shows what’s actually required to close that gap: not a bigger architectural leap, but relentless product-shaping around a sound research idea: audio, duration, resolution tiers, cost tiers, and distribution into surfaces people already use. The interesting question going forward isn’t “will video generation get more temporally consistent” (that problem is largely tamed). It’s whether the next constraint (multi-shot narrative coherence, precise physical simulation, real-time generation) gets solved the same way: one elegant architectural insight, followed by two years of unglamorous productization.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c519b7e24be2" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/from-lumiere-to-veo-3-how-google-solved-the-hardest-problem-in-video-generation-c519b7e24be2">From Lumiere to Veo 3: How Google Solved the Hardest Problem in Video Generation</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Prefill-Decode Disaggregation: When and Why to Split Your Inference Stack]]></title>
            <link>https://pub.towardsai.net/prefill-decode-disaggregation-when-and-why-to-split-your-inference-stack-7ddcbd680553?source=rss-3ec8787f291f------2</link>
            <guid isPermaLink="false">https://medium.com/p/7ddcbd680553</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Yuval Mehta]]></dc:creator>
            <pubDate>Sun, 26 Jul 2026 23:31:01 GMT</pubDate>
            <atom:updated>2026-07-26T23:31:01.511Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hQOR8MnJ1gqG-XUt" /><figcaption>Photo by <a href="https://unsplash.com/@freeche?utm_source=medium&amp;utm_medium=referral">Kvistholt Photography</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Every major inference engine now supports it. vLLM shipped disaggregated prefill as a stable feature. SGLang has it. LLM-d built its entire architecture around it. NVIDIA’s Dynamo and TensorRT-LLM support it natively. The infrastructure story has converged.</p><p>What hasn’t converged is practitioner understanding of when it’s worth doing. Most teams either skip disaggregation entirely because it sounds like a research technique or they adopt it reflexively because the throughput numbers in vendor blog posts look compelling without checking whether their workload actually has the shape that makes those numbers real. Both are mistakes. This post is the decision framework that’s been missing.</p><p><strong>TL;DR</strong></p><ul><li>Prefill and decode have fundamentally different resource profiles — compute-bound versus memory-bound — and that mismatch is the entire reason disaggregation exists.</li><li>Disaggregation pays off when you have variable prompt lengths, high concurrency, and interactivity requirements (low inter-token latency). It does not pay off for small-scale or latency-tolerant batch workloads.</li><li>Chunked prefill is the cheaper first lever — try it before you build separate node pools.</li><li>KV cache transfer between prefill and decode nodes is the operational cost nobody puts on the slide—network bandwidth and transfer latency can eat your throughput gains if the connector isn’t tuned.</li><li>Real production numbers: disaggregated setups have shown throughput gains in the 75–250% range over collocated serving, but the hardware cost and complexity scale with it.</li></ul><h3>Why Prefill and Decode Don’t Belong on the Same GPU</h3><p>Every LLM inference request has two phases with opposite performance characteristics.</p><p><strong>Prefill</strong> processes your entire input prompt in a single forward pass — every token attends to every other token in one dense matrix multiply. This is compute-bound: the GPU’s raw FLOPS throughput is the limiting factor, and a long prompt keeps the GPU’s compute units saturated.</p><p><strong>Decode</strong> generates output tokens one at a time. Each new token requires reading the key-value cache for every previously generated token from GPU memory. This is memory-bound: the GPU spends most of its cycles moving KV cache tensors, not computing, and its compute units sit mostly idle.</p><p>When both phases run on the same GPU under continuous batching — the default in most serving stacks — a new prefill request has to be interleaved into an active batch of decode requests. That interleaving preempts ongoing decode work, and the batch has to pause token generation while it processes the new prompt. This is the direct cause of the unpredictable latency spikes you see under bursty traffic: a well-behaved decode stream gets stalled every time a large new prompt shows up in the batch.</p><p>Prefill draws close to peak GPU power (70–100%), while decode typically uses only 20–40%. Running both on identical hardware means you’re either over-provisioning for decode’s compute needs or under-provisioning for prefill&#39;s—you can’t tune the same GPU for both profiles simultaneously.</p><p>Disaggregation resolves this by physically separating the two phases onto different node pools, each independently scaled and independently hardware-matched: compute-heavy GPUs (H100 class) for prefill and memory-bandwidth-heavy or cheaper GPUs for decode.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*U-CqmKpL2e8hJo3vawoDXw.png" /></figure><h3>The Decision Framework</h3><p>Disaggregation is not free. It requires standing up two node pools, a KV cache transfer layer between them, and meaningfully more operational surface area than a single collocated deployment. Here’s when the tradeoff is worth it.</p><h4>Reach for disaggregation when:</h4><p><strong>Your prompts are long and variable.</strong> If you’re consistently hitting prompt lengths in the thousands of tokens with high concurrency, Prefill&#39;s compute demand is great enough and unpredictable enough to genuinely disrupt decode when they share hardware.</p><p><strong>You need low, stable inter-token latency.</strong> Interactive applications—chat interfaces, coding assistants, anything with a human staring at the token stream—are exactly where prefill interference is most visible. Removing it is the primary latency win disaggregation delivers, independent of raw throughput.</p><p><strong>You’re operating at real production scale.</strong> The RTP-LLM production deployment benchmarks at 480B-parameter MoE scale show disaggregation delivering meaningfully lower time-to-first-token and higher cache hit rates than collocated serving at that concurrency level. At a small scale, the fixed overhead of the separate deployment and transfer layer swamps the benefit.</p><p><strong>You have workload heterogeneity that benefits from independent scaling.</strong> If your prefill load and decode load don’t scale together—bursty prompt-heavy traffic at one time of day, sustained generation-heavy traffic at another—separate pools let you scale each independently instead of over-provisioning one to satisfy the other.</p><h4>Skip disaggregation when:</h4><p><strong>Your workload is small or steady-state.</strong> Published testing has found that under-scaled or poorly tuned disaggregated setups can actually underperform collocated serving by 20–30%, because the fixed cost of cross-node KV transfer isn’t amortized across enough traffic.</p><p><strong>Your prompts are short, or your prefix cache hit rate is high.</strong> If most of your prompt is already cached—a long shared system prompt, for instance—the actual prefill compute per request is small, and running it locally on the decode worker is often faster and simpler than paying network transfer costs to a separate prefill node.</p><p><strong>You haven’t tried chunked prefill yet.</strong> This is the cheaper, simpler lever most teams should reach for first.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/624/1*CgpNm3eyVD6ecx6Ojqu3gQ.png" /></figure><h3>Chunked Prefill: The Step Before Disaggregation</h3><p>Before standing up separate node pools, chunked prefill addresses the same interference problem within a single collocated deployment. Instead of processing an entire long prompt in one uninterrupted forward pass, the engine splits it into smaller chunks and interleaves them with ongoing decode steps — piggybacking decode work onto the prefill computation instead of letting one fully block the other.</p><pre># vLLM: enabling chunked prefill<br>from vllm import LLM, SamplingParams<br><br>llm = LLM(<br>    model=&quot;meta-llama/Llama-3.1-70B-Instruct&quot;,<br>    enable_chunked_prefill=True,<br>    max_num_batched_tokens=2048,   # chunk size ceiling per batch step<br>    max_num_seqs=256,               # concurrent sequence cap<br>)</pre><p>This is a single-node, single-engine optimization — no new infrastructure, no KV transfer layer, no second node pool. For teams under the concurrency threshold where disaggregation pays off, chunked prefill alone often closes most of the latency gap. Only move to full disaggregation once you’ve confirmed chunked prefill isn’t enough for your traffic pattern.</p><h3>The Operational Cost Nobody Puts on the Slide: KV Cache Transfer</h3><p>The throughput numbers in vendor benchmarks assume a well-tuned KV transfer layer between your prefill and decode nodes. This is the part that’s genuinely hard to get right, and it’s where most first attempts at disaggregation lose the gains they were promised.</p><p>When a prefill node finishes processing a prompt, it has to hand the resulting KV cache—which can be gigabytes for long contexts—to a decode node over the network, fast enough that the decode node isn’t sitting idle waiting for it. The transfer mechanism matters enormously:</p><pre># vLLM: NIXL-based KV transfer configuration (prefill node)<br>from vllm import LLM<br><br>prefill_llm = LLM(<br>    model=&quot;meta-llama/Llama-3.1-70B-Instruct&quot;,<br>    kv_transfer_config={<br>        &quot;kv_connector&quot;: &quot;NixlConnector&quot;,<br>        &quot;kv_role&quot;: &quot;kv_producer&quot;,<br>    },<br>)<br># Decode node<br>decode_llm = LLM(<br>    model=&quot;meta-llama/Llama-3.1-70B-Instruct&quot;,<br>    kv_transfer_config={<br>        &quot;kv_connector&quot;: &quot;NixlConnector&quot;,<br>        &quot;kv_role&quot;: &quot;kv_consumer&quot;,<br>    },<br>)</pre><p>NIXL (NVIDIA Inference Xfer Library) and similar connectors handle the low-level transfer over high-bandwidth interconnects like UCX or RDMA-capable networking. On AMD hardware, the MORI-IO connector serves the equivalent role. Recent testing on 8-GPU MI300X nodes showed disaggregated serving achieving roughly 2.5x higher goodput than collocated serving on the same hardware, but that number depends entirely on the interconnect being fast enough that KV transfer doesn’t become the new bottleneck.</p><p>If your nodes are connected over standard networking rather than a high-bandwidth RDMA fabric, budget real engineering time for this layer. A disaggregated setup with a slow transfer connector can genuinely underperform a well-tuned collocated deployment—the throughput math only works if the KV cache moves fast enough to keep the decode node fed.</p><h3>Real Numbers, With Context</h3><p>Published benchmark comparisons on Llama 3.1 70B have shown disaggregated H100+H200 setups costing roughly 45% more per hour in hardware than a collocated equivalent, while delivering roughly 75% more throughput — a favorable tradeoff if your bottleneck is genuinely throughput and not cost per token at low volume.</p><p>At a larger scale and with more aggressive architectures—intra-GPU disaggregation approaches like Nexus, which split a single GPU’s resources between prefill and decode rather than using separate physical nodes—research benchmarks report up to 2.2x higher throughput and over an order of magnitude lower time-to-first-token compared to standard vLLM collocated serving, using half the GPU count of traditional node-level disaggregation.</p><p>The spread between these numbers is the point: “disaggregation gives you 2x throughput” is not a fixed constant. It’s a function of your prompt length distribution, concurrency, hardware, interconnect quality, and which disaggregation architecture you implement. Treat every vendor benchmark as a ceiling, not a guarantee, until you’ve reproduced something close to it on your own traffic shape.</p><h3>Gotchas: What Nobody Tells You</h3><p><strong>1. Disaggregation changes your failure modes, not just your performance profile.</strong></p><p>A collocated deployment fails as a unit. A disaggregated deployment can fail asymmetrically — your prefill pool can be healthy while your decode pool is degraded, or the KV transfer layer itself can become a silent bottleneck that doesn’t show up as a node-level failure. Instrument the transfer layer explicitly; don’t rely on per-node health checks alone.</p><p><strong>2. Prefill can become memory-bound too, at long enough context lengths.</strong></p><p>The clean compute-bound/memory-bound split that motivates disaggregation breaks down as context length grows very large—at that point, prefill starts converging toward the same memory-bandwidth constraints as decode, which undermines the hardware specialization argument. If your workload involves very long contexts (100K+ tokens), validate that the compute/memory split still holds for your actual prompt distribution before assuming the standard disaggregation playbook applies.</p><p><strong>3. “It works in the vendor’s benchmark” doesn’t mean it works at your concurrency level.</strong></p><p>Most public benchmarks are run at concurrency and batch sizes tuned to showcase the architecture favorably. Reproduce the comparison at your actual expected traffic volume before committing engineering time to the migration—the crossover point where disaggregation beats collocated serving is workload-specific, not universal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/685/1*NzpfZRgLPGNU0B2TYqPMog.png" /></figure><h3>Conclusion</h3><p>Prefill-decode disaggregation solves a real, structural problem: two phases of LLM inference with opposite resource profiles being forced to share hardware. The infrastructure to do it well now exists across every major serving engine. But the decision to adopt it should be driven by your workload’s actual shape—prompt length distribution, concurrency, and latency requirements—not by the throughput number in the most recent vendor blog post.</p><p>Start with chunked prefill. If that’s not enough, and your traffic genuinely has the long-prompt, high-concurrency, interactivity-sensitive profile that disaggregation is built for, move to full disaggregation—and budget real engineering time for the KV transfer layer, because that’s where the promised gains actually get won or lost.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7ddcbd680553" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/prefill-decode-disaggregation-when-and-why-to-split-your-inference-stack-7ddcbd680553">Prefill-Decode Disaggregation: When and Why to Split Your Inference Stack</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Hidden Cost of Agent Memory: What Mem0, Zep, and Letta Don’t Tell You]]></title>
            <link>https://levelup.gitconnected.com/the-hidden-cost-of-agent-memory-what-mem0-zep-and-letta-dont-tell-you-3e9ff70e4ea6?source=rss-3ec8787f291f------2</link>
            <guid isPermaLink="false">https://medium.com/p/3e9ff70e4ea6</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Yuval Mehta]]></dc:creator>
            <pubDate>Mon, 20 Jul 2026 15:34:18 GMT</pubDate>
            <atom:updated>2026-07-20T15:34:18.135Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EHvjwVFMbrHONrLh" /><figcaption>Photo by <a href="https://unsplash.com/@thefredyjacob?utm_source=medium&amp;utm_medium=referral">Fredy Jacob</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Every agent memory comparison post follows the same template: a feature table, a benchmark score, and a winner. Mem0 wins on latency, Zep on temporal reasoning, and Letta on context management. Pick one, ship it, move on.</p><p>None of those posts tells you what happens six months in, when your memory layer has absorbed hundreds of thousands of writes, and you discover, usually from a support ticket, not a changelog, that the architecture you built on has quietly shifted underneath you.</p><p>This post is a map of the failure modes that actually show up in production across the three dominant memory architectures, plus the hot/cold path design that handles all three cleanly.</p><p><strong>TL;DR</strong></p><ul><li>Mem0’s 2026 architecture update replaced the queryable graph relations interface with built-in entity linking, a breaking change for anyone doing custom multi-hop reasoning over that graph.</li><li>Temporal staleness is a structural risk in any memory system that separates ingestion from indexing — Zep’s graph construction pipeline is the clearest example, but the pattern generalizes.</li><li>Memory write latency compounds badly when writes are synchronous, and on the agent’s response path, the fix is a hot/cold split, not a faster database.</li><li>Letta’s context-paging model trades latency variability for information fidelity — know which one your product actually needs before you pick it.</li></ul><h3>Failure Mode 1: Mem0’s Graph Interface Isn’t What It Was</h3><p>Mem0 originally offered a queryable knowledge graph — entities and relationships extracted from conversation, stored as traversable triplets, backed by an external graph store like Neo4j. If you were doing custom reasoning over that graph — multi-hop queries like “find everyone connected to this project who hasn’t been contacted in 30 days” — you built it on top of that relations field.</p><p>In their 2026 architecture revision, Mem0 replaced external graph store support with built-in entity linking — entities are now extracted during the add() step and stored in a parallel entity collection rather than a queryable graph. The relations field is gone; entity relationships now influence retrieval ranking but can no longer be traversed directly.</p><p>For most use cases, this is a genuine improvement — teams get entity-aware retrieval without the overhead of running a Neo4j instance. But if your product depended on graph traversal — not just “retrieve relevant memories” but “walk the relationship chain to answer a structural question” — this is a silent regression you only notice once multi-hop queries stop returning what they used to.</p><pre># Pre-2026 pattern: direct graph traversal<br>def find_connected_entities(memory_client, entity: str, hops: int = 2):<br>    # This assumed a queryable `relations` graph existed<br>    graph = memory_client.get_graph(entity_id=entity)<br>    return graph.traverse(max_hops=hops)  # No longer available<br><br># Post-2026 pattern: you have to rebuild traversal yourself<br>def find_connected_entities_v2(memory_client, entity: str, hops: int = 2):<br>    # Entity linking now only influences retrieval ranking.<br>    # If you need actual graph traversal, maintain your own<br>    # graph layer alongside Mem0, populated from its entity extraction output.<br>    related = memory_client.search(<br>        query=entity,<br>        filters={&quot;entity_boost&quot;: True},<br>    )<br>    # This is ranked retrieval, not traversal - you&#39;d need to<br>    # build your own adjacency structure from extracted entities<br>    # if multi-hop reasoning is a hard product requirement.<br>    return related</pre><p>The lesson isn’t “don’t use Mem0.” It’s: treat a vendor’s graph interface as a moving part, not a foundation. Pin versions, watch changelogs, and if multi-hop reasoning is core to your product, own that layer directly rather than depending on a hosted abstraction that can restructure underneath you.</p><h3>Failure Mode 2: Temporal Staleness in Graph-Construction Pipelines</h3><p>Systems that build a knowledge graph from conversation — rather than storing flat vector embeddings — typically do so asynchronously. The conversation happens, then a background process extracts entities, resolves relationships, and updates the graph. This is a reasonable design: synchronous graph construction on the response path would be far too slow.</p><p>The cost is a stale window. Independent testing has found that immediate post-ingestion retrieval can fail on graph-heavy memory systems — correct answers only became retrievable hours later, once background graph processing completed. For a real-time application, that’s a serious constraint, not a footnote.</p><p>This isn’t unique to one vendor — it’s structural to any architecture that decouples ingestion from indexing. The fix isn’t eliminating the async pipeline; it’s designing your application layer to know about the staleness window rather than assume immediate consistency.</p><pre>class MemoryWithStalenessAwareness:<br>    def __init__(self, memory_client, staleness_window_seconds: int = 300):<br>        self.client = memory_client<br>        self.staleness_window = staleness_window_seconds<br>        self.recent_writes = {}  # in-process cache for the staleness window<br><br>  def add(self, content: str, user_id: str) -&gt; None:<br>        self.client.add(content, user_id=user_id)<br>        # Cache recent writes locally so retrieval doesn&#39;t miss them<br>        # while the background graph pipeline catches up<br>        self.recent_writes.setdefault(user_id, []).append({<br>            &quot;content&quot;: content,<br>            &quot;timestamp&quot;: time.time(),<br>        })<br>        self._prune_stale(user_id)<br>    def search(self, query: str, user_id: str) -&gt; list[dict]:<br>        graph_results = self.client.search(query, user_id=user_id)<br>        # Supplement with anything written inside the staleness window<br>        # that might not have propagated to the graph yet<br>        recent = self._get_recent(user_id)<br>        return self._merge_and_dedupe(graph_results, recent)<br>    def _prune_stale(self, user_id: str) -&gt; None:<br>        cutoff = time.time() - self.staleness_window<br>        self.recent_writes[user_id] = [<br>            w for w in self.recent_writes.get(user_id, [])<br>            if w[&quot;timestamp&quot;] &gt; cutoff<br>        ]</pre><p>This is a workaround, not a fix — but it’s the right response to a structural property of the system you’re building on. Forcing synchronous consistency out of an async pipeline usually means fighting the architecture rather than working with it.</p><h3>Failure Mode 3: Write Latency Compounds When It’s on the Hot Path</h3><p>The default integration pattern for most memory libraries puts the memory write directly in your agent’s response path: user sends a message, agent processes it, agent calls memory.add(), agent responds. This is simple, and it&#39;s wrong for anything beyond a prototype.</p><p>Every memory write involves at minimum one LLM call for fact extraction, often a second for conflict resolution against existing memories, and the storage write itself. That’s not a 5ms Postgres insert — it’s frequently 500ms–2s of LLM round-trip time, sitting directly in your response latency budget.</p><pre># Hot-path pattern (what most teams ship first, and regret)<br>async def handle_message(user_message: str, user_id: str) -&gt; str:<br>    response = await agent.generate(user_message)<br>    await memory_client.add(user_message, user_id=user_id)  # blocks response<br>    await memory_client.add(response, user_id=user_id)       # blocks response<br>    return response  # user waited for both memory writes<br><br># Hot/cold split (what production actually needs)<br>async def handle_message_v2(user_message: str, user_id: str) -&gt; str:<br>    response = await agent.generate(user_message)<br>    <br>    # Fire memory writes into a background queue - don&#39;t block the response<br>    await memory_write_queue.enqueue({<br>        &quot;user_message&quot;: user_message,<br>        &quot;agent_response&quot;: response,<br>        &quot;user_id&quot;: user_id,<br>        &quot;timestamp&quot;: time.time(),<br>    })<br>    <br>    return response  # user gets response immediately<br># Separate worker process drains the queue<br>async def memory_write_worker(queue: MemoryWriteQueue, memory_client):<br>    async for item in queue.consume():<br>        try:<br>            await memory_client.add(item[&quot;user_message&quot;], user_id=item[&quot;user_id&quot;])<br>            await memory_client.add(item[&quot;agent_response&quot;], user_id=item[&quot;user_id&quot;])<br>        except Exception as e:<br>            await queue.dead_letter(item, error=e)  # never silently drop writes</pre><p>The architecture that holds up at scale is a hot/cold split: a fast in-context cache for recent turns that answers immediate continuity needs, and a cold path — the actual memory layer — that processes writes asynchronously for long-horizon retrieval. The agent reads from both: hot path for “what did we just discuss,” cold path for “what do you know about me broadly.” Neither blocks the other, and the user never pays LLM-extraction latency on their response. It’s the same cache-aside pattern used in traditional backends — except your “cache miss” cost is an LLM call, not a database query.</p><h3>Failure Mode 4: Letta’s Paging Model Has a Different Failure Shape</h3><p>Letta (formerly MemGPT) takes a structurally different approach: instead of extracting facts into a separate store, it manages an OS-inspired virtual memory abstraction, paging information between the active context window and external storage.</p><p>This sidesteps graph construction entirely, but introduces its own failure mode: the recursive summarization and hierarchical compression the paging model relies on can introduce latency variability and lose information fidelity, particularly for tasks needing precise textual recall over gist-level continuity.</p><p>In practice, Letta-style systems fail differently than Mem0 or Zep: not “the fact wasn’t retrievable yet” but “the fact came back compressed and lossy.” Fine for a support agent recalling an account tier; risky for a legal or medical context where exact wording matters. Test for it explicitly rather than assuming paging preserves fidelity.</p><h3>Choosing an Architecture: A Decision Frame, Not a Winner</h3><ul><li><strong>You need fast semantic recall with low operational overhead, and multi-hop graph reasoning isn’t core to your product</strong>: Mem0’s current entity-linking model is a reasonable default. Just don’t build critical logic on the assumption of a traversable graph.</li><li><strong>You need temporally precise reasoning — “what changed and when” — and can tolerate a staleness window with proper handling</strong>: Zep’s temporal graph model is purpose-built for this, provided your application layer accounts for the async indexing gap.</li><li><strong>You need precise long-context recall without building a separate memory store, and can tolerate compression-related fidelity loss on less-critical recalls</strong>: Letta’s paging model fits, especially for conversational continuity use cases.</li></ul><p>None of these is categorically better. Each embeds a different tradeoff between latency, fidelity, and architectural complexity — and the tradeoff that doesn’t show up in any benchmark table is what breaks when you’re 500,000 memory writes into production, which is exactly when it matters most.</p><h3>Gotchas: What Nobody Tells You</h3><p><strong>1. Silent regressions in hosted memory layers don’t show up in your test suite.</strong></p><p>Mem0’s graph interface change didn’t break integration tests for most teams — it broke a specific class of query that many teams weren’t testing explicitly because it “just worked” before. If you depend on a specific capability of a memory vendor’s architecture, write an explicit test for that capability, not just for “memory retrieval returns something relevant.”</p><p><strong>2. Memory write failures are invisible unless you make them loud.</strong></p><p>Because memory writes are often fire-and-forget in production (correctly, per the hot/cold pattern above), a failed write doesn’t crash anything — it just means a fact silently never gets remembered. Always route failed writes to a dead-letter queue with alerting, not a swallowed exception.</p><p><strong>3. Benchmark scores on memory retrieval accuracy don’t capture staleness windows.</strong></p><p>A system can score well on retrieval accuracy benchmarks while still failing your product’s actual latency requirements, because most public benchmarks measure whether the right memory is <em>eventually</em> retrievable — not whether it’s retrievable within the window your product needs. Test retrieval latency under your actual write-then-read timing, not just retrieval correctness in isolation.</p><h3>Conclusion</h3><p>The comparison-post format optimizes for a decision you make once. Production memory architecture is a decision you live with continuously — through vendor updates that change interfaces underneath you, staleness windows that only matter at real write volume, and latency budgets that only bite with real users. Mem0, Zep, and Letta are all legitimate choices; none of them are “solved” in the sense that you pick one and stop thinking about the architecture around it.</p><p>The teams that don’t get burned build the hot/cold split regardless of vendor, test the specific capability their product depends on instead of trusting “it retrieves relevant memories,” and treat a vendor’s architecture docs as a snapshot, not a contract.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3e9ff70e4ea6" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/the-hidden-cost-of-agent-memory-what-mem0-zep-and-letta-dont-tell-you-3e9ff70e4ea6">The Hidden Cost of Agent Memory: What Mem0, Zep, and Letta Don’t Tell You</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Reward Design Is the Hard Part: Building Verifiable Rewards for Tool-Using Agents]]></title>
            <link>https://pub.towardsai.net/reward-design-is-the-hard-part-building-verifiable-rewards-for-tool-using-agents-f99c5c38f9b3?source=rss-3ec8787f291f------2</link>
            <guid isPermaLink="false">https://medium.com/p/f99c5c38f9b3</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Yuval Mehta]]></dc:creator>
            <pubDate>Sat, 11 Jul 2026 12:31:00 GMT</pubDate>
            <atom:updated>2026-07-11T12:31:00.931Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZO5F388PeA4XSiul" /><figcaption>Photo by <a href="https://unsplash.com/@toddquackenbush?utm_source=medium&amp;utm_medium=referral">Todd Quackenbush</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>RLVR works beautifully for math. Generate a response, extract the final answer, compare it to ground truth, and assign 1 or 0. No reward model to train, no human preference data to collect, no reward hacking against a learned proxy, just a deterministic check. This is why DeepSeek-R1-Zero could go from 15.6% to 77.9% on AIME with nothing but GRPO and a verifier.</p><p>Now try to apply the same idea to an agent that searches the web, calls three APIs, reads the results, and books a flight. What’s the verifier? “Did it book the right flight” is a valid terminal reward, but it’s the only signal across a 15-step trajectory, and most of those steps had nothing to do with correctness in a way you can check with a string comparison.</p><p>This is where most teams doing agentic RL get stuck. The RLVR playbook that worked for math and code doesn’t transfer cleanly to tool-using agents, and the honest answer is that reward design here is closer to systems engineering than to machine learning.</p><p><strong>TL;DR</strong></p><ul><li>Outcome-only rewards are catastrophically sparse for multi-step tool-use trajectories — you need a process-level signal, or your agent never learns intermediate behavior.</li><li>Turn-level verifiable rewards (tool executed correctly, output well-formed, partial progress detectable) give you a dense signal without sacrificing verifiability.</li><li>LLM-judge rewards are necessary for non-verifiable steps but introduce their own reward-hacking surface — mitigate with rubrics, not vibes.</li><li>Reward hacking in agentic settings is subtler than in math/code: agents learn to game your verifier’s blind spots (calling tools without using their output, retrying until a lucky pass, gaming format checks).</li><li>Guidance-based RL (teacher hints during training) is emerging as the practical answer to sparse-reward stalls in software engineering and search agents.</li></ul><h3>Why Outcome-Only Rewards Break Down for Agents</h3><p>RLVR is the standard method for improving LLM reasoning and has shown strong results in verifiable domains like math and competitive programming, but its effectiveness drops sharply in agentic environments. The reason is structural, not incidental.</p><p>In a math problem, the trajectory is short (a few hundred tokens of reasoning) and the final answer fully determines correctness. In a tool-using agent trajectory, you might have 10–20 turns, each involving a tool call, an observation, and a reasoning step. A single terminal reward has to somehow propagate credit back across all of those turns — and with GRPO-style group-relative advantages, if every rollout in a group fails the same way, you get zero gradient signal despite non-zero variation in <em>how</em> they failed.</p><pre># Naive agentic RLVR — this is the failure pattern<br>def compute_reward(trajectory: list[dict], ground_truth: str) -&gt; float:<br>    final_answer = trajectory[-1][&quot;content&quot;]<br>    return 1.0 if final_answer.strip() == ground_truth.strip() else 0.0<br><br># Problem: an agent that calls the right tool with the right arguments<br># but misparses the response gets the same reward (0) as an agent<br># that never calls any tool at all. No signal differentiates them.</pre><p>This is the sparse reward problem, and it’s the reason RLVR training in multi-turn tool-interaction settings suffers from reward hacking when using a purely outcome-based reward — the policy finds degenerate shortcuts because the reward surface gives it no reason not to.</p><h3>Turn-Level Verifiable Rewards: The First Fix</h3><p>The most direct fix is decomposing the terminal reward into intermediate, still-verifiable signals. For tool-calling agents, verifiable turn-level rewards can be constructed from tool execution success and evidence of progress toward the answer. Concretely:</p><p>A tool execution reward awards partial credit when the tool call is correctly formatted, and the environment doesn’t return an error. A separate reward checks whether the correct answer appears anywhere in the tool’s returned results, not whether the agent has already synthesized the final answer, just whether the retrieval step surfaced the right information.</p><pre>def turn_level_reward(turn: dict, ground_truth_answer: str) -&gt; float:<br>    reward = 0.0<br>    <br>    # Tool execution reward: was the tool called correctly?<br>    if turn.get(&quot;tool_call_well_formed&quot;) and not turn.get(&quot;tool_error&quot;):<br>        reward += 0.2<br>    <br>    # Progress reward: does the tool&#39;s output contain the answer?<br>    tool_output = turn.get(&quot;tool_result&quot;, &quot;&quot;)<br>    if ground_truth_answer.lower() in tool_output.lower():<br>        reward += 0.5<br>    <br>    return reward<br><br>def trajectory_reward(trajectory: list[dict], ground_truth: str) -&gt; float:<br>    # Sum turn-level rewards, then add the outcome reward at the end<br>    turn_rewards = sum(<br>        turn_level_reward(t, ground_truth) for t in trajectory<br>    )<br>    final_answer = trajectory[-1][&quot;content&quot;]<br>    outcome_reward = 1.0 if ground_truth.lower() in final_answer.lower() else 0.0<br>    <br>    return turn_rewards + outcome_reward</pre><p>This gives the policy gradient signal at every step where the agent did something checkable, instead of only at the end. It’s a meaningful improvement over outcome-only rewards, but it only covers what you can verify with string matching or execution success — which is a fraction of what actually matters in a good trajectory (did the agent choose an efficient path, did it avoid redundant tool calls, did it reason coherently between steps).</p><h3>Process Reward Models: Dense Signal Without a Verifier</h3><p>For behaviors that can’t be checked with a rule, the alternative is a learned process reward model (PRM) that scores intermediate steps. Process Reward Models automatically extract intermediate milestones from demonstrations to provide dense progress estimations, addressing the long-horizon sparsity problem without requiring every step to be independently verifiable by a rule.</p><p>The tradeoff is real: a PRM is a trained model, which means it can be gamed, it requires its own training data (usually derived from expert demonstrations or successful trajectories), and it adds inference cost to your RL loop since you’re scoring every intermediate step during rollout collection.</p><p>An alternative to a trained PRM is a rubric-based LLM judge — scoring each predicted step against a structured, multi-dimensional rubric rather than a single scalar, which reduces the judge’s tendency to collapse a complex behavior into a single noisy number. If the judge fails to produce a valid score, the safe default is to zero out that reward rather than guess — a silent bad score is worse than a visibly failed one you can filter out downstream.</p><pre>RUBRIC_DIMENSIONS = [<br>    &quot;tool_choice_appropriateness&quot;,<br>    &quot;argument_correctness&quot;,<br>    &quot;result_interpretation&quot;,<br>    &quot;progress_toward_goal&quot;,<br>    &quot;reasoning_coherence&quot;,<br>]<br><br>def llm_judge_reward(step: dict, context: dict, judge_client) -&gt; float:<br>    &quot;&quot;&quot;<br>    Score a single agent step against a structured rubric.<br>    Returns a normalized [0, 1] reward, or 0.0 if the judge fails<br>    to produce a parseable score (fail-safe, not fail-silent).<br>    &quot;&quot;&quot;<br>    scores = []<br>    for dimension in RUBRIC_DIMENSIONS:<br>        response = judge_client.score(<br>            step=step,<br>            context=context,<br>            dimension=dimension,<br>            scale=&quot;1-5&quot;,<br>        )<br>        parsed = try_parse_score(response)<br>        if parsed is None:<br>            return 0.0  # fail-safe default, logged for offline review<br>        scores.append(parsed)<br>    <br>    return sum(scores) / (len(scores) * 5)  # normalize to [0, 1]</pre><h3>The Reward Hacking Problem Is Different Here</h3><p>In math RLVR, reward hacking looks like format exploitation — the model learns to produce output that pattern-matches the expected answer format without actually solving the problem. In agentic RLVR, the exploits are structurally different and harder to catch:</p><p><strong>Tool-call theater</strong>: the agent learns to call tools because tool-calling is rewarded, without ever using the returned information in its final answer. Your turn-level reward for “tool executed successfully” is satisfied; the agent has learned nothing about actually using tools.</p><p><strong>Retry gaming</strong>: if your environment allows retries and your reward only checks the final state, the policy learns to retry blindly until something passes, rather than reasoning about why a previous attempt failed.</p><p><strong>Judge gaming</strong>: if you’re using an LLM judge for non-verifiable dimensions, the policy can learn surface patterns that the judge rewards — verbose reasoning that <em>sounds</em> thorough without being correct, or hedged language that scores well on “reasoning coherence” without committing to a checkable claim.</p><p>The fix for all three is the same discipline: audit your reward function the way you’d audit a metric that’s about to become a target. Ask “what’s the cheapest way to maximize this reward without doing the task” before you start training, not after you notice the model doing it.</p><pre>def audit_reward_function(reward_fn, sample_trajectories: list[dict]) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Run cheap, degenerate trajectories against your reward function<br>    before training. If any of these score highly, your reward has a hole.<br>    &quot;&quot;&quot;<br>    adversarial_probes = {<br>        &quot;empty_reasoning_valid_tool_call&quot;: ...,   # tool call w/ no reasoning<br>        &quot;tool_call_ignored_output&quot;: ...,           # calls tool, ignores result<br>        &quot;verbose_but_wrong&quot;: ...,                  # long reasoning, wrong answer<br>        &quot;minimal_effort_lucky_guess&quot;: ...,         # skips steps, guesses right<br>    }<br>    return {<br>        name: reward_fn(probe) for name, probe in adversarial_probes.items()<br>    }<br>    # Any of these scoring above ~0.5 signals an exploitable reward shape.</pre><h3>Guidance-Based RL: Fixing Sparsity at the Source</h3><p>A more structural fix than reward decomposition alone is injecting guidance during training rather than relying purely on reward shape. Agent-RLVR incorporates agent guidance through a multi-agent framework where teacher guidance provides targeted information about failing patterns and expectations during RL training, functioning similarly to how a junior engineer gets pointed in the right direction by a senior teammate rather than being left to find the fix through unguided trial and error.</p><p>This matters because agentic RL settings characterized by multi-step, complex problem-solving lead to high failure rates even for frontier models, since the reward landscape is too sparse for effective training via conventional RLVR alone. No amount of reward decomposition fixes a setting where the model simply never stumbles into a successful trajectory during rollout collection. Guidance — hints, partial solutions, or corrective context injected mid-trajectory during training — increases the rate of successful rollouts without compromising the verifiability of the final reward signal, since the hints only affect exploration, not scoring.</p><p>The practical takeaway: if your agent’s task success rate during early RL rollouts is near zero, more reward engineering won’t save you. You need either an easier curriculum, guidance injection, or expert demonstration data to bootstrap from — reward shape can’t manufacture successful trajectories that never occur.</p><h3>Gotchas: What Nobody Tells You</h3><p><strong>1. Dense rewards can suppress exploration just as much as sparse ones.</strong></p><p>It’s tempting to over-engineer turn-level rewards until every plausible good behavior gets some credit. This can backfire: if partial credit is too generous, the policy converges on “safe” trajectories that collect small guaranteed rewards (call a tool, get 0.2, stop trying) rather than pushing for the larger reward that requires more exploration. Weight terminal task success meaningfully higher than intermediate process rewards — process rewards should shape behavior, not replace the incentive to actually finish the task.</p><p><strong>2. Reward hacking gets worse, not better, as your policy improves.</strong></p><p>A weak policy fails obviously. A policy that’s 80% of the way to competent finds subtler exploits, because it has enough capability to identify the reward function’s blind spots rather than stumbling past them. Re-run your adversarial reward audits periodically during training, not just once before you start.</p><p><strong>3. Non-determinism in tool environments breaks reward consistency.</strong></p><p>If your agent calls a live search API or a web tool, the same action can return different results across rollouts — meaning identical trajectories can receive different rewards. This adds noise that compounds with GRPO’s group-relative advantage computation, since your “identical” prompts aren’t producing directly comparable groups anymore. Where possible, use cached or replayed tool responses during RL training and reserve live environments for evaluation.</p><h3>Conclusion</h3><p>Reward design for tool-using agents is not a smaller version of reward design for math — it’s a different problem. Outcome-only verification, which is RLVR’s core strength in single-shot domains, becomes a liability once trajectories span multiple steps with tool calls in between. The fix isn’t one technique; it’s a layered system: turn-level verifiable rewards where you can get them, process reward models or rubric-based judges where you can’t, systematic reward auditing to catch hacking before it compounds, and guidance injection when sparsity is severe enough that no reward shape will manufacture a successful trajectory on its own.</p><p>The teams shipping real agentic RL systems in 2026 aren’t the ones with the cleverest single reward function. They’re the ones treating reward design as an evolving system with its own test suite — one that gets audited as rigorously as the policy it’s training.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f99c5c38f9b3" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/reward-design-is-the-hard-part-building-verifiable-rewards-for-tool-using-agents-f99c5c38f9b3">Reward Design Is the Hard Part: Building Verifiable Rewards for Tool-Using Agents</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LangGraph Checkpointing Is Not Free: A Production Postmortem]]></title>
            <link>https://pub.towardsai.net/langgraph-checkpointing-is-not-free-a-production-postmortem-398bc86861f4?source=rss-3ec8787f291f------2</link>
            <guid isPermaLink="false">https://medium.com/p/398bc86861f4</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Yuval Mehta]]></dc:creator>
            <pubDate>Tue, 30 Jun 2026 13:31:01 GMT</pubDate>
            <atom:updated>2026-06-30T13:31:01.369Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rqXfsvVhNq-LmxH2" /><figcaption>Photo by <a href="https://unsplash.com/@guerrillabuzz?utm_source=medium&amp;utm_medium=referral">GuerrillaBuzz</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The LangGraph docs make checkpointing look like a two-line concern. Add a PostgresSaver, wire it into your graph config, and you get persistence, resumable runs, human-in-the-loop pauses, and time-travel debugging. All of it is backed by a table in Postgres you already have.</p><p>Then you ship to production. Threads start accumulating. You push an update that adds a new field to your state schema. Half your running threads fail to deserialize on resume. You dig in and discover that interrupt/resume semantics behave differently when you have parallel branches. Your checkpoint table has grown to 40GB in six weeks with no retention policy. Welcome to checkpointing in production — the feature nobody warned you about.</p><p><strong>TL;DR</strong></p><ul><li>Checkpoint schema changes require explicit migration — LangGraph won’t handle it for you, and old threads will silently break.</li><li>interrupt_before / interrupt_after have undefined behavior during mid-fan-out in parallel subgraph execution.</li><li>Thread-scoped checkpointing doesn’t compose cleanly with multi-tenant systems — you’ll need user-scoped indexing and a naming convention from day one.</li><li>Every graph step writes a full checkpoint blob to Postgres. This is in your hot path. At scale, it matters.</li><li>MemorySaver does not reproduce serialization bugs. Your dev/prod environment mismatch is hiding failures.</li></ul><h3>What Checkpointing Actually Does</h3><p>Before the failure modes, a precise model of what’s happening. LangGraph’s checkpointing system persists the state of a graph execution at each node boundary. The checkpoint captures the full graph state—every key in your TypedDict, the pending tasks, and the execution position and stores it under a (thread_id, checkpoint_id) pair.</p><pre>┌──────────────────────────────────────────────────────────┐<br>│                     LangGraph Runtime                    │<br>│                                                          │<br>│  Node A ──▶ [CHECKPOINT] ──▶ Node B ──▶ [CHECKPOINT]    │<br>│                  │                           │           │<br>│                  ▼                           ▼           │<br>│         ┌────────────────────────────────────────┐       │<br>│         │          PostgresSaver                 │       │<br>│         │  thread_id | checkpoint_id | state_blob│       │<br>│         └────────────────────────────────────────┘       │<br>└──────────────────────────────────────────────────────────┘</pre><p>This gives you three capabilities:</p><ul><li><strong>Fault tolerance</strong>: a thread that crashes mid-execution can resume from the last saved checkpoint.</li><li><strong>Human-in-the-loop</strong>: you can interrupt execution before a node, surface state to a human, collect input, and then resume.</li><li><strong>Time travel</strong>: you can replay from any past checkpoint, which is invaluable for debugging.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/540/1*t6GGt9LX-VGXA1wRP0hCfA.png" /><figcaption>AI Generated</figcaption></figure><p>Setting up PostgresSaver looks like this:</p><pre>from langgraph.checkpoint.postgres import PostgresSaver<br>from psycopg import Connection<br><br>DB_URI = &quot;postgresql://user:password@localhost:5432/mydb&quot;<br>with Connection.connect(DB_URI) as conn:<br>    checkpointer = PostgresSaver(conn)<br>    checkpointer.setup()  # Creates checkpoint tables if they don&#39;t exist<br>graph = my_compiled_graph.compile(checkpointer=checkpointer)<br>config = {&quot;configurable&quot;: {&quot;thread_id&quot;: &quot;thread-abc-123&quot;}}<br>result = graph.invoke({&quot;messages&quot;: [...]}, config=config)</pre><p>Simple. Now here’s what breaks.</p><h3>Failure Mode 1: State Schema Migration Is Entirely on You</h3><p>Your graph state is a TypedDict. You store it as a blob in Postgres at each checkpoint. When you add a new field to that TypedDict — or remove one, or change its type — every checkpoint written before that change now deserializes into a shape your code no longer expects.</p><pre># Week 1: your state<br>class AgentState(TypedDict):<br>    messages: list[AnyMessage]<br>    tool_calls: list[dict]<br><br># Week 3: you add context tracking<br>class AgentState(TypedDict):<br>    messages: list[AnyMessage]<br>    tool_calls: list[dict]<br>    user_context: dict        # New field<br>    execution_trace: list[str]  # New field</pre><p>Any thread that has an existing checkpoint in Postgres will resume in this new schema. Depending on how your graph accesses user_context — if a node assumes it&#39;s present and reads it directly, you&#39;ll get a KeyError at runtime, with no migration error, no warning, and no obvious connection to the schema change that caused it.</p><p>LangGraph doesn’t have a built-in migration system. The fix is to treat your state schema like a database schema from day one:</p><pre>class AgentState(TypedDict):<br>    messages: list[AnyMessage]<br>    tool_calls: list[dict]<br>    # Always provide defaults for new fields to maintain backward compat<br>    user_context: Annotated[dict, lambda x, y: y]  # reducer<br>    execution_trace: Annotated[list[str], operator.add]<br><br>def safe_node(state: AgentState) -&gt; dict:<br>    # Defensive access: never assume new fields are present in resumed threads<br>    ctx = state.get(&quot;user_context&quot;, {})<br>    trace = state.get(&quot;execution_trace&quot;, [])<br>    # ...</pre><p>More importantly, maintain a version key in your state. When a thread resumes, check the version. If it’s behind, run an explicit migration step as the first node — or fail loudly with a message that’s actually debuggable.</p><h3>Failure Mode 2: Interrupt/Resume Breaks Under Parallel Execution</h3><p>interrupt_before and interrupt_after let you pause graph execution at a node boundary, hand off to a human, collect input, and resume. The pattern is central to human-in-the-loop agent design.</p><p>It works cleanly in sequential graphs. It becomes undefined in graphs with parallel branches.</p><pre># Sequential — interrupt works as expected<br>graph = StateGraph(AgentState)<br>graph.add_node(&quot;plan&quot;, planner_node)<br>graph.add_node(&quot;review&quot;, review_node)      # interrupt_before this<br>graph.add_node(&quot;execute&quot;, executor_node)<br>graph.compile(checkpointer=checkpointer, interrupt_before=[&quot;review&quot;])</pre><p>Now introduce parallelism via LangGraph’s Send API:</p><pre># Parallel fan-out — interrupt semantics break<br>def router(state: AgentState) -&gt; list[Send]:<br>    return [<br>        Send(&quot;worker&quot;, {&quot;task&quot;: t}) for t in state[&quot;tasks&quot;]<br>    ]<br><br>graph.add_conditional_edges(&quot;plan&quot;, router)<br>graph.add_node(&quot;worker&quot;, worker_node)     # 3 workers run in parallel<br>graph.add_node(&quot;review&quot;, review_node)     # interrupt_before here</pre><p>If interrupt_before=[&quot;review&quot;] fires after all workers complete, you&#39;re fine. But if one worker is slow and the runtime checkpoints mid-fan-out, you now have a checkpoint that represents a partial parallel execution state. When you resume, the runtime may re-execute completed workers, skip failed ones, or resume from an inconsistent intermediate. The behavior depends on your checkpoint granularity and which worker states were actually persisted.</p><p>The rule of thumb that holds up in production: <strong>don’t place interrupts downstream of fan-out nodes unless you’ve explicitly verified what partial state looks like</strong>. For human-in-the-loop patterns, place interrupts before the fan-out, not after.</p><h3>Failure Mode 3: Thread Scoping Doesn’t Compose with Multi-Tenancy</h3><p>Every checkpoint is scoped to a thread_id, which is a plain string. LangGraph has no concept of a user, a tenant, or an organization — that&#39;s your responsibility to build.</p><p>The instinct is to encode tenant context in the thread ID:</p><pre>thread_id = f&quot;user-{user_id}:task-{task_id}&quot;<br>config = {&quot;configurable&quot;: {&quot;thread_id&quot;: thread_id}}</pre><p>This works for routing individual requests. It breaks the moment you need to answer questions like “show me all active threads for user X” or “cancel every pending task for tenant Y.”</p><p>The checkpoint table looks like this under the hood:</p><pre>-- LangGraph&#39;s checkpoint table (simplified)<br>CREATE TABLE checkpoints (<br>    thread_id    TEXT NOT NULL,<br>    checkpoint_id UUID NOT NULL,<br>    parent_id    UUID,<br>    state        BYTEA,         -- full serialized state<br>    created_at   TIMESTAMPTZ DEFAULT now(),<br>    PRIMARY KEY (thread_id, checkpoint_id)<br>);</pre><p>There’s no user_id column. Querying &quot;all threads for user X&quot; means either a LIKE scan on thread_id (slow at scale) or maintaining a separate lookup table that you keep in sync.</p><p>The cleaner production pattern:</p><pre># Maintain a thread registry alongside checkpoints<br>class ThreadRegistry:<br>    def create_thread(self, user_id: str, task_type: str) -&gt; str:<br>        thread_id = f&quot;{uuid4()}&quot;  # opaque ID — no user info encoded<br>        # Store (thread_id, user_id, task_type, created_at) in your own table<br>        self.db.insert(&quot;thread_registry&quot;, {<br>            &quot;thread_id&quot;: thread_id,<br>            &quot;user_id&quot;: user_id,<br>            &quot;task_type&quot;: task_type,<br>        })<br>        return thread_id</pre><p>Separate the concern of thread identity (LangGraph’s job) from thread ownership (your job). This also future-proofs you for access control — checking whether a user can access a thread doesn’t require parsing string IDs.</p><h3>Failure Mode 4: Checkpoints Are in Your Hot Path</h3><p>This one is easy to miss because it only becomes visible at scale. By default, LangGraph writes a checkpoint after every node execution. For a 12-node graph processing 500 concurrent threads, that’s 6,000 checkpoint writes per graph execution cycle — all synchronous, all going to Postgres.</p><p>A healthy Postgres instance handles checkpoint writes in the 2–10ms range. That adds latency to every node transition, compounding across a graph’s depth. A 12-node graph with 5ms average checkpoint latency is adding 60ms of pure I/O overhead per execution—before any actual compute.</p><p>At 500 concurrent executions, you’re also hitting Postgres with ~500 concurrent writes at each node boundary. Watch your connection pool. The default pool sizes in most application configs assume a different request pattern.</p><p>Two levers:</p><pre># Option 1: Async checkpointer for non-blocking writes (LangGraph supports this)<br>from langgraph.checkpoint.postgres.aio import AsyncPostgresSaver<br><br>async with await AsyncPostgresSaver.from_conn_string(DB_URI) as checkpointer:<br>    graph = compiled.compile(checkpointer=checkpointer)<br># Option 2: Checkpoint selectively - not every node needs persistence<br># Only checkpoint at true recovery points: after expensive operations,<br># before human-in-the-loop steps, at subgraph boundaries<br>graph.add_node(&quot;expensive_op&quot;, expensive_node)   # checkpoint here<br>graph.add_node(&quot;cheap_transform&quot;, cheap_node)    # skip or batch</pre><p>Also, implement a retention policy immediately. There’s no built-in TTL on checkpoints. A 15-node graph running 1,000 threads/day generates ~15,000 checkpoint rows/day. Each checkpoint blob can be 5–50KB, depending on state complexity. At 50KB average, that’s 750MB/day with zero retention — 270GB/year. Set up a scheduled job that deletes checkpoints older than your recovery window.</p><pre>-- Run this daily; keep only the last 7 days<br>DELETE FROM checkpoints<br>WHERE created_at &lt; NOW() - INTERVAL &#39;7 days&#39;<br>  AND thread_id NOT IN (<br>    SELECT thread_id FROM thread_registry WHERE status = &#39;active&#39;<br><br>  );</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/902/1*06_EL_Hru69JnhrALIVy9Q.png" /><figcaption>AI generated</figcaption></figure><h3>Gotchas: What Nobody Tells You</h3><p><strong>1. </strong><strong>MemorySaver is not a faithful test double.</strong></p><p>In development, you almost certainly use MemorySaver — it&#39;s in-process, fast, and requires no infrastructure. The problem: MemorySaver stores Python objects directly; PostgresSaver serializes them to bytes and deserializes them on resume. Any custom object in your state that doesn&#39;t serialize cleanly (Pydantic models with validators that fire on construction, objects with non-default __reduce__ methods, or LangChain tool instances) will pass every test with MemorySaver and blow up in production.</p><p>Run at least part of your integration test suite against a real PostgresSaver pointed at a test database. The cost of the infrastructure is one Docker container. The cost of discovering this in prod is considerably higher.</p><p><strong>2. </strong><strong>thread_id is stringly-typed all the way down.</strong></p><p>config[&quot;configurable&quot;][&quot;thread_id&quot;] is a raw string. There&#39;s no typed ThreadId wrapper, no compile-time check, and no runtime validation that the thread you&#39;re trying to resume actually exists. A typo in thread construction silently creates a new thread rather than resuming the intended one. If you&#39;re constructing thread IDs programmatically, centralize that logic in a single function and write a test for it.</p><p><strong>3. Time travel has a correctness hazard, not just a cost.</strong></p><p>Replaying from a past checkpoint is invaluable for debugging. It’s dangerous in production if your graph has side effects that have already been executed—external API calls, database writes, and emails sent. Replaying from checkpoint N-5 will re-execute those side effects unless your nodes are explicitly idempotent. This is not a LangGraph problem; it’s a general replay problem. But LangGraph’s time-travel UI makes it easy to trigger accidentally. Document which nodes have side effects and add idempotency keys before you enable time-travel tooling for your team.</p><h3>Conclusion</h3><p>Checkpointing is LangGraph’s highest-leverage production feature and its least-understood operational dependency. Teams that treat it as an afterthought—writing in PostgresSaver, shipping, and moving on—consistently hit the same set of failures: schema migration surprises, interrupt edge cases in parallel graphs, unbounded checkpoint storage, and serialization bugs that only surface under real persistence.</p><p>The teams that ship it cleanly have one thing in common: they treat checkpointing like a database—with migrations, retention policies, access control patterns, and integration tests that run against real Postgres. The abstraction is good enough that you don’t need to understand its internals to use it. But you do need to understand what you’re building on top of it.</p><p>LangGraph’s persistence layer is still maturing. The async checkpointer, scoped storage APIs, and checkpoint garbage collection tools are all evolving. That’s expected for a system moving this fast. What shouldn’t catch you off guard is the operational gap between what the docs show and what production demands. Now you know where the edges are.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=398bc86861f4" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/langgraph-checkpointing-is-not-free-a-production-postmortem-398bc86861f4">LangGraph Checkpointing Is Not Free: A Production Postmortem</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[GRPO in Production: The Failure Modes Nobody Writes About]]></title>
            <link>https://levelup.gitconnected.com/grpo-in-production-the-failure-modes-nobody-writes-about-5d59c3fc9c3b?source=rss-3ec8787f291f------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d59c3fc9c3b</guid>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[reinforcement-learning]]></category>
            <dc:creator><![CDATA[Yuval Mehta]]></dc:creator>
            <pubDate>Mon, 22 Jun 2026 17:19:40 GMT</pubDate>
            <atom:updated>2026-06-22T17:19:40.677Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W5zSbhl8-jWteknY" /><figcaption>Photo by <a href="https://unsplash.com/@edge2edgemedia?utm_source=medium&amp;utm_medium=referral">Edge2Edge Media</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>GRPO is now the default RL algorithm for post-training LLMs. DeepSeek-R1 popularized it, and it’s since been adopted in Nemotron 3 Super, Qwen-Math derivatives, and dozens of open-source reasoning models. The explainer posts that followed all show the happy path: sample G responses per prompt, compute group-relative advantages, update the policy. Clean, elegant, cheaper than PPO because there’s no critic network.</p><p>What those posts don’t show is what happens at step 400 when your training loss goes flat, your rewards stop improving, and your model starts generating increasingly confident, increasingly wrong outputs. This is a post about those moments.</p><p><strong>TL;DR</strong></p><ul><li>GRPO’s group-relative advantage computation has three silent failure modes: entropy collapse, advantage collapse, and KL drift.</li><li>Advantage collapse is the most common and the least diagnosed — it happens whenever your rollout group returns uniform rewards, which is most of the time on hard prompts.</li><li>DAPO (Dynamic Sampling Policy Optimization) addresses all three with concrete algorithmic fixes you can apply to any GRPO training loop today.</li><li>Monitoring entropy and advantage variance as first-class training metrics is not optional at production scale.</li></ul><h3>What GRPO Actually Does</h3><p>GRPO (Group Relative Policy Optimization, introduced by DeepSeek in DeepSeekMath) is a policy gradient algorithm that gets rid of the critic model PPO requires. Instead of a learned value function estimating expected return, it estimates advantages by comparing responses <em>within a sampled group</em>.</p><p>For each prompt q, you sample G responses {y_1, ..., y_G} from the current policy π_θ, compute a scalar reward R_i for each, then normalize:</p><pre>Â_i = (R_i - mean(R)) / std(R)</pre><p>The policy is then updated to increase the probability of high-advantage responses and decrease low-advantage ones, subject to a KL penalty against a frozen reference model π_ref.</p><p>In PyTorch, the core training step looks roughly like this:</p><pre>import torch<br>import torch.nn.functional as F<br><br>def grpo_loss(<br>    log_probs: torch.Tensor,       # [G, seq_len] - current policy log probs<br>    old_log_probs: torch.Tensor,   # [G, seq_len] - log probs from rollout<br>    rewards: torch.Tensor,         # [G] - scalar rewards per response<br>    ref_log_probs: torch.Tensor,   # [G, seq_len] - reference model log probs<br>    epsilon: float = 0.2,<br>    beta: float = 0.04,<br>) -&gt; torch.Tensor:<br>    # Group-relative advantage: normalize within the sampled group<br>    advantages = (rewards - rewards.mean()) / (rewards.std() + 1e-8)<br>    advantages = advantages.unsqueeze(1)  # broadcast over sequence<br>    # PPO-style clipped surrogate objective<br>    ratio = torch.exp(log_probs - old_log_probs)<br>    clipped = torch.clamp(ratio, 1 - epsilon, 1 + epsilon)<br>    policy_loss = -torch.min(ratio * advantages, clipped * advantages).mean()<br>    # KL penalty against reference model<br>    kl_div = (old_log_probs - ref_log_probs).mean()<br>    <br>    return policy_loss + beta * kl_div</pre><p>This is clean. On math benchmarks with binary rewards (correct/incorrect), GRPO produces strong results — and eliminates the memory overhead of a value network, cutting training compute nearly in half versus PPO.</p><p>The problem is that this advantage formulation has sharp edges when your reward distribution stops cooperating.</p><h3>Failure Mode 1: Advantage Collapse</h3><p>This is the silent killer. It happens whenever all G responses in a group receive the <em>same</em> reward.</p><p>When std(R) ≈ 0the denominator Â_i = (R_i - mean(R)) / std(R) approaches zero. The 1e-8 The epsilon you add prevents a NaN, but what you get instead is a near-zero advantage for every response in the group. Zero advantage means zero gradient signal. The policy doesn&#39;t update. Training stalls.</p><p>When does this happen? Far more often than the GRPO paper implies:</p><ul><li><strong>Hard prompts</strong>: On genuinely difficult reasoning tasks, all G rollouts get reward = 0. No correct answer in the group means no contrast to learn from.</li><li><strong>Easy prompts</strong>: All G rollouts get a reward = 1. The model has already solved these — again, no contrast.</li></ul><p>As your model improves, the distribution of “easy” prompts grows. As you push into harder domains, the “impossible” set grows. You end up with a dumbbell distribution — and GRPO is starved of gradient signal at both ends.</p><pre>def diagnose_advantage_collapse(rewards_per_group: list[torch.Tensor]) -&gt; dict:<br>    &quot;&quot;&quot;<br>    Track the fraction of groups where advantage collapse occurs.<br>    Log this every N steps during training.<br>    &quot;&quot;&quot;<br>    collapsed = 0<br>    for group_rewards in rewards_per_group:<br>        if group_rewards.std() &lt; 1e-4:<br>            collapsed += 1<br>    <br>    collapse_rate = collapsed / len(rewards_per_group)<br>    return {<br>        &quot;advantage_collapse_rate&quot;: collapse_rate,<br>        # If this exceeds 0.3, your training is stalling. Act immediately.<br>    }</pre><p>DAPO’s fix: <strong>dynamic sampling</strong>. Filter out groups where all responses are correct or all are incorrect before computing the loss. Only train on groups with actual reward variance. This feels aggressive — you’re discarding a significant fraction of your rollout compute — but the alternative is updating the policy on noise.</p><h3>Failure Mode 2: Entropy Collapse</h3><p>Entropy collapse is slower-acting and harder to detect because your reward metrics can look fine while it’s happening.</p><p>As training progresses, the policy naturally concentrates probability mass on high-reward responses. This is the point of RL. But if entropy drops too fast, the policy loses diversity in its generations. You end up with a model that produces a narrow set of responses — and because GRPO’s group advantage is relative, a group of similar low-diversity responses gives you a similarly weak gradient signal.</p><p>Worse: once entropy collapses, it tends to stay collapsed. The policy has committed to a mode. Recovery requires either re-annealing the KL penalty or external injection of diversity.</p><pre>def token_entropy(log_probs: torch.Tensor) -&gt; float:<br>    &quot;&quot;&quot;<br>    Compute mean per-token entropy across a batch.<br>    Healthy range during early training: &gt; 2.0 nats.<br>    Alert threshold: &lt; 0.5 nats sustained for &gt; 50 steps.<br>    &quot;&quot;&quot;<br>    probs = log_probs.exp()<br>    entropy = -(probs * log_probs).sum(dim=-1).mean()<br>    return entropy.item()</pre><p>DAPO’s fix: an <strong>asymmetric KL clip</strong>. Standard GRPO clips the probability ratio symmetrically at [1-ε, 1+ε]. DAPO uses [1-ε_low, 1+ε_high] with ε_high &gt; ε_low — loosening the upper bound. This allows the policy to increase the probability of correct responses more aggressively, which counterintuitively <em>preserves entropy</em> because the model doesn&#39;t need to compress its output distribution as hard to capture the reward signal. More headroom means less pressure to collapse.</p><h3>Failure Mode 3: KL Drift</h3><p>The KL penalty in GRPO is meant to keep the updated policy close to the reference model (usually the SFT checkpoint you started from). In practice, the penalty coefficient β is a blunt instrument.</p><p>Set β too low and the policy drifts fast — you get reward hacking, degenerate outputs, and loss of the format and calibration that SFT gave you. Set it too high, and the penalty dominates the training signal — you&#39;re paying KL cost to stay close to a reference that&#39;s weaker than what you&#39;re trying to train toward. The model stops improving.</p><p>The standard approach of adding KL as a reward penalty (r_effective = r - β * KL(π || π_ref)) compounds this: you&#39;re shaping the reward signal itself, which interacts badly with group normalization. A group where one response has unusually high KL gets penalized in the reward, then again in the loss.</p><pre># STANDARD (problematic at scale):<br>kl_penalty = beta * (current_log_prob - ref_log_prob)<br>adjusted_reward = reward - kl_penalty  # KL baked into reward signal<br># DAPO-STYLE (cleaner):<br># Remove KL from the reward entirely.<br># Apply it as a direct loss term after advantage computation.<br># This decouples exploration pressure from reward normalization.<br>policy_loss = grpo_surrogate(log_probs, advantages)<br>kl_loss = F.kl_div(log_probs, ref_log_probs, reduction=&#39;batchmean&#39;, log_target=True)<br>total_loss = policy_loss + beta * kl_loss</pre><p>Decoupling KL from the reward lets you tune β on the loss side without distorting the reward distribution that advantage normalization depends on.</p><h3>Failure Mode 4 (Bonus): Token-Level vs. Sample-Level Loss Normalization</h3><p>This one is subtle but materially affects the reasoning model quality. Standard GRPO normalizes loss at the <strong>sample level</strong> — each response contributes equally to the gradient regardless of length.</p><p>This creates a structural bias against long chain-of-thought responses. A 2000-token reasoning trace is treated the same as a 200-token direct answer in terms of gradient weight. If the longer trace is what produced the correct answer, you’re underweighting the exact behavior you want to reinforce.</p><p>Token-level normalization, introduced in DAPO, divides the loss by the total number of tokens in the group rather than the number of responses:</p><pre>def token_level_loss(<br>    log_probs: torch.Tensor,      # [G, seq_len]<br>    advantages: torch.Tensor,     # [G, seq_len] — broadcast from [G]<br>    attention_mask: torch.Tensor, # [G, seq_len] — 1 for real tokens, 0 for padding<br>) -&gt; torch.Tensor:<br>    # Per-token policy loss<br>    per_token_loss = -log_probs * advantages * attention_mask<br>    <br>    # Normalize by total token count, not response count<br>    total_tokens = attention_mask.sum()<br>    return per_token_loss.sum() / total_tokens</pre><p>On math reasoning benchmarks, DAPO’s final configuration (50 AIME points) outperformed DeepSeek-R1-Zero-Qwen-32B (47 points) using 50% fewer training steps. Token-level normalization was one of the key contributors — not a minor tweak.</p><h3>Gotchas: What Nobody Tells You</h3><p><strong>1. Your reward model is probably the real bottleneck, not GRPO.</strong></p><p>All three GRPO failure modes are amplified by noisy rewards. If your verifier misclassifies 15% of responses, you’re injecting a false signal into every group. Advantage collapse gets worse because “hard” prompts, where the model is near threshold, flip between all-correct and all-incorrect groups unpredictably. Audit your reward model before diagnosing your training loop.</p><p><strong>2. GRPO failure modes compound across training stages.</strong></p><p>Entropy collapse in stage 1 (SFT warm-start) creates a fragile starting point that makes advantage collapse more likely in stage 2 (RL fine-tuning). Teams that report “GRPO didn’t work for us” often ran it on an SFT checkpoint that was already over-fitted to a narrow response format. Check the entropy of your SFT model before you start RL.</p><p><strong>3. The </strong><strong>G hyperparameter interacts non-linearly with all three failure modes.</strong></p><p>Increasing group size G reduces variance in advantage estimates and reduces collapse frequency — but also increases rollout cost linearly. Most production setups use G=8 or G=16. Going to G=32 can rescue a training run that&#39;s stuck, but the cost may not be worth it when dynamic sampling (dropping bad groups) achieves the same result at lower compute.</p><h3>Conclusion</h3><p>GRPO is genuinely a better default than PPO for LLM post-training: lower memory, comparable or better results, no critic. But the framing of “simpler than PPO” has led teams to underestimate its failure modes. Advantage collapse is structural — it’s not a bug you fix, it’s a property of group-relative normalization that you have to engineer around. Entropy collapse is slow and invisible until your loss curves look healthy, but your model has stopped generalizing.</p><p>The DAPO paper (Yu et al., 2025) is the most important engineering document in post-training right now, not because it’s theoretically novel, but because it maps each failure mode to a concrete fix: dynamic sampling, token-level normalization, asymmetric clip, and decoupled KL. You don’t have to adopt all of them. But you should understand all of them — because your training run will hit at least one.</p><p>The next frontier is agentic RL: multi-step tool-calling trajectories where reward is sparse, rollouts are expensive, and the assumption of “verifiable correct/incorrect” breaks down entirely. The failure modes are the same, just harder to detect. That’s a post for another day.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d59c3fc9c3b" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/grpo-in-production-the-failure-modes-nobody-writes-about-5d59c3fc9c3b">GRPO in Production: The Failure Modes Nobody Writes About</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Mechanistic Interpretability Is Having Its Moment: What Engineers Actually Need to Know]]></title>
            <link>https://pub.towardsai.net/mechanistic-interpretability-is-having-its-moment-what-engineers-actually-need-to-know-e4421f305f84?source=rss-3ec8787f291f------2</link>
            <guid isPermaLink="false">https://medium.com/p/e4421f305f84</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[ai-safety]]></category>
            <dc:creator><![CDATA[Yuval Mehta]]></dc:creator>
            <pubDate>Mon, 08 Jun 2026 22:01:00 GMT</pubDate>
            <atom:updated>2026-06-08T22:01:00.793Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>It just made MIT’s top-10 breakthrough technologies list. Anthropic, DeepMind, and OpenAI are all actively using it. Here’s what circuit-level analysis actually reveals — and why it matters for anyone building on LLMs.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*m3Jqh4lY6ZGfsTWn" /><figcaption>Photo by <a href="https://unsplash.com/@aoddeh?utm_source=medium&amp;utm_medium=referral">Ahmad Odeh</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>For most of its history, mechanistic interpretability occupied a narrow corner of AI safety research. The papers were technically rigorous, and the findings were genuinely surprising — individual neurons encode multiple unrelated concepts simultaneously; specific attention heads implement recognizable algorithms; models plan when writing poetry. But the work felt distant from production engineering. Interesting, not actionable.</p><p>That changed in 2025 and 2026. MIT Technology Review named mechanistic interpretability one of its ten breakthrough technologies of 2026. Anthropic applied attribution graphs to Claude 3.5 Haiku — a production model serving millions of users — and open-sourced the circuit tracing tooling that made it possible. DeepMind’s Gemma Scope 2 scaled sparse autoencoder analysis up to 27 billion parameters. OpenAI used chain-of-thought monitoring, a direct product of interpretability research, to catch a frontier model cheating on coding evaluations in real time.</p><p>The gap between “interpretability research” and “production engineering concern” is closing faster than most practitioners realize. This post is the bridge: what’s actually been discovered, how the tools work, and what it means for teams building on top of these models today.</p><p><strong>TL;DR</strong></p><ul><li>Mechanistic interpretability reverse-engineers neural networks to understand how they compute — not just what they output.</li><li>Two core concepts: features (human-interpretable directions in activation space) and circuits (causal pathways connecting features to outputs).</li><li>Sparse autoencoders decompose polysemantic neurons into interpretable features. Attribution graphs trace those features through a model’s computation.</li><li>Anthropic’s circuit tracing of Claude 3.5 Haiku revealed that the model plans ahead when writing poetry, processes concepts in language-independent circuits, and has persistent “reward model bias” features active during every assistant interaction.</li><li>For practitioners, the immediately relevant applications are: activation steering for behavior control, circuit-based debugging for failure mode investigation, and probe classifiers for real-time monitoring.</li></ul><h3>The Problem That Started Everything: Polysemanticity</h3><p>To understand why mechanistic interpretability exists, you need to understand the problem it’s trying to solve: neurons in neural networks don’t mean one thing.</p><p>In early interpretability research, the optimistic hypothesis was that individual neurons would specialize — one neuron for “cat,” another for “France,” another for “negation.” Reality is messier. A feature is an internal representation inside a neural network that corresponds to a human-interpretable concept — but features are not single neurons. Modern neural networks use superposition — they pack far more features into their neurons than there are neurons available.</p><p>This phenomenon — called polysemanticity — means a single neuron might activate for “curved shapes,” “sexual content,” “the number 17,” and “French text” simultaneously. These aren’t loosely related; they’re genuinely unconnected concepts sharing the same computational real estate. When you ask, “Why did the model produce this output?” looking at which neurons fired gives you a confusing mixture of unrelated signals.</p><p>Superposition arises because the model is under pressure to represent more concepts than it has neurons. It packs multiple sparse, mostly-non-overlapping features into the same neuron, exploiting the fact that any given forward pass will only activate a small fraction of features at once. The solution isn’t to train models differently — it’s to find a way to decompose the neuron’s activity into its constituent features.</p><p>That decomposition is what sparse autoencoders do.</p><h3>The Core Tools: Sparse Autoencoders and Attribution Graphs</h3><h4>Sparse Autoencoders (SAEs)</h4><p>A sparse autoencoder is a secondary neural network trained to decompose the activation vectors of a language model into a larger set of sparsely-active features. The key constraint is sparsity: on any given forward pass, only a small number of features should be active. This constraint forces the SAE to find the underlying structure — the distinct concepts packed into superposition — rather than just memorizing the activations.</p><p>python</p><pre># Conceptual SAE structure — not the full training setup, but the architecture<br>import torch<br>import torch.nn as nn<br><br>class SparseAutoencoder(nn.Module):<br>    def __init__(self, d_model: int, d_hidden: int, sparsity_coeff: float = 1e-3):<br>        super().__init__()<br>        # Encoder: model activations → sparse feature space<br>        self.encoder = nn.Linear(d_model, d_hidden)<br>        # Decoder: sparse features → reconstructed activations<br>        self.decoder = nn.Linear(d_hidden, d_model)<br>        self.sparsity_coeff = sparsity_coeff<br>    <br>    def forward(self, x: torch.Tensor) -&gt; tuple[torch.Tensor, torch.Tensor]:<br>        # Encode to sparse features (ReLU enforces non-negativity)<br>        features = torch.relu(self.encoder(x))<br>        # Reconstruct activations from features<br>        x_reconstructed = self.decoder(features)<br>        return features, x_reconstructed<br>    <br>    def loss(self, x: torch.Tensor) -&gt; torch.Tensor:<br>        features, x_reconstructed = self.forward(x)<br>        # Reconstruction loss: SAE should faithfully represent the activations<br>        reconstruction_loss = (x - x_reconstructed).pow(2).mean()<br>        # Sparsity loss: most features should be zero at any given time<br>        sparsity_loss = self.sparsity_coeff * features.abs().mean()<br>        return reconstruction_loss + sparsity_loss</pre><p>The features the SAE learns tend to be interpretable. Anthropic’s monosemanticity work on Claude 3 Sonnet found specific activation patterns for concepts like “the Golden Gate Bridge,” “rhyming words,” “legal language,” and “smallness.” When researchers artificially amplified the Golden Gate Bridge feature, the model became obsessed with it — weaving it into every response regardless of the prompt. This wasn’t a parlor trick; it demonstrated that the feature was causally responsible for the model’s behavior around that concept.</p><p>DeepMind’s Gemma Scope 2 scaled SAE analysis up to 27 billion parameters, and the open-source Gemma Scope tooling has become the standard starting point for practitioners wanting to explore feature decomposition on publicly available models.</p><p>The current limitation is real and worth naming: SAE-reconstructed activations cause 10–40% performance degradation on downstream tasks, and replacing GPT-4 activations with 16-million-latent SAE reconstructions degrades performance to roughly 10% of original pretraining compute. SAEs capture much of the model’s computation, but not all of it. The field is aware of this — it’s an active research problem, not an unknown unknown.</p><h4>Attribution Graphs and Circuit Tracing</h4><p>SAEs decompose neurons into features. Attribution graphs trace how those features cause each other through the model’s computation — producing a map of which features influence which other features, all the way from input tokens to output logits.</p><p>Anthropic’s circuit tracing methodology uses Cross-Layer Transcoders (CLTs), which replace dense MLP activations with interpretable, sparsely active features across transformer layers. These features act as “replacement neurons,” enabling the construction of attribution graphs — linearized, local computational maps that describe how information flows from input tokens through intermediate reasoning circuits to produce specific outputs.</p><p>The results of applying this to Claude 3.5 Haiku were surprising in their specificity. Researchers demonstrated a simple example where the model performs “two-hop” reasoning to identify that “the capital of the state containing Dallas” is “Austin” — and could see and manipulate an internal step where the model represents “Texas.” In poem generation, the model doesn’t just generate the next word — it engages in a form of planning, both forward and backward.</p><p>The planning finding is worth dwelling on. When Claude writes a poem, attribution graphs reveal that features corresponding to the intended rhyme word activate <em>before</em> the model writes the intervening lines. The model is working backward from a target — holding a future state in its internal representation while generating text that leads toward it. This is a structure that “next-token prediction” as a description of LLM computation doesn’t obviously predict.</p><p>In mid-2025, Anthropic open-sourced its circuit tracing library via the circuit-tracer Python package. It supports popular open-weights models (Gemma-2-2B, Llama-3.2-1B, Qwen3-4B) and ships with a frontend hosted on Neuronpedia for interactive graph exploration.</p><pre>pip install circuit-tracer</pre><pre>from circuit_tracer import load_model_and_transcoder, get_attribution_graph<br># Load a supported open-weights model with its pretrained transcoder<br>model, transcoder = load_model_and_transcoder(&quot;gemma-2-2b&quot;)<br><br># Get the attribution graph for a specific prompt and token position<br>prompt = &quot;The capital of the state containing Dallas is&quot;<br>target_token_position = -1  # The next token (what the model will predict)<br><br>graph = get_attribution_graph(<br>    model=model,<br>    transcoder=transcoder,<br>    prompt=prompt,<br>    target_position=target_token_position,<br>    threshold=0.01  # Only include edges with attribution &gt; 1%<br>)<br># Inspect top features influencing the prediction<br>for node in graph.top_nodes(n=10):<br>    print(f&quot;Feature {node.id}: &#39;{node.interpretation}&#39; - attribution: {node.attribution:.3f}&quot;)<br># Visualize interactively (opens in Neuronpedia)<br>graph.visualize()</pre><h3>What’s Been Found in Production Models</h3><p>The findings from applying these tools to frontier models are the most practically significant part of the field. Three discoveries stand out for engineering teams.</p><h4>Planning and Multi-Step Reasoning Are Internal, Not Sequential</h4><p>The discovery that Claude plans when writing poetry extends to more structured reasoning. Attribution graphs revealed that Claude 3.5 Haiku processes concepts in language-independent circuits before translating to specific languages. When the model reasons about Paris as the capital of France, it represents this relationship in an abstract conceptual space that’s then translated to whatever language the output requires — not in language-specific representations throughout.</p><p>This has a practical implication for multilingual deployments: reasoning quality shouldn’t degrade with output language as dramatically as you might expect, because the core reasoning happens upstream of language selection. If your model performs worse in one language, the problem is more likely in the output translation layer than in the reasoning circuits.</p><h4>Reward Model Bias Features Are Always On</h4><p>Attribution graphs revealed that Claude harbors “reward model bias” features that activate constantly during assistant interactions. These features appear to represent something like “this is an assistant interaction, and I should behave accordingly” — they’re persistently active throughout assistant-mode responses, not just when reward-relevant decisions are made.</p><p>This is direct empirical evidence for a behavior that alignment researchers had theorized but not observed directly: models trained with RLHF develop persistent internal representations of “being evaluated” that influence their behavior throughout interactions. For engineering teams, this isn’t necessarily alarming — it may be part of why RLHF-trained models are more helpful and less erratic than base models. But it’s a data point to hold when reasoning about edge cases where the model’s “assistant” behavior might conflict with the actual task.</p><h4>Hallucination Has Detectable Internal Structure</h4><p>Attribution graphs have been used to study hallucination inhibition mechanisms in Claude 3.5 Haiku — finding features that activate specifically when the model is uncertain and suppressing confident outputs. This is interpretability research at its most directly actionable: if you can identify features that correspond to “the model doesn’t actually know this,” you can build probes that detect this state at runtime.</p><p>This is the foundation for what’s becoming called “production probe classifiers” — lightweight linear classifiers trained on model activations to detect specific internal states: uncertainty, reasoning about evaluation, intent to take a particular class of action.</p><h3>Three Immediately Practical Applications</h3><h4>1. Activation Steering for Behavior Control</h4><p>If you can identify a feature that represents a specific concept or behavior, you can amplify or suppress it by adding a scaled version of the feature direction to the model’s activations during a forward pass. This is called activation steering.</p><pre>import torch<br>from transformer_lens import HookedTransformer<br><br>model = HookedTransformer.from_pretrained(&quot;gemma-2-2b&quot;)<br>def get_feature_direction(sae, feature_id: int) -&gt; torch.Tensor:<br>    &quot;&quot;&quot;Get the decoder direction for a specific SAE feature.&quot;&quot;&quot;<br>    return sae.W_dec[feature_id].clone()<br>def steer_activation(<br>    model,<br>    prompt: str,<br>    feature_direction: torch.Tensor,<br>    layer: int,<br>    alpha: float = 20.0  # Steering strength; negative to suppress<br>) -&gt; str:<br>    &quot;&quot;&quot;<br>    Add a scaled feature direction to model activations at a specific layer.<br>    Positive alpha amplifies the feature; negative alpha suppresses it.<br>    &quot;&quot;&quot;<br>    hook_name = f&quot;blocks.{layer}.hook_resid_post&quot;<br>    <br>    def steering_hook(value, hook):<br>        value[:, :, :] += alpha * feature_direction<br>        return value<br>    <br>    with model.hooks(fwd_hooks=[(hook_name, steering_hook)]):<br>        tokens = model.to_tokens(prompt)<br>        output = model.generate(tokens, max_new_tokens=100)<br>    <br>    return model.to_string(output[0])<br># Example: suppress a specific behavior feature<br>uncertainty_feature = get_feature_direction(sae, feature_id=1423)  # hypothetical<br>steered_output = steer_activation(<br>    model=model,<br>    prompt=&quot;Tell me about quantum computing&quot;,<br>    feature_direction=uncertainty_feature,<br>    layer=12,<br>    alpha=-15.0  # Suppress uncertainty features → more confident output<br>)</pre><p>Goodfire AI has productized this approach, offering an SDK that exposes feature-level steering on top of frontier models without requiring you to manage the underlying SAEs. For teams that want to adjust model behavior in specific, narrow domains without fine-tuning, activation steering is now a viable option.</p><h4>2. Probe Classifiers for Runtime Monitoring</h4><p>Train a lightweight logistic regression on model activations to detect specific internal states in real time. This is faster and cheaper than a second LLM call and more reliable than output-level heuristics.</p><pre>from sklearn.linear_model import LogisticRegression<br>from sklearn.model_selection import train_test_split<br>import numpy as np<br><br>def collect_activations(model, prompts: list[str], layer: int) -&gt; np.ndarray:<br>    &quot;&quot;&quot;Collect residual stream activations at a specific layer.&quot;&quot;&quot;<br>    activations = []<br>    for prompt in prompts:<br>        tokens = model.to_tokens(prompt)<br>        _, cache = model.run_with_cache(tokens)<br>        # Take the activation at the last token position<br>        act = cache[f&quot;blocks.{layer}.hook_resid_post&quot;][0, -1, :].detach().numpy()<br>        activations.append(act)<br>    return np.array(activations)<br># Build a probe for &quot;model is uncertain about this claim&quot;<br># Collect activations for known-uncertain and known-confident prompts<br>uncertain_prompts = [...]    # prompts where model hallucinated<br>confident_prompts = [...]    # prompts where model was verifiably correct<br>labels = [1] * len(uncertain_prompts) + [0] * len(confident_prompts)<br>all_prompts = uncertain_prompts + confident_prompts<br>X = collect_activations(model, all_prompts, layer=16)<br>y = np.array(labels)<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)<br>probe = LogisticRegression(max_iter=1000)<br>probe.fit(X_train, y_train)<br>print(f&quot;Probe accuracy: {probe.score(X_test, y_test):.3f}&quot;)<br># At inference time: check before returning response<br>def predict_with_uncertainty_check(model, probe, prompt: str, layer: int) -&gt; dict:<br>    activations = collect_activations(model, [prompt], layer)<br>    uncertainty_score = probe.predict_proba(activations)[0, 1]<br>    response = model.generate(...)<br>    return {<br>        &quot;response&quot;: response,<br>        &quot;uncertainty_score&quot;: uncertainty_score,<br>        &quot;flag_for_review&quot;: uncertainty_score &gt; 0.7<br>    }</pre><h4>3. Circuit-Based Failure Mode Investigation</h4><p>When your model produces a wrong or unexpected output, attribution graphs let you trace exactly which features were responsible. This is debugging at the computational level — not “the model said X when it should have said Y” but “feature 2341 (representing concept Z) had anomalously high attribution to this output, which explains the error.”</p><p>Lindsey et al. described ten investigations of model behavior using attribution graphs and intervention experiments, surfacing mechanisms in Claude 3.5 Haiku involved in behaviors like multi-step reasoning, planning, and hallucination inhibition. For engineering teams, the same methodology applies to specific failure modes in your domain: collect examples where the model fails, run attribution graphs on those examples, and look for patterns in which features are anomalously active.</p><h3>The Honest Limitations</h3><p>The fully understood “grokking” model has a single layer and processes numbers from 0 to 112. Production of large language models has hundreds of layers, billions of parameters, and processes the full complexity of human language. The gap between what we can explain and what we ship is enormous.</p><p>The risk is “streetlight interpretability” — we only understand the small parts that are easy to see, not necessarily the parts that matter most. The behaviors that have been explained through circuit analysis (two-digit addition, line breaks, poetry rhyming, two-hop reasoning) were selected in part because they were tractable. Explaining complex multi-step agentic reasoning or creative writing at the circuit level may be qualitatively harder — not just more work, but a different class of problem.</p><p>The SAE degradation problem also deserves honest acknowledgment. When you replace a model’s activations with SAE-reconstructed activations, you lose performance. The SAE is capturing most of the structure, but not all of it. Any technique built on top of SAEs — activation steering, probe classifiers, circuit tracing — operates on a slightly lossy representation of the model’s actual computation. For most practical applications, this is acceptable. For safety-critical monitoring, it’s a limitation to factor in.</p><h3>Gotchas Nobody Tells You</h3><p><strong>Feature interpretations are post-hoc human labels, not ground truth.</strong> When a paper says “we found a feature representing the Golden Gate Bridge,” what they mean is: this feature activates in contexts that humans, on inspection, associate with the Golden Gate Bridge. The feature doesn’t know it represents the Golden Gate Bridge. The interpretation is a hypothesis about what the feature does, validated by intervention experiments. Good interpretability research validates through perturbation (suppress the feature, see if the behavior changes). Bad interpretability research stops at the label.</p><p><strong>Activation steering degrades on out-of-distribution steering strengths.</strong> At low steering magnitudes, you get targeted behavior modification. At high magnitudes, you start producing incoherent outputs as the steered activations move outside the model’s training distribution. Finding the right steering strength requires empirical calibration per feature and per deployment context. There’s no universal safe range.</p><p><strong>The open-source tooling only covers open-weights models.</strong> circuit-tracer supports Gemma-2-2B, Llama-3.2-1B, and Qwen3-4B. If you&#39;re deploying Claude via API or GPT-4o, you can&#39;t run attribution graphs on production traffic — you don&#39;t have activation access. Probe classifiers can be built on open-weights proxies and may transfer partially, but interpretability findings from one model family don&#39;t automatically generalize to another.</p><h3>Conclusion</h3><p>Mechanistic interpretability matters to engineers now for the same reason that networking internals matter to web developers: you can build things without understanding the layer below, right up until you can’t. When your model produces a failure mode you can’t debug from outputs alone, when you need runtime monitoring that’s cheaper than a second LLM call, when you need targeted behavior modification without a full fine-tuning run — that’s when the layer below becomes relevant.</p><p>Anthropic’s work points toward a future where interpretability is not a luxury but a security primitive — as fundamental to AI system security as TLS is to web security. The progression from sparse autoencoders to circuit tracing to production probe classifiers follows the same maturation arc seen in every other security domain: research technique → validated methodology → deployable control.</p><p>The deployable control phase has started. The tooling is open-source, the findings are replicable across labs, and the gap between research and production engineering is the smallest it has ever been. You don’t need to understand every nuance of superposition to start using probe classifiers for uncertainty detection or activation steering for domain adaptation.</p><p>What you do need is to stop treating these models as black boxes by assumption. The box is opening.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e4421f305f84" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/mechanistic-interpretability-is-having-its-moment-what-engineers-actually-need-to-know-e4421f305f84">Mechanistic Interpretability Is Having Its Moment: What Engineers Actually Need to Know</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>