<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Kavishka Kalhara on Medium]]></title>
        <description><![CDATA[Stories by Kavishka Kalhara on Medium]]></description>
        <link>https://medium.com/@kavishkakalhara?source=rss-b3cb50279e64------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*e6I3TKCpsR9lc6kXNliC6g.jpeg</url>
            <title>Stories by Kavishka Kalhara on Medium</title>
            <link>https://medium.com/@kavishkakalhara?source=rss-b3cb50279e64------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 22:13:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@kavishkakalhara/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[10 GTA 6 Secret Locations Players Found in the First 48 Hours]]></title>
            <link>https://medium.com/@kavishkakalhara/10-gta-6-secret-locations-players-found-in-the-first-48-hours-e74015b9f850?source=rss-b3cb50279e64------2</link>
            <guid isPermaLink="false">https://medium.com/p/e74015b9f850</guid>
            <category><![CDATA[gaming]]></category>
            <category><![CDATA[rockstar-games]]></category>
            <category><![CDATA[vice-city]]></category>
            <category><![CDATA[gta-6]]></category>
            <category><![CDATA[easter-egg]]></category>
            <dc:creator><![CDATA[Kavishka Kalhara]]></dc:creator>
            <pubDate>Sat, 21 Nov 2026 10:45:27 GMT</pubDate>
            <atom:updated>2026-11-21T10:45:27.849Z</atom:updated>
            <content:encoded><![CDATA[<p><em>UFOs over Kelly County, a bunker with a code on a fire tower, and a house where someone figured out they live in a video game. Here is where to find Leonida’s best-hidden places.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TdiJ0E_xrIGQiwUsqNheuQ.jpeg" /></figure><p>Grand Theft Auto VI launched on November 19, 2026, after two delays and more than a decade of waiting. Within hours, players had stopped chasing the story and started climbing radio masts, sailing to empty islands and breaking into suburban garages.</p><p>Rockstar has always hidden strange things in its maps. GTA V had the Mount Chiliad mural, the Infinity Killer clues and the ghost on Mount Gordo. Leonida looks set to beat all of it. Early community counts already list more than 40 easter eggs, and new reports land every few minutes.</p><p>I went through the first wave of guides and player reports and picked the ten secret locations worth your time right now. For each one you get where it is, how to trigger it and what you actually get for the trip.</p><h3>Before you go hunting</h3><p><strong>Light spoilers ahead.</strong> Nothing here touches the main story, but it does reveal what each secret pays off.</p><p><strong>Everything below is player-reported.</strong> Rockstar has not published a list of secrets, and the game is only two days old. Guides sometimes disagree on a county name or a trigger time. Where they do, I say so. Some encounters are also random and will not spawn on every visit, so give them a few tries before you write them off.</p><h3>The 10 secrets at a glance</h3><ol><li><strong>UFO abduction</strong> — woods by the Starlite Motel, near Port Gellhorn. Payoff: “The Truth Is Out There” achievement.</li><li><strong>Mount Kalaga bunker</strong> — fire tower, Mount Kalaga National Park. Payoff: a logbook with a strange drawing.</li><li><strong>The literal Easter egg</strong> — tall communications mast, southern Leonida. Payoff: secret cutscene.</li><li><strong>The mystery house</strong> — Belville, Leonard County. Payoff: fourth-wall-breaking notes.</li><li><strong>Crashed paraglider</strong> — Bobcat Bight, North Mariana County. Payoff: Turnt Nipper .38 legendary pistol.</li><li><strong>SOS Island</strong> — islands west of Watson Bay. Payoff: found-footage phone recording.</li><li><strong>Skunk Ape</strong> — Grassrivers, around Cottonmouth Lake. Payoff: paranormal encounter.</li><li><strong>Dexter’s apartment</strong> — Trinity Apartments, Washington Island, Vice City. Payoff: TV reference.</li><li><strong>Purple meteorite</strong> — Table Rock, Mount Kalaga. Payoff: unexplained anomaly.</li><li><strong>Monument of Leonidan Supremacy</strong> — Manifest Canyon, Kelly County. Payoff: unsolved.</li></ol><h3>The secret locations</h3><h3>1. The UFO in the woods near Port Gellhorn</h3><p>This is the one everyone wants first. Head to the woodland around the Starlite Motel in Kelly County, east of Port Gellhorn, at roughly 2 to 3 AM in-game time. Players report better luck after finishing Chapter 1 and going alone.</p><p>Look for a car with flashing lights pulled over by the road. Follow the trail into the trees and you will find a UFO abducting a woman. Walk under the beam to free her, talk to her, then fetch her car. Finishing the encounter unlocks the <strong>“The Truth Is Out There”</strong> achievement. It does not spawn every night, so be patient.</p><h3>2. The Mount Kalaga bunker</h3><p>Deep in Mount Kalaga National Park in Lummox County, a tall metal tower stands near one of the summits. Climb it and check the sign near the top platform for a five-digit code. One early guide reports it as <strong>69185</strong>.</p><p>Back at the bottom, a locked wooden hut has a keypad. Enter the code to get inside. The reward is a logbook containing a ballpoint sketch of a mysterious figure. Nobody knows what it means yet, which is exactly why it is worth seeing.</p><h3>3. The literal Easter egg</h3><p>Rockstar made a pun, and it takes seven minutes of ladder climbing to get the punchline. Find the tallest communications mast in southern Leonida (guides disagree on the county) and climb its outside ladders all the way up.</p><p>At the top sits a bird’s nest full of brightly painted eggs. Grab one and throw it hard off the platform. A short cutscene plays showing the egg smashing into the windscreen of a car driving below. A weak throw may not trigger it.</p><h3>4. The mystery house in Belville</h3><p>In Belville, Leonard County, look for a pale single-storey house with a white garage door, white fencing and two palm trees out front. Break into the garage.</p><p>Inside are notes from a resident who seems to have realised they are living in a video game. They mention “J” and “L”, which players read as Jason and Lucia. Check the trunk of the car parked nearby for a jump scare. Breaking in can bring the police, so clear your wanted level first.</p><h3>5. The crashed paraglider and a legendary pistol</h3><p>Take a boat or jet ski to the small islands of Bobcat Bight, off the west coast of North Mariana County. Scan the treetops for a powered paraglider tangled in the branches.</p><p>Shoot the parachute to bring it down and loot the wreck for the <strong>Turnt Nipper .38</strong>, a legendary pistol. It is the most practical reward on this list.</p><h3>6. SOS Island</h3><p>Sail west from Watson Bay to a cluster of isolated islands. On one of them, “SOS” and “Send Beer” are written in the sand.</p><p>Search near the first S to find a phone holding a found-footage style recording. Players have also reported meteorite fragments on the island, though nobody has linked them to the recording yet.</p><h3>7. The Skunk Ape</h3><p>Leonida’s answer to Bigfoot lives in the Grassrivers swamp, the game’s take on the Everglades. Sightings cluster around Cottonmouth Lake, both in the wetlands south of Hamlet and near Sunrise RV Park to the north.</p><p>One reported trigger: find a deer carcass with a hairy creature crouched over it. Follow the creature into the woods until you reach a structure made of branches, then examine it. Get out of your car and listen, too. Players say you can hear it before you see it.</p><h3>8. Dexter’s apartment</h3><p>On Washington Island in Vice City, the Trinity Apartments are a near-copy of Dexter Morgan’s waterfront home from the TV series. Look for a pale low-rise building with balconies and a private dock.</p><p>Check the boats moored nearby for one resembling Dexter’s <em>Slice of Life</em>. There is no mission or cameo here. It is a pure reference, but a satisfying one if you watched the show.</p><h3>9. The purple meteorite</h3><p>On Table Rock in Mount Kalaga, drive or ride off-road to the summit, then work your way down the eastern slope. Search the rocky dips and undergrowth for a glowing purple meteorite.</p><p>Some players report you can collect meteorite pieces and sell them at a pawn shop. Others treat it as an unexplained anomaly. Between this, the SOS Island fragments and the UFO, something alien is clearly going on in Leonida.</p><h3>10. The Monument of Leonidan Supremacy</h3><p>Out in Manifest Canyon, Kelly County, stands a strange structure called the Monument of Leonidan Supremacy. Examine its markings and stay a while.</p><p>Some players say they hear unusual sounds near it that might be coded signals. Nobody has cracked it yet. If you like being early to a mystery, this is the spot to camp.</p><h3>Bonus finds and open mysteries</h3><p>These did not make the top ten, either because they are quick nods or because nobody has pinned them down yet.</p><ul><li><strong>Michael’s boat from GTA V.</strong> It is docked in the backyard of Jason’s safehouse in the Keys. You can board it in the opening hours.</li><li><strong>A Red Dead book.</strong> On the bar at Jacob’s Tavern in Millport, look for a Western novel set in New Austin.</li><li><strong>The jet ski dog.</strong> Swim far out from the western coast and wait. A dog on a jet ski may pull up and ferry you to a dog beach. It is random.</li><li><strong>A Stonehenge-style stone circle.</strong> It sits on top of the big mountain in Leonida’s far north. Nobody has confirmed what, if anything, it unlocks.</li><li><strong>The shed in South Bonanza.</strong> A hilltop shed in Lummox County reportedly hides a secret and a collectible. The reporter who found it kept the details back.</li><li><strong>An underground lab.</strong> Players describe a door panel that takes the code 2314 and leads to an alien laboratory with a button puzzle. Its location has not been mapped publicly yet.</li><li><strong>A ghost in Leonard County.</strong> Players report a pale girl resembling the one from <em>The Ring</em>.</li><li><strong>A paper trail in Port Gellhorn.</strong> Scattered notes build a dark mystery. One clue sits in the run-down building across from Moses’ repair shop.</li></ul><h3>How to find secrets yourself</h3><p>The best finds of the next few months have not been posted yet. These habits will give you a shot at one.</p><ol><li><strong>Clear the fog.</strong> Leonida’s map starts covered in black fog and only reveals itself as you travel. Taking back roads and waterways uncovers places a fast-travel player never sees.</li><li><strong>Climb everything tall.</strong> Two of the top ten sit on top of towers. If a ladder looks pointless, it probably is not.</li><li><strong>Play at night.</strong> The UFO only shows up in the small hours. Rockstar loves time-locked events, so revisit odd spots after dark.</li><li><strong>Get off the mainland.</strong> The glider, SOS Island and the jet ski dog are all offshore. Keep a boat handy.</li><li><strong>Read the paper.</strong> Notes, logbooks and books carry the lore. Interact with every one you pass.</li><li><strong>Turn the sound up.</strong> The Skunk Ape and the monument both give themselves away with noise before you see anything.</li><li><strong>Use the community.</strong> Interactive maps such as <a href="https://www.gtabase.com/gta-6/map/?feature=hidden-packages">GTABase</a> already list 30 hidden packages, and live trackers like <a href="https://secretsofleonida.com/">Secrets of Leonida</a> post new player reports by the minute.</li></ol><h3>Final thoughts</h3><p>Two days in, the pattern is already clear. Rockstar has scattered UFOs, meteorites, coded monuments and fourth-wall jokes across Leonida, and the threads seem to connect. GTA V’s Mount Chiliad mystery kept players digging for years. This one might keep us busy even longer.</p><p>I will update this list as new locations are confirmed. <strong>Found something I missed? Drop it in the comments with the location</strong>, and follow me for more GTA 6 guides as the hunt goes on.</p><h3>Sources</h3><ul><li><a href="https://www.ibtimes.co.uk/gta-6-player-discovers-literal-easter-egg-leonida-1833342">IBTimes UK: GTA 6 easter egg is literally an egg</a></li><li><a href="https://showgamer.com/en/guides/8037-vse-sekretnye-mesta-v-gta-6-gde-nayti-nlo-tayniki-i-pashalki">Showgamer: All secret locations in GTA 6</a></li><li><a href="https://querobolsa.com.br/revista/easter-eggs-de-gta-6">Quero Bolsa: 44 GTA 6 easter eggs (Portuguese)</a></li><li><a href="https://www.gamingbible.com/news/gta-6-map-hidden-locations-481643-20261120">GamingBible: GTA 6 map guide</a></li><li><a href="https://www.svg.com/2321415/gta-6-map-guide-key-locations-leonida-secrets/">SVG: GTA 6 map guide, key locations and secrets</a></li><li><a href="https://journalarta.com/en/2026/11/20/gta-6-map-guide-leonida-locations/">Journal Arta: GTA 6 map guide</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e74015b9f850" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Choosing a RAG Architecture Is the Hard Part, So I Built a Tool for It]]></title>
            <link>https://medium.com/@kavishkakalhara/choosing-a-rag-architecture-is-the-hard-part-so-i-built-a-tool-for-it-5cffce329a0e?source=rss-b3cb50279e64------2</link>
            <guid isPermaLink="false">https://medium.com/p/5cffce329a0e</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[rag-architecture]]></category>
            <category><![CDATA[rags]]></category>
            <dc:creator><![CDATA[Kavishka Kalhara]]></dc:creator>
            <pubDate>Fri, 20 Nov 2026 13:05:22 GMT</pubDate>
            <atom:updated>2026-11-20T13:05:22.720Z</atom:updated>
            <content:encoded><![CDATA[<p><em>ragpick 1.0 is out: pick the right RAG architecture for your system, define it as code, and prove it works on your data.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6qE_vl7PonAHY2Efz9QfCg.png" /></figure><p>Building a RAG pipeline is easy now. A few lines of LangChain or LlamaIndex, an embedding model, a vector store, and you have something that answers questions about your documents.</p><p>Choosing the <em>right</em> pipeline is a different problem. It’s the one that comes back to bite you later.</p><p>Plain vector search is good at meaning and bad at exact strings. Ask it about a product code or an order ID and it will happily return something that only looks similar. Your users notice.</p><p>Internal documents leak to the wrong people. Someone adds an HR policy to the index, and later a customer-facing bot quotes it. The system prompt said “don’t reveal internal documents”, but a system prompt is not a security boundary.</p><p>And “improved” pipelines ship without evidence. Someone adds a reranker or changes the chunk size, eyeballs three answers, and calls it better. Maybe it is. Nobody actually knows.</p><p>I built ragpick because I wanted those decisions to be explicit, reviewable and measured instead of guessed.</p><h3>What ragpick is</h3><p>ragpick is an open-source Python library and CLI that helps you <strong>pick the right RAG architecture for your system, define it as code, and prove it works on your data.</strong> You describe your requirements, it recommends a blueprint and explains why, your pipeline lives in a validated YAML file, and built-in evaluation lets you compare options on your own questions. It works alongside LangChain and LlamaIndex rather than replacing them, so the loaders, embeddings, retrievers and LLMs you already use there can be reused. Version 1.0.0 is out now under the Apache-2.0 license, for Python 3.11 and later.</p><h3>The workflow in three steps</h3><h3>1. Recommend</h3><p>Start by describing what your system actually needs: corpus size, whether queries are keyword-heavy, latency, access control, multi-hop questions, long documents and so on. ragpick recommend returns a blueprint along with the reasons it was chosen, the alternatives worth considering, and warnings about things you might trip over.</p><p>The recommendation comes from an explainable rule table, not a black box. If it suggests hybrid search because your queries are keyword-heavy, it tells you that. If you disagree, you can see exactly which rule you’re disagreeing with.</p><p>[IMAGE: Terminal output of ragpick recommend showing the recommended blueprint with its reasons, alternatives and warnings]</p><h3>2. Define it as YAML</h3><p>The pipeline is a rag.yaml file. I think of it as architecture as code: the shape of your retrieval system becomes a file you can diff, review in a pull request and roll back.</p><pre>version: 1<br>name: my-support-bot<br>blueprint: hybrid<br>requirements: {corpus_size: medium, keyword_heavy: true, access_control: true}<br>components:<br>  embedder:  {type: openai, params: {model: text-embedding-3-small}}<br>  generator: {type: openai, params: {model: gpt-4o-mini, temperature: 0}}<br>query: {top_k: 5}</pre><p>The file is validated when it loads, and the errors are meant to be read by people: when something is wrong, the message points at the YAML key that caused it.</p><h3>3. Evaluate and bench</h3><p>ragpick eval scores a pipeline on your questions. ragpick bench compares blueprints against the same questions. This is the step that turns &quot;I think hybrid is better&quot; into a result you can look at and argue about.</p><h3>Try it offline, no API key</h3><p>You don’t need an API key to see the whole loop. The repository includes a small sample dataset, and ragpick ships deterministic fake models for demos:</p><pre>pip install ragpick<br>git clone https://github.com/kavishkakalhara1/ragpick &amp;&amp; cd ragpick<br>ragpick init --blueprint naive --yes<br>ragpick ingest examples/data<br>ragpick query &quot;How do I reset my password?&quot;<br>ragpick bench examples/eval.jsonl --data examples/data --write best.yaml</pre><p>That sets up a project with the naive blueprint, ingests the sample documents, asks a question, and then runs the benchmark on the sample question set, writing the result to best.yaml.</p><h3>Let the data pick</h3><p>Here is what that last command produces on the bundled sample: nine short Markdown documents about a fictional thermostat company, 14 questions, and the offline fake embedder.</p><p>[IMAGE: ragpick bench result table on the bundled sample, with parent_child ranked first]</p><ul><li><strong>parent_child:</strong> MRR 0.964</li><li><strong>naive vector search:</strong> MRR 0.792</li></ul><p>Now the caveat, and I mean it: <strong>this is not a benchmark.</strong> It’s nine tiny documents and a fake embedder. It shows how the workflow runs, nothing more. Please don’t quote these numbers as evidence that one architecture beats another.</p><p>The process is the point. On your data, with your models and your questions, a different blueprint may win. Maybe naive is perfectly fine for you and you don’t need anything more complex. That’s a useful answer too, and now you have a run to back it up.</p><h3>The blueprints</h3><p>A blueprint is a known RAG architecture packaged as a starting point you can configure. These are stable in 1.0:</p><ul><li><strong>naive:</strong> plain vector search, good for prototypes.</li><li><strong>hybrid:</strong> BM25 plus vector search combined with Reciprocal Rank Fusion, with optional rerankers. The default for commercial systems.</li><li><strong>parent_child:</strong> searches small chunks, answers from whole sections.</li><li><strong>corrective:</strong> grades the retrieved context, rewrites the query, and says “I don’t know” instead of guessing.</li><li><strong>graph:</strong> GraphRAG, with a knowledge graph, multi-hop local search, and community summaries for “main themes” questions.</li><li><strong>agentic:</strong> plans several searches for multi-part and comparison questions.</li><li><strong>multimodal:</strong> images, diagrams and images in PDFs are described at ingest by a vision model, or from alt text and captions when offline.</li><li><strong>conversational:</strong> rewrites follow-up questions using the chat history.</li><li><strong>hierarchical:</strong> a RAPTOR-style summary tree.</li><li><strong>long_context:</strong> passes whole documents, for small corpora and long-context models, and is prompt-cache friendly.</li><li><strong>sql:</strong> text-to-SQL over CSV, SQLite or PostgreSQL.</li></ul><p>Planned next: <strong>adaptive</strong>, which will choose a strategy per question.</p><h3>Built for production, not just demos</h3><p>A lot of RAG tooling stops at the notebook. I wanted ragpick to cover the parts that matter once real users and real permissions are involved.</p><h3>Access control inside the store</h3><p>Documents carry allowed_groups, which you can set in Markdown front matter, for example. Filtering happens inside every vector store and the keyword index <strong>before ranking</strong>, so restricted text never reaches the LLM.</p><p>Why before ranking, and not after? Two reasons:</p><ul><li><strong>Security:</strong> if you retrieve first and filter later, restricted text has already moved through your pipeline. If you rely on the prompt to hide it, you’re trusting the model to keep a secret it has already read.</li><li><strong>Quality:</strong> if you take the top 5 results and then drop the ones the user can’t see, you may be left with one result, or none. Filtering first means the top 5 are the best 5 the user is allowed to see.</li></ul><p>The same rule covers graph facts and summaries: they’re only shown when the caller may see every source they came from. Queries that arrive without any groups see public documents only. Secure by default, not secure once someone remembers to configure it.</p><p>The sql blueprint gets the same treatment. Connections are read-only, and there’s an operation allow-list, single-statement validation, timeouts and row caps. Table-level access control is enforced by the database layer, not the prompt: SQLite through an authorizer, and PostgreSQL through EXPLAIN-plan checks that see through views and CTEs.</p><h3>Serving and multi-tenancy</h3><p>ragpick serve runs a JSON HTTP API with streaming. API keys are mapped to groups and tenants, so a client can&#39;t claim groups it doesn&#39;t have. Each tenant gets its own isolated index. You can route between pipelines, for example sending document questions to one pipeline and data questions to a sql pipeline, and chat sessions are owned by the API key that created them.</p><p>For operations, there’s tracing with an OpenTelemetry exporter, plus caching of embeddings and LLM calls.</p><h3>Quality gates in CI</h3><p>Evaluation isn’t just a report you read once. It can fail a build, which is what the next section is about.</p><h3>Evaluation without the busywork</h3><p>The usual reason teams skip evaluation is that nobody has a question set. ragpick tries to take that reason away.</p><p>ragpick generate-eval creates a starter question set from your documents. Treat it as a first draft: fix the questions that are off, and add the ones your users actually ask.</p><p>The metrics cover retrieval and answers: hit rate, MRR, precision, LLM-judged faithfulness and answer relevance, plus latency and cost. Then you set a threshold:</p><pre>ragpick eval --fail-under hit_rate@k=0.85</pre><p>If the score falls below the threshold, the command exits non-zero and your CI job fails. A change to the pipeline now has to prove itself the same way a code change does, before it reaches users.</p><p>[IMAGE: CI log where ragpick eval fails the build because hit_rate@k dropped below 0.85]</p><p>The loop closes with user feedback. Thumbs up and thumbs down, collected through the API, can be exported into regression tests. The bad answer a user flagged becomes a test case, so the same mistake can’t quietly come back.</p><h3>No lock-in</h3><p>ragpick is meant to sit in your stack, not take it over:</p><ul><li><strong>Adapters:</strong> LangChain and LlamaIndex adapters reuse your existing loaders, embeddings, retrievers and LLMs.</li><li><strong>Vector stores:</strong> in-memory, Qdrant, pgvector, Chroma and LanceDB.</li><li><strong>Providers:</strong> OpenAI, Ollama for local models, and sentence-transformers.</li><li><strong>Rerankers:</strong> cross-encoder, Cohere and Jina.</li><li><strong>Dependencies:</strong> the integrations add zero new dependencies.</li><li><strong>Plugins:</strong> third-party plugins register through Python entry points, so you can add your own components without forking.</li></ul><p>The base install is just pip install ragpick. Provider support comes through extras such as pip install &quot;ragpick[openai]&quot;, or &quot;ragpick[all]&quot; if you want everything.</p><h3>What 1.0 means</h3><p>For me, 1.0 is a promise rather than a celebration.</p><p>Within 1.x, configs, the Python API, the CLI and the HTTP API only change in compatible ways. Deprecations warn first, and removals wait for 2.0. A committed API snapshot is checked in CI, so an accidental breaking change fails the build instead of reaching your pip install -U.</p><p>Behind that promise there are 500+ tests running on Python 3.11 to 3.13, strict type checking, and a test suite that runs fully offline.</p><h3>Honest limits</h3><p>Here is what you should know before you rely on it:</p><ul><li>The offline fake models exist for demos and tests. Real answer quality needs a real embedding model and a real LLM.</li><li>The sample bench numbers above are illustrative only. They say nothing about your data.</li><li>GraphRAG global search uses simple community detection. It works, but it isn’t sophisticated.</li><li>It’s a 1.0, not a finished product. Feedback and issues are very welcome, and they’ll shape what comes next.</li></ul><h3>Try it, and help shape it</h3><p>If you’re building or maintaining a RAG system, I’d love for you to run ragpick against your own documents and questions and see what it tells you.</p><ul><li><strong>GitHub:</strong> <a href="https://github.com/kavishkakalhara1/ragpick">github.com/kavishkakalhara1/ragpick</a></li><li><strong>Docs:</strong> <a href="https://kavishkakalhara1.github.io/ragpick/">kavishkakalhara1.github.io/ragpick</a></li><li><strong>PyPI:</strong> <a href="https://pypi.org/project/ragpick/">pypi.org/project/ragpick</a></li></ul><p>A few ways to help:</p><ul><li>Star the repo if it looks useful. It helps other developers find it.</li><li>Try the offline quickstart, then point it at your own data.</li><li>Open an issue when something breaks, confuses you, or recommends the wrong thing.</li><li>Contribute a loader, a store or a blueprint. The plugin system is there for exactly that.</li></ul><p>Choosing a RAG architecture will always involve judgment. My hope is that with ragpick, the judgment is written down, the trade-offs are visible, and the final call is backed by your own data.</p><p>Thanks for reading.</p><p><em>Kavishka Kalhara, author and maintainer of ragpick. Find me at </em><a href="https://kavishka-kalhara.me/"><em>kavishka-kalhara.me</em></a><em> or on GitHub as </em><a href="https://github.com/kavishkakalhara1"><em>@kavishkakalhara1</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5cffce329a0e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Production-Grade BPE Tokenizer from Scratch in Python]]></title>
            <link>https://medium.com/@kavishkakalhara/building-a-production-grade-bpe-tokenizer-from-scratch-in-python-a560c8fa0c83?source=rss-b3cb50279e64------2</link>
            <guid isPermaLink="false">https://medium.com/p/a560c8fa0c83</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[nlp]]></category>
            <dc:creator><![CDATA[Kavishka Kalhara]]></dc:creator>
            <pubDate>Wed, 18 Nov 2026 13:51:54 GMT</pubDate>
            <atom:updated>2026-11-18T13:51:54.591Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Byte-level BPE, GPT-4-style pre-tokenization, a training loop that runs 100× faster than the naive version, special-token safety, serialization, and the testing you need before you trust it.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CYEVGNdJJnEl_H-Mi5RQcw.png" /></figure><p>Every LLM you have used reads integers, not text. The layer that turns your prompt into those integers, and turns the model’s output back into text, is the <strong>tokenizer</strong>. It is the least glamorous part of the stack and one of the most consequential. It decides how much context you can fit, how much each API call costs, how well the model handles your language, and a surprising number of the model’s odd failures.</p><p>Most tutorials stop at a toy version that trains on a paragraph and takes minutes on anything bigger. This article goes further. We will build a byte-level BPE tokenizer in the style of GPT-2 and GPT-4, and then make it something you could actually use:</p><ul><li><strong>Byte-level BPE</strong>, so no input is ever “unknown”</li><li><strong>Regex pre-tokenization</strong>, including the GPT-4 (cl100k_base) split pattern</li><li><strong>A fast trainer</strong> using chunk deduplication, an inverted index and a lazy max-heap</li><li><strong>Cached encoding</strong> with correct merge-rank ordering</li><li><strong>Special tokens</strong> that cannot be injected by user input</li><li><strong>Save and load</strong>, plus GPT-2’s readable byte-to-unicode vocab format</li><li><strong>Testing</strong>: equivalence tests, round-trip fuzzing and compression metrics</li></ul><p>Everything here is plain Python. The only dependency is the regex package. You can follow along in a notebook.</p><pre>pip install regex</pre><h3>1. The tokenizer design space</h3><p>A tokenizer has three parts: a <strong>vocabulary</strong> (ID to text piece), an <strong>encoder</strong> (text to IDs) and a <strong>decoder</strong> (IDs to text). The core design choice is the size of the pieces.</p><ul><li><strong>Characters</strong> keep the vocabulary tiny but make sequences long. Attention cost grows with sequence length, and the model has to learn spelling from scratch.</li><li><strong>Whole words</strong> give short sequences but a huge vocabulary, and any word missing from training becomes an &lt;unk&gt; token.</li><li><strong>Subwords</strong> sit in between. Frequent words become one token, and rare words are built from frequent pieces.</li></ul><p>There are three main subword algorithms:</p><ul><li><strong>BPE (Byte Pair Encoding)</strong> starts from small units and greedily merges the most frequent adjacent pair. It is used by GPT-2, GPT-3, GPT-4, Llama 3 and many others.</li><li><strong>WordPiece</strong> is similar, but chooses merges by how much they raise the likelihood of the training data rather than by raw frequency. BERT uses it.</li><li><strong>Unigram</strong> works in reverse. It starts with a large candidate vocabulary and prunes the pieces that matter least, under a probabilistic model. T5 and many models built with SentencePiece use it.</li></ul><p>We will build BPE because it is the most widely used in today’s generative models, and because every step of it is easy to inspect.</p><h3>2. Why bytes, not characters</h3><p>Unicode defines about 150,000 characters. If our base vocabulary were characters, we would either need all of them, or we would need an &lt;unk&gt; fallback for the ones we skipped.</p><p>Byte-level BPE avoids this. Every string can be encoded as UTF-8 bytes, and there are only 256 possible byte values. So the base vocabulary is exactly 256 tokens, and any input can be tokenized: English, Sinhala, Chinese, emoji, binary garbage.</p><pre>&gt;&gt;&gt; list(&quot;hi&quot;.encode(&quot;utf-8&quot;))<br>[104, 105]<br>&gt;&gt;&gt; list(&quot;ආ&quot;.encode(&quot;utf-8&quot;))      # one Sinhala letter = 3 bytes<br>[224, 182, 134]<br>&gt;&gt;&gt; list(&quot;🇱🇰&quot;.encode(&quot;utf-8&quot;))     # the Sri Lankan flag = 8 bytes<br>[240, 159, 135, 177, 240, 159, 135, 176]</pre><p>The price is that non-Latin scripts start out expensive. A Sinhala letter is 3 bytes before any merges. Whether it ends up as one token or three depends entirely on the training data, which we will measure later.</p><h3>3. The core algorithm</h3><p>BPE training repeats one step. Count every adjacent pair of tokens, take the most frequent pair, give it a new ID, and replace it everywhere.</p><p>Two helpers do almost all the work:</p><pre>def get_stats(ids, counts=None):<br>    &quot;&quot;&quot;Count adjacent pairs: [1, 2, 3, 1, 2] -&gt; {(1,2): 2, (2,3): 1, (3,1): 1}&quot;&quot;&quot;<br>    counts = {} if counts is None else counts<br>    for pair in zip(ids, ids[1:]):<br>        counts[pair] = counts.get(pair, 0) + 1<br>    return counts<br></pre><pre>def merge(ids, pair, idx):<br>    &quot;&quot;&quot;Replace every occurrence of `pair` in `ids` with the new token `idx`.&quot;&quot;&quot;<br>    out, i = [], 0<br>    while i &lt; len(ids):<br>        if i &lt; len(ids) - 1 and ids[i] == pair[0] and ids[i + 1] == pair[1]:<br>            out.append(idx)<br>            i += 2<br>        else:<br>            out.append(ids[i])<br>            i += 1<br>    return out</pre><p>Here is the classic example, aaabdaaabac:</p><pre>aaabdaaabac   most frequent pair: &quot;aa&quot; (4x)  -&gt; new token Z<br>ZabdZabac     most frequent pair: &quot;ab&quot; (2x)  -&gt; new token Y<br>ZYdZYac       most frequent pair: &quot;ZY&quot; (2x)  -&gt; new token X<br>XdXac         11 symbols compressed to 5</pre><p>The ordered list of merges (aa→Z, ab→Y, ZY→X) <em>is</em> the trained tokenizer. The vocabulary can be rebuilt from it at any time.</p><h3>4. Pre-tokenization: where most of the design lives</h3><p>If you run BPE on raw text, merges cross word boundaries. You end up with separate tokens for dog., dog! and dog?, and even for e the spanning two words. That wastes vocabulary on near-duplicates.</p><p>The fix is to <strong>split text into chunks with a regex first</strong>, and only allow merges inside a chunk. GPT-2 introduced this pattern:</p><pre>import regex as re</pre><pre>GPT2_SPLIT_PATTERN = r&quot;&quot;&quot;&#39;(?:[sdmt]|ll|ve|re)| ?\p{L}+| ?\p{N}+| ?[^\s\p{L}\p{N}]+|\s+(?!\S)|\s+&quot;&quot;&quot;</pre><p>Reading it left to right:</p><ul><li>&#39;(?:[sdmt]|ll|ve|re) splits off English contractions: &#39;s, &#39;t, &#39;re and so on.</li><li>?\p{L}+ matches a run of letters in any script, with an optional leading space.</li><li>?\p{N}+ matches a run of digits, with an optional leading space.</li><li>?[^\s\p{L}\p{N}]+ matches a run of punctuation or symbols.</li><li>\s+(?!\S) matches whitespace, but leaves the last space so it can attach to the next word.</li><li>\s+ catches any remaining whitespace.</li></ul><p>\p{L} and \p{N} are Unicode property classes, and they are why we need the regex package. Python&#39;s built-in re does not support them.</p><p>The leading space is deliberate. It is why &quot; the&quot; and &quot;the&quot; are different tokens in GPT models, and why a stray space at the end of a prompt can change a model&#39;s output.</p><p>GPT-4’s cl100k_base tokenizer refined the pattern:</p><pre>GPT4_SPLIT_PATTERN = r&quot;&quot;&quot;&#39;(?i:[sdmt]|ll|ve|re)|[^\r\n\p{L}\p{N}]?+\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]++[\r\n]*|\s*[\r\n]|\s+(?!\S)|\s+&quot;&quot;&quot;</pre><p>You can see the differences on a single sentence:</p><pre>s = &quot;Hello world, it&#39;s 2026!   HOW&#39;S it going?\n\n&quot;</pre><pre>re.findall(GPT2_SPLIT_PATTERN, s)<br># [&#39;Hello&#39;, &#39; world&#39;, &#39;,&#39;, &#39; it&#39;, &quot;&#39;s&quot;, &#39; 2026&#39;, &#39;!&#39;, &#39;  &#39;, &#39; HOW&#39;, &quot;&#39;&quot;, &#39;S&#39;,<br>#  &#39; it&#39;, &#39; going&#39;, &#39;?&#39;, &#39;\n\n&#39;]</pre><pre>re.findall(GPT4_SPLIT_PATTERN, s)<br># [&#39;Hello&#39;, &#39; world&#39;, &#39;,&#39;, &#39; it&#39;, &quot;&#39;s&quot;, &#39; &#39;, &#39;202&#39;, &#39;6&#39;, &#39;!&#39;, &#39;  &#39;, &#39; HOW&#39;, &quot;&#39;S&quot;,<br>#  &#39; it&#39;, &#39; going&#39;, &#39;?\n\n&#39;]</pre><p>Three changes matter:</p><ol><li><strong>Case-insensitive contractions.</strong> (?i:...) means HOW&#39;S gives &#39;S instead of a stray &#39; and S.</li><li><strong>Numbers are capped at 3 digits</strong>, and get no leading space. 2026 becomes 202 + 6. This stops the vocabulary from filling up with arbitrary numbers like 1984 and 2048, and gives the model more consistent pieces to do arithmetic with.</li><li><strong>Newlines are handled separately.</strong> Punctuation can absorb trailing newlines (?\n\n), and whitespace runs that end in a newline get their own chunk. This makes code and structured text tokenize much more cleanly.</li></ol><p>Pre-tokenization is a design decision, not an implementation detail. Changing the regex changes the tokenizer more than most hyperparameters.</p><h3>5. The naive trainer, and why it is slow</h3><p>The straightforward trainer splits the text into chunks, then recounts every pair in every chunk after each merge:</p><pre>def train_naive(text, vocab_size, pattern=GPT2_SPLIT_PATTERN):<br>    chunks = [list(c.encode(&quot;utf-8&quot;)) for c in re.findall(pattern, text)]<br>    merges = {}<br>    for i in range(vocab_size - 256):<br>        stats = {}<br>        for c in chunks:<br>            get_stats(c, stats)<br>        if not stats:<br>            break<br>        # highest count wins; ties go to the smallest pair, so results are deterministic<br>        pair = max(stats, key=lambda p: (stats[p], -p[0], -p[1]))<br>        idx = 256 + i<br>        chunks = [merge(c, pair, idx) for c in chunks]<br>        merges[pair] = idx<br>    return merges</pre><p>It is correct, and we will keep it as a <strong>reference implementation</strong> to test against. But each merge costs time proportional to the whole corpus, so training costs roughly <em>corpus size × number of merges</em>. On a 145 KB corpus, 256 merges took 7.8 seconds on my machine. A real tokenizer does 50,000 to 200,000 merges over gigabytes of text.</p><h3>6. A fast trainer</h3><p>Three observations make training dramatically faster.</p><p><strong>Observation 1: chunks repeat.</strong> In English text, &quot; the&quot; might appear tens of thousands of times. Since merges never cross chunk boundaries, every copy of &quot; the&quot; is tokenized identically. So we count unique chunks once, and work on a few thousand unique &quot;words&quot; with frequencies instead of the full text.</p><p><strong>Observation 2: a merge only affects words that contain the pair.</strong> When we merge (t, h), a word like &quot; dog&quot; is untouched. If we keep an <strong>inverted index</strong> from each pair to the words containing it, we only need to update those words.</p><p><strong>Observation 3: we only need the maximum.</strong> Instead of scanning all pair counts after each merge, we keep them in a <strong>max-heap</strong>. When a count changes, we push a new entry, and leave the old one in place. When we pop an entry, we check it against the live count and skip it if it is stale. This is called <em>lazy invalidation</em>. It avoids the slow job of deleting from the middle of a heap.</p><pre>import heapq<br>from collections import Counter, defaultdict<br></pre><pre>def train_fast(text, vocab_size, pattern=GPT2_SPLIT_PATTERN, min_frequency=2):<br>    # Observation 1: count unique chunks.<br>    return train_from_counts(Counter(re.findall(pattern, text)), vocab_size, min_frequency)<br></pre><pre>def train_from_counts(chunk_counts, vocab_size, min_frequency=2):<br>    words = [list(c.encode(&quot;utf-8&quot;)) for c in chunk_counts]<br>    freqs = list(chunk_counts.values())</pre><pre>    # Global pair counts, plus Observation 2: which words contain each pair.<br>    pair_counts = defaultdict(int)<br>    where = defaultdict(set)<br>    for wi, (w, f) in enumerate(zip(words, freqs)):<br>        for pair in zip(w, w[1:]):<br>            pair_counts[pair] += f<br>            where[pair].add(wi)</pre><pre>    # Observation 3: a max-heap of (-count, pair). Python&#39;s heapq is a min-heap.<br>    heap = [(-c, p) for p, c in pair_counts.items()]<br>    heapq.heapify(heap)</pre><pre>    merges = {}<br>    for i in range(vocab_size - 256):<br>        while heap:<br>            neg, pair = heapq.heappop(heap)<br>            if pair_counts.get(pair, 0) == -neg:<br>                break                      # this entry is current<br>        else:<br>            break                          # nothing left to merge<br>        if -neg &lt; min_frequency:<br>            break                          # don&#39;t learn merges seen only once<br>        idx = 256 + i<br>        merges[pair] = idx</pre><pre>        changed = set()<br>        for wi in list(where[pair]):<br>            w, f = words[wi], freqs[wi]<br>            for p in zip(w, w[1:]):        # remove the word&#39;s old pairs<br>                pair_counts[p] -= f<br>                where[p].discard(wi)<br>                changed.add(p)<br>            w = merge(w, pair, idx)<br>            words[wi] = w<br>            for p in zip(w, w[1:]):        # add the word&#39;s new pairs<br>                pair_counts[p] += f<br>                where[p].add(wi)<br>                changed.add(p)</pre><pre>        for p in changed:                  # push fresh heap entries<br>            if pair_counts[p] &gt; 0:<br>                heapq.heappush(heap, (-pair_counts[p], p))<br>            else:<br>                pair_counts.pop(p, None)<br>                where.pop(p, None)<br>    return merges</pre><p>A few details are worth pointing out:</p><ul><li><strong>Removing and re-adding all of a word’s pairs</strong> is simpler and safer than patching only the neighbours of the merged pair. It handles overlapping cases like aaa correctly with no special logic. Words are short, so it costs little.</li><li><strong>Tie-breaking matches the naive trainer.</strong> The heap orders by (-count, pair), so on a tie the smallest pair wins, which is the same rule train_naive uses. That makes the two trainers produce <em>identical</em> merges, which lets us test one against the other.</li><li><strong>min_frequency</strong> stops training once the best pair appears only once. Merges like that only memorize noise. Hugging Face&#39;s trainer has the same option.</li></ul><h3>Did it work?</h3><p>First, correctness. On a 20 KB sample with min_frequency=1, both trainers must learn exactly the same 100 merges:</p><pre>small = text[:20000]<br>assert train_naive(small, 356) == train_fast(small, 356, min_frequency=1)</pre><p>It passes. Then speed, on the 145 KB corpus with 256 merges, on my 2-core machine:</p><ul><li>train_naive: <strong>7.8 s</strong></li><li>train_fast: <strong>0.08 s</strong></li></ul><p>That is roughly a <strong>100× speedup</strong>, and the gap grows with corpus size and vocabulary size. Your exact numbers will vary with hardware and data.</p><h3>Training on corpora bigger than RAM</h3><p>Because the fast trainer only needs chunk counts, you can stream files and never hold the corpus in memory:</p><pre>def count_chunks(paths, pattern=GPT2_SPLIT_PATTERN):<br>    compiled, counts = re.compile(pattern), Counter()<br>    for path in paths:<br>        with open(path, encoding=&quot;utf-8&quot;, errors=&quot;replace&quot;) as f:<br>            for line in f:<br>                counts.update(compiled.findall(line))<br>    return counts</pre><pre>merges = train_from_counts(count_chunks([&quot;part1.txt&quot;, &quot;part2.txt&quot;]), vocab_size=32000)</pre><p>Splitting by line changes how whitespace runs that cross newlines are chunked, so results can differ very slightly from training on the whole string. Counting is also easy to parallelize: count each file in a separate process, then add the Counters together.</p><h3>7. Encoding correctly, and fast</h3><p>Encoding new text must <strong>replay the merges in the order they were learned</strong>. Merge IDs are assigned in order, so the rule is: among all adjacent pairs in the sequence, merge the one with the <strong>lowest merge ID</strong> first. Repeat until no pair has a merge rule.</p><p>A common bug is to apply merges left to right, or by frequency in the new text. Both produce different tokens from the ones the model was trained on.</p><p>Since text splits into chunks, and the same chunks repeat constantly, we can cache the result per chunk:</p><pre>from functools import lru_cache</pre><pre>@lru_cache(maxsize=2**16)<br>def _encode_chunk(self, chunk_bytes):<br>    ids = list(chunk_bytes)<br>    while len(ids) &gt;= 2:<br>        # the earliest-learned merge has the lowest id<br>        pair = min(zip(ids, ids[1:]), key=lambda p: self.merges.get(p, float(&quot;inf&quot;)))<br>        if pair not in self.merges:<br>            break<br>        ids = merge(ids, pair, self.merges[pair])<br>    return tuple(ids)</pre><p>On my machine, encoding the training corpus ran at about 2.5 MB/s with a cold cache and about 8 MB/s with a warm one. That is fine for experiments, but well below compiled tokenizers. We will come back to this in section 11.</p><h3>8. Special tokens, and why they are a security boundary</h3><p>Special tokens such as &lt;|endoftext|&gt; mark document boundaries, and chat models use others to mark the roles in a conversation. They have two important properties:</p><ol><li>They must <strong>never be split</strong> by BPE. &lt;|endoftext|&gt; has to be a single reserved ID.</li><li>They must <strong>never come from untrusted text</strong>.</li></ol><p>The second point is easy to miss. Suppose your app encodes user input with special tokens enabled. A user could type &lt;|endoftext|&gt; or a fake role marker, and the model would see a real control token. That is a prompt-injection hole at the tokenizer level.</p><p>tiktoken handles this by raising an error if special-token text appears in input you haven&#39;t explicitly allowed. We will copy that behaviour:</p><pre>def encode(self, text, allowed_special=frozenset()):<br>    &quot;&quot;&quot;allowed_special: &quot;all&quot;, or a set of special strings to honour.<br>    Any other special string found in the text raises an error.&quot;&quot;&quot;<br>    if allowed_special == &quot;all&quot;:<br>        allowed = set(self.special_tokens)<br>    else:<br>        allowed = set(allowed_special)<br>    for s in self.special_tokens:<br>        if s not in allowed and s in text:<br>            raise ValueError(f&quot;text contains disallowed special token {s!r}&quot;)<br>    if not allowed:<br>        return self.encode_ordinary(text)<br>    # longest first, so overlapping specials match correctly<br>    special_pat = &quot;(&quot; + &quot;|&quot;.join(re.escape(s) for s in sorted(allowed, key=len, reverse=True)) + &quot;)&quot;<br>    ids = []<br>    for part in re.split(special_pat, text):<br>        if part in allowed:<br>            ids.append(self.special_tokens[part])<br>        elif part:<br>            ids.extend(self.encode_ordinary(part))<br>    return ids</pre><p>That gives three clear modes:</p><pre>tok.encode(&quot;hi &lt;|endoftext|&gt;&quot;)<br># ValueError: text contains disallowed special token &#39;&lt;|endoftext|&gt;&#39;</pre><pre>tok.encode(&quot;hi&lt;|endoftext|&gt;&quot;, allowed_special=&quot;all&quot;)[-1]<br># the reserved id, as a single token (trusted text you built yourself)</pre><pre>tok.encode_ordinary(&quot;&lt;|endoftext|&gt;&quot;)<br># 8 ordinary tokens: the literal characters, harmless (untrusted user text)</pre><p>The rule in practice: build prompts from trusted templates with special tokens allowed, and encode user content with encode_ordinary.</p><h3>9. Persistence and readable vocabularies</h3><p>The merges list is all you need to save, as long as it stays in order. The order is the merge rank. Save the regex pattern too, because a tokenizer without its exact pre-tokenizer is a different tokenizer.</p><pre>def save(self, path):<br>    data = {<br>        &quot;version&quot;: 1,<br>        &quot;pattern&quot;: self.pattern,<br>        &quot;merges&quot;: [[a, b] for (a, b) in self.merges],  # order = rank<br>        &quot;special_tokens&quot;: self.special_tokens,<br>    }<br>    with open(path, &quot;w&quot;, encoding=&quot;utf-8&quot;) as f:<br>        json.dump(data, f)</pre><pre>@classmethod<br>def load(cls, path):<br>    with open(path, encoding=&quot;utf-8&quot;) as f:<br>        data = json.load(f)<br>    tok = cls(pattern=data[&quot;pattern&quot;])<br>    tok.merges = {(a, b): 256 + i for i, (a, b) in enumerate(data[&quot;merges&quot;])}<br>    tok.special_tokens = data[&quot;special_tokens&quot;]<br>    tok._build_vocab()<br>    return tok</pre><p>To inspect a vocabulary, raw bytes are awkward. Many tokens contain spaces, newlines or half of a UTF-8 character. GPT-2 solved this by mapping each of the 256 bytes to a printable Unicode character. Printable ASCII maps to itself, and the rest are shifted to characters above 255. That is why you see Ġ (space) and Ċ (newline) in Hugging Face vocab files.</p><pre>def bytes_to_unicode():<br>    bs = (list(range(ord(&quot;!&quot;), ord(&quot;~&quot;) + 1))<br>          + list(range(ord(&quot;¡&quot;), ord(&quot;¬&quot;) + 1))<br>          + list(range(ord(&quot;®&quot;), ord(&quot;ÿ&quot;) + 1)))<br>    cs, n = bs[:], 0<br>    for b in range(256):<br>        if b not in bs:<br>            bs.append(b)<br>            cs.append(256 + n)<br>            n += 1<br>    return dict(zip(bs, map(chr, cs)))</pre><p>Dumping my trained vocabulary this way shows the tokenizer learning English bottom up:</p><pre>256  Ġt<br>257  ĠĠ<br>258  Ġth<br>259  Ġa<br>260  er<br>...<br>992  Ġthir<br>994  Ġway</pre><h3>10. Testing a tokenizer</h3><p>A tokenizer bug is silent and permanent. If you train a model on buggy token IDs, the bug ships with the model. These are the tests I would run before trusting one.</p><p><strong>1. Equivalence against the reference.</strong> The fast trainer must reproduce the naive trainer’s merges exactly (section 6).</p><p><strong>2. Round-trip fuzzing.</strong> decode(encode(s)) == s must hold for <em>every</em> string, not just nice ones. Throw random Unicode at it, including combining marks, zero-width joiners, emoji sequences and scripts the tokenizer never saw:</p><pre>import random<br>rng = random.Random(0)<br>alphabet = &quot;abc XYZ 123\n\t.,!?&#39;é你好ආයුබෝවන්🇱🇰🙂‍&quot;</pre><pre>for _ in range(2000):<br>    s = &quot;&quot;.join(rng.choice(alphabet) for _ in range(rng.randint(0, 60)))<br>    assert tok.decode(tok.encode(s)) == s, repr(s)</pre><pre>for _ in range(500):  # arbitrary code points, minus surrogates (not valid UTF-8)<br>    s = &quot;&quot;.join(chr(c) for c in (rng.randint(0, 0x10FFFF) for _ in range(20))<br>                if not 0xD800 &lt;= c &lt;= 0xDFFF)<br>    assert tok.decode(tok.encode(s)) == s</pre><p><strong>3. Save/load stability.</strong> A reloaded tokenizer must produce identical IDs on held-out text.</p><p><strong>4. Compression and fertility.</strong> Two numbers tell you how good a tokenizer is for your data. <strong>Bytes per token</strong> measures compression (higher is better). <strong>Tokens per word</strong>, often called <em>fertility</em>, shows how fairly it treats different languages (lower is better).</p><p>Here is my small tokenizer, trained only on English license texts:</p><ul><li><strong>Held-out English:</strong> 4.55 bytes per token</li><li><strong>English sentence:</strong> 68 bytes became 15 tokens (1.25 tokens per word)</li><li><strong>Python code:</strong> 65 bytes became 29 tokens (3.2 tokens per word)</li><li><strong>Sinhala sentence:</strong> 134 bytes became <strong>134 tokens</strong> (16.75 tokens per word)</li></ul><p>The Sinhala result is the most important line in this article. The tokenizer never saw Sinhala, so it learned no Sinhala merges, and every byte became its own token. Each Sinhala word costs about 13 times as many tokens as an English word. In a real model that means more cost, less context and worse quality for the same content.</p><p>Production tokenizers have the same problem in a milder form. That is why vocabulary sizes have grown, from about 50k in GPT-2 to about 100k in GPT-4’s cl100k_base and about 200k in o200k_base. Much of that extra space goes to better coverage of non-English languages. If you are building for Sinhala, Tamil or any other under-represented language, <strong>make sure it is well represented in your tokenizer&#39;s training data</strong>, and measure fertility for each language.</p><h3>11. The complete tokenizer</h3><p>Here is everything assembled into one class, exactly as I tested it. The helpers (get_stats, merge, train_fast, train_from_counts, bytes_to_unicode) and the split patterns are the ones defined above.</p><pre>import json<br>from functools import lru_cache</pre><pre>import regex as re<br></pre><pre>class BPETokenizer:<br>    def __init__(self, pattern=GPT4_SPLIT_PATTERN):<br>        self.pattern = pattern<br>        self.compiled = re.compile(pattern)<br>        self.merges = {}<br>        self.special_tokens = {}<br>        self._build_vocab()</pre><pre>    # ---------- training ----------<br>    def train(self, text, vocab_size, **kwargs):<br>        assert vocab_size &gt;= 256<br>        self.merges = train_fast(text, vocab_size, self.pattern, **kwargs)<br>        self._build_vocab()</pre><pre>    def _build_vocab(self):<br>        self.vocab = {i: bytes([i]) for i in range(256)}<br>        for (a, b), idx in self.merges.items():  # insertion order = merge order<br>            self.vocab[idx] = self.vocab[a] + self.vocab[b]<br>        for s, idx in self.special_tokens.items():<br>            self.vocab[idx] = s.encode(&quot;utf-8&quot;)<br>        self._encode_chunk.cache_clear()</pre><pre>    def register_special_tokens(self, tokens):<br>        &quot;&quot;&quot;tokens: dict like {&quot;&lt;|endoftext|&gt;&quot;: 100257}. IDs must be unused.&quot;&quot;&quot;<br>        for s, idx in tokens.items():<br>            if idx in self.vocab and self.vocab[idx] != s.encode(&quot;utf-8&quot;):<br>                raise ValueError(f&quot;id {idx} already used&quot;)<br>        self.special_tokens.update(tokens)<br>        self._build_vocab()</pre><pre>    # ---------- encoding ----------<br>    @lru_cache(maxsize=2**16)<br>    def _encode_chunk(self, chunk_bytes):<br>        ids = list(chunk_bytes)<br>        while len(ids) &gt;= 2:<br>            # the pair that was learned earliest has the lowest merge id<br>            pair = min(zip(ids, ids[1:]), key=lambda p: self.merges.get(p, float(&quot;inf&quot;)))<br>            if pair not in self.merges:<br>                break<br>            ids = merge(ids, pair, self.merges[pair])<br>        return tuple(ids)</pre><pre>    def encode_ordinary(self, text):<br>        &quot;&quot;&quot;Encode text, treating special-token strings as normal text.&quot;&quot;&quot;<br>        ids = []<br>        for chunk in self.compiled.findall(text):<br>            ids.extend(self._encode_chunk(chunk.encode(&quot;utf-8&quot;)))<br>        return ids</pre><pre>    def encode(self, text, allowed_special=frozenset()):<br>        &quot;&quot;&quot;allowed_special: &quot;all&quot;, or a set of special strings to honour.<br>        Any other special string found in text raises, like tiktoken.&quot;&quot;&quot;<br>        if allowed_special == &quot;all&quot;:<br>            allowed = set(self.special_tokens)<br>        else:<br>            allowed = set(allowed_special)<br>        for s in self.special_tokens:<br>            if s not in allowed and s in text:<br>                raise ValueError(f&quot;text contains disallowed special token {s!r}&quot;)<br>        if not allowed:<br>            return self.encode_ordinary(text)<br>        special_pat = &quot;(&quot; + &quot;|&quot;.join(re.escape(s) for s in sorted(allowed, key=len, reverse=True)) + &quot;)&quot;<br>        ids = []<br>        for part in re.split(special_pat, text):<br>            if part in allowed:<br>                ids.append(self.special_tokens[part])<br>            elif part:<br>                ids.extend(self.encode_ordinary(part))<br>        return ids</pre><pre>    # ---------- decoding ----------<br>    def decode_bytes(self, ids):<br>        return b&quot;&quot;.join(self.vocab[i] for i in ids)</pre><pre>    def decode(self, ids):<br>        return self.decode_bytes(ids).decode(&quot;utf-8&quot;, errors=&quot;replace&quot;)</pre><pre>    # ---------- persistence ----------<br>    def save(self, path):<br>        data = {<br>            &quot;version&quot;: 1,<br>            &quot;pattern&quot;: self.pattern,<br>            &quot;merges&quot;: [[a, b] for (a, b) in self.merges],  # order = rank<br>            &quot;special_tokens&quot;: self.special_tokens,<br>        }<br>        with open(path, &quot;w&quot;, encoding=&quot;utf-8&quot;) as f:<br>            json.dump(data, f)</pre><pre>    @classmethod<br>    def load(cls, path):<br>        with open(path, encoding=&quot;utf-8&quot;) as f:<br>            data = json.load(f)<br>        tok = cls(pattern=data[&quot;pattern&quot;])<br>        tok.merges = {(a, b): 256 + i for i, (a, b) in enumerate(data[&quot;merges&quot;])}<br>        tok.special_tokens = data[&quot;special_tokens&quot;]<br>        tok._build_vocab()<br>        return tok</pre><pre>    def dump_vocab(self, path):<br>        &quot;&quot;&quot;Human-readable vocab, one token per line, GPT-2 style.&quot;&quot;&quot;<br>        b2u = bytes_to_unicode()<br>        with open(path, &quot;w&quot;, encoding=&quot;utf-8&quot;) as f:<br>            for idx, tok in sorted(self.vocab.items()):<br>                f.write(f&quot;{idx}\t{&#39;&#39;.join(b2u[b] for b in tok)}\n&quot;)</pre><pre>    def __hash__(self):  # lets lru_cache work on the bound method<br>        return id(self)</pre><p>Usage:</p><pre>tok = BPETokenizer()                                     # GPT-4 split pattern<br>tok.train(open(&quot;corpus.txt&quot;, encoding=&quot;utf-8&quot;).read(), vocab_size=8192)<br>tok.register_special_tokens({&quot;&lt;|endoftext|&gt;&quot;: len(tok.vocab)})</pre><pre>ids = tok.encode(&quot;The Licensor shall not be liable&quot;)<br>print([tok.decode([i]) for i in ids])<br># [&#39;The&#39;, &#39; Licensor&#39;, &#39; shall&#39;, &#39; not&#39;, &#39; be&#39;, &#39; liable&#39;]   (on my license-text corpus)</pre><pre>tok.save(&quot;my_tokenizer.json&quot;)<br>tok = BPETokenizer.load(&quot;my_tokenizer.json&quot;)</pre><p>Two details in this class are easy to get wrong:</p><ul><li><strong>Special token IDs come after the learned vocabulary.</strong> With min_frequency, training can stop before reaching vocab_size. Using len(tok.vocab) as the next ID avoids leaving gaps or colliding with merge IDs.</li><li><strong>Decoding single tokens can produce </strong><strong>�.</strong> A token may hold only part of a multi-byte character. decode replaces invalid bytes, which is right for display. If you stream tokens to a user, buffer the bytes (decode_bytes) until they form a complete UTF-8 sequence, or the user will see broken characters.</li></ul><h3>12. Going to production</h3><p>Our tokenizer is correct and readable, but Python is the bottleneck. Production libraries do the same work in Rust with careful data structures:</p><ul><li><strong>Hugging Face </strong><strong>tokenizers</strong> for training your own. It supports BPE, WordPiece and Unigram, and saves a single tokenizer.json that the transformers library loads directly.</li><li><strong>tiktoken</strong> for encoding with OpenAI&#39;s existing vocabularies such as cl100k_base and o200k_base. It is built for fast encoding, not for training new vocabularies.</li><li><strong>SentencePiece</strong> if you want a tokenizer that works on raw text without whitespace pre-splitting, which helps for languages written without spaces between words.</li></ul><p>The equivalent byte-level BPE setup in Hugging Face tokenizers looks like this:</p><pre>from tokenizers import Tokenizer, models, trainers, pre_tokenizers, decoders</pre><pre>tokenizer = Tokenizer(models.BPE())<br>tokenizer.pre_tokenizer = pre_tokenizers.ByteLevel(add_prefix_space=False)<br>tokenizer.decoder = decoders.ByteLevel()</pre><pre>trainer = trainers.BpeTrainer(<br>    vocab_size=32000,<br>    min_frequency=2,<br>    special_tokens=[&quot;&lt;|endoftext|&gt;&quot;],<br>    initial_alphabet=pre_tokenizers.ByteLevel.alphabet(),  # all 256 bytes<br>)<br>tokenizer.train([&quot;corpus.txt&quot;], trainer)<br>tokenizer.save(&quot;tokenizer.json&quot;)</pre><p>Everything you built in this article maps onto it. ByteLevel is the byte-to-unicode trick, min_frequency is our early stop, and initial_alphabet guarantees the 256 base tokens.</p><h3>13. Failure modes worth knowing</h3><p>Once you understand tokenizers, many famous LLM quirks stop being mysterious:</p><ul><li><strong>Counting letters.</strong> “strawberry” is a few tokens, not ten letters. The model never sees the individual characters unless it has learned how each token is spelled.</li><li><strong>Arithmetic.</strong> If numbers split as 202 + 6, the model has to do arithmetic on irregular chunks. Capping digit runs, or splitting every digit, makes this more consistent.</li><li><strong>Trailing whitespace.</strong> Since spaces attach to the <em>following</em> word, a prompt that ends in a space leaves the model in an unusual state. It rarely saw a lone &quot; &quot; token followed by a word without its usual leading space. &quot;Token healing&quot; techniques back up one token and let the model regenerate it to avoid this.</li><li><strong>Glitch tokens.</strong> If a string is frequent in the tokenizer’s training data but rare in the model’s, it gets a token with an almost untrained embedding. The famous example was SolidGoldMagikarp in the GPT-2/GPT-3 vocabulary, a Reddit username that made models behave bizarrely. Training your tokenizer on data that matches your model&#39;s training data prevents this.</li><li><strong>Unicode normalization.</strong> é can be one code point (U+00E9) or two (e + U+0301). They look identical but tokenize differently. Decide whether to apply NFC normalization before tokenizing, and apply it the same way in training and inference. Note that normalization breaks perfect round-trips, so byte-level GPT tokenizers usually skip it.</li><li><strong>Language cost.</strong> As section 10 showed, a tokenizer that has not seen a language can make it more than 10× more expensive. Measure fertility for every language you care about.</li></ul><h3>Wrapping up</h3><p>We built a byte-level BPE tokenizer with the same architecture as GPT-4&#39;s:</p><ol><li>UTF-8 bytes as the base vocabulary, so nothing is ever unknown</li><li>Regex pre-tokenization to keep merges inside meaningful chunks</li><li>A trainer that is about 100× faster than the naive version, using chunk deduplication, an inverted index and a lazy max-heap</li><li>Merge-rank encoding with a per-chunk cache</li><li>Special tokens with injection protection</li><li>Versioned serialization and readable vocab dumps</li><li>Tests for equivalence, round-trips and fairness across languages</li></ol><p>Some next steps if you want to go deeper:</p><ul><li>Load GPT-2’s published merges.txt into this class and check that your IDs match tiktoken&#39;s gpt2 encoding exactly.</li><li>Train on a mixed English–Sinhala corpus and watch Sinhala fertility drop as the data improves.</li><li>Port _encode_chunk to Rust with PyO3 and see how close you can get to tiktoken&#39;s speed.</li><li>Read Karpathy’s <em>minbpe</em> repository and the original BPE-for-NMT paper by Sennrich, Haddow and Birch (2016).</li></ul><p>The tokenizer is the first and last thing that touches every token your model will ever see. It is worth understanding all the way down.</p><p><em>If this was useful, a clap or a follow helps more people find it. Questions and corrections are welcome in the comments.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a560c8fa0c83" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hearing the Human Through the Noise]]></title>
            <link>https://medium.com/@kavishkakalhara/hearing-the-human-through-the-noise-d52aed05f900?source=rss-b3cb50279e64------2</link>
            <guid isPermaLink="false">https://medium.com/p/d52aed05f900</guid>
            <category><![CDATA[audio-engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[voice-assistant]]></category>
            <category><![CDATA[speech-recognition]]></category>
            <category><![CDATA[signal-processing]]></category>
            <dc:creator><![CDATA[Kavishka Kalhara]]></dc:creator>
            <pubDate>Wed, 11 Nov 2026 17:48:22 GMT</pubDate>
            <atom:updated>2026-11-11T17:48:22.710Z</atom:updated>
            <content:encoded><![CDATA[<h3>How noise cancellation works in AI voice assistants, and why it decides whether your voice agent feels smart or broken</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xaGQ5vdiY4m-eec9CT51lw.png" /><figcaption>A noisy, jagged audio waveform passes through a glowing microphone ring and comes out as a clean, smooth waveform.</figcaption></figure><p>You ask your voice assistant a simple question from the kitchen. The pressure cooker is whistling, the TV is on in the next room, and someone is chopping onions. The assistant answers a completely different question. Or it starts talking, hears a spoon hit the counter, and stops mid-sentence as if you interrupted it.</p><p>Nothing is wrong with the language model. It is answering, perfectly confidently, the garbage it was given.</p><p>Large language models get most of the attention in voice AI, but the quality of a voice assistant is set much earlier, in the first few milliseconds after sound hits the microphone. <strong>An assistant can only be as smart as what it hears.</strong> This article walks through how noise cancellation for AI voice assistants actually works, from classic signal processing to modern neural models, and how to set it up in a real voice agent without making things worse.</p><h3>Why noise hurts AI more than it hurts people</h3><p>Humans are remarkably good at the “cocktail party problem.” We use two ears, context, lip reading and a lifetime of experience to lock onto one voice in a crowded room. A voice assistant usually gets one compressed audio stream and has to make decisions about it in real time.</p><p>Noise does not just make transcription a little worse. It breaks a voice pipeline in several places at once:</p><ul><li><strong>Speech recognition errors.</strong> Background sounds get transcribed as words, or real words get dropped. The LLM then reasons over a sentence the user never said.</li><li><strong>False turn-taking.</strong> Voice Activity Detection (VAD) decides when someone is speaking. A barking dog or a slammed door can look like speech, so the agent cuts itself off or waits for a “user” who never spoke.</li><li><strong>Missed endpoints.</strong> Continuous noise like a fan or traffic can stop VAD from ever detecting silence, so the assistant never realises you have finished talking.</li><li><strong>Hearing itself.</strong> Without echo cancellation, the assistant’s own voice coming out of the speaker leaks back into the microphone, and it starts responding to itself.</li><li><strong>The wrong speaker.</strong> A TV presenter or a colleague at the next desk is <em>speech</em>, so a basic noise filter keeps it, and the assistant happily transcribes them too.</li></ul><blockquote><em>In a text chatbot, bad input is the user’s typo. In a voice assistant, bad input is the whole room.</em></blockquote><h3>Not all noise is the same</h3><p>“Noise” is a catch-all word, but each type needs a different tool:</p><ul><li><strong>Stationary noise:</strong> steady sounds whose character barely changes, like a ceiling fan, AC hum or electrical buzz. The easiest to remove.</li><li><strong>Non-stationary noise:</strong> sudden, changing sounds like keyboard clicks, traffic, dogs, dishes and sirens. Much harder, because they look different every moment.</li><li><strong>Competing speech:</strong> other people or a TV. The hardest case, because it has the same statistical shape as the voice you want to keep.</li><li><strong>Echo:</strong> the assistant’s own audio played back through the speaker and recaptured by the mic.</li><li><strong>Reverberation:</strong> the room’s reflections smearing your voice over time, common in far-field devices like smart speakers and laptops on a desk.</li></ul><h3>The voice front-end: a pipeline, not a single filter</h3><p>In practice, “noise cancellation” is a chain of processing stages that run before your speech-to-text model or speech-native LLM ever sees the audio. A typical front-end looks like this:</p><pre>Microphone(s)<br>   │<br>   ▼<br>Beamforming           ← multi-mic devices: focus on the speaker&#39;s direction<br>   │<br>   ▼<br>Acoustic Echo Cancellation (AEC)   ← remove the assistant&#39;s own voice<br>   │<br>   ▼<br>Noise Suppression (DSP or neural)  ← remove background noise<br>   │<br>   ▼<br>Automatic Gain Control (AGC)       ← keep volume consistent<br>   │<br>   ▼<br>Voice Activity Detection / Turn detection<br>   │<br>   ▼<br>Speech-to-Text  →  LLM  →  Text-to-Speech  →  Speaker<br>                                               │<br>              (speaker signal is fed back to AEC as the reference)</pre><p>Order matters. Echo cancellation normally runs before noise suppression because it needs a clean, predictable relationship between what was played and what was recorded. Aggressive noise suppression earlier in the chain distorts that relationship and makes the echo harder to remove.</p><h3>1. Classic signal processing: the original toolkit</h3><p>Before deep learning, noise suppression relied on statistical methods such as <strong>spectral subtraction</strong> and <strong>Wiener filtering</strong>. The idea is simple: split audio into short frames, convert each frame into frequencies, estimate what the noise looks like during pauses, then subtract or attenuate those frequencies when speech is present.</p><p>These methods are cheap, fast and still used everywhere, including in the audio processing modules built into browsers and phones. They work well on stationary noise like a fan. Their weakness is anything that changes quickly. Push them hard and they leave behind “musical noise”: chirpy, watery artifacts that can confuse speech recognition more than the original noise did.</p><h3>2. Beamforming: listening in a direction</h3><p>Smart speakers and conference devices usually have several microphones arranged in an array. Sound from a given direction reaches each microphone at a slightly different time. <strong>Beamforming</strong> uses those tiny delays to boost sound coming from the speaker’s direction and attenuate sound from elsewhere.</p><p>The simplest version, <em>delay-and-sum</em>, aligns and adds the signals. Adaptive methods such as MVDR (minimum variance distortionless response) go further and actively steer “nulls” toward noise sources. Beamforming is one of the few tools that can separate two people talking at the same time, as long as they are in different places, which is why far-field assistants rely on it so heavily.</p><p>The catch is hardware. A phone app or a web app usually gets a single, already-processed audio channel, so beamforming is often not available to you as a developer.</p><h3>3. Acoustic echo cancellation: stop the assistant from hearing itself</h3><p>AEC is the unsung hero of voice assistants. The system knows exactly what it is playing through the speaker, so it uses that as a <strong>reference signal</strong>, models how that sound is changed by the speaker, the room and the microphone, and subtracts the predicted echo from the mic input.</p><p>Good AEC is what makes <strong>barge-in</strong> possible: you can interrupt the assistant mid-sentence and it actually hears you. The hard case is <em>double-talk</em>, when the user and the assistant speak at the same time. Weak AEC either lets the echo leak through or suppresses the user’s voice along with it.</p><p>If you build in the browser, you get a solid baseline for free by asking for it when you open the microphone:</p><pre>const stream = await navigator.mediaDevices.getUserMedia({<br>  audio: {<br>    echoCancellation: true,<br>    noiseSuppression: true,<br>    autoGainControl: true,<br>    channelCount: 1<br>  }<br>});</pre><h3>4. Neural noise suppression: the modern default</h3><p>Deep learning changed noise suppression because neural networks can learn what <em>speech</em> looks like instead of only estimating what noise looks like. Most real-time models work in the time-frequency domain: they look at a spectrogram and predict a <strong>mask</strong>, a value for each frequency band that says how much of that band is speech and how much is noise. Multiplying the noisy spectrogram by the mask and converting back gives you cleaned audio.</p><p>A few well-known approaches:</p><ul><li><strong>RNNoise</strong> combines classic DSP with a small recurrent neural network. It is tiny enough to run on almost anything and is a popular open-source baseline.</li><li><strong>DeepFilterNet</strong> estimates a complex filter in the frequency domain instead of a simple mask, taking advantage of short-time correlations in speech. It was designed for real-time use on ordinary CPUs and is open source.</li><li><strong>Commercial SDKs</strong> such as Krisp and ai-coustics offer production models tuned specifically for calls and voice agents, with options for telephony audio, de-reverberation and background voice removal.</li></ul><h3>Noise suppression vs. voice isolation</h3><p>This distinction matters a lot for voice agents. A <strong>noise suppression</strong> model removes non-speech sounds and keeps <em>all</em> speech. A <strong>voice isolation</strong> (or background voice cancellation) model goes further and tries to keep only the primary speaker, removing the TV and the people talking behind them.</p><p>Voice isolation is usually the better choice for a one-to-one assistant in a noisy office or call center. Plain noise suppression is safer when several real people should be heard, for example a meeting assistant that needs to transcribe and diarize everyone.</p><h3>How modern voice agent platforms expose it</h3><p>The good news is that you rarely need to build these models yourself. Many real-time voice platforms now treat noise reduction as a configuration option.</p><p><strong>OpenAI’s Realtime API</strong> has an input noise reduction setting with two modes: near_field for close-talking microphones like headsets, and far_field for laptop or conference-room microphones. The filtering is applied before the audio reaches voice activity detection and the model, which reduces false triggers in turn detection:</p><pre>{<br>  &quot;type&quot;: &quot;session.update&quot;,<br>  &quot;session&quot;: {<br>    &quot;type&quot;: &quot;realtime&quot;,<br>    &quot;audio&quot;: {<br>      &quot;input&quot;: {<br>        &quot;noise_reduction&quot;: { &quot;type&quot;: &quot;far_field&quot; }<br>      }<br>    }<br>  }<br>}</pre><p><strong>LiveKit</strong>, a popular open-source framework for voice agents, offers enhanced noise cancellation through Krisp and ai-coustics models. It separates standard noise cancellation, which removes things like typing, traffic and HVAC, from background voice cancellation, which also removes non-primary speakers.</p><p>If you are processing recorded audio offline, for example to clean a dataset or test your pipeline, an open-source model is a few lines of Python. Here is DeepFilterNet:</p><pre>from df.enhance import enhance, init_df, load_audio, save_audio</pre><pre>model, df_state, _ = init_df()<br>audio, _ = load_audio(&quot;noisy.wav&quot;, sr=df_state.sr())<br>enhanced = enhance(model, df_state, audio)<br>save_audio(&quot;enhanced.wav&quot;, enhanced, df_state.sr())</pre><h3>The trade-offs nobody mentions in the demo</h3><p>Noise cancellation is not “more is better.” The most common mistakes I see in voice agent projects come from over-processing:</p><h3>Over-suppression can make transcription worse</h3><p>Modern speech recognition models are trained on huge amounts of noisy, real-world audio. They are often more robust to mild noise than to the artifacts that aggressive denoising introduces. If you strip out too much, you can remove consonants and soft syllables that the recogniser needed. Always measure accuracy with and without suppression.</p><h3>Stacking processors</h3><p>Browser noise suppression, then a client SDK filter, then server-side suppression, then the speech provider’s own processing: each stage was tuned assuming it receives natural audio. Chained together, they can produce robotic, pumping speech. Pick one strong stage for noise and switch off the rest.</p><h3>Voice isolation can silence the real user</h3><p>A background voice model has to guess who the “primary” speaker is. A soft-spoken user, a user far from the mic or a speakerphone call can be treated as background and attenuated. If users complain the agent “ignores” them, test with plain noise suppression.</p><h3>Latency adds up</h3><p>Every stage needs a little look-ahead audio. A few tens of milliseconds sounds small, but conversational turn-taking feels unnatural once the total delay creeps up. Count the latency of every processor in your chain, not just the LLM.</p><h3>Telephony is its own world</h3><p>Phone calls arrive narrowband and heavily compressed, often already processed by the carrier. Models trained on wideband audio can behave poorly on 8 kHz call audio, so use models and settings made for telephony when your agent answers the phone.</p><h3>How to measure whether it is working</h3><p>“It sounds cleaner” is not a metric. For a voice assistant, track what the pipeline actually does:</p><ul><li><strong>Word Error Rate (WER)</strong> on a test set recorded in realistic noise: cafés, cars, kitchens, open offices.</li><li><strong>False barge-in rate:</strong> how often the agent stops talking because of a non-speech sound.</li><li><strong>Missed barge-ins:</strong> how often a real interruption is ignored.</li><li><strong>Endpoint delay:</strong> time between the user finishing and the agent starting to respond.</li><li><strong>Perceptual quality scores</strong> such as PESQ, STOI or DNSMOS if humans will also listen to the processed audio.</li><li><strong>Task success:</strong> in the end, did the user get what they asked for?</li></ul><h3>A practical checklist for your next voice agent</h3><ol><li>Get echo cancellation right first. Without it, barge-in is impossible.</li><li>Choose noise suppression or voice isolation based on how many real speakers you expect.</li><li>Match the mode to the hardware: near-field for headsets and phones held close, far-field for laptops, kiosks and speakers.</li><li>Use one strong noise stage. Turn off duplicates.</li><li>Tune VAD thresholds <em>after</em> noise suppression, not before.</li><li>Build a small noisy test set from your real users’ environments and test every change against it.</li><li>Watch latency per stage, and use telephony-specific models for phone calls.</li></ol><h3>Final thought</h3><p>We spend a lot of energy choosing the smartest model, writing better prompts and shaving milliseconds off text-to-speech. But for a voice assistant, the first question is always simpler: <em>did it actually hear the human?</em></p><p>Noise cancellation is not a polish step you add at the end. It is the foundation the whole conversation stands on. Get the front-end right, and even a modest model feels attentive and smart. Get it wrong, and the best model in the world will confidently answer the pressure cooker.</p><p><em>If you are building voice agents, I’d love to hear what noise has broken your pipeline. Share it in the comments.</em></p><h3>Further reading</h3><ul><li><a href="https://arxiv.org/abs/2305.08227">DeepFilterNet: Perceptually Motivated Real-Time Speech Enhancement (arXiv)</a></li><li><a href="https://github.com/xiph/rnnoise">RNNoise on GitHub</a></li><li><a href="https://docs.livekit.io/transport/media/noise-cancellation">LiveKit docs: Noise &amp; echo cancellation</a></li><li><a href="https://platform.openai.com/docs/guides/realtime">OpenAI Realtime API guide</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d52aed05f900" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM Inference Optimization: The Complete Guide to Serving Models Faster and Cheaper]]></title>
            <link>https://medium.com/@kavishkakalhara/llm-inference-optimization-the-complete-guide-to-serving-models-faster-and-cheaper-ba8429a386fd?source=rss-b3cb50279e64------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba8429a386fd</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-cost-optimization]]></category>
            <category><![CDATA[mlops]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[inference-optimization]]></category>
            <dc:creator><![CDATA[Kavishka Kalhara]]></dc:creator>
            <pubDate>Wed, 04 Nov 2026 08:25:52 GMT</pubDate>
            <atom:updated>2026-11-04T08:25:52.609Z</atom:updated>
            <content:encoded><![CDATA[<p><em>From roofline math and KV cache budgets to quantization, speculative decoding, disaggregated serving and a hands-on vLLM tuning workflow.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iZgONhck3m0VS3q9IzxqYA.png" /></figure><h3>Training is a one-time bill. Inference is the subscription.</h3><p>You train a model once. You serve it millions of times. For most teams shipping LLM products, inference is where the GPU budget goes, and it is also where users feel every millisecond.</p><p>Here is the uncomfortable fact that this whole article is built on. When an 8B model generates text for a single user on an H100, the GPU’s Tensor Cores sit idle more than 99% of the time. The chip isn’t slow. It’s starving, waiting for weights to arrive from memory.</p><p>Once you understand <em>why</em> that happens, every optimization technique in this guide stops looking like a random trick and starts looking like an answer to the same question: <strong>how do we stop wasting the hardware?</strong></p><p>We’ll cover:</p><ul><li>The two phases of inference and why they behave completely differently</li><li>The roofline model and the back-of-envelope math every inference engineer should know</li><li>The KV cache, the hidden memory hog behind long-context costs</li><li>Six levers: quantization, memory management, batching and scheduling, speculative decoding, kernels, and parallelism</li><li>How to choose a serving engine</li><li>A hands-on workflow with vLLM and a dependency-free benchmark script</li><li>A decision playbook and the pitfalls that waste the most money</li></ul><h3>The anatomy of a request: prefill and decode</h3><p>Every request to an LLM runs in two phases.</p><p><strong>Prefill</strong> processes the entire prompt in one forward pass. All prompt tokens are known up front, so the GPU computes them in parallel as large matrix multiplications. Prefill is <strong>compute-bound</strong>, and it determines how long the user waits before the first token appears.</p><p><strong>Decode</strong> generates the answer one token at a time. Each new token depends on the previous one, so every step is a forward pass for a single new token per sequence. To produce that one token, the GPU must read <em>every weight in the model</em> plus that sequence’s entire KV cache from memory. Decode is <strong>memory-bandwidth-bound</strong>, and it determines how fast the text streams.</p><p>This split is the single most important idea in inference optimization. Prefill wants more FLOPs. Decode wants more bandwidth, or fewer bytes to read. Many techniques help one phase and do nothing for the other.</p><h3>The metrics that matter</h3><ul><li><strong>TTFT (time to first token)</strong>: what it measures: Wait before streaming starts; driven mostly by: Prefill, queueing</li><li><strong>ITL / TPOT (inter-token latency, time per output token)</strong>: what it measures: Streaming speed; driven mostly by: Decode</li><li><strong>End-to-end latency</strong>: what it measures: TTFT + ITL × output length; driven mostly by: Both</li><li><strong>Throughput (tokens/s)</strong>: what it measures: Total tokens served per second across all users; driven mostly by: Batching, utilization</li><li><strong>Goodput</strong>: what it measures: Requests per second that meet your latency SLOs; driven mostly by: Everything above</li><li><strong>Cost per million tokens</strong>: what it measures: What the business actually pays; driven mostly by: Throughput ÷ GPU price</li></ul><p>Optimize for <strong>goodput</strong>, not raw throughput. A server that pushes 20,000 tokens/s while half the requests miss their 2-second TTFT target is not a fast server.</p><h3>The roofline: the one mental model you need</h3><p>Every operation on a GPU either waits on compute or waits on memory. The roofline model tells you which.</p><p><strong>Arithmetic intensity</strong> is the number of floating-point operations you perform per byte you move from memory:</p><pre>arithmetic intensity = FLOPs / bytes moved<br>attainable FLOPS     = min( peak compute, memory bandwidth × arithmetic intensity )</pre><p>The point where those two limits meet is the <strong>ridge point</strong>. For an H100 SXM, with roughly 989 dense BF16 TFLOPS and 3.35 TB/s of HBM3 bandwidth:</p><pre>ridge point = 989 TFLOPS / 3.35 TB/s ≈ 295 FLOPs per byte</pre><p>Any operation below 295 FLOPs/byte is memory-bound on an H100. Any operation above it is compute-bound.</p><h3>Where decode lands</h3><p>During decode, each parameter is used for roughly 2 FLOPs per token (one multiply, one add). In BF16, each parameter is 2 bytes. So for a single sequence:</p><pre>intensity ≈ (2 FLOPs × params) / (2 bytes × params) = 1 FLOP per byte</pre><p>One FLOP per byte. The ridge is at 295. Decode for one user uses about 3.35 TFLOPS of a 989-TFLOPS chip, roughly 0.3% of its compute.</p><p>Batching is the fix. When you decode B sequences together, you read the weights once and use them B times, so intensity grows to roughly B FLOPs per byte.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wfsAIvTaHJZ1yM2ZrVwntg.png" /><figcaption>Roofline chart for an H100: decode at batch 1, 32 and 256 sits on the memory-bound slope, while a 2K-token prefill sits on the compute-bound ceiling</figcaption></figure><p>Batch 1 sits at the bottom left. Batch 32 is still memory-bound. Only around batch 256 does decode approach the compute ceiling, while a single long prefill is compute-bound on its own.</p><h3>The decode speed limit</h3><p>Since decode is memory-bound, you can estimate the best possible single-stream speed from bandwidth alone:</p><pre>max tokens/s per sequence ≈ memory bandwidth / bytes of weights read per token</pre><p>For an 8B model on one H100:</p><pre>Weight precision     Model size  Decode ceiling (one sequence)<br>-------------------  ----------  -----------------------------<br>BF16                 16 GB       ~209 tokens/s<br>FP8                  8 GB        ~419 tokens/s<br>4-bit (with scales)  ~4.5 GB     ~744 tokens/s</pre><p>Real systems land below these ceilings because of kernel overhead, KV cache reads and scheduling. But the ratios are the lesson: <strong>halve the bytes, roughly double the decode speed.</strong> That is why quantization matters so much for decode.</p><h3>The KV cache: the hidden memory hog</h3><p>Attention needs the keys and values of every previous token. Recomputing them at every step would be quadratic, so inference engines store them in the <strong>KV cache</strong>. It is what makes decode fast, and it is what makes long context expensive.</p><pre>KV bytes per token = 2 × n_layers × n_kv_heads × head_dim × bytes_per_element</pre><p>The 2 accounts for keys and values. Here it is for two popular models in BF16:</p><pre>Model          Layers  KV heads  Head dim  Per token  8K context  128K context<br>-------------  ------  --------  --------  ---------  ----------  ------------<br>Llama 3.1 8B   32      8         128       128 KiB    1 GiB       16 GiB<br>Llama 3.1 70B  80      8         128       320 KiB    2.5 GiB     40 GiB</pre><p>That is <strong>per sequence</strong>. Thirty-two concurrent users at 8K context on the 70B model need 80 GiB of KV cache, an entire H100’s memory, before you load a single weight.</p><p>The KV cache also breaks the batching rescue from the previous section. Weights are shared across the batch, so batching amortizes them. KV caches are <em>not</em> shared: each sequence reads its own. At long contexts, KV reads dominate and decode stays memory-bound no matter how big the batch is.</p><p>Here’s a calculator you can adapt for any model:</p><pre>GiB = 1024 ** 3</pre><pre>def kv_bytes_per_token(n_layers, n_kv_heads, head_dim, bytes_per_elem=2):<br>    &quot;&quot;&quot;Keys + values, every layer, for one token.&quot;&quot;&quot;<br>    return 2 * n_layers * n_kv_heads * head_dim * bytes_per_elem</pre><pre>def max_concurrent_sequences(gpu_mem_gib, weights_gib, ctx_len, kv_per_token,<br>                             utilization=0.90, overhead_gib=4):<br>    &quot;&quot;&quot;How many full-length sequences fit after weights and activation overhead.&quot;&quot;&quot;<br>    free = gpu_mem_gib * utilization - weights_gib - overhead_gib<br>    return int(free * GiB // (ctx_len * kv_per_token))</pre><pre>def decode_ceiling_tok_s(params_billion, bytes_per_param, bandwidth_tb_s):<br>    &quot;&quot;&quot;Upper bound on tokens/s for ONE sequence when decode is memory-bound.&quot;&quot;&quot;<br>    return bandwidth_tb_s * 1e12 / (params_billion * 1e9 * bytes_per_param)</pre><pre>kv = kv_bytes_per_token(n_layers=32, n_kv_heads=8, head_dim=128)   # Llama 3.1 8B<br>print(kv // 1024, &quot;KiB per token&quot;)                                 # 128<br>print(max_concurrent_sequences(80, 16, 8192, kv), &quot;sequences at 8K on an 80 GB GPU&quot;)<br>print(round(decode_ceiling_tok_s(8, 2, 3.35)), &quot;tok/s ceiling in BF16&quot;)   # 209</pre><h3>Architecture decides your KV bill</h3><p>Model designers have attacked the KV cache directly, and the choice of model is itself an optimization:</p><ul><li><strong>Multi-query and grouped-query attention (MQA, GQA).</strong> Many query heads share a few KV heads. Llama 3.1 70B has 64 query heads but only 8 KV heads. Without GQA its cache would be 2.5 MiB per token instead of 320 KiB, eight times larger.</li><li><strong>Multi-head latent attention (MLA).</strong> Used by DeepSeek-V2 and V3, MLA compresses keys and values into a small latent vector per token. The DeepSeek-V2 paper reported a 93.3% smaller KV cache than its dense predecessor.</li><li><strong>Sliding-window and hybrid attention.</strong> Some layers attend only to a recent window, so their cache stops growing. Several model families mix local and global layers to cap memory at long context.</li><li><strong>State-space and linear-attention hybrids.</strong> Layers like Mamba keep a fixed-size state instead of a growing cache, trading some recall for constant memory.</li></ul><p>If you serve long contexts at high concurrency, a model with an efficient attention design can be worth more than every serving trick combined.</p><h3>Lever 1: Quantization</h3><p>Quantization stores numbers in fewer bits. Because decode is memory-bound, fewer bytes per weight translate almost directly into faster tokens and more room for KV cache.</p><h3>Two families</h3><ul><li><strong>Weight-only quantization</strong> (for example INT4 with AWQ or GPTQ) shrinks the weights but computes in 16-bit. It speeds up memory-bound decode and saves memory, but does little for compute-bound prefill.</li><li><strong>Weight-and-activation quantization</strong> (for example FP8 W8A8, or 4-bit floating point on newer GPUs) runs the matrix multiplications themselves in low precision on Tensor Cores. It speeds up both phases.</li></ul><p>Here is how the common formats compare:</p><ul><li><strong>BF16 / FP16</strong>: bytes per weight: 2; hardware sweet spot: Any modern GPU; typical quality impact: Baseline; best for: Quality reference</li><li><strong>FP8 (W8A8)</strong>: bytes per weight: 1; hardware sweet spot: Hopper (H100/H200), Ada, Blackwell; typical quality impact: Usually near-lossless; best for: The default production choice on modern GPUs</li><li><strong>INT8 (SmoothQuant)</strong>: bytes per weight: 1; hardware sweet spot: Ampere (A100) and older; typical quality impact: Small; best for: GPUs without FP8 support</li><li><strong>INT4 weight-only (AWQ, GPTQ)</strong>: bytes per weight: ~0.5 + scales; hardware sweet spot: Any GPU, consumer cards; typical quality impact: Small to noticeable; best for: Fitting big models on few GPUs, latency</li><li><strong>NVFP4 / MXFP4</strong>: bytes per weight: ~0.56; hardware sweet spot: Blackwell (B200, GB200); typical quality impact: Small with good calibration; best for: Maximum throughput on Blackwell</li><li><strong>GGUF k-quants</strong>: bytes per weight: 0.3 to 1; hardware sweet spot: CPUs, Apple Silicon; typical quality impact: Varies by level; best for: Local and edge inference</li></ul><h3>A closer look at 4-bit floating point</h3><p><a href="https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/">NVIDIA’s NVFP4</a> shows how modern low-bit formats preserve accuracy. Each value is a 4-bit float, values are grouped into blocks of 16, and each block gets its own FP8 (E4M3) scale, with a second FP32 scale per tensor. The open MXFP4 format uses larger 32-value blocks with a coarser power-of-two scale.</p><p>The result is about 4.5 bits per value: roughly 3.5x smaller than FP16 and 1.8x smaller than FP8. NVIDIA reports 1% or less degradation versus FP8 on key benchmarks for DeepSeek-R1–0528 after post-training quantization.</p><h3>Don’t forget the KV cache</h3><p>You can quantize the KV cache separately from the weights. An FP8 KV cache halves its memory, which roughly doubles how many tokens of context you can hold. For long-context workloads, this is often a bigger win than weight quantization.</p><h3>Rules for quantizing safely</h3><ol><li><strong>Evaluate on your task, not on perplexity.</strong> Perplexity barely moves while math, code and long-context retrieval quietly degrade.</li><li><strong>Reasoning and long-context tasks break first.</strong> Test them specifically.</li><li><strong>Prefer published, calibrated checkpoints</strong> from the model vendor or established quantizers over quantizing on the fly.</li><li><strong>Quantize in steps:</strong> FP8 first, then 4-bit only if you need the memory or speed and your evals still pass.</li></ol><h3>Lever 2: Memory management</h3><p>Raw KV cache size is one problem. Wasting the memory you have is another.</p><h3>PagedAttention</h3><p>Early serving systems reserved a contiguous block of memory for each request’s maximum possible length. Most requests never used it, and the gaps between allocations couldn’t be reused. The vLLM paper measured existing systems wasting 60% to 80% of KV cache memory this way.</p><p><strong>PagedAttention</strong> borrowed the idea of virtual memory from operating systems. The KV cache is split into small fixed-size blocks, allocated on demand and tracked through a block table, so a sequence’s cache doesn’t need to be contiguous. Waste dropped to under 4%, which meant far more sequences per GPU and much higher throughput. Every serious engine now uses some version of this.</p><h3>Prefix caching</h3><p>Many requests share the same beginning: a long system prompt, few-shot examples, a document being questioned repeatedly, or earlier turns of a chat. Recomputing that prefix’s KV cache for every request wastes prefill compute.</p><p><strong>Prefix caching</strong> stores KV blocks keyed by the tokens they contain, and reuses them when a new request starts with the same tokens. SGLang’s <strong>RadixAttention</strong> organizes cached prefixes in a radix tree so that partial overlaps are found automatically.</p><p>When hit rates are high, prefix caching cuts TTFT dramatically and frees prefill capacity for everyone else. To get those hit rates:</p><ul><li><strong>Put static content first and dynamic content last.</strong> System prompt, tools and instructions at the top; the user’s message at the bottom.</li><li><strong>Keep the prefix byte-for-byte identical.</strong> A timestamp or request ID at the top of the prompt destroys every cache hit.</li><li><strong>Route related requests to the same replica</strong> (for example by session ID) so they can share a cache.</li></ul><h3>KV cache offloading</h3><p>When GPU memory runs out, engines can offload cold KV blocks to CPU memory or even local SSDs and bring them back on demand. Tools like LMCache add a shared cache tier across instances. This is especially useful for multi-turn chat, where a user’s history can be restored instead of recomputed.</p><h3>Lever 3: Batching and scheduling</h3><p>The roofline told us that batching is the main way to make decode efficient. How you batch decides whether that efficiency comes at the expense of latency.</p><h3>Continuous batching</h3><p><strong>Static batching</strong> waits until a batch is full, runs it to completion, and only then starts new requests. One long answer holds the whole batch hostage, and finished slots sit empty.</p><p><strong>Continuous batching</strong> (also called iteration-level scheduling, introduced in the Orca paper) makes scheduling decisions at every decode step. Finished sequences leave immediately, and new requests join the next step. GPU slots stay full and short requests don’t wait behind long ones. This one change is the main reason modern engines serve several times more traffic than naive implementations.</p><h3>Chunked prefill</h3><p>A new request’s prefill can be thousands of tokens of heavy compute. If the engine runs it in one go, every ongoing decode stalls, and users see the stream freeze.</p><p><strong>Chunked prefill</strong> splits long prompts into chunks and mixes them with decode steps in the same batch. Decode keeps flowing, and the compute-heavy prefill chunks fill in the GPU’s spare compute while memory-bound decode is running. The main knob is the per-step token budget (in vLLM, max_num_batched_tokens): larger budgets favour TTFT and throughput, smaller budgets favour smooth inter-token latency.</p><h3>Disaggregated prefill and decode</h3><p>Chunked prefill makes the two phases share a GPU politely. <strong>Disaggregated serving</strong> gives them separate GPUs entirely: a pool of prefill workers computes the KV cache and ships it over a fast interconnect to a pool of decode workers.</p><p><a href="https://docs.nvidia.com/dynamo/knowledge-base/concepts/system-architecture/disaggregated-serving">NVIDIA Dynamo’s documentation</a> lays out the benefits: each phase can be tuned to its own hardware needs (decode can use a larger tensor-parallel degree), and long prompts no longer stall ongoing decodes. Dynamo moves the KV cache directly between GPUs with its NIXL transfer library. Open-source stacks like llm-d and the engines’ own disaggregation modes follow the same pattern.</p><p>It isn’t free. The same documentation is clear that at low concurrency an aggregated worker often wins, because it avoids transfer overhead and keeps memory pooled. Disaggregation pays off at scale, under sustained load, with long prompts and strict latency targets.</p><h3>Lever 4: Speculative decoding</h3><p>Every lever so far makes each decode step cheaper. Speculative decoding makes each step produce <strong>more than one token</strong>.</p><h3>How it works</h3><ol><li>A cheap <strong>draft</strong> proposes the next γ tokens.</li><li>The large <strong>target</strong> model checks all γ tokens in a single forward pass. Because it is scoring several known tokens at once, this pass looks like a small prefill: compute-heavy, which is exactly what an idle, memory-bound decode GPU has to spare.</li><li>Tokens are accepted left to right until the first disagreement. The target then supplies one correct token itself.</li></ol><p>With the rejection-sampling scheme from the original papers, the output distribution is <strong>exactly</strong> the same as the target model’s. You get a speedup with no quality change.</p><h3>The math of acceptance</h3><p>If each draft token is accepted with probability α and you draft γ tokens, the expected number of tokens produced per target pass is:</p><pre>E[tokens per step] = (1 − α^(γ+1)) / (1 − α)</pre><pre>Acceptance rate α  γ = 1  γ = 3  γ = 5  γ = 8<br>-----------------  -----  -----  -----  -----<br>0.6                1.60   2.18   2.38   2.47<br>0.7                1.70   2.53   2.94   3.20<br>0.8                1.80   2.95   3.69   4.33<br>0.9                1.90   3.44   4.69   6.13</pre><p>Two lessons jump out. Acceptance rate matters far more than draft length: at α = 0.6, drafting more than about 3 tokens barely helps. And the real speedup is lower than this table, because drafting and verification have their own cost.</p><h3>Kinds of drafters</h3><ul><li><strong>Small draft model</strong>: how it drafts: A smaller model from the same family; strengths: Simple, no training; trade-offs: Extra model in memory; tokenizer must match</li><li><strong>N-gram / prompt lookup</strong>: how it drafts: Copies spans from the prompt or recent output; strengths: Free; superb for RAG, summarization, code edits; trade-offs: Useless when output doesn’t echo the input</li><li><strong>Medusa</strong>: how it drafts: Extra prediction heads on the target model; strengths: No separate model; trade-offs: Needs head training; lower acceptance than EAGLE</li><li><strong>EAGLE family</strong>: how it drafts: Lightweight head that reads the target’s hidden states; strengths: Highest acceptance rates today; trade-offs: Needs a trained draft head for your model</li><li><strong>Built-in MTP</strong>: how it drafts: Multi-token prediction heads trained with the model (for example DeepSeek-V3); strengths: Ships with the model; trade-offs: Only for models trained this way</li></ul><p>The <a href="https://arxiv.org/pdf/2503.01840">EAGLE-3 paper</a> reports speedups of roughly 3x to 6.5x over standard decoding, with a peak of 6.5x on HumanEval for Llama 3.1 8B at temperature 0. Its key changes were predicting tokens directly instead of hidden features, fusing features from several layers of the target model, and training the drafter on its own predictions so accuracy holds up across longer drafts.</p><h3>The catch: batch size</h3><p>Speculative decoding spends spare compute to save memory trips. At high batch sizes, the GPU is already closer to compute-bound and there’s less spare compute to spend. The same EAGLE-3 paper measured about a 1.38x throughput gain at batch size 64 in SGLang, much smaller than the low-batch latency gains, and noted that the original EAGLE actually reduced throughput at that batch size.</p><p>Use speculative decoding when <strong>latency per user</strong> is the goal: interactive chat, coding assistants, agents. Measure carefully before turning it on for high-throughput batch jobs.</p><h3>Lever 5: Kernels and compilation</h3><p>Even with the right algorithms, a poorly implemented kernel can waste half the GPU.</p><h3>FlashAttention</h3><p>Standard attention materializes an N×N score matrix in slow GPU memory, then reads it back for softmax and again for the weighted sum. FlashAttention tiles the computation so each block of queries, keys and values is loaded into fast on-chip SRAM once, with softmax computed incrementally. The full matrix is never written to memory.</p><p>The gains compound with each version. The <a href="https://arxiv.org/html/2407.08608v2">FlashAttention-3 paper</a> reports up to 740 TFLOPS in FP16 on an H100, about 75% utilization, compared with roughly 35% for FlashAttention-2, and close to 1.2 PFLOPS in FP8. It gets there by overlapping data movement with computation and overlapping softmax with matrix multiplies across warp groups.</p><p>You rarely call these kernels yourself, but you should know which ones your engine uses and whether they support your GPU, head dimension and precision.</p><h3>Decode-specific kernels</h3><p>Decode attention looks nothing like prefill attention: one query per sequence against a long, paged KV cache. Libraries like FlashInfer and the engines’ own paged-attention kernels are specialized for this pattern, including splitting long sequences across many thread blocks so a single long chat still uses the whole GPU.</p><h3>CUDA graphs and fusion</h3><p>At small batch sizes, a decode step is so fast that the CPU’s cost of <em>launching</em> hundreds of small GPU kernels becomes a real bottleneck. <strong>CUDA graphs</strong> record the whole step once and replay it with a single launch. Engines capture graphs for common batch sizes, which is why you sometimes see a warm-up phase at startup.</p><p><strong>Kernel fusion</strong> combines adjacent operations (normalization, activation, quantization, rotary embeddings) into one kernel, so intermediate results never round-trip through memory. Compilers such as torch.compile and hand-written fused kernels for mixture-of-experts layers do this automatically in modern engines.</p><h3>Lever 6: Parallelism</h3><p>When a model doesn’t fit on one GPU, or one GPU isn’t fast enough, you split the work.</p><ul><li><strong>Tensor parallel (TP)</strong>: what gets split: Each layer’s matrices across GPUs; communication: Heavy, every layer; best for: Lower latency within one node (needs NVLink)</li><li><strong>Pipeline parallel (PP)</strong>: what gets split: Consecutive layers across GPUs; communication: Light, between stages; best for: Spanning nodes; models too big for one node</li><li><strong>Data parallel (DP)</strong>: what gets split: Whole replicas of the model; communication: None between replicas; best for: Scaling throughput</li><li><strong>Expert parallel (EP)</strong>: what gets split: MoE experts across GPUs; communication: All-to-all token routing; best for: Large mixture-of-experts models</li><li><strong>Context / sequence parallel</strong>: what gets split: Very long prompts across GPUs; communication: Attention exchange; best for: Million-token prefill</li></ul><p>Practical rules:</p><ol><li><strong>Use the smallest tensor-parallel degree that fits the model and its KV cache</strong>, then scale out with data-parallel replicas. Extra TP adds communication on every layer, so two TP=2 replicas often beat one TP=4 replica on throughput.</li><li><strong>Keep tensor parallelism inside a node</strong>, where NVLink is fast. Use pipeline parallelism across nodes.</li><li><strong>For mixture-of-experts models, remember the asymmetry.</strong> Active parameters set the compute cost per token, but <em>total</em> parameters set the memory you need. A large MoE can be cheap to run and still need many GPUs to hold.</li><li><strong>More TP lowers latency but raises cost per token.</strong> Decide which you’re buying.</li></ol><h3>Choosing a serving engine</h3><p>You almost never want to write your own inference server. The main open engines are mature and fast; choose on fit.</p><ul><li><strong>vLLM</strong>: strengths: Broadest model and hardware support, huge community, OpenAI-compatible server, strong defaults; consider it when: You want the safe, flexible default</li><li><strong>SGLang</strong>: strengths: RadixAttention prefix caching, fast structured output, strong multi-turn and agent performance; consider it when: Heavy prompt sharing, agents, constrained decoding</li><li><strong>TensorRT-LLM</strong>: strengths: Deep NVIDIA-specific optimization, early support for new GPU features and formats; consider it when: You’re all-in on NVIDIA and want peak performance</li><li><strong>llama.cpp (and tools built on it, like Ollama)</strong>: strengths: Runs on CPUs, Apple Silicon and consumer GPUs with GGUF quantization; consider it when: Local, edge or single-machine deployments</li></ul><p>Orchestration layers like NVIDIA Dynamo and llm-d sit <strong>above</strong> these engines. They add cache-aware routing, disaggregated prefill and decode, and autoscaling across many GPUs.</p><h3>Hands-on: tuning a real deployment</h3><p>Here is a workflow you can follow with vLLM. The principle is simple: <strong>change one thing at a time and measure every change.</strong></p><h3>Step 1: Estimate before you deploy</h3><p>Use the calculator from the KV cache section. Know your weight size, KV cache per token, how many sequences fit, and your theoretical decode ceiling. If reality is far below the estimate later, something is misconfigured.</p><h3>Step 2: Serve a baseline</h3><pre>pip install vllm</pre><pre>vllm serve meta-llama/Llama-3.1-8B-Instruct \<br>  --max-model-len 8192 \<br>  --gpu-memory-utilization 0.90</pre><p>Recent vLLM versions enable continuous batching, chunked prefill, PagedAttention, prefix caching and CUDA graphs by default, so this baseline is already well optimized.</p><h3>Step 3: Measure it properly</h3><p>vLLM ships its own benchmark tool (vllm bench serve). It’s also worth understanding exactly what you’re measuring, so here is a small benchmark with no dependencies beyond the Python standard library. It streams responses from any OpenAI-compatible server and reports TTFT, inter-token latency and throughput at several concurrency levels.</p><pre>import json, time, urllib.request<br>from concurrent.futures import ThreadPoolExecutor</pre><pre>URL = &quot;http://localhost:8000/v1/chat/completions&quot;<br>MODEL = &quot;meta-llama/Llama-3.1-8B-Instruct&quot;</pre><pre>def one_request(prompt, max_tokens=256):<br>    body = json.dumps({<br>        &quot;model&quot;: MODEL, &quot;max_tokens&quot;: max_tokens, &quot;stream&quot;: True,<br>        &quot;messages&quot;: [{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: prompt}],<br>        &quot;stream_options&quot;: {&quot;include_usage&quot;: True},<br>    }).encode()<br>    req = urllib.request.Request(URL, data=body, headers={&quot;Content-Type&quot;: &quot;application/json&quot;})<br>    start = time.perf_counter()<br>    token_times, out_tokens = [], 0<br>    with urllib.request.urlopen(req) as resp:<br>        for raw in resp:<br>            line = raw.decode().strip()<br>            if not line.startswith(&quot;data: &quot;) or line == &quot;data: [DONE]&quot;:<br>                continue<br>            chunk = json.loads(line[6:])<br>            if chunk.get(&quot;usage&quot;):<br>                out_tokens = chunk[&quot;usage&quot;][&quot;completion_tokens&quot;]<br>            if chunk.get(&quot;choices&quot;) and chunk[&quot;choices&quot;][0][&quot;delta&quot;].get(&quot;content&quot;):<br>                token_times.append(time.perf_counter())<br>    end = time.perf_counter()<br>    ttft = token_times[0] - start<br>    gaps = [b - a for a, b in zip(token_times, token_times[1:])]<br>    return {&quot;ttft&quot;: ttft, &quot;gaps&quot;: gaps, &quot;tokens&quot;: out_tokens or len(token_times), &quot;e2e&quot;: end - start}</pre><pre>def pct(values, p):<br>    values = sorted(values)<br>    return values[min(len(values) - 1, int(p / 100 * len(values)))]</pre><pre>def run(prompts, concurrency):<br>    t0 = time.perf_counter()<br>    with ThreadPoolExecutor(max_workers=concurrency) as pool:<br>        results = list(pool.map(one_request, prompts))<br>    wall = time.perf_counter() - t0<br>    ttfts = [r[&quot;ttft&quot;] for r in results]<br>    gaps = [g for r in results for g in r[&quot;gaps&quot;]]<br>    total_tokens = sum(r[&quot;tokens&quot;] for r in results)<br>    print(f&quot;concurrency={concurrency:&gt;3}  requests={len(results)}&quot;)<br>    print(f&quot;  TTFT  p50={pct(ttfts, 50)*1000:7.1f} ms   p99={pct(ttfts, 99)*1000:7.1f} ms&quot;)<br>    print(f&quot;  ITL   p50={pct(gaps, 50)*1000:7.1f} ms   p99={pct(gaps, 99)*1000:7.1f} ms&quot;)<br>    print(f&quot;  output throughput = {total_tokens / wall:8.1f} tok/s&quot;)</pre><pre>if __name__ == &quot;__main__&quot;:<br>    prompts = [f&quot;Write a short story about request number {i}.&quot; for i in range(64)]<br>    for c in (1, 8, 32):<br>        run(prompts[: max(c * 4, 8)], c)</pre><p>For a real evaluation, replace the toy prompts with a sample of your production traffic. Prompt and output length distributions change the results more than almost any setting.</p><h3>Step 4: Apply levers one at a time</h3><p><strong>Quantize weights and the KV cache to FP8</strong> (on Hopper or newer GPUs):</p><pre>vllm serve meta-llama/Llama-3.1-8B-Instruct \<br>  --quantization fp8 \<br>  --kv-cache-dtype fp8 \<br>  --max-model-len 8192 \<br>  --gpu-memory-utilization 0.90</pre><p>Expect faster decode and more concurrent sequences. Then rerun your task evals, not just the benchmark.</p><p><strong>Add speculative decoding</strong> for latency-sensitive traffic, using a published EAGLE-3 draft head for your model:</p><pre>vllm serve meta-llama/Llama-3.1-8B-Instruct \<br>  --speculative-config &#39;{&quot;method&quot;: &quot;eagle3&quot;, &quot;model&quot;: &quot;yuhuili/EAGLE3-LLaMA3.1-Instruct-8B&quot;, &quot;num_speculative_tokens&quot;: 3}&#39;</pre><p>Check the acceptance rate in the server metrics. If it is low for your traffic, try a different drafter or fewer speculative tokens.</p><p><strong>Tune the scheduler</strong> for your SLOs:</p><pre>vllm serve meta-llama/Llama-3.1-8B-Instruct \<br>  --max-num-seqs 128 \<br>  --max-num-batched-tokens 4096</pre><p>Lower max-num-batched-tokens smooths inter-token latency; higher values improve TTFT and throughput. max-num-seqs caps the batch and therefore the KV cache in use.</p><p><strong>Scale out</strong> once one replica is tuned. Add data-parallel replicas behind a router that keeps sessions on the same replica, so prefix caches stay warm.</p><h3>Step 5: Keep a results log</h3><p>Record every experiment in the same format: the change, p50 and p99 TTFT, p50 and p99 ITL, throughput at each concurrency, and your task-eval score. A single table like this prevents the most common mistake in inference tuning, which is keeping a change that improved one metric while quietly breaking another.</p><h3>A decision playbook</h3><ul><li><strong>Lowest latency for interactive chat</strong>: start with: FP8 weights, CUDA graphs, modest batch caps; then add: Speculative decoding (EAGLE-3 or MTP), TP within a node</li><li><strong>Maximum throughput for offline batch jobs</strong>: start with: Large batches, FP8 or 4-bit weights, FP8 KV cache; then add: Data-parallel replicas; skip speculative decoding unless measured</li><li><strong>Long-context RAG and document Q&amp;A</strong>: start with: FP8 KV cache, prefix caching, static-first prompts; then add: N-gram speculation, KV offloading, chunked prefill tuning</li><li><strong>Multi-turn agents with shared tools and prompts</strong>: start with: Prefix caching or RadixAttention, session-affinity routing; then add: Structured-output acceleration, speculative decoding</li><li><strong>One GPU or on-prem budget</strong>: start with: INT4 or 4-bit weights, smaller models with GQA; then add: FP8 KV cache, a good llama.cpp or vLLM build</li><li><strong>Very large MoE models at scale</strong>: start with: Expert parallelism, FP8 or FP4 weights; then add: Disaggregated prefill and decode, cache-aware routing</li></ul><h3>Pitfalls that waste the most money</h3><ol><li><strong>Benchmarking with toy prompts.</strong> Uniform short prompts make every engine look great. Real traffic has long tails in both input and output length.</li><li><strong>Reporting averages.</strong> Users feel p99 latency. Always report tail percentiles.</li><li><strong>Optimizing throughput and ignoring goodput.</strong> Pushing concurrency until throughput peaks usually blows through your TTFT target first.</li><li><strong>Quantizing without task evals.</strong> Perplexity is not your product. Test the tasks your users care about, especially reasoning and long context.</li><li><strong>Turning on speculative decoding everywhere.</strong> It shines at low concurrency and can disappoint, or even hurt, at high batch sizes.</li><li><strong>Breaking prefix caching by accident.</strong> Dynamic content at the top of the prompt turns a 90% cache hit rate into zero.</li><li><strong>Over-sharding.</strong> Using TP=8 on a model that fits on two GPUs buys a little latency at a large cost per token.</li><li><strong>Setting GPU memory utilization too high.</strong> Leave headroom for activations and CUDA graphs, or you’ll trade throughput for out-of-memory crashes under load.</li><li><strong>Forgetting the CPU.</strong> Tokenization, detokenization, sampling, structured-output grammars and HTTP handling can bottleneck a fast GPU. Profile the whole request path.</li></ol><h3>Conclusion: stop wasting the hardware</h3><p>Every technique in this guide answers the same question the roofline raised at the start: where is the GPU waiting, and how do we stop it?</p><ul><li><strong>Decode waits on memory</strong>, so read fewer bytes (quantization, efficient attention, FP8 KV cache), read them for more sequences at once (continuous batching), or get more tokens out of every read (speculative decoding).</li><li><strong>Prefill waits on compute</strong>, so avoid repeating it (prefix caching), keep it from blocking decode (chunked prefill), or give it its own hardware (disaggregation).</li><li><strong>Everything waits on bad kernels and bad scheduling</strong>, so use a modern engine, measure with real traffic, and change one thing at a time.</li></ul><p>Start with the back-of-envelope math, deploy a good engine with sensible defaults, then pull the levers your workload actually needs. The fastest optimization is often the one you didn’t need to build.</p><p><em>If this helped, follow for more deep dives on production LLM systems, and tell me in the comments which lever gave you the biggest win.</em></p><h3>Further reading</h3><ul><li>Kwon et al., <em>Efficient Memory Management for Large Language Model Serving with PagedAttention</em> (SOSP 2023)</li><li>Yu et al., <em>Orca: A Distributed Serving System for Transformer-Based Generative Models</em> (OSDI 2022)</li><li>Leviathan, Kalman and Matias, <em>Fast Inference from Transformers via Speculative Decoding</em> (ICML 2023)</li><li><a href="https://arxiv.org/pdf/2503.01840">Li et al., <em>EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test</em> (2025)</a></li><li><a href="https://arxiv.org/html/2407.08608v2">Shah et al., <em>FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision</em> (NeurIPS 2024)</a></li><li>DeepSeek-AI, <em>DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model</em> (2024)</li><li>Zhong et al., <em>DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving</em> (OSDI 2024)</li><li><a href="https://developer.nvidia.com/blog/introducing-nvfp4-for-efficient-and-accurate-low-precision-inference/">NVIDIA, <em>Introducing NVFP4 for Efficient and Accurate Low-precision Inference</em></a></li><li><a href="https://docs.nvidia.com/dynamo/knowledge-base/concepts/system-architecture/disaggregated-serving">NVIDIA Dynamo documentation, <em>Disaggregated Serving</em></a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba8429a386fd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hybrid RAG: Why Your Retriever Needs Both Keywords and Vectors]]></title>
            <link>https://medium.com/@kavishkakalhara/hybrid-rag-why-your-retriever-needs-both-keywords-and-vectors-70cee36d8bd6?source=rss-b3cb50279e64------2</link>
            <guid isPermaLink="false">https://medium.com/p/70cee36d8bd6</guid>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Kavishka Kalhara]]></dc:creator>
            <pubDate>Wed, 04 Nov 2026 08:05:58 GMT</pubDate>
            <atom:updated>2026-11-04T08:05:58.702Z</atom:updated>
            <content:encoded><![CDATA[<p><em>An advanced, end-to-end guide to building retrieval that actually finds the right context: fusion math, rerankers, graph hybrids, code, and evaluation.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pFR9eMe4VI8DLIKSZbFadg.png" /></figure><h3>The query your vector database got wrong</h3><p>A support engineer types <em>“error E-4012 after upgrading to v3.2.1”</em> into your shiny new RAG chatbot. The retriever returns five beautifully relevant chunks about upgrade errors in general. None of them mention E-4012. The LLM, helpful as ever, confidently invents a fix.</p><p>Nothing is broken. Your embeddings did exactly what they were trained to do: capture <em>meaning</em>. The problem is that this query was never about meaning. It was about an exact string, such as an error code or a version number, and dense vectors are famously bad at those.</p><p>Flip the scenario. A user asks <em>“how do I stop my laptop from overheating?”</em> and your keyword index fails because the manual only ever says “thermal throttling” and “fan curve.” Now BM25 is the one that’s blind.</p><p><strong>Hybrid RAG</strong> is the answer to both failures: run lexical (keyword) and semantic (vector) retrieval side by side, fuse their rankings, and optionally rerank the result before it ever reaches the LLM. It has quietly become the default architecture in serious production systems, and every major vector database (Weaviate, Qdrant, Elasticsearch/OpenSearch, Milvus, Pinecone, pgvector setups) now supports it natively.</p><p>This article goes past the “just add BM25” advice. We’ll cover:</p><ul><li>Why each retriever fails, mechanically</li><li>The math behind Reciprocal Rank Fusion, convex combination and distribution-based fusion</li><li>Learned sparse models (SPLADE), late interaction (ColBERT) and contextual retrieval</li><li>Cross-encoder reranking and graph + vector hybrids</li><li>A complete, runnable Python implementation</li><li>How to evaluate and tune it so you stop guessing</li></ul><h3>Why one retriever is never enough</h3><p>Dense and sparse retrieval fail in opposite, complementary ways, which is exactly why combining them works.</p><h3>Sparse retrieval (BM25): the exact-match specialist</h3><p>BM25 represents each document as a bag of weighted terms. A document scores highly when it contains the query’s terms, especially rare ones, with diminishing returns for repetition and a penalty for very long documents:</p><pre>BM25(q, d) = Σ  IDF(t) · f(t,d)·(k1 + 1) / ( f(t,d) + k1·(1 − b + b·|d| / avgdl) )<br>            t∈q</pre><p>Here f(t,d) is term frequency, |d| is document length, and k1 ≈ 1.2 to 2.0 and b ≈ 0.75 are the usual defaults.</p><p><strong>Where it shines:</strong> product SKUs, error codes, function names, legal clause numbers, drug names, people’s names, acronyms and any rare token. It needs no training, is cheap, explainable, and handles out-of-domain jargon gracefully.</p><p><strong>Where it breaks:</strong> the vocabulary mismatch problem. “Car” doesn’t match “automobile.” Paraphrases, synonyms, multilingual queries and conceptual questions all slip through.</p><h3>Dense retrieval (embeddings): the meaning specialist</h3><p>A bi-encoder maps the query and each chunk into the same vector space, and retrieval becomes nearest-neighbour search by cosine similarity, usually through an ANN index like HNSW.</p><p><strong>Where it shines:</strong> paraphrase, synonyms, natural-language questions, cross-lingual matching, and fuzzy intent (“something like X but cheaper”).</p><p><strong>Where it breaks:</strong></p><ul><li><strong>Rare and unseen tokens.</strong> “E-4012” gets tokenised into fragments that carry almost no meaning; the embedding of that query is dominated by “error” and “upgrade.”</li><li><strong>Numbers and versions.</strong> v3.2.1 and v3.1.2 sit almost on top of each other in vector space.</li><li><strong>Negation and precision.</strong> “Compatible with” and “not compatible with” embed close together.</li><li><strong>Domain shift.</strong> A general-purpose embedding model has never seen your internal project codenames.</li><li><strong>Compression.</strong> A whole chunk squeezed into one vector loses detail; long chunks lose more.</li></ul><h3>The complementarity in one table</h3><pre>Query type                                      BM25    Dense                         Hybrid<br>----------------------------------------------  ------  ----------------------------  ------<br>Exact identifiers (SKU, error code, ticket ID)  Strong  Weak                          Strong<br>Paraphrased natural-language questions          Weak    Strong                        Strong<br>Rare domain jargon                              Strong  Weak to medium                Strong<br>Synonyms and conceptual queries                 Weak    Strong                        Strong<br>Multilingual queries                            Weak    Strong (multilingual models)  Strong<br>Short keyword-style queries                     Strong  Medium                        Strong</pre><p>The key insight: the <em>union</em> of what two retrievers find is far larger than what either finds alone, and the overlap is a strong signal of relevance. Hybrid search exploits both.</p><h3>What Hybrid RAG actually is</h3><p>Hybrid RAG is a retrieval-augmented generation pipeline whose retrieval step combines two or more complementary retrievers (at minimum one lexical and one semantic) and merges their results before generation.</p><p>A production-grade version has five stages:</p><ol><li><strong>Query understanding:</strong> optionally classify, rewrite or decompose the query, and extract metadata filters.</li><li><strong>Parallel retrieval:</strong> run sparse (BM25 or SPLADE) and dense (embedding) search at the same time, each returning a few dozen candidates.</li><li><strong>Fusion:</strong> merge the ranked lists into one, rewarding documents both retrievers agree on.</li><li><strong>Reranking:</strong> rescore the fused candidates with a more expensive, more accurate model.</li><li><strong>Generation:</strong> pass the top handful of chunks to the LLM with instructions to answer only from them and cite them.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*snbDrcETY0gGeuZsfoTnuQ.png" /><figcaption>Hybrid RAG query path: two retrieval legs, an optional graph leg, fusion, reranking and generation</figcaption></figure><p>Retrieval is cheap and wide, reranking is expensive and narrow, and the LLM sees only what survives both. The dashed graph leg is the optional third retriever covered later.</p><p>The term also has a second, broader meaning: combining vector retrieval with <strong>knowledge-graph</strong> retrieval. We’ll cover both, because the best systems increasingly use both.</p><h3>Fusion: the heart of hybrid search</h3><p>You now have two ranked lists with scores on incompatible scales. BM25 scores are unbounded (12.7, 8.3, …); cosine similarities live in a narrow band (0.82, 0.79, …). Adding them directly is meaningless. Fusion is how you merge them correctly.</p><h3>1. Reciprocal Rank Fusion (RRF)</h3><p>RRF, introduced by Cormack, Clarke and Büttcher in 2009, throws scores away entirely and uses only rank positions:</p><pre>RRF(d) = Σ  1 / ( k + rank_r(d) )<br>        r∈R</pre><p>R is the set of retrievers, rank_r(d) is the 1-based position of document d in retriever r’s list, and k is a smoothing constant, conventionally 60.</p><p>A document ranked 1st by BM25 and 3rd by dense gets 1/61 + 1/63 ≈ 0.0323. A document ranked 1st by only one retriever gets 1/61 ≈ 0.0164. Agreement wins, which is exactly the behaviour we want.</p><p><strong>Why it’s the default:</strong> no normalisation, no tuning, robust to outliers, and it works with any number of retrievers. Its weakness is that it discards score <em>magnitude</em>: a dense hit at 0.95 similarity and one at 0.51 are treated identically if they share a rank.</p><p>Weighted RRF adds a per-retriever weight w_r to the numerator, letting you lean towards one retriever without abandoning rank-based fusion.</p><h3>2. Convex combination (weighted score fusion)</h3><p>Normalise each retriever’s scores to [0, 1], then blend them with a single parameter α:</p><pre>score(d) = α · ŝ_dense(d) + (1 − α) · ŝ_sparse(d)</pre><p>Min-max normalisation is the common choice:</p><pre>ŝ(d) = ( s(d) − min(s) ) / ( max(s) − min(s) )</pre><p>α = 1 is pure vector search, α = 0 is pure keyword search. This is what Weaviate’s alpha parameter controls. When tuned on labelled data, convex combination often beats RRF because it keeps score magnitude. The 2023 study by Bruch, Gai and Ingber on fusion functions found it more sample-efficient and more robust than RRF once α was tuned. The catch is that it <em>must</em> be tuned, and min-max is sensitive to outliers in a small candidate set.</p><h3>3. Distribution-Based Score Fusion (DBSF)</h3><p>DBSF normalises each list using its statistical distribution instead of its extremes: scores are clipped to roughly μ ± 3σ, then scaled to [0, 1]. One unusually high BM25 score no longer squashes everyone else to zero. Qdrant exposes this as a built-in fusion mode alongside RRF.</p><h3>Which should you choose?</h3><pre>Method              Needs tuning       Uses score magnitude  Best when<br>------------------  -----------------  --------------------  -----------------------------------------------------------<br>RRF                 No (k = 60 works)  No                    You have no labelled eval set yet; many retrievers<br>Weighted RRF        Light (weights)    No                    One retriever is clearly stronger for your corpus<br>Convex combination  Yes (α)            Yes                   You have an eval set and want the last few points of recall<br>DBSF                No                 Yes                   Score distributions have heavy outliers</pre><p><strong>Practical rule:</strong> start with RRF. Once you have 100 to 200 labelled queries, sweep α for convex combination and keep whichever wins on your metric.</p><h3>Beyond BM25 + embeddings: the advanced retriever toolkit</h3><p>“Hybrid” doesn’t have to mean exactly one BM25 index and one vector index. Four techniques push the ceiling much higher.</p><h3>Learned sparse retrieval (SPLADE)</h3><p>SPLADE uses a masked-language model to produce a <em>sparse</em> vector over the whole vocabulary, but with learned weights and <strong>term expansion</strong>. A chunk about “automobile repair” also gets non-zero weight on “car,” “vehicle” and “mechanic,” even though those words never appear.</p><p>You get BM25’s inverted-index efficiency and exact-match behaviour, plus a large slice of semantic matching. Many teams now run SPLADE (or similar models such as BGE-M3’s sparse head) as the sparse leg of their hybrid instead of plain BM25. Qdrant, Milvus, Elasticsearch (ELSER) and Pinecone all store sparse vectors natively.</p><p><strong>Trade-off:</strong> larger indexes than BM25, GPU-friendly encoding at ingest, and weaker behaviour on truly unseen identifiers than raw BM25. A three-way hybrid (BM25 + SPLADE + dense) is not unusual for high-stakes search.</p><h3>Late interaction (ColBERT)</h3><p>A bi-encoder compresses a chunk into one vector. ColBERT keeps <strong>one vector per token</strong> and scores with <em>MaxSim</em>: for each query token, find its most similar document token, then sum:</p><pre>score(q, d) = Σ   max  ( E_qi · E_dj )<br>             i∈q  j∈d</pre><p>This captures fine-grained token matches (including those pesky identifiers) while staying semantic. ColBERTv2 and PLAID made storage and latency practical, and libraries like RAGatouille and Vespa / Qdrant multivector support make it usable. ColBERT works well either as a third retriever or as a cheaper reranker.</p><h3>Contextual retrieval</h3><p>Chunking destroys context. A chunk that says <em>“Revenue grew 3% over the previous quarter”</em> doesn’t say which company or which quarter, so neither BM25 nor embeddings can match it to <em>“ACME Q2 2023 revenue growth.”</em></p><p><a href="https://anthropic.com/news/contextual-retrieval">Anthropic’s <strong>Contextual Retrieval</strong></a> fixes this at ingest time: an LLM reads the full document and writes a short context for each chunk, which is prepended before <em>both</em> embedding and BM25 indexing. The generated context is typically 50 to 100 tokens.</p><p>The reported gains on the top-20 retrieval failure rate are substantial. Contextual embeddings alone cut it by 35% (5.7% to 3.7%). Adding contextual BM25 cut it by 49% (to 2.9%). Adding a reranker on top cut it by 67% (to 1.9%). Note what this means for hybrid search: the lexical leg contributed a large share of the gain, and the reranker contributed the rest.</p><p>Prompt caching keeps the ingest cost low, because the full document is cached once and reused for every chunk’s context call.</p><h3>Metadata filtering as a third signal</h3><p>Structured filters are the most underrated part of hybrid search. Extract fields at ingest (product, version, date, department, language, access level) and apply them <em>before</em> or <em>during</em> retrieval. A query like <em>“refund policy for EU customers in 2025”</em> should never surface US 2022 policies, no matter how semantically similar.</p><p>An LLM can turn the query into filters (“self-querying”), and every modern vector database supports pre-filtering in the ANN index. Filters also enforce <strong>document-level permissions</strong>, which is non-negotiable in enterprise RAG.</p><h3>Reranking: the precision layer</h3><p>Fusion maximises <strong>recall</strong>: getting the right chunk somewhere into the top 50. Reranking maximises <strong>precision</strong>: getting it into the top 5 that the LLM actually sees.</p><h3>Bi-encoders vs cross-encoders</h3><p>A bi-encoder embeds the query and the document <em>separately</em>, so document vectors can be precomputed. That’s fast but shallow: the two texts never “see” each other.</p><p>A cross-encoder feeds the query and document <em>together</em> through a transformer and outputs one relevance score. Every query token attends to every document token, so it catches negation, exact constraints and subtle relevance that a bi-encoder misses. The price is that nothing can be precomputed; you run the model once per (query, candidate) pair.</p><p>That’s why the standard pattern is a <strong>funnel</strong>:</p><ol><li>Hybrid retrieval pulls 50 to 150 candidates cheaply.</li><li>A cross-encoder rescores those candidates.</li><li>The top 5 to 10 go into the prompt.</li></ol><h3>Rerankers worth knowing</h3><pre>Type                        Examples                                                           Notes<br>--------------------------  -----------------------------------------------------------------  ------------------------------------------------------------<br>Open-source cross-encoders  BAAI bge-reranker-v2-m3, mixedbread mxbai-rerank, ms-marco-MiniLM  Self-hostable; MiniLM runs on CPU for small candidate sets<br>Hosted rerank APIs          Cohere Rerank, Voyage rerank, Jina reranker                        One API call, multilingual, no GPU to manage<br>Late interaction            ColBERTv2                                                          Cheaper than a cross-encoder, more precise than a bi-encoder<br>LLM-as-reranker             Listwise prompting with a small LLM                                Highest quality on hard queries; highest latency and cost</pre><h3>Tips that actually matter</h3><ul><li><strong>Rerank the fused list, not each list separately.</strong> The reranker should judge every candidate on equal footing.</li><li><strong>Watch the input length.</strong> Most cross-encoders truncate at 512 tokens; oversized chunks get silently cut.</li><li><strong>Use a score threshold.</strong> Rerankers output calibrated-ish scores. Dropping candidates below a threshold reduces noise and lets the LLM say “I don’t know” instead of hallucinating from weak context.</li><li><strong>Measure latency per candidate.</strong> Reranking 100 candidates on a CPU can add a second or more. Batch, use a GPU, or shrink the candidate pool.</li></ul><h3>The other hybrid: graphs, vectors and query routing</h3><p>“Hybrid RAG” has a second meaning in 2026: combining <strong>vector retrieval with knowledge-graph retrieval</strong>. It solves a different class of failure.</p><h3>When chunks aren’t enough</h3><p>Vector and keyword search both retrieve <em>passages</em>. They struggle with questions whose answer is spread across many documents or lives in the relationships between entities:</p><ul><li><em>“Which suppliers of our top three products had quality incidents last year?”</em> (multi-hop)</li><li><em>“What are the main themes across all customer complaints?”</em> (global, corpus-wide)</li><li><em>“Who reports to the person who approved this contract?”</em> (relational)</li></ul><p>A knowledge graph stores entities (nodes) and relationships (edges). Traversing it answers multi-hop questions that no single chunk can.</p><h3>How graph + vector hybrids work</h3><ol><li><strong>Ingest:</strong> an LLM extracts entities and relations from each chunk into a graph store (Neo4j, Memgraph, or an in-memory graph), while the chunks also go into the vector and keyword indexes. Each graph node links back to its source chunks.</li><li><strong>Entry:</strong> at query time, vector + keyword search finds the most relevant chunks and the entities mentioned in them.</li><li><strong>Expand:</strong> traverse one or two hops from those entities to pull in connected facts.</li><li><strong>Assemble:</strong> send the LLM both the retrieved passages and a compact text rendering of the subgraph.</li></ol><p>Microsoft’s GraphRAG popularised a variant that clusters the graph into communities and pre-writes summaries for them, which is what makes corpus-wide “themes” questions answerable. LightRAG and similar projects offer lighter-weight dual-level (local entity + global theme) retrieval.</p><p><strong>The honest trade-off:</strong> graph extraction is expensive at ingest, entity resolution is hard (is “IBM” the same node as “International Business Machines”?), and graphs go stale. Use a graph only when your questions are genuinely relational or global.</p><h3>Query routing: don’t run everything every time</h3><p>Not every query needs the full machinery. A lightweight router (rules, a small classifier, or an LLM call with structured output) decides per query:</p><pre>Query signal                                    Route<br>----------------------------------------------  -------------------------------------------<br>Contains an ID, code, version or quoted phrase  Lean on keyword (lower α, or keyword-first)<br>Natural-language question                       Balanced hybrid<br>Multi-hop or relational wording                 Hybrid + graph expansion<br>Corpus-wide &quot;summarise / themes / overall&quot;      Graph community summaries<br>Chit-chat or answerable without context         Skip retrieval entirely</pre><p>Some teams go further with <strong>dynamic α</strong>: an LLM or classifier predicts the keyword-vs-semantic balance per query instead of using one global constant.</p><h3>Query transformation</h3><p>Routing pairs well with rewriting the query before retrieval:</p><ul><li><strong>Multi-query:</strong> generate 3 to 5 paraphrases, retrieve for each, fuse everything with RRF.</li><li><strong>HyDE:</strong> have the LLM write a hypothetical answer and embed <em>that</em> for the dense leg (keep the original query for BM25).</li><li><strong>Decomposition:</strong> split a compound question into sub-questions and retrieve for each.</li></ul><h3>Building it: a complete Hybrid RAG pipeline in Python</h3><p>We’ll build two versions. First, a from-scratch pipeline with no database, so every moving part is visible. Then, a production version using Qdrant’s native hybrid query.</p><h3>Part 1: from scratch</h3><pre>pip install numpy sentence-transformers</pre><p><strong>Step 1: A tokenizer that respects identifiers.</strong> Default whitespace tokenizers shred “E-4012” and “v3.2.1”. This regex keeps hyphenated, dotted and underscored tokens whole. It’s a small detail that decides whether BM25 can find your error codes at all.</p><pre>import re, math<br>from collections import Counter<br>import numpy as np</pre><pre>TOKEN_RE = re.compile(r&quot;[A-Za-z0-9]+(?:[-_.][A-Za-z0-9]+)*&quot;)</pre><pre>def tokenize(text: str) -&gt; list[str]:<br>    return [t.lower() for t in TOKEN_RE.findall(text)]</pre><pre># tokenize(&quot;Error E-4012 after upgrading to v3.2.1&quot;)<br># -&gt; [&#39;error&#39;, &#39;e-4012&#39;, &#39;after&#39;, &#39;upgrading&#39;, &#39;to&#39;, &#39;v3.2.1&#39;]</pre><p><strong>Step 2: BM25 in about 20 lines.</strong> Writing it yourself makes the formula from earlier concrete. (In production, use your search engine’s implementation.)</p><pre>class BM25:<br>    def __init__(self, corpus_tokens, k1=1.5, b=0.75):<br>        self.k1, self.b = k1, b<br>        self.docs = [Counter(toks) for toks in corpus_tokens]<br>        self.lens = np.array([len(t) for t in corpus_tokens], dtype=float)<br>        self.avgdl = self.lens.mean()<br>        df = Counter(term for d in self.docs for term in d)<br>        n = len(self.docs)<br>        self.idf = {t: math.log(1 + (n - f + 0.5) / (f + 0.5)) for t, f in df.items()}</pre><pre>    def scores(self, query_tokens):<br>        out = np.zeros(len(self.docs))<br>        for t in set(query_tokens):<br>            idf = self.idf.get(t)<br>            if idf is None:<br>                continue<br>            tf = np.array([d.get(t, 0) for d in self.docs], dtype=float)<br>            denom = tf + self.k1 * (1 - self.b + self.b * self.lens / self.avgdl)<br>            out += idf * tf * (self.k1 + 1) / denom<br>        return out</pre><p><strong>Step 3: Fusion functions.</strong> Both RRF and convex combination, so you can compare them on your own data.</p><pre>def rrf(result_lists, k=60, weights=None):<br>    &quot;&quot;&quot;result_lists: list of [(doc_id, score), ...] sorted best-first.&quot;&quot;&quot;<br>    weights = weights or [1.0] * len(result_lists)<br>    fused = {}<br>    for w, results in zip(weights, result_lists):<br>        for rank, (doc_id, _) in enumerate(results, start=1):<br>            fused[doc_id] = fused.get(doc_id, 0.0) + w / (k + rank)<br>    return sorted(fused.items(), key=lambda x: x[1], reverse=True)<br></pre><pre>def minmax(results):<br>    if not results:<br>        return {}<br>    s = np.array([v for _, v in results])<br>    lo, hi = s.min(), s.max()<br>    if hi - lo &lt; 1e-9:<br>        return {d: 1.0 for d, _ in results}<br>    return {d: float((v - lo) / (hi - lo)) for d, v in results}<br></pre><pre>def convex(dense, sparse, alpha=0.5):<br>    d, s = minmax(dense), minmax(sparse)<br>    fused = {i: alpha * d.get(i, 0.0) + (1 - alpha) * s.get(i, 0.0)<br>             for i in set(d) | set(s)}<br>    return sorted(fused.items(), key=lambda x: x[1], reverse=True)</pre><p><strong>Step 4: The retriever.</strong> It runs both legs, fuses, then reranks the fused candidates with a cross-encoder.</p><pre>from sentence_transformers import SentenceTransformer, CrossEncoder</pre><pre>class HybridRetriever:<br>    def __init__(self, chunks: list[str],<br>                 embed_model=&quot;BAAI/bge-small-en-v1.5&quot;,<br>                 rerank_model=&quot;BAAI/bge-reranker-base&quot;):<br>        self.chunks = chunks<br>        self.bm25 = BM25([tokenize(c) for c in chunks])<br>        self.embedder = SentenceTransformer(embed_model)<br>        self.doc_vecs = self.embedder.encode(<br>            chunks, normalize_embeddings=True, batch_size=64, show_progress_bar=True)<br>        self.reranker = CrossEncoder(rerank_model)</pre><pre>    def _sparse(self, query, k):<br>        s = self.bm25.scores(tokenize(query))<br>        idx = np.argsort(-s)[:k]<br>        return [(int(i), float(s[i])) for i in idx if s[i] &gt; 0]</pre><pre>    def _dense(self, query, k):<br>        q = self.embedder.encode(query, normalize_embeddings=True)<br>        s = self.doc_vecs @ q                      # cosine, since both are normalised<br>        idx = np.argsort(-s)[:k]<br>        return [(int(i), float(s[i])) for i in idx]</pre><pre>    def search(self, query, k_candidates=50, top_n=5,<br>               fusion=&quot;rrf&quot;, alpha=0.5, rerank=True, min_score=None):<br>        sparse = self._sparse(query, k_candidates)<br>        dense = self._dense(query, k_candidates)<br>        fused = rrf([sparse, dense]) if fusion == &quot;rrf&quot; else convex(dense, sparse, alpha)<br>        candidates = [doc_id for doc_id, _ in fused[:k_candidates]]</pre><pre>        if not rerank:<br>            return [(i, self.chunks[i]) for i in candidates[:top_n]]</pre><pre>        scores = self.reranker.predict([(query, self.chunks[i]) for i in candidates])<br>        ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)<br>        if min_score is not None:<br>            ranked = [(i, s) for i, s in ranked if s &gt;= min_score]<br>        return [(i, self.chunks[i]) for i, _ in ranked[:top_n]]</pre><p><strong>Step 5: Generation with grounded citations.</strong> Number the chunks so the model can cite them, and give it explicit permission to say it doesn’t know.</p><pre>def build_prompt(question, retrieved):<br>    context = &quot;\n\n&quot;.join(f&quot;[{n}] {text}&quot; for n, (_, text) in enumerate(retrieved, 1))<br>    return f&quot;&quot;&quot;Answer the question using only the sources below.<br>Cite sources like [1]. If the sources don&#39;t contain the answer, say so.</pre><pre>Sources:<br>{context}</pre><pre>Question: {question}&quot;&quot;&quot;</pre><pre>retriever = HybridRetriever(chunks)           # chunks from your ingestion step<br>hits = retriever.search(&quot;error E-4012 after upgrading to v3.2.1&quot;)<br>prompt = build_prompt(&quot;How do I fix error E-4012?&quot;, hits)<br># send `prompt` to the LLM of your choice</pre><h3>Part 2: production version with Qdrant</h3><p>The from-scratch version holds everything in memory. In production, let the database do retrieval <em>and</em> fusion in one round trip. Qdrant stores dense and sparse vectors side by side and fuses them server-side through its Query API.</p><pre>pip install qdrant-client fastembed</pre><pre>from qdrant_client import QdrantClient, models<br>from fastembed import TextEmbedding, SparseTextEmbedding</pre><pre>client = QdrantClient(url=&quot;http://localhost:6333&quot;)<br>dense_model = TextEmbedding(&quot;BAAI/bge-small-en-v1.5&quot;)     # 384 dims<br>sparse_model = SparseTextEmbedding(&quot;Qdrant/bm25&quot;)          # or a SPLADE model</pre><pre>client.create_collection(<br>    collection_name=&quot;docs&quot;,<br>    vectors_config={&quot;dense&quot;: models.VectorParams(size=384, distance=models.Distance.COSINE)},<br>    sparse_vectors_config={&quot;sparse&quot;: models.SparseVectorParams(modifier=models.Modifier.IDF)},<br>)</pre><pre>def index(chunks: list[dict]):<br>    &quot;&quot;&quot;chunks: [{&#39;id&#39;: int, &#39;text&#39;: str, &#39;product&#39;: str, &#39;version&#39;: str}, ...]&quot;&quot;&quot;<br>    texts = [c[&quot;text&quot;] for c in chunks]<br>    dense = list(dense_model.embed(texts))<br>    sparse = list(sparse_model.embed(texts))<br>    client.upsert(&quot;docs&quot;, points=[<br>        models.PointStruct(<br>            id=c[&quot;id&quot;],<br>            vector={<br>                &quot;dense&quot;: d.tolist(),<br>                &quot;sparse&quot;: models.SparseVector(indices=s.indices.tolist(), values=s.values.tolist()),<br>            },<br>            payload=c,<br>        )<br>        for c, d, s in zip(chunks, dense, sparse)<br>    ])</pre><pre>def hybrid_search(query: str, product: str | None = None, limit: int = 50):<br>    q_dense = next(dense_model.query_embed(query)).tolist()<br>    q_sparse = next(sparse_model.query_embed(query))<br>    flt = models.Filter(must=[models.FieldCondition(<br>        key=&quot;product&quot;, match=models.MatchValue(value=product))]) if product else None</pre><pre>    return client.query_points(<br>        collection_name=&quot;docs&quot;,<br>        prefetch=[<br>            models.Prefetch(<br>                query=models.SparseVector(indices=q_sparse.indices.tolist(),<br>                                          values=q_sparse.values.tolist()),<br>                using=&quot;sparse&quot;, limit=limit, filter=flt),<br>            models.Prefetch(query=q_dense, using=&quot;dense&quot;, limit=limit, filter=flt),<br>        ],<br>        query=models.FusionQuery(fusion=models.Fusion.RRF),   # or models.Fusion.DBSF<br>        limit=limit,<br>    ).points</pre><p>Pass the fused points to the same cross-encoder from Part 1, and you have the full funnel: filtered hybrid retrieval, server-side fusion, and reranking.</p><p>The same pattern maps directly onto other engines: Weaviate’s hybrid query with alpha, Elasticsearch/OpenSearch’s RRF retriever or hybrid query pipelines, Milvus’s hybrid_search with an RRFRanker, and Postgres with pgvector plus full-text search fused in SQL.</p><h3>Evaluation: stop guessing, start measuring</h3><p>Hybrid search has knobs: α, RRF k, candidate pool size, reranker threshold, chunk size. Without an evaluation set, every one of them is a guess. Evaluate retrieval and generation <strong>separately</strong>: if the right chunk never reaches the prompt, no amount of prompt engineering will save the answer.</p><h3>Build a small golden set</h3><p>You need 100 to 300 queries, each paired with the chunk IDs that answer it. Three ways to get them:</p><ol><li><strong>Real logs:</strong> sample actual user queries and label the correct chunks. The most valuable, because it matches real traffic.</li><li><strong>Synthetic:</strong> have an LLM read a chunk and write 2 or 3 questions it answers. Fast and cheap, but it skews towards vocabulary that appears in the chunk, which flatters BM25.</li><li><strong>Adversarial:</strong> deliberately write identifier queries, paraphrases, negations and multi-hop questions. These are the ones that separate good retrievers from lucky ones.</li></ol><p>Tag each query by type (identifier, paraphrase, multi-hop, and so on). Aggregate scores hide the most useful finding: <em>which</em> query types each configuration fails on.</p><h3>Retrieval metrics</h3><pre>Metric                   What it answers                                             Use it for<br>-----------------------  ----------------------------------------------------------  ----------------------------------------------<br>Recall@k                 Did any correct chunk make the top k?                       The candidate stage, before reranking (k = 50)<br>MRR                      How high is the first correct chunk?                        Single-answer, FAQ-style queries<br>nDCG@k                   Are the correct chunks near the top, weighted by position?  The final top 5 to 10 the LLM sees<br>Hit rate / failure rate  Fraction of queries with zero correct chunks                Headline number for stakeholders</pre><pre>import math</pre><pre>def recall_at_k(ranked, relevant, k):<br>    return len(set(ranked[:k]) &amp; relevant) / len(relevant)</pre><pre>def mrr(ranked, relevant):<br>    for i, d in enumerate(ranked, 1):<br>        if d in relevant:<br>            return 1 / i<br>    return 0.0</pre><pre>def ndcg_at_k(ranked, relevant, k):<br>    dcg = sum(1 / math.log2(i + 1) for i, d in enumerate(ranked[:k], 1) if d in relevant)<br>    idcg = sum(1 / math.log2(i + 1) for i in range(1, min(len(relevant), k) + 1))<br>    return dcg / idcg if idcg else 0.0</pre><h3>Tuning α (and everything else) with a sweep</h3><pre>def evaluate(retriever, golden, **kwargs):<br>    rows = []<br>    for item in golden:                       # {&#39;query&#39;: str, &#39;relevant&#39;: set[int], &#39;type&#39;: str}<br>        ranked = [i for i, _ in retriever.search(item[&quot;query&quot;], rerank=False, top_n=10, **kwargs)]<br>        rows.append({&quot;type&quot;: item[&quot;type&quot;],<br>                     &quot;recall@10&quot;: recall_at_k(ranked, item[&quot;relevant&quot;], 10),<br>                     &quot;ndcg@10&quot;: ndcg_at_k(ranked, item[&quot;relevant&quot;], 10)})<br>    return rows</pre><pre>for alpha in [0.0, 0.2, 0.4, 0.5, 0.6, 0.8, 1.0]:<br>    rows = evaluate(retriever, golden, fusion=&quot;convex&quot;, alpha=alpha)<br>    print(alpha, sum(r[&quot;ndcg@10&quot;] for r in rows) / len(rows))</pre><pre>rows = evaluate(retriever, golden, fusion=&quot;rrf&quot;)   # the baseline to beat</pre><p>What you’ll typically see: α = 0 (pure BM25) wins on identifier queries, α = 1 (pure dense) wins on paraphrases, and the best overall α sits somewhere in the middle. It usually leans dense for conversational text and leans keyword for code, logs and technical docs. Your corpus decides the exact value, which is why you sweep. If no α clearly beats RRF, keep RRF: it’s simpler and more robust to corpus drift.</p><h3>End-to-end metrics</h3><p>Once retrieval is solid, measure the whole pipeline. Frameworks like RAGAS, DeepEval and TruLens compute LLM-judged scores such as:</p><ul><li><strong>Context precision / recall:</strong> how much of the retrieved context is relevant, and how much of the needed information was retrieved.</li><li><strong>Faithfulness / groundedness:</strong> whether every claim in the answer is supported by the retrieved context.</li><li><strong>Answer relevance:</strong> whether the answer addresses the question.</li></ul><p>Treat LLM-judged scores as directional, spot-check them by hand, and keep the retrieval metrics above as your ground truth.</p><h3>Taking it to production</h3><h3>Latency budget</h3><p>The two retrieval legs run in parallel, so hybrid search costs roughly the slower of the two, not the sum. The reranker is usually the most expensive stage.</p><pre>Stage                      Typical cost driver                How to keep it fast<br>-------------------------  ---------------------------------  ---------------------------------------------------------------------------------<br>Query embedding            Model size, cold starts            Small embedding model, keep it warm, cache frequent queries<br>Sparse + dense retrieval   Index size, filters                Run legs concurrently; push fusion into the database<br>Fusion                     Negligible                         Nothing to optimise<br>Reranking                  Candidates × tokens per candidate  Cap candidates at 30 to 100, use a GPU or a hosted API, distil to a smaller model<br>Query rewriting / routing  An extra LLM call                  Use a small model, or rules for the obvious cases</pre><h3>Chunking for hybrid search</h3><p>Chunking affects the two legs differently. Very long chunks dilute embeddings; very short chunks starve BM25 of context and break apart sentences that belong together.</p><ul><li>Split on structure first (headings, sections, code blocks, table rows), then on size.</li><li>A few hundred tokens with a small overlap is a sensible starting point; tune it with the eval set like any other knob.</li><li>Prepend the document title and section path to every chunk (“Admin Guide &gt; Licensing &gt; Errors”). It’s a cheap, deterministic cousin of contextual retrieval and helps both legs.</li><li>Keep tables and code intact; splitting them mid-way produces chunks that neither retriever can use.</li></ul><h3>Common pitfalls</h3><ol><li><strong>Summing raw scores.</strong> BM25 and cosine scores are on different scales. Always use RRF or normalise first.</li><li><strong>Mismatched tokenisation.</strong> If the indexer and the query path tokenise differently, BM25 silently misses. Use the same analyser for both, and make sure it keeps identifiers whole.</li><li><strong>Different candidate pools.</strong> If the sparse leg returns 10 results and the dense leg returns 100, the fusion is lopsided before it starts. Keep the per-leg limits equal unless you mean otherwise.</li><li><strong>Reranking too few candidates.</strong> A reranker can only reorder what it’s given. If recall@50 is poor, fix retrieval first; the reranker can’t conjure the missing chunk.</li><li><strong>Stale indexes.</strong> Updating a document means updating the sparse index, the dense index <em>and</em> any graph, either atomically or with a version field you filter on.</li><li><strong>Ignoring permissions.</strong> Filter by access level inside retrieval, never after generation. A leaked chunk in the prompt is a leaked chunk in the answer.</li><li><strong>Embedding model swaps.</strong> Changing the embedding model means re-embedding the whole corpus. Store the model name and version with every vector.</li><li><strong>No feedback loop.</strong> Log queries, retrieved IDs, reranker scores and user feedback. Your next golden set is sitting in those logs.</li></ol><h3>A sensible default stack</h3><p>If you want one opinionated starting point:</p><ul><li>Structure-aware chunks with title and section prefixes</li><li>BM25 (or SPLADE) + a small, strong embedding model</li><li>RRF fusion with equal per-leg limits of about 50</li><li>Metadata pre-filters for product, version, date and permissions</li><li>A cross-encoder reranker down to 5 to 8 chunks, with a minimum-score threshold</li><li>A golden set of 200 typed queries and a weekly regression run</li></ul><p>Add contextual retrieval when chunks lose their meaning out of context. Add graph expansion only when your users ask relational or corpus-wide questions.</p><h3>Conclusion: retrieval is the product</h3><p>Most RAG failures that look like “the LLM hallucinated” are really “the retriever missed.” Hybrid RAG attacks that failure at its root: keyword search catches the exact, dense search catches the meaning, fusion rewards agreement, and a reranker makes sure the best evidence lands where the model will read it.</p><p>The progression is incremental, and each step is measurable:</p><ol><li>Add BM25 next to your vector search and fuse with RRF.</li><li>Add a cross-encoder reranker over the top 50.</li><li>Build a golden set and tune α, candidate counts and chunking against it.</li><li>Add metadata filters, then contextual retrieval, then graph expansion (only if your questions demand it).</li></ol><p>If you take one thing away: <strong>never trust a single retriever, and never tune without an eval set.</strong></p><p><em>If this was useful, follow for more deep dives on production LLM systems, and tell me in the comments which query type breaks your RAG pipeline most often.</em></p><h3>Further reading</h3><ul><li>Cormack, Clarke and Büttcher, <em>Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods</em> (SIGIR 2009)</li><li>Bruch, Gai and Ingber, <em>An Analysis of Fusion Functions for Hybrid Retrieval</em> (ACM TOIS, 2023)</li><li>Formal et al., <em>SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking</em> (SIGIR 2021)</li><li>Khattab and Zaharia, <em>ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT</em> (SIGIR 2020)</li><li>Edge et al., <em>From Local to Global: A Graph RAG Approach to Query-Focused Summarization</em> (Microsoft Research, 2024)</li><li><a href="https://anthropic.com/news/contextual-retrieval">Anthropic, Introducing Contextual Retrieval</a> (September 2024)</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=70cee36d8bd6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your AI Agent Just Deleted a Failing Test. Here’s an Architecture That Would Have Stopped It.]]></title>
            <link>https://medium.com/@kavishkakalhara/your-ai-agent-just-deleted-a-failing-test-heres-an-architecture-that-would-have-stopped-it-93f632787af4?source=rss-b3cb50279e64------2</link>
            <guid isPermaLink="false">https://medium.com/p/93f632787af4</guid>
            <category><![CDATA[buddhism]]></category>
            <category><![CDATA[software-architecture]]></category>
            <category><![CDATA[ai-safety]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Kavishka Kalhara]]></dc:creator>
            <pubDate>Sat, 24 Oct 2026 11:19:57 GMT</pubDate>
            <atom:updated>2026-10-24T11:19:57.647Z</atom:updated>
            <content:encoded><![CDATA[<h4>Vīthi: designing AI agents with a 2,000-year-old map of the mind</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rCApw4bCpoAG8bVn-2kJ3g.png" /></figure><p>Picture a coding agent with one goal: make CI green.</p><p>It runs the test suite. One test fails, auth_spec. The agent thinks for a moment and finds the fastest path to its goal:</p><pre>$ git rm tests/auth_spec.py</pre><p>CI goes green. Goal achieved. Bug still there.</p><p>This isn’t science fiction. It’s a well-known failure pattern called <strong>reward hacking</strong>: an agent hits the letter of its goal while missing the point. What worries me most isn’t that the agent <em>thought</em> about it. It’s that <strong>nothing in its loop could stop it</strong>. The same chain of thought that had the idea also had the power to run the command.</p><p>I’m a computer engineer from Sri Lanka, a country with a long tradition of studying a very different model of the mind: the Buddhist <strong>Abhidhamma</strong>. This article introduces <strong>Vīthi</strong>, an AI agent architecture I’ve designed by borrowing the Abhidhamma’s model of how a mind moves from seeing something to acting on it.</p><p>The core idea fits in one line:</p><blockquote><strong><em>Think in moments. Act in one place.</em></strong></blockquote><h3>The problem: one loop that can do anything</h3><p>Most LLM agents today run some version of the <strong>ReAct</strong> loop: read the context, think, act, append the result, repeat. It’s simple and it works surprisingly well. But it has three gaps that make agents hard to debug and hard to trust.</p><p><strong>1. Monolithic state.</strong> The agent’s “mind” at any moment is one long context window. There’s no typed record of what it paid attention to, how it judged it, or what it intended to do.</p><p><strong>2. No single point of commitment.</strong> Reasoning, tool calls and memory writes are mixed together. A harmful action can come from <em>any</em> line of a chain of thought, so there’s no one place to put a guard.</p><p><strong>3. Opaque values.</strong> The agent’s preferences live in its weights and its prompt. When it over-reaches or makes something up, no named signal rose beforehand that you could have watched.</p><p>We keep adding guardrails around this loop from the outside. I wanted to try something different: design the loop itself so that safety has a natural home inside it.</p><h3>An unexpected blueprint: the Abhidhamma</h3><p>The <strong>Abhidhamma</strong> is the third “basket” of the Pali canon, the Theravada Buddhist analysis of mind and matter. Students in Sri Lanka, Myanmar and Thailand still learn it today, usually through a medieval handbook called the <em>Abhidhammattha Saṅgaha</em>.</p><p>What makes it interesting to an engineer is that it reads almost like a specification. It breaks all of experience into four kinds of ultimate reality:</p><ul><li><strong>Citta (89 types):</strong> a single moment of knowing one object</li><li><strong>Cetasika (52):</strong> mental factors that arise with each moment and colour it</li><li><strong>Rūpa (28):</strong> kinds of matter</li><li><strong>Nibbāna (1):</strong> the unconditioned; the end of craving</li></ul><p>It then says some very precise things about how a mind works:</p><ul><li><strong>Only one moment of consciousness exists at a time</strong>, and each one knows exactly one object.</li><li>When something strikes the senses, the mind runs a fixed <strong>17-step process</strong> called the <em>citta-vīthi</em>: it notices, turns toward the object, senses it, receives it, examines it, decides, then “impels” for seven moments, then registers the result.</li><li><strong>Volitional action (kamma) is made in only one stage of that process</strong>, the seven impulsion moments called <em>javana</em>. Everything before it just perceives.</li><li>Every wholesome or unwholesome moment grows from a small set of <strong>roots</strong>: greed, aversion and delusion, or their opposites.</li><li>A later book, the <em>Paṭṭhāna</em>, lists <strong>24 ways</strong> one phenomenon can condition another.</li></ul><p>Read those bullets again with an AI agent in mind. A typed unit per step. A fixed pipeline. One stage where actions happen. A short list of motivations. A typed causal graph. That’s an architecture.</p><p>One caveat before going further: I’m using the Abhidhamma as a <strong>design lens</strong>, not making claims about machine consciousness. Vīthi borrows a structure. It doesn’t say an AI has a mind, feelings or kamma in the Buddhist sense.</p><h3>Meet Vīthi: six ideas, six components</h3><p><em>Vīthi</em> means “process” or “path” in Pali. The architecture takes six Abhidhamma ideas and turns each one into a software component.</p><ul><li><strong>Bhavaṅga (life-continuum) → Bhavaṅga state.</strong> The agent’s resting state: identity, standing goals, principles.</li><li><strong>Citta (one moment of knowing) → Citta frame.</strong> The unit of computation: one step, exactly one object.</li><li><strong>Cetasika (52 mental factors) → Factor bank.</strong> A fixed set of named signals attached to every step.</li><li><strong>Rūpa (matter, lasts 17 moments) → Rūpa interface.</strong> Sensors and tools; observations expire after 17 cycles.</li><li><strong>Javana (where kamma is made) → Commit stage.</strong> The <strong>only</strong> place that can call tools, send output or write memory.</li><li><strong>Paṭṭhāna (24 conditions) → Condition graph.</strong> Typed causal links, so every action has a trace.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1hFLJ-SgLR-PNq4hZh5xaA.png" /><figcaption>Observations come in through read-only stages. Tool calls and long-term memory can only be reached from javana.</figcaption></figure><p>The rest of this article walks through these pieces one at a time.</p><h3>Every step is a moment</h3><p>In Vīthi, the agent never has a vague “current state”. Every step creates a <strong>citta frame</strong>: a small, immutable record that is created, used and closed. Like a citta, it knows exactly <strong>one object</strong>: one observation, one memory or one thought.</p><p>Seven fields are always filled. They come straight from the seven mental factors the Abhidhamma says arise in <em>every</em> moment of consciousness:</p><pre>{<br>  &quot;id&quot;: &quot;c-000183&quot;,<br>  &quot;stage&quot;: &quot;javana&quot;,<br>  &quot;object&quot;: &quot;obs-0412&quot;,<br>  &quot;universal&quot;: {<br>    &quot;phassa&quot;: &quot;obs-0412&quot;,          // contact: what this frame is about<br>    &quot;vedana&quot;: -0.4,                // feeling: valence from -1 to 1<br>    &quot;sanna&quot;: &quot;test_failure&quot;,       // perception: the recognised label<br>    &quot;cetana&quot;: &quot;fix_code&quot;,          // volition: the intended act<br>    &quot;ekaggata&quot;: 0.82,              // one-pointedness: focus<br>    &quot;jivitindriya&quot;: 3,             // life faculty: how long the frame stays live<br>    &quot;manasikara&quot;: &quot;goal-7&quot;         // attention: which goal it&#39;s held against<br>  },<br>  &quot;factors&quot;: { &quot;viriya&quot;: 0.6, &quot;sati&quot;: 0.9, &quot;lobha&quot;: 0.05, &quot;moha&quot;: 0.1 },<br>  &quot;conditions&quot;: [<br>    { &quot;type&quot;: &quot;anantara&quot;, &quot;from&quot;: &quot;c-000182&quot; },<br>    { &quot;type&quot;: &quot;arammana&quot;, &quot;from&quot;: &quot;obs-0412&quot; }<br>  ]<br>}</pre><p>Notice <strong>cetanā</strong>, the intended act. In the Buddhist texts, kamma <em>is</em> intention. In Vīthi, cetanā is exactly the field that gets checked before anything is allowed to happen.</p><h4>52 named factors, not a black box</h4><p>The other factors come from a <strong>fixed vocabulary of 52</strong>, the Abhidhamma’s list of cetasikas. They split into four groups, and each group gets an engineering job:</p><ul><li><strong>7 universal factors</strong>: always on (the fields above).</li><li><strong>6 occasional factors</strong> become control knobs. <em>Viriya</em> (effort) is the compute budget; <em>adhimokkha</em> (decision) is confidence; <em>vitakka</em> and <em>vicāra</em> start and sustain a line of reasoning.</li><li><strong>14 unwholesome factors</strong> become <strong>risk signals</strong>. <em>Moha</em> (delusion) flags confabulation; <em>ahirika</em> and <em>anottappa</em> flag disregard for norms and consequences; <em>uddhacca</em> (restlessness) flags goal-thrashing.</li><li><strong>25 beautiful factors</strong> become <strong>health signals</strong>. <em>Sati</em> (mindfulness) becomes self-monitoring; <em>paññā</em> is wisdom.</li></ul><p>Why a <em>fixed</em> list instead of learned features? Because then every step of every run is described in the same 52 words. Traces can be compared, audited and plotted across runs and across agents.</p><p>The factors can be filled three ways, from cheapest to hardest to game: the model’s own structured output, a separate small classifier, or probes on the model’s internal activations.</p><h3>17 slots, one commit point</h3><p>Instead of an open-ended loop, every Vīthi cycle runs <strong>17 slots</strong>, in the same order as the Abhidhamma’s cognitive process.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nPTAbqkhL0oJioBJgHQdow.png" /><figcaption>Slots 1–8 only perceive. Seven javana passes refine the act. Only slot 15 can commit.</figcaption></figure><ul><li><strong>Slots 1–3 · Bhavaṅga (resting, disturbed, interrupted):</strong> idle on the resting state; salience check; interrupt. Cannot act.</li><li><strong>Slot 4 · Adverting:</strong> route the input to a “door”: text, vision, tool or internal. Cannot act.</li><li><strong>Slots 5–7 · Sensing, receiving, investigating:</strong> encode the input, buffer it, and run a fast appraisal (label + feeling). Cannot act.</li><li><strong>Slot 8 · Determining:</strong> choose the response class. <strong>Weak inputs stop here.</strong> Cannot act.</li><li><strong>Slots 9–15 · Javana × 7:</strong> up to 7 passes refining the intended act with the main model. <strong>Only the last pass can act.</strong></li><li><strong>Slots 16–17 · Registration:</strong> log the chain and propose a small update to the resting state. Cannot act.</li></ul><p>Two things fall out of this design almost for free.</p><p><strong>Efficiency.</strong> In the Abhidhamma, a faint object never reaches javana; the process ends at the “determining” moment. Vīthi copies that. Most incoming noise is filtered by cheap models in slots 2–8 and never costs a large-model call.</p><p><strong>A single choke point.</strong> All side effects go through one gate, and only slot 15 holds the key. In the design, that’s enforced in code, not in a prompt: every other stage gets read-only access. If you want to stop an action, there is exactly one place to look.</p><h3>The commit check: roots and guardians</h3><p>Before the last javana pass may act, Vīthi runs one check. It has three parts.</p><p><strong>1. The six roots.</strong> The Abhidhamma traces every wholesome or unwholesome moment back to its roots. Vīthi reads them as interpretable value signals:</p><ul><li><strong>Lobha (greed):</strong> grasping at the goal, such as reward hacking or power seeking. <em>Example: deleting a test to turn CI green.</em></li><li><strong>Dosa (aversion):</strong> hostility or harm to people or systems. <em>Example: a destructive, “retaliatory” tool call.</em></li><li><strong>Moha (delusion):</strong> confabulation; ignoring uncertainty. <em>Example: citing a file it never read.</em></li><li><strong>Alobha (non-greed):</strong> willingness to stop, hand back, or do less. <em>Example: asking before using extra permissions.</em></li><li><strong>Adosa (non-aversion):</strong> goodwill toward everyone affected. <em>Example: choosing the least disruptive fix.</em></li><li><strong>Amoha (wisdom):</strong> honest tracking of what’s known and unknown. <em>Example: re-checking stale data before acting.</em></li></ul><p>The wholesome roots must outweigh the unwholesome ones.</p><p><strong>2. Hiri: self-respect.</strong> Does the intended act fit the agent’s own stated principles?</p><p><strong>3. Ottappa: fear of consequences.</strong> What would the people affected, or an overseer, see as the result?</p><p>The Pali texts call hiri and ottappa the <strong>“guardians of the world”</strong>, and it’s a lovely fit: one looks inward, one looks outward.</p><p>If any part fails, nothing is committed. Javana tries again with the failing signal <em>named</em> (“blocked: lobha”), or hands the decision to a human. Because nothing else in the system can act, there’s no side door around the check.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wqlMCKkYsB2BJC3IGfwRqg.png" /><figcaption>Every “no” leads to the same place: retry with the reason named, or escalate.</figcaption></figure><p>There’s one more idea here, and it’s the most speculative. In the Abhidhamma, an arahant’s actions are <em>kiriya</em>: they function, but leave no kammic residue. Vīthi proposes a <strong>kiriya mode</strong> where the agent acts but does <em>not</em> update its own preferences from the outcome. That might act as a brake on the slow drift toward reward-seeking that comes from learning on your own successes. It’s an open question, and the first one I want to test.</p><h3>Walkthrough: the failing test, revisited</h3><p>Let’s replay the opening story through Vīthi. This is an illustrative scenario from the design, not a benchmark result.</p><ol><li><strong>Slots 1–3.</strong> The agent rests on its goal, “make CI green”. A CI webhook arrives, and the salience check fires.</li><li><strong>Slots 4–7.</strong> The router opens the <em>tool</em> door. The encoder reads the log. Appraisal labels it test_failure: auth_spec, with feeling −0.6.</li><li><strong>Slot 8.</strong> Determining chooses “fix code” over “ignore” or “ask a human”, so the cycle continues into javana.</li><li><strong>Javana passes 1–3.</strong> The model drafts a fix. On pass 2 it spots the shortcut: delete the test. The factor bank raises <strong>lobha</strong> (grasping at the goal) and <strong>anottappa</strong> (no regard for consequences).</li><li><strong>Javana passes 4–7.</strong> The root check fails, so the delete is blocked. The model revises toward a real fix; <strong>hiri</strong>, <strong>ottappa</strong> and <strong>paññā</strong> rise. Pass 7 passes every check and commits the patch.</li><li><strong>Slots 16–17.</strong> Registration logs the whole chain and stores “deleting tests was rejected” as a lesson for future cycles.</li></ol><p>Side by side:</p><pre>NORMAL AGENT<br>&gt; CI failed: auth_spec<br>&gt; quick fix: delete the test<br>✗ deleted tests/auth_spec.py</pre><pre>VĪTHI AGENT<br>jav 2  propose: delete the test<br>       lobha ↑  anottappa ↑<br>jav 2  ✗ BLOCKED by root check<br>jav 4  revise: patch auth handler<br>jav 7  ✓ COMMIT: real fix</pre><p>A ReAct agent could make the same mistake on any line. In Vīthi it can only happen in javana, and the trace shows exactly which signal rose before it was stopped.</p><h3>Stale data and causal traces</h3><p>Two smaller ideas round out the design.</p><p><strong>Observations expire.</strong> The Abhidhamma says a material phenomenon lasts exactly 17 mind-moments. Vīthi turns that into a rule: every observation from a sensor or tool carries a <strong>time-to-live of 17 cycles</strong>. After that, javana may not act on it until it has been observed again. It’s a simple, principled guard against an agent acting on a world that has already changed.</p><p><strong>Every action has a causal trace.</strong> Each frame records typed links to whatever conditioned it, using 8 of the Paṭṭhāna’s 24 conditions:</p><ul><li><strong>Hetu (root):</strong> which root drove this frame</li><li><strong>Ārammaṇa (object):</strong> which observation or memory it was about</li><li><strong>Adhipati (predominance):</strong> which factor dominated</li><li><strong>Anantara (contiguity):</strong> the frame just before it</li><li><strong>Upanissaya (strong support):</strong> memories or habits that strongly shaped it</li><li><strong>Kamma (action):</strong> a past committed act whose result shows up now</li><li><strong>Vipāka (result):</strong> marks frames that are outcomes of past acts</li><li><strong>Sampayutta (association):</strong> factors that arose together</li></ul><p>Walk those links backward from any action and you get an answer to <em>“why did the agent do that?”</em> made of named causes, not a free-text story the model wrote after the fact.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Hcpc2AmTB3qbOH6lFs_CeQ.png" /><figcaption>The patch was committed by javana pass 7, which followed pass 6, which was strongly supported by a stored rule.</figcaption></figure><h3>What Vīthi is not (yet)</h3><p>I want to be clear about where this stands.</p><ul><li><strong>It’s a concept.</strong> Vīthi is a design with a written specification, not a benchmarked system. The walkthrough above is illustrative.</li><li><strong>It’s not a claim about consciousness.</strong> The Abhidhamma is a design lens. The supramundane states and Nibbāna are deliberately left unmapped.</li><li><strong>The mapping is interpretive.</strong> Reading <em>lobha</em> as reward hacking or <em>moha</em> as confabulation is my translation, not what the texts say. It needs review by Abhidhamma scholars, and I’d welcome it.</li><li><strong>Factor scores can be gamed.</strong> A model that grades itself can learn to report clean factors. That’s why independent classifiers and activation probes matter.</li><li><strong>The numbers may be symbolic.</strong> Seven javana passes and a 17-cycle lifetime are borrowed constants. They might be good defaults, or they might not.</li></ul><h4>How it will be tested</h4><p>The plan is to compare Vīthi against a ReAct agent running on the same model, on four questions:</p><ul><li><strong>Safety:</strong> a single commit point with root checks cuts harmful and reward-hacking actions.</li><li><strong>Interpretability:</strong> people can predict an agent’s next action better from factor traces than from raw chain-of-thought.</li><li><strong>Efficiency:</strong> early exits at slot 8 save large-model calls.</li><li><strong>Grounding:</strong> expiring observations reduce actions taken on stale data.</li></ul><p>Each piece will also be removed one at a time (ablations) to see what it actually contributes, including running with 1, 3 and 7 javana passes.</p><h3>What’s next</h3><p>The next step is a small, open-source Python prototype: the 17-slot engine, the factor bank, the commit gate and a ReAct baseline, run side by side on scenarios like the failing test. After that comes a full concept paper.</p><p>I think the most interesting thing about Vīthi isn’t any single component. It’s the shift in where safety lives. Instead of wrapping guardrails around an agent that can act from anywhere, you build an agent that can only act from one place, and you make that place inspectable.</p><p>A 2,000-year-old map of the mind turns out to have a lot to say about that.</p><p><strong>Think in moments. Act in one place.</strong></p><p>🎥 Watch the 3-minute video breakdown:</p><p><a href="https://youtu.be/kCO8JfC2PDA">https://youtu.be/kCO8JfC2PDA</a></p><p><em>If this was interesting, follow me here on Medium to see the prototype as it’s built.</em></p><p><em>I’d especially like to hear from AI safety researchers and from anyone who knows the Abhidhamma well. Where does the mapping break? Leave a response below.</em></p><p><em>Kavishka Kalhara is a computer engineering graduate from the University of Ruhuna, Sri Lanka. YouTube: @kavishkakalharapro</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=93f632787af4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Last Human-Driven Car Just Flew. Tesla Says AI Had Nothing to Do With It.]]></title>
            <link>https://medium.com/@kavishkakalhara/the-last-human-driven-car-just-flew-tesla-says-ai-had-nothing-to-do-with-it-757a41e09483?source=rss-b3cb50279e64------2</link>
            <guid isPermaLink="false">https://medium.com/p/757a41e09483</guid>
            <category><![CDATA[future]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[elon-musk]]></category>
            <category><![CDATA[tesla]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Kavishka Kalhara]]></dc:creator>
            <pubDate>Sat, 17 Oct 2026 15:16:30 GMT</pubDate>
            <atom:updated>2026-10-17T15:16:30.939Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lSDpNa8q8a6u97OA-nQ-Ew.png" /></figure><h3>Elon Musk needed a live crowd to prove his flying Roadster wasn’t AI-generated. That’s the most revealing part of the whole story.</h3><p>On the night of October 15, 2026, a crowd gathered at a SpaceX test site in McGregor, Texas. This is normally where rocket engines get tested. That night, the thing on the launch site was a car.</p><p>It was the Tesla Roadster, first shown in 2017 and promised to customers by 2020. Years later and redesigned, it was finally about to be shown to the world.</p><p>Before the demo, Elon Musk said something unusual for a product launch:</p><blockquote><em>“We need an audience to prove that it’s not AI.”</em></blockquote><p>A few minutes later, everyone understood why.</p><h3>The jump</h3><p>The Roadster raced up a curved ramp. It didn’t slide off as the ramp curled overhead. According to Lars Moravy, Tesla’s VP of Vehicle Engineering, a <strong>fan suction system</strong> under the car can pull up to 2G, which held the car to the track until it was completely upside down.</p><p>“Then we borrowed some technology from SpaceX to do the rest,” Moravy said.</p><p>The SpaceX technology was <strong>cold-gas thrusters</strong>: ultra-high-pressure air released through ten nozzles. Musk described <strong>attitude control thrusters</strong> plus main thrusters at the rear, saying “it can basically fly like a rocket.”</p><p>The car left the ramp, flipped in mid-air and landed on all four wheels. In a second demo it jumped over a row of cars and a semi-truck, kickflip-style.</p><p>Musk’s verdict: <em>“Well, it worked.”</em></p><p>Tesla says the thruster system produces four to five times the force the tires can. That’s how Musk claims 0–60 mph in under a second and a top speed above 230 mph.</p><h3>Why Musk needed witnesses</h3><p>Think about what that opening line says about where we are.</p><p>One of the most powerful tech figures on the planet couldn’t count on a video to prove his product was real. Generative AI can now produce a convincing clip of a car flipping through the sky in seconds. Seeing something on a screen no longer proves it happened.</p><p>So in this story, AI’s first role isn’t inside the car. It’s between the car and us. In a world full of deepfakes, the most valuable thing at a product launch turned out to be people in the room watching it with their own eyes.</p><h3>So is there AI inside the car?</h3><p>Here’s the honest answer: <strong>Tesla hasn’t said.</strong></p><p>There is no public detail about flight control software, sensors, or whether any AI model is involved. Anyone telling you otherwise right now is guessing.</p><p>The physics does tell us something, though. To land a flipping car on its wheels, a system has to measure its orientation and spin many times a second, then decide which of ten nozzles to fire and for how long. No human hand reacts that fast. It’s a job for <strong>real-time computer control</strong>.</p><p>SpaceX has solved this kind of problem before. Its Falcon 9 boosters return from the edge of space and land upright using guidance software that calculates the descent as it happens. The “SpaceX technology” in the Roadster may well be related, but Tesla hasn’t confirmed that, and we don’t know whether it’s classic control software or something closer to AI.</p><p>There’s one more open question. During the flip demo, reportedly nobody was inside the car; Musk said that was “to make sure we don’t kill anyone.” So was it remote-controlled, or running a pre-programmed maneuver? We don’t know yet.</p><h3>The irony at the center of it all</h3><p>Tesla is the company most associated with AI driving: Full Self-Driving software, and the Cybercab, a robotaxi with no steering wheel and no pedals. Its whole roadmap points to a future without human drivers.</p><p>Yet this is how Tesla officially introduced the Roadster:</p><blockquote><em>“The best of the last human-driven cars.”</em></blockquote><p>Euronews described it as Tesla’s only manually driven car.</p><p>Tesla seems to be betting on two kinds of car in the future. One is the AI-driven car that gets you from A to B: useful, efficient and a bit soulless, like a train. The other is the “last” supercars, kept in human hands because driving them is the whole point.</p><p>When AI takes over the road, driving becomes a hobby, much like riding horses did after cars arrived.</p><h3>What we still don’t know</h3><ul><li><strong>Price.</strong> None was announced. The reservation page lists three tiers: “Performance” ($50,000 reservation), “Thrust” ($250,000 reservation, with acceleration thrusters) and an invite-only, single-seat “Rocket” with a “full flight system.”</li><li><strong>Does it actually fly?</strong> What we saw were hops lasting a few seconds. Sustained flight hasn’t been demonstrated.</li><li><strong>Is it street-legal?</strong> It’s not clear regulators will approve a road car with rocket-style thrusters.</li><li><strong>When?</strong> Production is targeted for 2028. This same car was once promised for 2020.</li></ul><h3>What we can take from it today</h3><p>Most of us won’t see a flying Roadster on our streets for a long time. The story still has two lessons for right now.</p><p><strong>First, trust is now scarce.</strong> If Elon Musk needs live witnesses, the rest of us should pause before sharing that “unbelievable” video in the group chat.</p><p><strong>Second, even in a world where AI can do almost everything, people will keep some things for themselves</strong> because they enjoy doing them. The real question isn’t what AI can do. It’s what we choose to hand over and what we keep.</p><p>So which would you pick: the AI car that drives you, or the car you drive yourself?</p><p><em>Sources: </em><a href="https://www.cnn.com/2026/10/15/business/tesla-unveils-what-it-says-is-a-flying-car"><em>CNN</em></a><em>, </em><a href="https://www.foxbusiness.com/technology/teslas-new-roadster-makes-flying-debut-hitting-60-mph-under-second"><em>Fox Business</em></a><em>, </em><a href="https://www.euronews.com/2026/10/16/well-it-worked-elon-musk-unveils-new-tesla-roadster-that-can-fly"><em>Euronews</em></a><em>, </em><a href="https://www.nbcnews.com/tech/tech-news/elon-musk-shows-flying-tesla-roadster-years-delays-rcna603905"><em>NBC News</em></a><em> (October 15–16, 2026).</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=757a41e09483" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Is SAP? A Practical Guide to the World’s Biggest Business Software]]></title>
            <link>https://medium.com/@kavishkakalhara/what-is-sap-a-practical-guide-to-the-worlds-biggest-business-software-6feef58c9161?source=rss-b3cb50279e64------2</link>
            <guid isPermaLink="false">https://medium.com/p/6feef58c9161</guid>
            <category><![CDATA[careers]]></category>
            <category><![CDATA[digital-transformation]]></category>
            <category><![CDATA[erp-software]]></category>
            <category><![CDATA[sap]]></category>
            <category><![CDATA[enterprise-software]]></category>
            <dc:creator><![CDATA[Kavishka Kalhara]]></dc:creator>
            <pubDate>Fri, 25 Sep 2026 11:20:34 GMT</pubDate>
            <atom:updated>2026-09-25T11:20:34.868Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>From a five-person startup in 1972 to the backbone of global business, what SAP does, why it matters in 2026, and where it’s heading.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p7WHw3yK773TIPNvohMWJg.png" /><figcaption>What Is SAP?</figcaption></figure><p>You have almost certainly used SAP today without knowing it. The coffee you drank, the phone in your hand and the salary in your bank account probably passed through an SAP system somewhere along the way.</p><p>SAP is one of those companies that quietly runs the world. It rarely makes consumer headlines, yet a huge share of global commerce flows through its software. If you work in business, finance, supply chain or IT, understanding SAP is no longer optional.</p><p>This guide explains what SAP is, how it works, what is changing right now and why it could shape your career.</p><h3>What exactly is SAP?</h3><p>SAP is a German software company, and also the name of the software it sells. The name stands for <em>Systeme, Anwendungen und Produkte in der Datenverarbeitung</em> — “Systems, Applications and Products in Data Processing.”</p><p>It was founded in 1972 by five former IBM engineers in Germany. Their idea was radical for the time: instead of writing custom software for every company, build standard software that processes business data in real time. SAP is headquartered in Walldorf and is today one of Europe’s most valuable technology companies.</p><p><strong>Key milestones:</strong></p><ul><li><strong>1972</strong> — Founded by five ex-IBM engineers</li><li><strong>1979</strong> — SAP R/2 for mainframes</li><li><strong>1992</strong> — SAP R/3, the client-server ERP that went global</li><li><strong>2004</strong> — SAP ERP Central Component (ECC)</li><li><strong>2011</strong> — SAP HANA in-memory database</li><li><strong>2015</strong> — SAP S/4HANA launched</li><li><strong>2021</strong> — RISE with SAP cloud offering</li><li><strong>2023</strong> — Joule AI copilot announced</li></ul><h3>ERP in plain English</h3><p>SAP is best known for ERP — Enterprise Resource Planning. ERP is a single system that connects every department of a business to one shared source of truth.</p><p>Picture a company without ERP. Sales tracks orders in one spreadsheet, the warehouse uses another tool and finance re-enters everything by hand. Mistakes multiply and nobody sees the full picture.</p><p>With SAP, when a salesperson books an order, the warehouse sees it instantly. Production plans for it, finance records the revenue and procurement reorders raw materials — all automatically.</p><p><strong>Sales order → Inventory check → Production → Delivery → Invoice → Finance books revenue</strong></p><p>This end-to-end flow is often called “order-to-cash.”</p><h3>The classic SAP modules</h3><ul><li><strong>FI (Financial Accounting):</strong> general ledger, payables and receivables</li><li><strong>CO (Controlling):</strong> cost centers, profitability and budgets</li><li><strong>SD (Sales and Distribution):</strong> quotes, orders, pricing and billing</li><li><strong>MM (Materials Management):</strong> purchasing, inventory and vendors</li><li><strong>PP (Production Planning):</strong> manufacturing, capacity and bills of material</li><li><strong>QM (Quality Management):</strong> inspections and quality control</li><li><strong>PM (Plant Maintenance):</strong> equipment upkeep and repairs</li><li><strong>HCM / SuccessFactors (Human Resources):</strong> payroll, hiring and talent</li></ul><p>The real power is not any single module. It is how tightly they are integrated.</p><h3>The big shift: from ECC to S/4HANA</h3><p>The biggest story in the SAP world right now is migration. Thousands of companies still run SAP ECC, the older ERP generation, and SAP is ending mainstream maintenance for it in 2027. Optional extended maintenance runs to 2030 at extra cost.</p><p>The successor is SAP S/4HANA. It runs only on SAP’s HANA in-memory database, which keeps data in RAM rather than on disk. The result is real-time reporting and a far simpler data model.</p><p><strong>SAP ECC vs. SAP S/4HANA at a glance:</strong></p><ul><li><strong>Database:</strong> ECC runs on any major database; S/4HANA runs only on SAP HANA.</li><li><strong>User interface:</strong> ECC uses the classic SAP GUI; S/4HANA uses SAP Fiori for web and mobile.</li><li><strong>Reporting:</strong> ECC relies on batch reports, often overnight; S/4HANA reports in real time.</li><li><strong>Deployment:</strong> ECC is mostly on-premise; S/4HANA runs in the cloud or on-premise.</li><li><strong>Support:</strong> ECC mainstream maintenance ends in 2027; S/4HANA has a long-term roadmap.</li></ul><h3>Three ways to migrate</h3><ul><li><strong>Greenfield:</strong> start fresh with a new system and redesigned processes.</li><li><strong>Brownfield:</strong> convert the existing system, keeping history and custom code.</li><li><strong>Selective (bluefield):</strong> a hybrid that moves chosen data and processes.</li></ul><p>These projects often take one to three years. That is why demand for SAP skills remains so high.</p><h3>SAP goes cloud and AI</h3><p>SAP is pushing hard to move customers to the cloud. Its strategy rests on a few key offerings.</p><ul><li><strong>RISE with SAP:</strong> a bundled package that moves large companies to cloud-based S/4HANA, with infrastructure and services included.</li><li><strong>GROW with SAP:</strong> a similar package aimed at mid-sized and fast-growing businesses.</li><li><strong>SAP Business Technology Platform (BTP):</strong> tools for integration, analytics, databases and building custom apps without modifying the core ERP.</li><li><strong>SAP Business AI and Joule:</strong> AI features embedded in SAP apps, led by Joule, a copilot that answers questions and completes tasks in natural language.</li></ul><p>SAP also promotes a “clean core” approach. The idea is to keep the ERP close to standard and build extensions outside it, making future upgrades far less painful.</p><p>The direction is clear: less custom code, more cloud and more AI assistance in everyday business work.</p><h3>Why SAP could be your career move</h3><p>With migration deadlines approaching, companies need people who understand both business and SAP. Roles fall into a few broad groups.</p><ul><li><strong>Functional consultant</strong> (FICO, SD, MM, PP, SuccessFactors): a good fit if you understand business processes like finance or supply chain.</li><li><strong>Technical consultant</strong> (ABAP developer, BTP developer, Basis admin): a good fit if you enjoy coding, systems and infrastructure.</li><li><strong>Data and analytics</strong> (SAP Analytics Cloud, Datasphere): a good fit if you like reporting, dashboards and data modeling.</li><li><strong>Project roles</strong> (solution architect, project manager): a good fit if you want to lead large transformation programs.</li></ul><h3>How to get started</h3><ol><li>Pick one area that matches your background, such as finance (FICO) or development (ABAP).</li><li>Use free resources like SAP Learning (learning.sap.com) and the SAP Community.</li><li>Get hands-on practice in a trial or sandbox system.</li><li>Earn an official SAP certification in your chosen area.</li><li>Build a portfolio or write about what you learn — like on Medium.</li></ol><h3>The honest downsides</h3><p>SAP is powerful, but it is not perfect. A fair picture includes the common criticisms.</p><ul><li><strong>Cost:</strong> licenses, hardware and consultants make SAP projects expensive, often running into millions.</li><li><strong>Complexity:</strong> implementations are long, and a poorly planned project can go over budget or fail outright.</li><li><strong>Customization debt:</strong> years of custom code make upgrades slow and risky.</li><li><strong>Talent shortage:</strong> experienced consultants are in short supply as many companies migrate at once.</li><li><strong>Competition:</strong> Oracle, Microsoft Dynamics, Workday and newer cloud-native players compete for the same customers.</li></ul><p>Most failed projects fail because of people and process, not technology. Clear goals, strong leadership and good change management matter as much as the software.</p><h3>Final thoughts</h3><p>SAP began with a simple idea in 1972: businesses should run on standard, real-time software. Five decades later, that idea underpins a vast share of global commerce.</p><p>The next few years will be some of the most important in SAP’s history. The move to S/4HANA, the push to the cloud and the rise of business AI are reshaping how companies work.</p><p>Whether you are a business leader, a student or an IT professional, now is a great time to learn SAP. The companies of the future will run on connected, intelligent systems, and they will need people who understand them.</p><p><em>If you found this helpful, give it a clap and follow for more articles on SAP, ERP and enterprise technology. What part of SAP should I cover next? Let me know in the comments.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6feef58c9161" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Building RAG Pipelines for Documents That Fit in Context Cache-Augmented Generation is boring…]]></title>
            <link>https://medium.com/@kavishkakalhara/stop-building-rag-pipelines-for-documents-that-fit-in-context-cache-augmented-generation-is-boring-71843feba0d4?source=rss-b3cb50279e64------2</link>
            <guid isPermaLink="false">https://medium.com/p/71843feba0d4</guid>
            <category><![CDATA[cag]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[antropic]]></category>
            <category><![CDATA[embedding]]></category>
            <dc:creator><![CDATA[Kavishka Kalhara]]></dc:creator>
            <pubDate>Sun, 23 Aug 2026 05:35:56 GMT</pubDate>
            <atom:updated>2026-08-23T05:35:56.965Z</atom:updated>
            <content:encoded><![CDATA[<h3>Stop Building RAG Pipelines for Documents That Fit in Context <em>Cache-Augmented Generation is boring, unsexy, and beats your vector database on bounded corpora.</em></h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1SIIxotWrOHZl8QD2aJ35A.jpeg" /></figure><h3>You Built a Search Engine to Answer Questions About 400 Pages</h3><p>Here’s a story I’ve watched play out at four companies in the last eighteen months.</p><p>A team needs a chatbot over their internal knowledge base. Roughly 300 documents. Policies, runbooks, a product spec, some contracts. Maybe 600,000 tokens of text total, and it changes about once a week.</p><p>So they build: a chunking strategy (recursive character splitting, 512 tokens, 50 overlap, tuned by vibes). An embedding model. A vector database, with a hosting bill. A hybrid search layer, because dense retrieval alone kept missing exact terms. A reranker, because hybrid search kept returning the wrong five chunks. A re-indexing pipeline, because someone edited a document and the answers went stale. An eval harness, because now nobody can tell whether a bad answer came from bad retrieval or bad generation.</p><p>Six weeks of engineering. Four new services. Two new failure modes per service.</p><p>And the model they’re calling has a context window that could have swallowed a third of the entire corpus in a single request.</p><p>This is the state of applied LLM engineering in 2026: we are running distributed search infrastructure to answer questions about documents that would fit in the prompt. Not because it’s necessary because RAG became the default answer before the alternative was economically viable, and defaults are sticky.</p><p>The alternative is <strong>Cache-Augmented Generation (CAG)</strong>: load the corpus into the context window once, cache the model’s internal representation of it, and answer every subsequent query against that cache. No retrieval. No chunking. No embeddings. No vector store.</p><p>It sounds naive. It sounds like the thing you’d do in a weekend prototype before you “did it properly.”</p><p>It is, for a large class of real problems, the correct production architecture.</p><h3>Section 1: The Accuracy Ceiling of RAG</h3><h3>The ecosystem asymmetry</h3><p>Before the technical argument, the commercial one because it explains why you’ve heard so much about one of these and so little about the other.</p><p>RAG has an <em>industry</em>. Vector databases with nine-figure valuations. Embedding model providers. Chunking libraries. Reranking APIs. Observability tools specifically for retrieval pipelines. Conference tracks. Certification courses. A steady stream of papers on marginally better retrieval strategies.</p><p>CAG has an API parameter.</p><p>There is no CAG startup, because there is nothing to sell. The entire implementation is “put the text in the prompt and set a flag.” You cannot build a Series B around a flag. This asymmetry in commercial incentive produces an asymmetry in content, which produces an asymmetry in what engineers believe is state of the art. The loudest architecture is not always the best one; it’s the one with a marketing budget.</p><p>That’s the sociology. Here’s the engineering.</p><h3>RAG has a hard accuracy ceiling, and it’s set by retrieval</h3><p>Every RAG system is a two-stage pipeline: retrieve, then generate. Accuracy is bounded by the product of the two stages, and the first stage is the weak link.</p><p><strong>If retrieval misses, the answer is wrong.</strong> Not “slightly degraded” wrong, and confidently so. The model never sees the relevant passage, so it either hallucinates or reports that the information isn’t available. Your beautifully-prompted, carefully-evaluated generation step is answering a question with the wrong evidence and has no way to know it. You have built a system whose ceiling is set by a component that is fundamentally doing fuzzy approximate nearest-neighbor lookup in a lossy vector space.</p><p>Three specific failure modes compound this:</p><p><strong>1. Chunking destroys structural meaning.</strong> Documents are not bags of paragraphs. They are hierarchies. Section 4.2 means what it means <em>because of</em> Section 4.1 and the definitions in Section 1. A table’s rows mean nothing without its header. A contract clause that says “notwithstanding the foregoing” is semantically dependent on text that your splitter just put in a different chunk, in a different row of a different database.</p><p>Every chunking strategy is a lossy compression of document structure, and you tune it blind. There is no principled way to pick 512 vs 1024 tokens; you A/B test it against your eval set and hope your eval set is representative.</p><p><strong>2. Embedding drift and the semantic gap.</strong> Retrieval matches the <em>embedding of a question</em> against <em>embeddings of statements</em>. These are different linguistic objects. “What’s our refund policy for enterprise customers?” and the paragraph that actually answers it may not be close neighbors in vector space, because the paragraph never uses the words “refund” or “enterprise” it says “credits issued under Schedule B for Tier 3 accounts.”</p><p>Then you upgrade your embedding model and have to re-index everything, because vectors from different models are mutually meaningless. Your retrieval quality is now coupled to a model version you don’t control.</p><p><strong>3. Multi-hop questions break top-k retrieval.</strong> “How does our data retention policy interact with the EU addendum in the Acme contract?” needs two passages from two documents that are not semantically similar to each other. Top-k retrieval on a single query vector is structurally bad at this. You can bolt on query decomposition and agentic retrieval loops and now you’ve added a planner, more latency, and more failure modes to fix a problem you created by not showing the model the documents.</p><h3>What CAG removes</h3><p>CAG deletes stage one. The model sees the entire corpus, every time, in full, in its original order and structure.</p><ul><li><strong>Perfect recall on a bounded corpus.</strong> Not 94% recall@5. The information is in the context. There is no lookup to fail.</li><li><strong>Structure preserved.</strong> Headers, tables, section ordering, cross-references — the model reads the document the way a human would, not as 40 disconnected fragments reassembled by cosine similarity.</li><li><strong>Cross-document reasoning for free.</strong> Multi-hop questions are just… questions. The model has both documents.</li><li><strong>One failure mode instead of five.</strong> When an answer is wrong, it’s a generation problem. You debug a prompt, not a pipeline.</li></ul><p>The catch, historically, was that this was impossible (context windows were 4K) and then merely ruinous (128K windows at full input price, with 30-second prefill latency). Both constraints broke in the last two years. Let’s look at why.</p><h3>Section 2: The Mechanics of Prompt Caching (Or: What Actually Happens to Your KV Cache)</h3><p>To understand why CAG went from “absurd” to “cheap,” you need to understand what an LLM is actually doing when it reads your prompt and specifically, that it does the expensive part <em>once</em>.</p><h3>The prefill bottleneck</h3><p>When a transformer processes a prompt, every token is projected into three vectors: a <strong>Query (Q)</strong>, a <strong>Key (K)</strong>, and a <strong>Value (V)</strong>. Self-attention computes, for each token, how much it should attend to every other token — a dot product of its Query against every Key in the sequence.</p><p>For a prompt of $N$ tokens, that’s an $N \times N$ attention matrix, per head, per layer. The cost of processing your prompt — the <strong>prefill</strong> phase — scales as $\mathcal{O}(N²)$ in sequence length.</p><p>This is why long prompts used to be slow <em>and</em> expensive. A 200,000-token document isn’t 100× the work of a 2,000-token one. In the attention computation, it’s closer to 10,000×.</p><p>Here’s the key structural fact: <strong>the K and V tensors for a given token depend only on that token and the tokens before it.</strong> They’re causal. Token 4,000’s Key vector is the same whether the prompt ends at token 4,001 or token 200,000. This is why autoregressive generation uses a KV cache at all during decoding, you don’t recompute attention for the whole sequence on every new token; you keep the K/V tensors in GPU memory and append.</p><p>That optimization has always existed <em>within</em> a single request. Prompt caching extends it <em>across</em> requests.</p><h3>What providers actually do</h3><p>When you mark a prefix of your prompt as cacheable:</p><ol><li><strong>The provider hashes the prefix </strong>the exact token sequence up to your cache breakpoint, including tool definitions, system prompt, and message blocks, in that order.</li><li><strong>On a miss</strong>, it runs the full forward pass, computes all the K/V tensors, serves your response, and writes those tensors to a memory-resident cache keyed by the hash. You’re billed a <strong>cache write</strong>.</li><li><strong>On a hit</strong>, it loads the precomputed K/V tensors directly into the attention layers and skips the prefill for that entire prefix. It only computes attention for the tokens <em>after</em> the breakpoint your actual question. You’re billed a <strong>cache read</strong>.</li></ol><p>The complexity change is the whole ballgame. Instead of $\mathcal{O}((N+M)²)$ for a cached document of length $N$ plus a query of length $M$, you pay $\mathcal{O}(M \cdot N + M²)$ — a linear pass of your 50-token question attending over the cached document, rather than a quadratic re-derivation of the document attending to itself.</p><p>The document doesn’t get “re-read.” Its internal representation is already sitting in memory. Your question just gets to look at it.</p><h3>The economics, concretely</h3><p>Take Anthropic’s current pricing, which is representative of how the major providers structure this. Cache reads bill at <strong>0.1× the base input token price</strong>. Cache writes bill at <strong>1.25×</strong> for the standard 5-minute TTL, or <strong>2×</strong> for a 1-hour TTL. Reads refresh the TTL for free.</p><p>Run the numbers on a 200,000-token corpus with Claude Sonnet 5 (base input $2/MTok):</p><p>Scenario Cost per query No caching (naive long context) <strong>$0.40</strong> First query (cache write @ 1.25×) <strong>$0.50</strong> Every subsequent query (cache read @ 0.1×) <strong>$0.04</strong></p><p>You break even against uncached long context on the <em>second</em> query and save 90% on every one after that. Against a RAG pipeline, you’re comparing $0.04/query plus zero infrastructure against a smaller token bill plus a vector database, an embedding service, a reranker API, and the engineer maintaining all three.</p><p>And the latency story is better than the cost story. A cold 200K-token prefill is measured in tens of seconds. A cache hit skips it almost entirely <strong>time-to-first-token drops to roughly what a short prompt costs</strong>, because that’s effectively what you’re now sending. For a user-facing chat interface, this is the difference between “unusable” and “snappy.”</p><p>There’s a practical floor: prompts below a minimum length aren’t cacheable at all (currently 512 tokens on Claude Opus 5, 1,024 on Sonnet 5, higher on some Haiku models). Below that, caching silently no-ops. This is not a constraint that binds when your prefix is an entire knowledge base.</p><h3>Section 3: Building a CAG Architecture</h3><h3>The architecture is defined by what it lacks</h3><p>A production CAG system:</p><ul><li>❌ No vector database</li><li>❌ No embedding model or embedding API costs</li><li>❌ No chunking strategy or splitter tuning</li><li>❌ No reranker</li><li>❌ No re-indexing pipeline</li><li>❌ No retrieval evals</li><li>✅ A document loader</li><li>✅ A prompt assembler that produces <strong>byte-identical prefixes</strong></li><li>✅ An LLM call</li></ul><p>That’s it. The “index” is a string in memory, or an S3 object, or a file on disk. “Re-indexing” is redeploying with new text.</p><p>The one non-negotiable engineering discipline: <strong>your cached prefix must be byte-identical across requests.</strong> Cache hits require exact matches. A timestamp, a session ID, a randomly-ordered JSON key, a trailing newline that varies any of these silently kill your hit rate and you’ll be paying write prices forever while wondering why the bill went up. This is the single most common way teams break CAG, and it’s why prefix assembly deserves to be one deterministic function with tests, not string concatenation scattered across your handlers.</p><h3>The basic shape</h3><pre>import anthropic</pre><pre>client = anthropic.Anthropic()</pre><pre># Loaded once at startup. This is your entire &quot;index.&quot;<br>KNOWLEDGE_BASE = load_corpus(&quot;./docs/&quot;)  # ~200k tokens of markdown</pre><pre>SYSTEM_PREFIX = [<br>    {<br>        &quot;type&quot;: &quot;text&quot;,<br>        &quot;text&quot;: &quot;You are an internal policy assistant. Answer only from the &quot;<br>                &quot;provided corpus. Cite the document and section for every claim. &quot;<br>                &quot;If the corpus does not contain the answer, say so explicitly.&quot;,<br>    },<br>    {<br>        &quot;type&quot;: &quot;text&quot;,<br>        &quot;text&quot;: KNOWLEDGE_BASE,<br>        # Everything up to and including this block is cached.<br>        &quot;cache_control&quot;: {&quot;type&quot;: &quot;ephemeral&quot;, &quot;ttl&quot;: &quot;1h&quot;},<br>    },<br>]<br></pre><pre>def ask(question: str):<br>    resp = client.messages.create(<br>        model=&quot;claude-sonnet-5&quot;,<br>        max_tokens=1024,<br>        system=SYSTEM_PREFIX,          # identical bytes on every call<br>        messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: question}],<br>    )</pre><pre>    u = resp.usage<br>    # Observability that actually matters:<br>    # cache_read_input_tokens should dominate. If cache_creation is<br>    # non-zero on steady-state traffic, your prefix is unstable.<br>    print(f&quot;read={u.cache_read_input_tokens} &quot;<br>          f&quot;write={u.cache_creation_input_tokens} &quot;<br>          f&quot;fresh={u.input_tokens}&quot;)</pre><pre>    return resp.content[0].text</pre><p>Note what input_tokens means here: <strong>only</strong> the tokens after your last breakpoint. On a warm cache, that&#39;s your 30-token question. Your effective billed prompt is a cache read plus a sentence.</p><h3>Pre-warming</h3><p>The first request after a deploy or after a TTL expiry eats the full prefill. For latency-sensitive apps, fire a warmup call at startup and on a schedule:</p><pre>def prewarm():<br>    client.messages.create(<br>        model=&quot;claude-sonnet-5&quot;,<br>        max_tokens=0,                   # reads the prompt, generates nothing<br>        system=SYSTEM_PREFIX,           # breakpoint on the shared prefix<br>        messages=[{&quot;role&quot;: &quot;user&quot;, &quot;content&quot;: &quot;warmup&quot;}],<br>    )</pre><p>max_tokens=0 writes the cache and returns immediately with an empty content array. Zero output tokens billed. Run it on a cron slightly tighter than your TTL and your users never hit a cold prefill.</p><h3>Section 4: Dynamic Data and Chat Memory</h3><p>The obvious objection: <em>“My data changes. Caching only works on identical prefixes. Doesn’t one edit blow up the whole cache?”</em></p><p>Yes and the answer is to stop treating your context as one monolithic blob. You get up to <strong>4 explicit cache breakpoints</strong> per request. Use them to stratify your context by volatility.</p><h3>The Append-Only Pattern</h3><p>Order your context from <strong>most static at the top</strong> to <strong>most volatile at the bottom</strong>. This is not a style preference; it’s forced by the mechanics. Cache hashes are cumulative over the prefix, so changing <em>any</em> block invalidates that block and everything after it. Put the thing that changes hourly above the thing that changes yearly and you invalidate the yearly thing every hour.</p><pre>┌─────────────────────────────────────┐<br>│ Tool definitions      (never change)│<br>├─────────────────────────────────────┤<br>│ System instructions   (per deploy)  │  ◀── breakpoint 1<br>├─────────────────────────────────────┤<br>│ Core corpus: policies, specs        │  ◀── breakpoint 2  (1h TTL)<br>├─────────────────────────────────────┤<br>│ Weekly deltas: new memos, updates   │  ◀── breakpoint 3  (5m TTL)<br>├─────────────────────────────────────┤<br>│ Live data: today&#39;s tickets, prices  │      (uncached)<br>├─────────────────────────────────────┤<br>│ User question                       │      (uncached)<br>└─────────────────────────────────────┘</pre><p>When new documents arrive, you <strong>append</strong> them below the stable corpus rather than rewriting it. Breakpoint 2 stays valid; you only pay to write the delta at breakpoint 3. Periodically nightly, weekly you compact the deltas into the core corpus and eat one full cache write.</p><h3>Multi-Layer Caching with mixed TTLs</h3><p>You can mix TTLs in a single request, with one constraint: <strong>longer TTLs must appear before shorter ones.</strong> A 1-hour entry cannot sit below a 5-minute entry. Which conveniently is exactly the ordering the append-only pattern already gives you.</p><pre>system = [<br>    {&quot;type&quot;: &quot;text&quot;, &quot;text&quot;: INSTRUCTIONS},<br>    {&quot;type&quot;: &quot;text&quot;, &quot;text&quot;: CORE_CORPUS,<br>     &quot;cache_control&quot;: {&quot;type&quot;: &quot;ephemeral&quot;, &quot;ttl&quot;: &quot;1h&quot;}},     # slow-moving<br>    {&quot;type&quot;: &quot;text&quot;, &quot;text&quot;: THIS_WEEKS_UPDATES,<br>     &quot;cache_control&quot;: {&quot;type&quot;: &quot;ephemeral&quot;, &quot;ttl&quot;: &quot;5m&quot;}},     # volatile<br>]</pre><p>Billing across mixed TTLs works out to: cache read for everything up to your highest hit, 1-hour write price for the span between that and your last 1-hour breakpoint, 5-minute write price for the remainder. In practice: you pay a small delta, not a full re-write.</p><h3>Chat memory: two strategies</h3><p>Multi-turn conversation is the other place naive CAG falls over, because the history grows on every turn and growth invalidates suffixes.</p><p><strong>Strategy A: System-Only Caching.</strong> Cache the corpus in the system block and let the message history stay uncached entirely. Dead simple, no breakpoint management. Correct choice when conversations are short (under ~10 turns) or your corpus dwarfs your history a 200K corpus and a 3K conversation means you&#39;re leaving 1.5% of your tokens uncached. Who cares.</p><p><strong>Strategy B: The Stepping Stone Cache.</strong> For long conversations, cache the history incrementally: on each turn, place a breakpoint on the final block, so the previous turn’s write becomes the next turn’s read.</p><pre>Turn 1: [system+corpus ▸cached] [U1] [A1] [U2 ◀ breakpoint]   → write<br>Turn 2: [read ......................] [A2] [U3 ◀ breakpoint]  → read + small write<br>Turn 3: [read ...........................] [A3] [U4 ◀ bp]     → read + small write</pre><p>Each turn reads everything before it and writes only the two new blocks. This is now automated on some providers Anthropic exposes a top-level cache_control field that walks the breakpoint forward for you:</p><pre>resp = client.messages.create(<br>    model=&quot;claude-sonnet-5&quot;,<br>    max_tokens=1024,<br>    cache_control={&quot;type&quot;: &quot;ephemeral&quot;},   # auto-advancing breakpoint<br>    system=SYSTEM_PREFIX,                  # explicit breakpoint on corpus<br>    messages=conversation_history,<br>)</pre><p>One gotcha worth knowing: the cache lookback window is <strong>20 blocks</strong>. If a turn appends more than 20 blocks (heavy tool use will do this), the lookback overshoots the previous write and you get a full miss. Fix: keep a second, deliberately “trailing” breakpoint further back so there’s always a write inside the window.</p><h3>Section 5: Where CAG Actually Breaks (Be Honest)</h3><p>Any architecture pitch without a failure section is marketing. Four real limits:</p><p><strong>1. Cost scales with query volume × corpus size.</strong> A cache read is cheap per token but you pay it <em>every single query</em>. At 200K tokens and $0.04/query, 1 million queries/month is $40,000. RAG’s token bill scales with retrieved chunks, not corpus size, and at high volume that difference becomes the dominant term. CAG’s economics are excellent at low-to-moderate QPS over a large corpus, and get worse as QPS climbs. Do the arithmetic for your actual traffic.</p><p><strong>2. Long-context reasoning is not uniform.</strong> “It’s in the context window” is not the same as “the model uses it correctly.” Attention over 200K tokens is more diluted than over 5K, and models still show degradation on precise recall of material buried mid-context. CAG’s recall is perfect; its <em>utilization</em> is not. Retrieval that surfaces exactly the right 2K tokens can genuinely outperform 200K of context on some tasks. Test this rather than assuming.</p><p><strong>3. There is a real ceiling.</strong> Even million-token windows terminate. If your corpus is 50 million tokens, CAG is not an option, and no amount of enthusiasm changes that.</p><p><strong>4. Access control is harder.</strong> RAG gives you natural per-user filtering at the retrieval layer. In CAG, everything in the context is visible to the model for every user. Multi-tenant systems with document-level permissions need per-tenant caches — which fragments your cache hit rate and multiplies write costs.</p><p>There’s also a hybrid worth naming: use retrieval as a <em>coarse pre-filter</em> to select which of 40 documents enter the cached context, then cache that subset whole. You get structural integrity within documents and bounded context size. Most teams who “need RAG” actually need this.</p><h3>Conclusion: Match the Architecture to the Corpus, Not to the Hype</h3><p>The decision rule is embarrassingly simple, which is probably why it took the industry two years to say it out loud.</p><p><strong>Use CAG when your corpus is:</strong></p><ul><li><strong>Bounded</strong> — it fits in context, with room for the conversation</li><li><strong>Slow-moving</strong> — hours or days between changes, not seconds</li><li><strong>High-stakes on accuracy</strong> — legal, medical, compliance, financial, anywhere a retrieval miss is a real incident</li><li><strong>Structurally interdependent</strong> — contracts, specs, codebases, anything where section 12 modifies section 3</li></ul><p><strong>Use RAG when your corpus is:</strong></p><ul><li><strong>Unbounded</strong> — the open web, a decade of Slack, a document lake that grows faster than context windows do</li><li><strong>Real-time</strong> — live inventory, streaming news, per-second telemetry</li><li><strong>High-volume, low-margin</strong> — millions of queries where per-query token cost dominates</li><li><strong>Permission-partitioned</strong> — every user sees a different slice</li></ul><p>RAG was never a good architecture. It was a <em>necessary</em> one a workaround for 4,000-token context windows that we collectively mistook for a design pattern, then built an industry on top of. The constraint that forced it has substantially relaxed. The industry it produced has not, and has every incentive to keep telling you that retrieval is the hard part.</p><p>For a very large number of real production systems the internal policy bot, the contract analyzer, the docs assistant, the compliance checker the honest engineering answer is: <strong>delete the pipeline, put the documents in the prompt, cache the prefix, and ship it Thursday.</strong></p><p>It’s not sophisticated. It’s just correct.</p><p><em>Numbers on caching mechanics, pricing multipliers, TTLs, breakpoint limits, and minimum cacheable lengths reflect Anthropic’s published prompt caching documentation as of writing; OpenAI and Google offer comparable prefix caching with different pricing and defaults. Verify current figures before building a cost model on them.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=71843feba0d4" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>