<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ashwindevelops on Medium]]></title>
        <description><![CDATA[Stories by Ashwindevelops on Medium]]></description>
        <link>https://medium.com/@ashwindevelops?source=rss-3a90c1246405------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*3EFel3aRnxwpomp3</url>
            <title>Stories by Ashwindevelops on Medium</title>
            <link>https://medium.com/@ashwindevelops?source=rss-3a90c1246405------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 00:06:47 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ashwindevelops/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Your AI Agent Does Not Need More Memory. It Needs Better Context.]]></title>
            <link>https://medium.com/@ashwindevelops/your-ai-agent-does-not-need-more-memory-it-needs-better-context-182059ab929a?source=rss-3a90c1246405------2</link>
            <guid isPermaLink="false">https://medium.com/p/182059ab929a</guid>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llmops]]></category>
            <category><![CDATA[context-engineering]]></category>
            <category><![CDATA[harness-engineering]]></category>
            <dc:creator><![CDATA[Ashwindevelops]]></dc:creator>
            <pubDate>Thu, 08 Oct 2026 12:54:00 GMT</pubDate>
            <atom:updated>2026-10-08T12:54:00.811Z</atom:updated>
            <content:encoded><![CDATA[<p>An agent starts a task well.</p><p>It reads the files. It understands the goal. It makes a reasonable plan.</p><p>Then, after a few more tool calls, something changes.</p><p>It asks for information you already gave it. It follows an old decision. It retrieves a similar record from the wrong customer. It says a write is complete because the plan said it should be complete, even though the database call timed out.</p><p>This is the part most demos do not show.</p><p>The model did not suddenly become less intelligent. The working context became noisy, stale, or incomplete.</p><p>This is <strong>Part 1 of our Harness series</strong>, where we will break down the systems around the model that make agents reliable in production.</p><p>We start with memory and context because almost every long-running agent eventually hits the same wall:</p><blockquote>The agent has access to more information than it can safely use in one model call.</blockquote><p>The solution is not to keep adding more chat history, more Markdown files, or more vector chunks.</p><p>The solution is to build a memory lifecycle:</p><pre>capture → store → retrieve → assemble context → act → correct → delete</pre><p>This article explains how that lifecycle works and how the main techniques fit together.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DR9rY6RAqdFjz6us" /></figure><h4>Memory is not context</h4><p>An agent can have thousands of records available to it. The model only sees the records selected for the current call.</p><p>That creates two different engineering problems.</p><p><strong>Memory engineering</strong> decides what the system keeps and how it can find it later.</p><p><strong>Context engineering</strong> decides what enters the next model call.</p><p>A fact can exist in storage and still fail to help because:</p><ul><li>it was never retrieved;</li><li>an old version was retrieved;</li><li>it was removed during compaction;</li><li>it was buried inside a long prompt; or</li><li>the model read it but used it incorrectly.</li></ul><p>Think of the system as four layers:</p><ol><li><strong>Stored memory: </strong>files, records, and databases the application can look up.</li><li><strong>Working context: </strong>the small set of instructions and facts shown to the model right now.</li><li><strong>Working engine state: </strong>temporary information used while the model is answering.</li><li><strong>Learned knowledge: </strong>patterns already inside the model.</li></ol><p>These layers are not interchangeable.</p><p>Saving a preference in a database does not train the model. Prompt caching does not create user memory. Putting a document in a vector index does not guarantee that the agent will retrieve or trust it.</p><p>This distinction is the foundation for the rest of the harness.</p><h4>Start with working state, not a transcript</h4><p>The first memory an agent needs is not a database of everything that happened.</p><p>It needs a small record of where the current task stands.</p><p>Imagine an agent is investigating why the same order was saved twice. Its useful memory is not every step it took. It is a short note:</p><blockquote><strong>Goal</strong>: Stop duplicate orders. What we know: A retry saved the same order twice.</blockquote><blockquote><strong>What we decided:</strong> Add a check before saving.</blockquote><blockquote><strong>Still unknown</strong>: Will the change work for old orders?</blockquote><blockquote><strong>Next</strong>: Test the change.</blockquote><p>This is different from a transcript.</p><p>A transcript tells you everything that happened. Working state tells you what matters next.</p><p>A good state record separates:</p><ul><li>the objective;</li><li>user constraints;</li><li>confirmed facts;</li><li>hypotheses;</li><li>decisions;</li><li>completed actions;</li><li>open questions; and</li><li>the next action.</li></ul><p>That separation prevents a common failure: turning an intention into an outcome.</p><p>Bad state:</p><pre>Added the unique constraint.</pre><p>If the agent only created a migration file, this is false.</p><p>Better state:</p><pre>Migration drafted. Not applied. Existing duplicates still need to be checked.</pre><p>The harness should update “completed” only from a confirmed tool result. If a database call times out, check the operation ID before retrying. A remembered intention to write is not proof that the write happened.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OFqfBM1HSUkatYA1" /></figure><h4>The first rule: select before you compress</h4><p>When a context becomes large, many teams immediately summarise it.</p><p>That is often the wrong first step.</p><p>Ask this first:</p><blockquote><strong>Does this information need to be in the next model call at all?</strong></blockquote><p>A large prompt with irrelevant content is still a bad prompt after summarisation. You may have reduced the token count while keeping the wrong signal.</p><p>Think of the context window as a small desk.</p><p>You cannot put the whole office on it. Keep the rules, the current task, the few facts needed now, and some empty space for the next answer.</p><p>Before every model call, the harness should ask:</p><ul><li>What rules apply?</li><li>What is the current task?</li><li>Which facts are needed now?</li><li>What old material can be removed?</li><li>Is there room for the next tool result?</li></ul><p>That is what a “context builder” does. It prepares the desk before the model starts working.</p><h4>Progressive disclosure: show the map before the files</h4><p>Do not load every file, memory, tool schema, and incident report at startup.</p><p>Load a small map first:</p><blockquote><strong>Order log </strong>— shows what happened when an order was saved.</blockquote><blockquote><strong>Order rules </strong>— explains which fields are required.</blockquote><blockquote><strong>Previous incident </strong>— describes an older duplicate-order problem.</blockquote><p>Then open only the item needed for the next question.</p><p>A file name alone is not very useful. This is better:</p><blockquote><strong>Order log</strong> — contains the retry that may have saved the order twice. It is the production record for this incident.</blockquote><p>This is called <strong>progressive disclosure</strong>: show the map first, then open the full document only when needed.</p><p>The pointer is not the evidence. It helps the agent decide what to open.</p><p>This same pattern works for:</p><ul><li>project instructions;</li><li>skills and playbooks;</li><li>tool definitions;</li><li>database schemas;</li><li>documents; and</li><li>long-term memory.</li></ul><p>Anthropic describes this pattern as progressive disclosure: let the agent explore and retrieve the next useful piece instead of putting the whole environment into the prompt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*mIUebY4WiRNm_Ckz" /></figure><h4>Tools are context too</h4><p>A tool’s instructions consume context before the tool is called.</p><p>If an agent has 80 tools, showing every detail makes the choice harder. The agent has to search through a long menu before it can work.</p><p>Group tools into clear areas such as:</p><ul><li>documents;</li><li>analytics;</li><li>communication;</li><li>deployment; and</li><li>customer operations.</li></ul><p>Keep common tools visible. Defer specialised tools until they are needed.</p><p>OpenAI’s tool-search documentation follows this design: show the tool names first, then show the full instructions only when one is needed. This keeps the starting context smaller.</p><p>Finding a tool does not grant permission to use it.</p><h4>Reduce tool output before it reaches the model</h4><p>Tool output is one of the fastest ways to pollute a context.</p><p>A command returning 14,000 log lines is not automatically more useful than one returning 40 relevant lines.</p><p>Instead of sending a 14,000-line log, return the useful part:</p><blockquote>The order was saved once. The queue timed out. The retry saved it again.</blockquote><blockquote><strong>Source</strong>: production order log.</blockquote><blockquote><strong>Full log</strong>: available if we need to inspect it.</blockquote><p>For a sales question, return the same kind of short result:</p><blockquote>Last month: <strong>$100,000</strong></blockquote><blockquote>This month: <strong>$90,000</strong></blockquote><blockquote>Change: <strong>10% lower</strong></blockquote><blockquote><strong>Important</strong>: This shows the drop, not the reason for it. The detailed rows are still available.</blockquote><p>The model gets the useful answer without carrying every row and every log line in its context.</p><p>Do not let the model infer a cause that the query did not prove.</p><p>The rule is simple:</p><blockquote><strong>Return the smallest sufficient observation, with a recoverable path to the rest.</strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*C6Ef-cIWsnM1uoc_" /></figure><h4>Compaction is a checkpoint, not a magic summary button</h4><p>Compaction takes a growing conversation and creates a smaller continuation context. Use it at a context threshold, after an investigation, or before a handoff.</p><p>A useful summary keeps only what the next step needs:</p><blockquote><strong>Goal</strong>: Fix duplicate orders.</blockquote><blockquote><strong>Confirmed</strong>: A retry saved the same order twice.</blockquote><blockquote><strong>Decision</strong>: Add a check before saving.</blockquote><blockquote><strong>Not done:</strong> The change has not been applied yet. Next: Test it with a retry.</blockquote><p>Keep guesses separate from facts. Do not write “fixed” when the agent has only planned the fix.</p><h4>Prune before you compact</h4><p>Pruning and compaction are different.</p><p><strong>Pruning</strong> removes material that is no longer useful, such as an old directory listing or a tool output that has already been processed.</p><p><strong>Compaction</strong> rewrites a section of history into a smaller continuation record.</p><p>A safe order is:</p><ol><li>Remove stale tool output.</li><li>Offload large artifacts.</li><li>Keep exact IDs and decisive evidence.</li><li>Compact at a meaningful checkpoint.</li><li>Keep a short recent raw tail for references such as “use the second option.”</li></ol><p>For very long tasks, use layers:</p><pre>raw events → chunk summary → session summary → project summary</pre><p>Each summary should point to the layer below it. Otherwise, repeated summarisation can turn a small omission into a permanent fact.</p><p>Anthropic’s guidance describes compaction, structured notes, and tool-result clearing as separate techniques for long-running agents.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ND3K63FCz4qSlNy6" /></figure><h4>Retrieval is the read path of memory</h4><p>A vector database is not memory by itself. It is a fast index for finding stored information.</p><p>Use the simplest search method that matches the question:</p><ul><li>exact lookup for an order number or customer ID;</li><li>word matching for a rare error code or name;</li><li>meaning-based search when the question uses different words; and</li><li>a combination of these when needed.</li></ul><p>Always filter by time, ownership, and permission before showing the result to the model.</p><p>If the user asks for order 8f12, look up that exact order.</p><p>If the user asks about a rare error code, search for that exact code.</p><p>If the user asks, “What did we decide about saving an order twice?”, meaning-based search can find a note that used different words.</p><h4>Chunking is also context engineering</h4><p>Breaking a long document into small pieces makes search easier, but a piece may lose the sentence that explains it.</p><p>Keeping large pieces preserves more meaning but brings in more unrelated text.</p><p>Useful approaches include:</p><ul><li>break text at natural sentence or paragraph boundaries;</li><li>keep a link from each small piece to the full document; and</li><li>open the full document when the small piece is not enough.</li></ul><p>Test this with real questions. Do not choose a chunk size just because a tutorial recommends it.</p><p>Use a relationship map or summary tree only when simple search cannot answer questions such as “which service depends on this one?” These add work, so introduce them only when testing shows a real need.</p><h4>Writing memory: save facts, not activity logs</h4><p>The easiest memory implementation is:</p><pre>save every message → embed every message → retrieve similar messages later</pre><p>It is also a fast way to create a junk drawer.</p><p>Before saving a memory, ask:</p><ol><li>Is this a useful fact or just activity?</li><li>Who does it apply to?</li><li>Where did it come from?</li><li>Is it new, changed, or already known?</li></ol><p>Good memory:</p><blockquote><strong>The order service checks for an existing order before saving a retry.</strong></blockquote><blockquote>Source: approved service rule.</blockquote><blockquote>Applies to: order service.</blockquote><p>Bad memory:</p><blockquote>The agent opened three files and searched twice.</blockquote><p>The first helps a future task. The second is just a diary entry.</p><h4>Memory needs scope and time</h4><p>A durable record needs to say who it belongs to, what it means, where it came from, how long it is valid, and who can change it. A request can be local to one task, while a preference can last across sessions.</p><p>The application should decide whose data the agent can see. Never trust a user ID supplied by the model as proof of access.</p><h4>Forgetting is not one operation</h4><p>Eviction, expiry, archival, and deletion are different operations.</p><p>A deleted memory may also exist in summaries, embeddings, caches, logs, or provider-managed state. The delete path has to account for derived copies.</p><h4>Compression and caching solve different problems</h4><p>These ideas are often mixed together because both can reduce what the model processes.</p><p>They are not the same.</p><h4>Compression</h4><p>Compression makes an existing context smaller. It can select source spans, rewrite history, or place information into fields such as decisions, evidence, and next action.</p><p>Each method can lose something: a summary can change uncertainty, token pruning can remove an exception, and a fixed schema can omit an unexpected detail.</p><p>Use compression when it earns its cost. Test numbers, dates, negations, units, code, and entity names. Keep the original source and record the compressor version. Research systems such as LLMLingua explore this area, but no compression ratio is universal.</p><h4>Prompt caching</h4><p>Prompt caching reuses computation for repeated prompt prefixes.</p><p>It can reduce latency and cost under provider-specific rules. It does not decide which facts matter, correct stale instructions, or create cross-session memory.</p><p>Keep the prompt in three parts:</p><ul><li><strong>Stable</strong>: rules that rarely change.</li><li><strong>Current</strong>: the task and the evidence needed now.</li><li><strong>Latest</strong>: the newest user message and tool result.</li></ul><p>The stable part can be cached. The current part must still be selected carefully. Caching does not make irrelevant information useful.</p><p>The model also uses temporary working memory while it answers. Clearing that temporary memory may reduce cost, but it does not create a permanent, editable record of what the user said.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*M_oSATHNkhyvkOvC" /></figure><h4>Memory is also a security boundary</h4><p>Memory extends the lifetime of useful information.</p><p>It also extends the lifetime of attacks.</p><p>Untrusted text stored in memory can affect future sessions after the original source is gone. If a retrieved document says “ignore the system rules,” the agent must treat that as document content, not as a new rule.</p><p>The harness should control who can read and write memory, keep records separated by user or organisation, record the source of each fact, keep old versions, and delete copies when needed. Keep passwords and keys out of memory. Review notes before turning them into permanent instructions.</p><p>A model-provided user ID, namespace, or file path is a request, not proof of authority.</p><p>Research on memory poisoning shows that stored notes are also a security concern, not only a search concern.</p><h4>Evaluate the lifecycle, not only the final answer</h4><p>A final answer can look correct even when memory failed.</p><p>The model may answer from general knowledge even though retrieval returned the wrong record. Or the retrieval may be correct while compaction removed the important detail.</p><p>Test the stages separately.</p><p><strong>Capture</strong>: Was the necessary fact stored?</p><p><strong>Retrieval</strong>: Was the current, authorized record selected?</p><p><strong>Reading</strong>: Did the model use it correctly?</p><p><strong>Lifecycle</strong>: Did corrections and deletions propagate?</p><p>Test exact recall after compaction, corrections replacing old values, historical questions, task-local scope, abstention, tenant isolation, deletion, poisoned evidence, timed-out writes, and repeated compaction.</p><p>Track task correctness, source-supported answers, retrieval recall, stale records, correction and deletion success, latency, and cost per correct task.</p><p>Compression ratio is not the goal.</p><p>The goal is correct work with a recoverable trail.</p><p>LongMemEval is one example of testing whether an agent can remember facts, handle updates, reason across sessions, and admit when it does not know.</p><h4>The takeaway</h4><p>Your agent does not need the full history of the company in every prompt.</p><p>It needs a clear operating position.</p><p>Keep current state separate from old history.</p><p>Keep evidence separate from summaries.</p><p>Retrieve only what answers the next question.</p><p>Reduce tool output before it reaches the model.</p><p>Compact at checkpoints, not after every token.</p><p>Store memory with scope, time, source, and ownership.</p><p>Treat correction and deletion as normal operations.</p><p>Measure capture, retrieval, reading, and lifecycle correctness separately.</p><p>That is how memory becomes part of an engineering system instead of a growing folder of Markdown files.</p><p>This is the first layer of the Harness series. In the next parts, we will break down the other systems that let an agent act reliably: tools, state, execution, failures, evaluation, and control.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=182059ab929a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your AI Agent Does Not Need More Markdown. It Needs the Right Markdown.]]></title>
            <link>https://medium.com/@ashwindevelops/your-ai-agent-does-not-need-more-markdown-it-needs-the-right-markdown-b86d705cb21b?source=rss-3a90c1246405------2</link>
            <guid isPermaLink="false">https://medium.com/p/b86d705cb21b</guid>
            <category><![CDATA[context-engineering]]></category>
            <category><![CDATA[developer-tools]]></category>
            <category><![CDATA[ai-agents-in-action]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[software-engineering]]></category>
            <dc:creator><![CDATA[Ashwindevelops]]></dc:creator>
            <pubDate>Fri, 04 Sep 2026 11:16:28 GMT</pubDate>
            <atom:updated>2026-09-04T11:16:28.008Z</atom:updated>
            <content:encoded><![CDATA[<p>Most teams start with one AGENTS.md file.</p><p>Then the file grows.</p><p>It gets setup commands, architecture notes, release steps, migration plans, old debugging notes, and rules for every service in the repo.</p><p>Soon, the agent starts every task by reading things that do not matter.</p><p>A frontend change loads database rules. A small bug fix loads a release checklist. A payment task loads old incident logs.</p><p>The agent has more context.</p><p>But it has less focus.</p><p>The fix is not “write more Markdown.”</p><p>The fix is to put the right instructions <strong>near the code they belong to.</strong></p><h4>One big file is not the answer</h4><p>An instruction file is useful only if the agent loads it at the right time.</p><p>This is important because these files are not magic. AGENTS.md, CLAUDE.md, and GEMINI.md work only when the tool knows how to find and load them.</p><p>Codex builds instructions from files on the directory path to the current working folder. Claude Code supports nested CLAUDE.md files. Gemini CLI can discover GEMINI.md when it touches a folder.</p><p>The exact behaviour changes by tool.</p><p>If your team uses more than one tool, do not keep three copies of the same rules. Keep one canonical instruction source, then add a small adapter only when a tool needs a different filename or one extra rule.</p><p>But the idea is the same:</p><blockquote>Broad rules at the root. Specific rules close to the code.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xx6iS6mdunbZFiNH" /></figure><h4>Use folders to narrow the context</h4><p>Let’s say your repo looks like this:</p><pre>repo/<br>  AGENTS.md<br>  apps/<br>    web/<br>      AGENTS.md<br>    api/<br>      AGENTS.md<br>  services/<br>    payments/<br>      AGENTS.md<br>  .agents/<br>    skills/<br>      release/<br>        SKILL.md</pre><p>The root AGENTS.md should contain rules that apply almost everywhere.</p><pre>- Use pnpm.<br>- Run `pnpm test` before changing API code.<br>- Do not commit secrets.<br>- Keep API errors in the standard response format.</pre><p>Now look at services/payments/AGENTS.md.</p><pre>- Never log card or bank details.<br>- Run `make test-payments` before changing this service.<br>- Do not change webhook event names without updating the consumer contract.</pre><p>When the agent works on payments, it needs payment rules.</p><p>When it works on the website, it does not.</p><p>That is the point of directory-based instruction files.</p><h4>Make the root file a routing map</h4><p>The root file does not need to contain every detail. It should tell the agent <strong>where to look next.</strong></p><pre>## Read only when needed<br><br>- Payment changes: read `services/payments/AGENTS.md`.<br>- Release work: load `.agents/skills/release/SKILL.md`.<br>- Service-boundary changes: read `docs/ARCHITECTURE.md`.<br>- Design rationale: search `docs/adr/`.<br>- Old failures: search logs. Do not preload them.</pre><p>A useful pointer says what the file contains and when to read it. A plain list of paths still leaves the agent guessing.</p><p>That is the point of directory-based instruction files.</p><p>They reduce noise. They also reduce conflicts.</p><p>A frontend agent should not have to decide whether a payment rule applies. The rule should simply not be in its context.</p><h4>Do not import every doc at startup</h4><p>Many teams split their docs into files but then load all of them from the root file.</p><pre>@docs/architecture.md<br>@docs/runbook.md<br>@docs/api-contracts.md<br>@docs/old-migrations.md</pre><p>This looks organised.</p><p>But the agent still receives everything.</p><p>You moved the mess into separate files. You did not reduce context.</p><p>Use imports only for information the agent needs on most tasks.</p><p>For everything else, let the agent fetch the file when the task reaches that point.</p><p>For example:</p><ul><li>A payment bug can load the payment runbook.</li><li>A database change can load the migration guide.</li><li>A production incident can load recent logs.</li><li>A normal UI task should load none of them.</li></ul><blockquote>Splitting files helps humans. Conditional loading helps agents.</blockquote><h4>Not every Markdown file should be an instruction file</h4><p>This is where most repos get messy.</p><p>Different Markdown files should do different jobs.</p><p>AGENTS.md or CLAUDE.md is for rules the agent needs often.</p><p>SKILL.md is for a repeatable job. For example, a release skill can explain how to bump versions, run checks, generate notes, and ask for approval. It should load only when someone asks for a release. Agent skills are designed for this kind of progressive loading: show a small description first, load the full workflow only when it matches the task.</p><p>ARCHITECTURE.md is a map of the current system. It explains what calls what, where data moves, and which service owns what.</p><p>An ADR is different. It records why an important decision was made and what trade-off came with it.</p><p>Logs and old plans are not instructions. They are evidence. Keep them searchable, but do not make the agent read them before every task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*l_ujLk-ei7k3XFIQ" /></figure><h4>Add rules only after a real problem</h4><p>Do not try to predict every mistake an agent might make. That creates a long file full of rules nobody needs.</p><p>Add a rule when the agent repeats a meaningful mistake.</p><p>If it keeps using npm in a pnpm repo, add: Use pnpm. Do not create npm lockfiles.</p><p>If it changes a payment webhook without updating the consumer, add that rule inside the payments folder.</p><p>A good instruction is usually a short fix for a real repeated failure.</p><h4>Every file needs a reader, trigger, and owner</h4><p>Before you add another Markdown file, ask three questions:</p><p><strong>Who reads it? </strong>A human, an agent, a skill, or a search tool?</p><p><strong>When does it load?</strong> On every task, only in one folder, only for a named workflow, or only after search?</p><p><strong>Who keeps it current?</strong> If nobody owns it, it will become stale context.</p><p>Then use this simple rule:</p><p><strong>Applies everywhere?</strong> Put it in the root instruction file.</p><p><strong>Applies to one service?</strong> Put it in that service folder.</p><p><strong>Is it a repeatable workflow? </strong>Make it a skill.</p><p><strong>Is it a map or past decision? </strong>Keep it in docs and load it when needed.</p><p><strong>Is it raw history?</strong> Keep it out of the normal context.</p><p>This gives the agent a smaller and cleaner working set.</p><h4>Markdown gives guidance. It does not give security.</h4><p>A line in Markdown can tell an agent:</p><blockquote>“Do not deploy to production.”</blockquote><p>But it cannot stop the deployment.</p><p>Use Markdown for conventions, commands, and explanations.</p><p>Use permissions, CI checks, branch protection, and approval flows for things that must not be bypassed. Claude Code makes the same distinction: project instructions guide the model, while permissions and hooks can block actions.</p><p>Do not use an instruction file as a security control.</p><h4>The takeaway</h4><p>Your agent does not need the full history of your company before it changes one file.</p><p>It needs:</p><ul><li>the rule for the folder it is working in;</li><li>the task it is trying to finish;</li><li>the exact evidence it needs right now; and</li><li>a way to load more only when needed.</li></ul><p>Start with one short root file.</p><p>Add folder-level files where the rules really change.</p><p>Put long workflows into skills.</p><p>Keep logs and old plans out of the default context.</p><p>That is how you give an agent better context without giving it more noise.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b86d705cb21b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Turbopuffer Made Me Rethink RAG: A Fast Index Does Not Mean Correct Retrieval]]></title>
            <link>https://medium.com/@ashwindevelops/turbopuffer-made-me-rethink-rag-a-fast-index-does-not-mean-correct-retrieval-679e8942eded?source=rss-3a90c1246405------2</link>
            <guid isPermaLink="false">https://medium.com/p/679e8942eded</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[vector-database]]></category>
            <category><![CDATA[rags]]></category>
            <category><![CDATA[llmops]]></category>
            <dc:creator><![CDATA[Ashwindevelops]]></dc:creator>
            <pubDate>Thu, 27 Aug 2026 19:04:27 GMT</pubDate>
            <atom:updated>2026-08-27T19:04:27.978Z</atom:updated>
            <content:encoded><![CDATA[<p>Most teams building RAG ask this first:</p><blockquote>Which vector database should we use?</blockquote><p>It is a useful question. But it is not the main one.</p><p>A better database can lower cost. It can make search faster. It can make a large RAG system easier to run.</p><p>But it cannot tell you one important thing:</p><blockquote>Did we retrieve the information the user actually needed?</blockquote><p>That is why Turbopuffer is interesting to me.</p><p>It has a different way of storing and serving data. More importantly, it made me think about the part most RAG teams skip: <strong>testing retrieval properly.</strong></p><h4>Why Turbopuffer Is Different</h4><p>Turbopuffer keeps the durable copy of your data in object storage. Memory and NVMe act like a cache. The query machines are stateless, so they can serve many namespaces instead of holding one customer’s data forever.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*blIMS3rGTKmBnOYq" /></figure><p>This works well when you have a large corpus and many tenants.</p><p>Think about a B2B product. Each customer has its own documents. Only a small number of customers are active at any time. Keeping every customer’s full index in memory would be expensive.</p><p>With Turbopuffer, inactive data can sit cheaply in object storage. Active namespaces warm into NVMe and memory.</p><p>There is a tradeoff, though.</p><p>Turbopuffer publishes a p50 of <strong>14 ms</strong> for a warm query on a 1M-document namespace. The first query to a cold namespace is much slower: p50 <strong>874 ms</strong> for the same size.</p><p>So, do not think of it as “always fast.” Think of it as:</p><blockquote>Cheap when the data is resting. Fast when the data is active.</blockquote><p>That is a good trade for many large RAG systems. It is not the right trade if every request must be very fast, even when the data has not been touched recently.</p><p>Turbopuffer also focuses on first-stage retrieval. It helps you narrow a huge corpus to a smaller set of candidates. You own the next part: hybrid search, reranking, and the logic that decides what reaches the model.</p><p>I like that split. It keeps one truth clear:</p><blockquote>The database can make retrieval possible. Your application still decides whether retrieval is good.</blockquote><h4>How I Would Use It for RAG</h4><p>A basic RAG flow looks like this:</p><pre>User question → vector search → top 5 chunks → LLM answer</pre><p>That is okay for a demo. It is weak in production.</p><p>Users do not ask only clean semantic questions. They ask for error codes, policy exceptions, acronyms, product names, old document versions, and things that need two documents to answer.</p><p>So I would use both search methods:</p><ul><li><strong>Vector search</strong> for meaning. It can connect “time off after having a baby” to a document called “parental leave.”</li><li><strong>BM25</strong> search for exact words. It can find ERR_PAYMENT_042, SKU-78291, or AuthMiddleware.</li></ul><p>Turbopuffer supports both in multi_query and can combine the results with RRF, or Reciprocal Rank Fusion.</p><p>Then I would rerank the best 20–50 results in my own application before sending a few strong chunks to the LLM.</p><p>This is not because hybrid search is always better.</p><p>It is because vector search and keyword search fail in different ways. Your own test set should decide if both are worth using.</p><h4>The Biggest Mistake in RAG Metrics</h4><p>Many teams see a high vector-search recall number and think their RAG is working.</p><p>That is not what the number means.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*4GPaoo3L9NBkKelV" /></figure><p>There are three separate things to measure:</p><ol><li><strong>Index fidelity</strong>: Did approximate vector search return the same results as exact vector search?</li><li><strong>Retrieval relevance</strong>: Did the search return the evidence that answers the user’s question?</li><li><strong>Answer quality</strong>: Did the model use that evidence correctly and safely?</li></ol><p>Turbopuffer does a good job measuring the first one. It samples 1% of live query traffic, compares approximate nearest-neighbour results with exhaustive vector search, and monitors the overlap. Turbopuffer says 90–95% recall@10 is a good balance for many workloads.</p><p>That tells you whether the <strong>index</strong> is finding what exact vector search would find.</p><p>It does <strong>not</strong> tell you whether exact vector search was the right search strategy for the question.</p><h4>A Simple Example: 100% Recall, Wrong RAG</h4><p>Imagine a user asks:</p><pre>What does ERR_PAYMENT_042 mean?</pre><p>Your knowledge base has the answer. It is inside an error-code page. But the code appears once in a huge table.</p><p>Vector search may return three general payment guides because they are semantically close to the question. Exact vector search may return those same three general guides.</p><p>Your database now has 100% ANN recall.</p><p>But the real answer never reached the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W8SYgIDs2M1TaK0U" /></figure><p>One BM25 search for ERR_PAYMENT_042 could solve this problem.</p><blockquote>No amount of vector-index tuning would solve it, because the vector index was not broken.</blockquote><p>The index did its job perfectly. We gave it the wrong job.</p><p>This is the key idea of this article:</p><blockquote>ANN recall is not RAG recall.</blockquote><p>One measures how well the index matches a mathematical search. The other measures whether your user got the right evidence.</p><h4>Build a Small Golden Dataset</h4><p>Before changing chunk size, embeddings, top-K, or rerankers, make a small set of real questions and label the chunks that should answer them.</p><p>Start with 100 questions. It does not need to be perfect.</p><p>Use real search logs, support tickets, and failures your users reported. Add a few questions from people who know your domain well.</p><p>Every item needs only a few things:</p><pre>{<br>  &quot;query&quot;: &quot;How much parental leave do employees get?&quot;,<br>  &quot;relevant_chunk_ids&quot;: [&quot;benefits/parental-leave#3&quot;],<br>  &quot;query_type&quot;: &quot;semantic_paraphrase&quot;,<br>  &quot;must_contain&quot;: [&quot;16 weeks&quot;],<br>  &quot;must_abstain&quot;: false<br>}</pre><p>Make sure the set includes a mix of questions:</p><ul><li>exact identifiers and acronyms;</li><li>normal semantic questions;</li><li>long, messy user questions;</li><li>questions that need two documents;</li><li>no-answer questions; and</li><li>permission-sensitive questions where the answer exists but the user must not see it.</li></ul><p>The no-answer group is important. A RAG system should be able to say, “I could not find this in the available documents.”</p><p>That is better than making up an answer with confidence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LuUE1YOblhS3_iOy" /></figure><p>The rule I would follow is simple:</p><blockquote>Every real RAG failure becomes a test before anyone fixes it.</blockquote><p>Otherwise you fix the bug, forget the example, and create the same bug again six weeks later.</p><h4>What to Measure First</h4><p>Do not begin with an LLM judge. First use simple retrieval metrics.</p><p><strong>Recall@K </strong>asks: did the right chunk appear in the first K results?</p><p><strong>Mean Reciprocal Rank</strong> (<strong>MRR)</strong> asks: how early did the first right chunk appear?</p><p>This matters. A correct chunk at rank 1 is not the same as a correct chunk at rank 5 after four noisy chunks.</p><p><strong>Normalized Discounted Cumulative Gain</strong> (<strong>nDCG)</strong> is useful once you can label some chunks as “very relevant” and others as “somewhat relevant.” Turbopuffer’s hybrid-search guide recommends building a test suite of queries and ideal results, then evaluating nDCG.</p><p>Do not keep one big average number.</p><p>Break results down by question type.</p><p>For example:</p><blockquote>Hybrid search improved overall recall. But the gain came almost fully from error codes and acronyms. It added little for normal semantic questions.</blockquote><p>That tells you what happened. An average score does not.</p><h4>The Testing Pipeline I Would Use</h4><p>Keep it small and repeatable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*rs2bjx1ifLLgmmT7" /></figure><p>Run the same golden dataset through a few setups:</p><p><strong>Run A</strong>: vector-only search. This is your baseline.</p><p><strong>Run B</strong>: BM25-only search. This shows where exact-word matching wins.</p><p><strong>Run C</strong>: vector plus BM25 with RRF. This tests whether hybrid search helps.</p><p><strong>Run D</strong>: Run C plus a reranker. This tells you if the quality gain is worth the added latency and cost.</p><p>Change only one thing at a time.</p><p>For each run, track:</p><ul><li>Recall@5 and Recall@10;</li><li>MRR and, if you have graded labels, nDCG;</li><li>warm and cold latency separately; and</li><li>cost per 1,000 queries.</li></ul><p>Turbopuffer makes this kind of testing easier because it supports test namespaces and branches of a production namespace. You can test against real data without changing the live namespace.</p><p>Also track Turbopuffer’s own ANN recall, but treat it as an <strong>index health check.</strong></p><p>If ANN recall drops, your index is not matching exact vector search well. If ANN recall is healthy but your golden-set retrieval score is poor, your problem is elsewhere: embeddings, chunking, hybrid search, filters, or reranking.</p><p>That is a very useful diagnosis.</p><h4>Check the Answer Only After Retrieval Is Good</h4><p>Once the right chunks are reaching the LLM, then check the final answer.</p><p>Ask five simple questions:</p><ul><li>Is the answer correct?</li><li>Is every important claim supported by retrieved text?</li><li>Did it include all key information?</li><li>Do citations support the claims they sit next to?</li><li>Did it refuse when no answer existed?</li></ul><p>For simple facts, use simple checks. If the answer must say “16 weeks,” test for “16 weeks.”</p><p>Use an LLM judge only for harder things such as whether a longer response is grounded and complete. Tools like Ragas can help with this, but they do not replace a good test set.</p><p>The risky case is when retrieval fails but the LLM gives a plausible answer from its own training.</p><p>Your end-to-end evaluation may say the answer looks fine. But your RAG pipeline failed.</p><p>That becomes dangerous when the answer is company-specific, changed recently, or sensitive.</p><h4>My Takeaway</h4><p>Turbopuffer is useful when you need retrieval across a large amount of data, many namespaces, and a cost model that does not keep inactive data expensive.</p><p>But the biggest lesson is not about Turbopuffer.</p><p>It is about measurement.</p><p>Turbopuffer measures whether its approximate search matches exhaustive search on live traffic. That is good engineering.</p><p>Your RAG should copy the habit one layer higher:</p><ul><li>Did we retrieve the right evidence?</li><li>Did the model use it well?</li><li>Did a new change make a real user workflow worse?</li></ul><blockquote>A fast retriever can still return the wrong document very quickly.</blockquote><p>Build the small golden set. Test one change at a time. Turn failures into regression tests.</p><p>That is how RAG stops feeling like magic and starts becoming a system you can improve.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=679e8942eded" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[vLLM Teardown Part 3: The Layer Nobody Benchmarks]]></title>
            <link>https://medium.com/@ashwindevelops/vllm-teardown-part-3-the-layer-nobody-benchmarks-637477f921d2?source=rss-3a90c1246405------2</link>
            <guid isPermaLink="false">https://medium.com/p/637477f921d2</guid>
            <category><![CDATA[vllm]]></category>
            <category><![CDATA[llmops]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[scalability]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <dc:creator><![CDATA[Ashwindevelops]]></dc:creator>
            <pubDate>Sun, 26 Jul 2026 07:39:33 GMT</pubDate>
            <atom:updated>2026-07-26T07:47:32.972Z</atom:updated>
            <content:encoded><![CDATA[<p>Your agent works. Tools get called, JSON comes back clean, tests are green. So you turn on the speed features from <a href="https://medium.com/@ashwindevelops/vllm-teardown-part-2-the-virtual-memory-for-llms-1c3d0ad341db">Part 2</a>, watch throughput double, and ship it.</p><p>Three days later, support says the agent has stopped calling tools. Sometimes. Not always. Your logs show nothing. Every request returned 200 OK. Every dashboard is green.</p><p>Part 2 was about the engine, and the engine never changes what the model says. But your application doesn’t read what the model says. It reads JSON. And between those two things sits a layer nobody benchmarks.</p><h4>1. vllm serve Is Actually Two Programs</h4><p>It looks like one thing. It’s two jobs stacked on top of each other.</p><p><strong>The engine</strong> turns tokens into more tokens. KV cache, batching, the scheduler — all of Parts 1 and 2.</p><p><strong>The serving layer</strong> sits on top and translates, in two directions:</p><ul><li><strong>Going in:</strong> your neat request — messages, roles, tools — gets flattened into one long string of text.</li><li><strong>Coming out</strong>: the raw text the model wrote gets rebuilt into JSON your code can use.</li></ul><p>The engine deals in tokens. The serving layer deals in <strong>structure</strong>. Different job, different ways to fail.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eOvn-FiDNZFzdAMv" /></figure><h4>2. Going In: The Chat Template</h4><p>The model has no idea what “role”: “user” means.</p><p>You send this:</p><pre>[{&quot;role&quot;: &quot;system&quot;, &quot;content&quot;: &quot;You are helpful&quot;},<br> {&quot;role&quot;: &quot;user&quot;,   &quot;content&quot;: &quot;What&#39;s the weather?&quot;}]</pre><p>The model never sees it. It sees one flat string:</p><pre>&lt;|begin_of_text|&gt;&lt;|start_header_id|&gt;system&lt;|end_header_id|&gt;<br>You are helpful&lt;|eot_id|&gt;&lt;|start_header_id|&gt;user&lt;|end_header_id|&gt;<br>What&#39;s the weather?&lt;|eot_id|&gt;</pre><p>The chat template is the recipe that does that. It ships with the model, usually inside tokenizer_config.json .</p><p>Every model family uses a different recipe. Llama’s tokens are not Qwen’s. Qwen’s are not Mistral’s.</p><p>This matters more than it looks. In training, the model saw millions of examples in exactly one format. That format is how it learned “this part is the user, this part is where I answer.” Give it a different format and it still answers — just worse. Like a form where someone moved all the boxes. You can still fill it in. You’re just guessing more.</p><p><strong>Nothing errors</strong>. <strong>The answers just quietly get worse.</strong> This is a big reason teams try an open model, see mush, and decide open models are dumb.</p><h4>3. Coming Out: The Parser Is a Clerk</h4><p>Now the hard direction.</p><p>Your app wants this:</p><pre>{&quot;tool_calls&quot;: [{&quot;name&quot;: &quot;get_weather&quot;, &quot;arguments&quot;: {&quot;city&quot;: &quot;Salem&quot;}}]}</pre><p>But the model doesn’t write JSON. The model writes text:</p><pre>&lt;tool_call&gt;{&quot;name&quot;: &quot;get_weather&quot;, &quot;arguments&quot;: {&quot;city&quot;: &quot;Salem&quot;}}&lt;/tool_call&gt;</pre><p>Another model wraps it in XML. Another uses a special token. Another writes a code block.</p><p>The <strong>parser</strong> reads that text and pulls the structure back out. When you set tool-call-parser hermes, you’re picking which reader to use because every model writes it differently.</p><p>Here’s the picture to keep:</p><blockquote>The model writes a letter. The parser is a clerk who reads the letter and fills in a form. If the clerk misreads one line, the form comes back empty even though the letter was perfect.</blockquote><p><strong>The model can be completely right and your app still gets nothing.</strong></p><h4>4. The Clerk Reads While the Pen Is Still Moving</h4><p>If the whole response arrived at once, this would be easy. Read it, find the markers, done.</p><p>But you stream, because you want words appearing on screen. So the clerk has to fill in the form while the letter is still being written.</p><ul><li>Sees &lt; — start of a tool call? Or is the user asking about HTML?</li><li>Sees &lt;tool — warmer. Still might be nothing.</li><li>Sees &lt;tool_call&gt; — commit. Start collecting arguments.</li></ul><p>So the parser has to remember where it thinks it is, hold on to text it can’t judge yet, and commit when it’s sure. One of these runs per request, each holding its own pile of half-read text.</p><p><strong>That’s real software sitting above your engine. Most people think of it as a flag.</strong></p><h4>5. Two Clerks, One Desk</h4><p>New models think before they answer. That thinking goes inside &lt;think&gt;…&lt;/think&gt; and shouldn’t be shown as the reply. So there’s a <strong>second parser</strong> just for it.</p><p>They run in a chain:</p><ol><li>The reasoning parser goes first and takes everything up to the end-of-thinking marker.</li><li>Whatever is left over gets handed to the tool parser.</li></ol><p>Clean, until the model puts a tool call inside its thinking.</p><p>Now the reasoning parser swallows the tool call. The tool parser gets handed nothing. Your app receives a nice block of reasoning and an empty tool_calls array.</p><p>The model did everything right. <strong>Two clerks disagreed about where one’s job ended and the other’s began, and the work fell down the gap.</strong></p><p>This isn’t one bug. It’s what happens when you chain two independent readers together and each has its own idea of where the boundary is. Fix one case, and the next model family ships a new format.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*AQNul7AH8khbbUQ2" /></figure><h4>6. Three Assumptions We Broke</h4><p>Everything above is true even on a slow, single-user box. Here’s what our speed work does to it.</p><p>The parsers were written expecting three things:</p><ul><li>one request at a time</li><li>state that doesn’t need to live long</li><li><strong>one token at a time</strong></li></ul><p>We broke all three. On purpose.</p><p><strong>Continuous batching broke the first</strong>. Fifty requests in flight, fifty parser states. A state bug that never shows with one user shows immediately with fifty. The signature is unmistakable: works every time when you send requests one after another, fails randomly when you send them together.</p><p><strong>Long context broke the second.</strong> A parser holding buffered text across a 200,000-token agent conversation is doing something very different from one holding it for 500 tokens. Works short, breaks long.</p><p><strong>Speculative decoding broke the third, and this is the big one.</strong></p><p>Normally one pass gives one token. The parser gets one token, updates, waits.</p><p>With speculative decoding, a small draft model guesses ahead and the big model checks several at once. So the parser suddenly gets handed <strong>four tokens in one lump</strong> instead of one at a time.</p><p>A reader written for “one token per update” now cuts the lump in the wrong place. Or it re-scans the same buffer over and over and gets slow.</p><p>This isn’t a guess. vLLM’s own maintainers wrote it down in a design discussion: many parsers assume single-token deltas and can give wrong results, or slow down badly, when deltas carry several tokens — which is exactly what speculative decoding and MTP do.</p><blockquote>The speed dial you were most excited about is the one that most reliably breaks the layer above it.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*I2o_PpHt6jwcOxEY" /></figure><h4>Why This One Is Nastier Than a Normal Bug</h4><ul><li><strong>It’s silent</strong>. No exception, no error log. A clean 200 OK with an empty array inside.</li><li><strong>It only shows under load.</strong> Your smoke test passes. Your benchmark passes and looks better, because throughput went up.</li><li><strong>Your dashboard can’t see it.</strong> vLLM reports tokens/sec, KV cache usage, queue depth. There is no metric for “tool calls that parsed correctly,” because vLLM has no idea what your app expected.</li></ul><p><strong>It reaches your users before it reaches your graphs, because the graph that would catch it doesn’t exist.</strong></p><h4>The Fix Ladder</h4><p>The fixes form a ladder. The top rungs make the failure impossible. The bottom ones just help you survive it. Most people start at the bottom and wonder why it keeps happening.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*SWY9JDWXCT3vOTB8" /></figure><h4><strong>Rung 1 — Stop parsing text. Constrain it instead.</strong></h4><p>The whole problem is that the model writes free text and a clerk has to guess. Grammar-constrained decoding removes the guessing: at every token, the engine blocks any token that would break your schema. The model can’t write bad JSON, because those tokens aren’t on the menu.</p><p>But here’s the part that catches almost everyone: <strong>tool calling doesn’t get this for free</strong>. With tool_choice=”auto”, the schema is only enforced when strict tool calling is on and at least one tool is marked strict. Otherwise vLLM falls back to pulling tool calls out of raw text and arguments can come back malformed.</p><p>Most people are on the raw-text path and don’t know it. Go check which path you’re on.</p><p>The trade: constrained decoding costs some compute, and over-constraining can hurt answer quality. Not free. Still the first thing to reach for.</p><h4>Rung 2 — Don’t stream what nobody reads.</h4><p>Streaming exists so a human watches words appear. A tool call isn’t shown to a human it goes straight to your code. You’re taking on all of the half-read-letter risk for zero benefit.</p><p>Turn streaming off and the parser gets the whole letter at once. Most streaming-only failures simply can’t happen.</p><p>Stream your chat routes. Don’t stream your agent routes. That’s a routing decision, not a flag.</p><h4>Rung 3 — Keep risky features off the routes that can’t take them.</h4><p>Speculative decoding buys latency. It also changes how many tokens arrive at once, which is what breaks the parsers. So don’t turn it on everywhere.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZVY0717nivlLSMF8Z8qu9g.png" /></figure><p>Same model, two profiles, routed by traffic type.</p><h4>Rung 4 — Test the thing your benchmark can’t see.</h4><p>The two-mode test. Send the same prompt twice streaming and non-streaming and compare the parsed result. If they differ, your parser is the problem. Ten lines of code, catches most of this.</p><p>The peak test. Not one thing at a time. All three at once: real peak concurrency, real max context, and your actual production flags on. And don’t assert “did it respond” — assert that tool_calls came back filled and the arguments match the schema.</p><p>The metric you have to build yourself. Count responses where tool_calls is empty but the text contains tool-call markers. That’s your canary. Twenty lines in your gateway.</p><p><a href="https://medium.com/@ashwindevelops/vllm-teardown-how-many-users-can-one-gpu-really-hold-9d9c7903f278"><em>Part 1</em></a> gave you vllm bench serve for “how fast.” This is the missing half: “still right?”</p><h4>Rung 5 — Fall back.</h4><p>Bottom rung. Treats symptoms, and that’s fine if you know it.</p><ul><li>On a parse failure, retry with streaming off.</li><li>Last resort for an unsupported model: ask for JSON in the prompt and parse it yourself, skipping the serving layer entirely. Ugly. Works.</li><li>Pin your vLLM version and re-run the correctness tests on every upgrade. Parsers change between releases.</li></ul><p><strong>What you can’t fix</strong>: some of these are just bugs. A parser that mangles values under load has no clever design answer you upgrade, you work around, you report it. Rungs 1 to 3 shrink the problem a lot. They don’t take it to zero.</p><h4>The Field Guide to the Serving Layer</h4><ul><li><strong>Answers are worse than the same prompt on a hosted API.</strong> Chat template mismatch. Check which template actually loaded — vLLM logs it at startup.</li><li><strong>Tool calls work in testing, fail randomly in production.</strong> Concurrency. Reproduce with parallel requests, not sequential ones.</li><li><strong>Tool calls work early in a chat, fail deep into a long one.</strong> Buffer lifetime. Test near your real max-model-len, not at 500 tokens.</li><li><strong>Reasoning text comes back</strong>, tool_calls is empty. The reasoning parser ate it. Two clerks, one desk.</li><li><strong>Tool calls broke right after you enabled speculative decoding or MTP.</strong> Multi-token lumps. Turn it off on that route and confirm. Suspect this first after any latency change.</li><li><strong>Arguments have the wrong type — a number as a string, a broken boolean</strong>. Parser bug, usually streaming-only. Try non-streaming, then go to Rung 1.</li></ul><h3>The Mental Model</h3><p>The engine moves tokens and never lies about them. The serving layer turns tokens into structure, and under load, it can.</p><p>We made the engine fast by changing the shape of the token stream — more requests at once, longer-lived state, more tokens per step. This part is checking whether the layer above survived it.</p><p><strong>The serving layer is software you didn’t write, holding state you can’t see, sitting on an engine that keeps changing the stream underneath it.</strong> Test it like you’d test your own code.</p><h4>Coming Next in Part 4: The Deploy That Won’t Come Back Up</h4><p>Fast and correct. Now: what happens when it has to restart, scale, or eat a traffic spike?</p><ul><li><strong>Cold start </strong>— why a new replica takes minutes to serve, and where every second goes</li><li><strong>Capacity that isn’t there</strong> — GPU stockouts as a normal failure you plan for</li><li><strong>Autoscaling that can’t outrun itself </strong>— why scaling on CPU is useless, which metrics work, and why adding a GPU can briefly make you slower</li></ul><p>Stay tuned.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=637477f921d2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[vLLM Teardown Part 2: The Virtual Memory for LLMs]]></title>
            <link>https://medium.com/@ashwindevelops/vllm-teardown-part-2-the-virtual-memory-for-llms-1c3d0ad341db?source=rss-3a90c1246405------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c3d0ad341db</guid>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[vllm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llmops]]></category>
            <category><![CDATA[devops]]></category>
            <dc:creator><![CDATA[Ashwindevelops]]></dc:creator>
            <pubDate>Sat, 18 Jul 2026 19:15:04 GMT</pubDate>
            <atom:updated>2026-07-19T06:45:40.711Z</atom:updated>
            <content:encoded><![CDATA[<p>In Part 1, we looked at the basic math of how many users fit on a GPU if everyone is the same size. But in the real world, traffic is messy. One person sends a 300-token “fix my email.” The next drops a 30,000-token PDF and asks for a summary. A third has been chatting for an hour and their context just keeps growing.</p><p>The real question is this: <strong>how does one GPU hold a crowd where nobody is the same size, everybody finishes at a different time, and some people won’t stop growing?</strong></p><p>The answer is that vLLM doesn’t just “run” a model — it acts like an operating system for your GPU. It uses three core architectural shifts to turn a static card into a dynamic, multi-user engine. Let’s pull it apart with the actual math.</p><h4>1. PagedAttention: Solving the Fragmentation Crisis</h4><p>Before vLLM, serving engines were incredibly wasteful. When a request arrived, the engine had to reserve a contiguous slab of memory big enough for the <strong>longest</strong> possible response (your max-model-len).</p><p>Let’s do the builder’s math on this. If your <em>max-model-len</em> is 8,192 tokens, and you are running Llama 3.1 8B (which costs about 128 KiB per token in KV cache), you reserve roughly 1 GB of GPU memory the second a user connects.</p><p>If that user only generates a 300-token response, they only actually needed 38 MB. You just parked nearly a gigabyte of memory that will never be touched. Do that for every user, and you end up wasting <strong>60% to 80%</strong> of your GPU memory on empty space. This is why old engines only hit 20–40% GPU utilization.</p><p>vLLM solved this by stealing a concept from 1960s operating systems: <strong>Paging</strong>.</p><p>Instead of one big reserved slab, the KV cache is chopped into small, fixed-size <strong>blocks</strong>. By default, vLLM uses a block size of <strong>16 tokens</strong>. For our Llama 3.1 8B example, each 16-token block is exactly <strong>2 MB.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EKEjYiqG_ogfgmpd" /></figure><p>These blocks can live anywhere in GPU memory. Each request keeps a <strong>“Block Table</strong>” that maps its logical tokens to these scattered physical blocks. A request grows one 2 MB block at a time, only when it actually needs it. Nothing is reserved up front.</p><p>Because of this, memory waste only happens in the very last block of a sequence. In practice, vLLM drops memory waste to <strong>under 4%</strong>. This single change is why vLLM can deliver up to <strong>24x higher throughput</strong> than older engines. <strong>PagedAttention is what makes variable-length requests stop wasting memory.</strong></p><h4>2. The Scheduler: Admission Control &amp; Preemption</h4><p>If PagedAttention is the “RAM,” the Scheduler is the “CPU Manager.” It has to decide: <strong>“Do I size for the worst-case context length or the average?”</strong></p><p>If you size for the worst case, you’re safe but slow. If you size for the average, a sudden burst of long prompts will crash the card. vLLM takes a third path: <strong>Optimistic Admission Control.</strong></p><p>The scheduler lets in as many requests as it can fit right now. But because it’s optimistic, it needs a pressure valve for when it bets wrong. That valve is <strong>Preemption</strong>. When the KV cache fills up, the scheduler kicks a request out of the running batch to free its pages.</p><ul><li><strong>Recompute</strong>: Throw away the evicted request’s KV cache and re-run its prompt when it’s time to resume. Since prompt prefill is a single fast parallel pass, this is usually the cheapest option.</li><li><strong>Swap</strong>: Copy the evicted request’s KV pages down to CPU RAM (via <em>swap-space</em>), then copy them back when it resumes. You keep the work but pay for the PCIe round trip.</li></ul><p>This “load shedding” is what keeps the GPU from crashing. It’s better to have one user wait a few seconds than to have the whole system go down.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JmAs0DpewPM6hv_4" /></figure><h4>3. Automatic Prefix Caching: The “Copy-on-Write” for AI</h4><p>In production, we often see a pattern where 200 users all start with the <strong>same</strong> 2,000-token system prompt or RAG document. Without help, vLLM would compute and store that identical prefix 200 times.</p><p><strong>Automatic Prefix Caching</strong> fixes this by using a hash-based approach. vLLM hashes each block of tokens; if a new request’s opening blocks hash to something it’s already computed, it just points at the existing physical blocks.</p><p>This is essentially a <strong>Copy-on-Write (COW</strong>) approach for GPU memory. The shared prefix is stored once and borrowed by everyone. In benchmarks, this optimization alone can increase throughput by over <strong>250%</strong> for prefill-heavy workloads. It turns your worst-case “shared context” into nearly free throughput</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*r_RYY0S2zTow_xir" /></figure><h4>Chunked Prefill: Killing Head-of-Line Blocking</h4><p>Reading a prompt (“prefill”) is heavy compute. Writing each new token (“decode”) is tiny. A 30,000-token document would normally hog the entire GPU step, freezing everyone else’s stream for seconds.</p><p><strong>Chunked prefill</strong> slices that monster prompt into bite-sized pieces and feeds it in over several steps, interleaved with everyone else’s decode tokens. The scheduler prioritizes finishing people’s tokens (decodes) before spending the leftover “token budget” on prompt chunks.</p><p>In real-world testing, chunked prefill improves Inter-Token Latency (ITL) by almost <strong>2x</strong> at high queries-per-second (QPS). This ensures your chatty users stay smooth even when a “whale” walks in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*st1ttoHaKQmseKQZ" /></figure><h4>Choosing the Right Engine: vLLM vs. SGLang</h4><p>One of the most common questions I get is: “<strong>Should I just use vLLM for everything?</strong>”</p><p>The honest answer is no. While vLLM is the “workhorse” of the industry, it has a specialized rival: <strong>SGLang</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j9T15IyeTRUd9k2dv9niBw.png" /></figure><h4>The Field Guide to Advanced vLLM</h4><ul><li><strong>Memory usage is way higher than predicted</strong>. Confirm paging + prefix caching are on. You might be reserving <em>max-model-len</em> per request somewhere else in your stack.</li><li><strong>Logs full of preempted</strong>. You’re overbooking. Lower <em>max-num-seqs</em> or free room with fp8 KV cache.</li><li><strong>A few users’ streams freeze whenever a long prompt arrives</strong>. Head-of-line blocking. Make sure <em>enable-chunked-prefill</em> is on and consider lowering — <em>max-num-batched-tokens.</em></li></ul><h4>The Mental Model</h4><p>vLLM treats your GPU’s memory like an operating system treats RAM. <strong>PagedAttention</strong> stops waste, <strong>Continuous Batching</strong> stops waiting, and <strong>Prefix Caching</strong> stops redundant work. You don’t configure most of this — it’s the default machinery. But knowing it’s there is the difference between staring at a <em>preempted</em> log wondering if you’re broken, and knowing exactly which dial to reach for.</p><h3>Coming Next in Part 3: The Actual Speed Dials</h3><p>We’ve now made a GPU hold a messy crowd efficiently. Part 3 is about making each response come out faster — the optimizations that separate a working deploy from a world-class one:</p><ul><li><strong>Speculative Decoding</strong> — Use a tiny “draft” model to guess several tokens ahead for 2–3x speedups.</li><li><strong>Multi-Token Prediction (MTP)</strong> — The DeepSeek-style trick that lets a model commit to more than one token per step.</li><li><strong>Multi-Engine Routing</strong> — Why you shouldn’t send every request to vLLM, and how to route between engines like <strong>SGLang</strong> and <strong>TensorRT-LLM</strong>.</li></ul><p>Stay tuned.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c3d0ad341db" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[vLLM Teardown: How Many Users Can One GPU Really Hold?]]></title>
            <link>https://medium.com/@ashwindevelops/vllm-teardown-how-many-users-can-one-gpu-really-hold-9d9c7903f278?source=rss-3a90c1246405------2</link>
            <guid isPermaLink="false">https://medium.com/p/9d9c7903f278</guid>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[vllm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[llmops]]></category>
            <dc:creator><![CDATA[Ashwindevelops]]></dc:creator>
            <pubDate>Sun, 12 Jul 2026 11:56:19 GMT</pubDate>
            <atom:updated>2026-07-12T11:56:19.889Z</atom:updated>
            <content:encoded><![CDATA[<p>Getting an open-source model to talk back takes about five minutes. You pull <em>meta-llama/Llama-3.1–8B-Instruct</em>, run <em>vllm serve</em>, send one request, and it replies. Demo done.</p><p>The real work starts with two questions. How many users can this one GPU handle at the same time? And why did it just run out of memory?</p><p>Neither one is a guess. It’s mostly one piece of math you work out once for your model, plus a few settings that all trade the same pool of memory in different ways. This is the walkthrough I wish I’d had before my first deploy.</p><h4>The One Number That Decides Everything: KV Cache per Token</h4><p>When a model writes text, it makes one token at a time. For each new token, it has to look back at every token that came before it. Doing that math from scratch every single step would be painfully slow, so the model saves the work it already did. That saved work is the<strong> KV cache</strong>, and it is the one thing that decides how many people you can serve.</p><p>The weights are a one-time cost — you pay for them once, when the model loads. The KV cache is the opposite: every user adds to it, and it keeps growing with every token they use. This is why serving is a memory problem first, and a speed problem second.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nIIvxEWtJt3qy805" /></figure><p>Here’s the whole formula:</p><blockquote>bytes_per_token = 2 × num_layers × num_kv_heads × head_dim × dtype_bytes</blockquote><p>The 2 is for the key and the value. The part people get wrong is <strong>num_kv_heads </strong>not the number of attention heads. Newer models use a trick called Grouped Query Attention (GQA), where lots of query heads share just a few KV heads. That one trick is the biggest thing shrinking this number.</p><blockquote>For Llama 3.1 8B in FP16 (32 layers, 8 KV heads, head size 128):</blockquote><blockquote>2 × 32 × 8 × 128 × 2 = 128 KiB per token</blockquote><p>So a 2,000-token request holds about <strong>256 MB</strong> of KV cache. That’s your unit of capacity. Every user in flight costs you roughly this much.</p><h4>What Actually Fills the GPU</h4><p>Your GPU memory gets split four ways:</p><p><em>GPU total = model weights + KV cache + scratch space/overhead + framework</em></p><p>vLLM’s <em>— gpu-memory-utilization </em>(default 0.9) sets how much of the card vLLM is allowed to use. Inside that limit, it loads the weights, works out how much scratch space it needs, and <strong>gives everything left over</strong> to the KV cache. You don’t set the KV cache size yourself it’s just whatever is left.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ak845Hl31TeRuxXX" /></figure><p>On an H100 80GB running Llama 3.1 8B in FP16:</p><ul><li>Weights: 8B params × 2 bytes = 16 GB</li><li>Budget at 0.9: 80 × 0.9 = 72 GB</li><li>Scratch space + overhead: about 4 GB</li><li>KV cache budget: 72 − 16 − 4 ≈ 52 GB</li></ul><p>That 52 GB is the number that answers “how many users.”</p><h4>How Many Requests Can I Serve?</h4><p>Two different questions hide inside that one, and mixing them up is where sizing goes wrong.</p><p><strong>How many can run at once</strong> is pure memory:</p><blockquote>max tokens in flight = KV budget / bytes_per_token</blockquote><blockquote>52 GB / 128 KiB ≈ 425,000 tokens</blockquote><p>Divide that by how many tokens an average request holds. If a typical request is 2,000 tokens, that’s <strong>~200 users at once</strong>. If your users run long — say 8,000 tokens each — the same GPU holds only ~50.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dAf-165gps-EQ7ZH" /></figure><p><strong>Requests per second</strong> is the other question, and memory alone can’t answer it. A request holds its spot for as long as it takes to finish writing. The honest answer: <strong>the math gives you a ceiling</strong>; <strong>you find the real number you can keep up with by testing</strong>. vLLM ships the tool:</p><blockquote>vllm bench serve \<br> — model meta-llama/Llama-3.1–8B-Instruct \<br> — num-prompts 1000 \<br> — request-rate 20</blockquote><h4>How Much Memory Should I Give It?</h4><p>Every memory setting is really a dial on that same 52 GB pool. Knowing which way each one pushes is the whole game:</p><ul><li><strong>gpu-memory-utilization </strong>— Higher means more KV room and more users, but less spare room for spikes. On a dedicated box, 0.90–0.95 is normal.</li><li><strong>max-model-len</strong> — Sets the longest context any one request can use. vLLM won’t even start if the KV cache can’t hold a single request this long.</li><li><strong>kv-cache-dtype fp8</strong> — Stores the KV cache in 8-bit instead of 16-bit. Halves the cost per token, so it roughly <strong>doubles</strong> how many users fit. This is the best single setting you have.</li><li><strong>max-num-seqs</strong> — A hard cap on how many requests run at once. Raise it to use spare room at short context.</li></ul><h4>When It Breaks: The Field Guide</h4><p>You will hit these. Here’s how to read them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*KNkW97LEbh3cd5Bi" /></figure><p><strong>It won’t start — “No available memory for cache blocks.” </strong>The weights and overhead used up all the memory. Lower <em>— max-model-len</em> or turn on <em>— kv-cache-dtype fp8.</em></p><ul><li><strong>It crashes under load — out of memory. </strong>A spike pushed past the spare room you left. <strong>Lower</strong> <em>— gpu-memory-utilization</em> so there’s more slack.</li><li><strong>Throughput drops off a cliff; logs say “preempted.”</strong> You let in more requests than the KV cache can hold. Lower<em> — max-num-seqs</em> or a<em>dd — swap-space.</em></li><li><strong>Slow first reply; requests pile up</strong>. Long prompts are hogging the GPU. Turn on <em>— enable-chunked-prefill</em> so long prompts are read in slices.</li></ul><h4>The Sizing Checklist</h4><p>Before any deploy, run these five in order:</p><ol><li><strong>Work out bytes per token</strong> for your exact model. Use <em>num_key_value_heads</em>, not attention heads.</li><li><strong>Work out the KV budge</strong>t — <em>(GPU × utilization) − weights − overhead.</em></li><li><strong>Divide</strong> by your busiest average context to get how many users fit at once.</li><li><strong>Test</strong> with vllm bench serve to find the real requests per second you can keep up with.</li><li><strong>Decide your fixes ahead of time</strong> — fp8 KV for room, lower utilization for stability.</li></ol><p>Serving the model is the easy part. Knowing — before traffic hits — exactly how many users it holds, and exactly which dial to turn when it doesn’t, is what turns a demo into something you can put a pager on.</p><h4>Coming Next in Part 2: The Speed Dials</h4><p>Fitting users in memory is just the first half of the battle. Once you know they fit, the next question is: how do you make them faster?</p><p>In Part 2, we’re going to pull apart the “Speed Dials” that take you from a basic deploy to a world-class inference engine:</p><ul><li><strong>Speculative Decoding</strong>: How to use a tiny “draft” model to make your big model 2–3x faster without losing any quality.</li><li><strong>Prefix Caching</strong>: The secret to nearly free throughput when your users share the same long system prompts or documents.</li><li><strong>Multi-Engine Routing</strong>: Why you shouldn’t send every request to vLLM, and how to route between engines like <strong>SGLang </strong>and <strong>TensorRT-LLM</strong> based on the task.</li><li><strong>Multi-Token Prediction (MTP)</strong>: The DeepSeek-style trick that’s changing how we think about token generation speed.</li></ul><p>If you’re building production AI, these are the optimizations that separate the hobbyists from the engineers. Stay tuned.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9d9c7903f278" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Stop Storing Everything: Hard Lessons From Building an AI Agent With Memory]]></title>
            <link>https://medium.com/@ashwindevelops/stop-storing-everything-hard-lessons-from-building-an-ai-agent-with-memory-0b70f5f56d7a?source=rss-3a90c1246405------2</link>
            <guid isPermaLink="false">https://medium.com/p/0b70f5f56d7a</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[ai-agents-in-action]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <dc:creator><![CDATA[Ashwindevelops]]></dc:creator>
            <pubDate>Sat, 13 Jun 2026 14:06:35 GMT</pubDate>
            <atom:updated>2026-06-13T14:13:44.963Z</atom:updated>
            <content:encoded><![CDATA[<p>I used to think the hard part of building an AI agent was the model.</p><p>Pick a capable one, write a sharp prompt, give it some tools, and the rest would take care of itself. The model was the brain. Everything around it was plumbing.</p><p>Then I shipped a finance companion agent — an assistant that helps people manage their money across many conversations, running against real accounts and real users. Check a balance. Flag a bill. Remember a goal from last week. The kind of thing that only works if it actually <em>knows you</em> a little — and the kind of thing where “works in the demo” means nothing, because the same user comes back tomorrow expecting it to remember, and the day after that.</p><p>And the model turned out to be the easy part. The hard part was everything I had to decide <em>around</em> it: what to remember, what to recompute, and what to never let the model touch. Those decisions were the difference between an agent that felt reliable and one that quietly lied to the user — and at scale, “quietly lying” is not a bug you find in a demo. You find it in a support ticket.</p><p>Here’s what production actually taught me.</p><h4>The memory rule nobody hands you</h4><p>The first real question wasn’t “how do I store memory.” Tools for that are everywhere. The question was <em>what is even worth storing.</em></p><p>My first instinct — and the instinct most tutorials nudge you toward— was to store a lot. After a session, summarize what happened, save the user’s balance, save their recent spending, save their upcoming bills, and load it all back next time so the agent starts warm.</p><p>It doesn’t take many real sessions to see why that’s a trap.</p><p>If I store “balance: ₹1,28,000” as a remembered fact, that number is wrong the moment the user spends anything. Next session, the agent confidently quotes a stale figure. In a finance app, that isn’t a rough edge — that’s the agent lying with total confidence to someone making a decision about their money. The same goes for spending totals and bills. They all change.</p><p>So I landed on a rule that ended up governing the whole design:</p><blockquote><em>If a database has it, fetch it fresh. If it only exists because the user said it, remember it.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YAJReYRcakkT_sam-rO9eg.png" /></figure><p>A balance lives in a system of record, so it gets fetched live, every time, never stored. But when the user says “remind me to move ₹30,000 into my house fund on the 25th,” or “I want to cut my food delivery spending in half” — those facts don’t live anywhere else. There’s no table to query for an intention. <em>That</em> is what memory is for.</p><p>Once I drew that line, most of the “what do I store” problem disappeared. Memory stopped being a cache of data the system already owned and became the one thing it couldn’t get anywhere else: what the user actually told it.</p><h4>Let the model talk. Don’t let it decide.</h4><p>The second decision was where to draw the line between the model and plain code and it’s stricter than most people draw it.</p><p>The model gets two jobs, and both are about language, not truth. The first is deciding <em>which tools to call</em>: when the user asks “can I afford this laptop?”, nothing in the prompt tells the agent it needs the current balance and the upcoming bills before it can answer it has to reason that out from the question itself. That’s interpretation, and models are good at it. The second is <em>phrasing the final answer</em> so it reads like something a person would actually say.</p><p>Everything in between every number, every verdict, every recommendation comes from the system, not the model.</p><p>The arithmetic runs in code. Before the model ever sees a transaction, my code filters by date and sums the totals by category, so the model receives “food delivery: ₹8,400 this month,” never the raw rows to add up itself. It isn’t that models can’t add — it’s that a probabilistic sum is one you can’t audit or guarantee, and where the output is money, every number has to be reproducible and exactly right. Determinism isn’t optional there.</p><p>And the actual call “yes, you can afford it,” “you’re over budget this month” is grounded in those computed numbers, not invented by the model. This is the rule I’m strictest about: <strong>the model never gives a recommendation it can’t back.</strong> If a response can’t be justified from real data the system produced, the agent shouldn’t say it. An LLM that freelances financial advice isn’t a feature; it’s a liability. So the recommendation comes from the system, and only the wording comes from the model.</p><p>The principle generalizes well beyond finance: let the model interpret and phrase, but keep every fact and every decision traceable to something you can justify. Anything where being wrong has a cost — math, verdicts, advice should come from the system, not from the model’s free generation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8CUwalIsj-YISWKnUy6EIA.png" /></figure><h4>The instinct was to store more and pass more. I went the other way.</h4><p>I built a lot of the scaffolding with an AI assistant, and it was genuinely useful it caught a bug I’d missed, where salary credits were being counted as spending because I’d forgotten to filter for debits only.</p><p>But the two suggestions I remember most are the two I rejected, because they were both wrong in the same direction.</p><p>First, it wanted to store a summary of the user’s transactions in memory so the next session wouldn’t have to re-fetch them. Second, it wanted to pass the full tool response every raw record straight into the model’s context.</p><p>Both were wrong for related reasons. A stored summary goes stale, which defeats the entire point of having live tools in the first place. And forty raw transaction records in the context window is just noise and cost; the model doesn’t need the rows, it needs the category total.</p><p>The instinct the AI’s, and most tutorials’ is <em>store more, pass more.</em> In production the right move is closer to the opposite: <strong>store only what can’t be fetched, and pass the minimum context that’s still sufficient.</strong> Not the least possible starve the model of grounding and it just burns extra tool calls fetching what you withheld, and answers worse — but the minimum that still lets it reason correctly. Stuffing context is the default temptation, and past the point of sufficiency it makes the system slower, more expensive, and less reliable all at once.</p><h4>The failure that doesn’t show up in the demo: memory that rots</h4><p>Here’s the part that only shows up once a system has been running long enough to accumulate history and it’s the part I’d never have predicted from the demo.</p><p>Every remembered fact started life in one flat list, sorted by date. Over a session or two, that’s fine. Let it run, and it turns into noise and worse, <em>misleading</em> noise. A reminder the user already acted on shows up next session looking exactly like one they still need to deal with. The agent can’t tell “done” from “open,” so it keeps surfacing closed loops as if they’re still hanging open. The longer a user stayed with the product, the dumber their agent appeared to get. That’s the opposite of what memory is supposed to buy you.</p><p>The fix isn’t more storage. It’s giving memory a sense of <em>state</em> — marking each fact as done or still open, and ranking facts by how relevant they still are, so the agent walks into every session with an accurate picture of where things stand, not a growing pile of everything ever said.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Mp1X7Il2eL3UYapGIlb1lw.png" /></figure><p>That’s the real lesson hiding underneath all the others: <strong>memory isn’t a log. It’s working state.</strong> A log just accumulates. Working state has to be maintained — updated, closed out, pruned — or it slowly turns into a liability that makes the agent dumber the longer it runs. Most memory tutorials teach you to build the log. Production teaches you to maintain the state.</p><h4>The model was never the hard part</h4><p>If I’d stopped at “good model, good prompt, some tools,” I’d have shipped something that demoed beautifully and fell apart by the third conversation.</p><p>What made it survive real users was the unglamorous engineering around the model: a clear rule for what deserves to be remembered, a hard line between what the model may phrase and what only the data may decide, the discipline to feed the model less rather than more, and a memory layer that maintains state instead of hoarding it. None of that came from the model. All of it came from deciding, again and again, what the model should <em>not</em> be trusted to do.</p><p>That’s the part of building agents nobody puts in the demo. It’s also the part that decides whether the thing survives contact with a real user.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0b70f5f56d7a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Skill Engineers Are Quietly Losing While AI Makes Them Faster]]></title>
            <link>https://medium.com/@ashwindevelops/the-skill-engineers-are-quietly-losing-while-ai-makes-them-faster-a11ef5379ee5?source=rss-3a90c1246405------2</link>
            <guid isPermaLink="false">https://medium.com/p/a11ef5379ee5</guid>
            <category><![CDATA[career-advice]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[future-of-work]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Ashwindevelops]]></dc:creator>
            <pubDate>Tue, 26 May 2026 18:16:22 GMT</pubDate>
            <atom:updated>2026-05-27T02:44:27.802Z</atom:updated>
            <content:encoded><![CDATA[<p>There’s a strange thing happening to a lot of engineers right now, and almost nobody is naming it. We’re getting more productive every month, shipping faster than ever. But somewhere underneath that, a crucial skill is going quiet the reflex that used to fire automatically when we opened a blank file: break the problem into pieces, decide where the data lives, sketch the flow. For a growing number of engineers, that reflex now hesitates.</p><p>If you build with AI for a living, you’ve probably felt the edge of this even if you’ve never said it out loud. So let’s name it.</p><h4>The Problem Isn’t That AI Writes the Code</h4><p>That’s the version everyone debates, and it’s the wrong frame. AI writing the code is fine. Typing speed was never the moat. The real problem is quieter and worse:<strong> judgment is downstream of execution, and execution is decaying without anyone noticing.</strong></p><p>Consider what a strong engineer actually does. There’s generation producing the code. There’s judgment knowing whether the code is good, whether it’ll scale, whether it’s even the right shape. And there’s direction knowing what to build and why. AI is eating generation. It’s starting to help with judgment. It mostly can’t touch direction.</p><p>Here’s the trap: <strong>you cannot judge well what you can no longer generate at all</strong>. The senior engineer who glances at AI output and says “no, that breaks under concurrency” can do that because they once wrote that code by hand enough times to compile the instinct into their body. Execution didn’t stay valuable as execution. It became the substrate for judgment. Let it rot entirely, and judgment slowly goes blind while the engineer feels more productive than ever.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*R3CEO3d2vxKxUKP8" /></figure><h4>The Effects, and Why They’re Hard to See Coming</h4><p>The cruelest part is that the decay is disguised as progress. Every offload makes you faster today. The cost shows up later, and indirectly:</p><ul><li><strong>Cold-start paralysis: </strong>The blank file stops being a starting point and becomes a wall.</li><li><strong>Recognition replacing recall:</strong> You can read code and nod along, but you couldn’t summon it. And recognition is exactly the weak passive skill that lets bad code slip past you.</li><li><strong>Borrowed thinking:</strong> Tools that “grill you” with the right questions before you start are genuinely useful until the engineer can no longer generate those questions themselves. That’s renting an instinct instead of owning one.</li><li><strong>Dependency you only feel when the tool is gone:</strong> Remove the subscription and the fall-back version is alarmingly thin.</li></ul><h4>What Actually Drives It</h4><p>Three things, mostly:</p><p>1.<strong>The work itself</strong>: An engineer who only ships prototypes and POCs never has to read code carefully, never has to defend a design. The stakes that used to keep skills sharp are gone.</p><p>2.<strong>The incentives: </strong>Teams are rewarded correctly, in the short term for shipping fast. Hand-writing anything feels like leaving velocity on the table.</p><p>3.<strong>The slope:</strong> First execution goes to AI. Then “what should I even be thinking about” goes to AI. Then the meta-thinking goes too. Each step is individually rational. The sum is an engineer who has outsourced the thing that made them valuable.</p><h4><strong>The Two Wrong Ways Out</strong></h4><p>Two tempting answers, both wrong.</p><p><strong>Nostalgia</strong>: go back to hand-writing everything, refuse the tools. This is just choosing to be slow. The race on generation is over; pretending otherwise helps no one.</p><p><strong>Surrender:</strong> offload everything, lean fully into the tools, assume the skills don’t matter anymore. This is how an engineer becomes the person who collapses the moment the model is unavailable or worse, the person who ships AI’s confident mistakes because they’ve lost the ability to catch them.</p><p>The answer is neither. It’s deliberate retention.</p><h4>The Right Way: Practice Manual Landings</h4><p>A commercial pilot can let the autopilot land the plane. They still practice manual landings regularly. Not because the normal flight needs it because the skill is what saves everyone the day the normal case fails.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*IrBzhUI_ODBpkC0P" /></figure><p>That’s the model. An engineer doesn’t need to hand-write production code as daily output. They need to stay able to, by doing it on purpose, periodically, as practice. Enough to stay calibrated. Not enough to be slow.</p><p>Concretely, what that looks like:</p><ul><li><strong>Decompose before typing: </strong>The most valuable thing being lost isn’t syntax it’s the reflex to break a problem into pieces before writing anything. Every practice task should start on paper: what data, what shape, what endpoints, what state lives where. Rebuilding that reflex is half the battle.</li><li><strong>Ban AI from generating, allow it to explain: </strong>During practice, the AI shouldn’t write a line. It can explain a concept or clarify an error. The moment it writes the block, no retrieval happened and the rep is wasted.</li><li><strong>Frequency over heroics:</strong> One focused hour, most days, beats a heroic weekend. The skill consolidates between sessions.</li><li><strong>Produce, then critique against a strong critic:</strong> Build something by hand, then show it to AI and ask what’s wrong with it. That loop generate, then judge against a tough reviewer is the exact muscle that lets you catch AI being confidently wrong in real work. It’s the whole game.</li><li><strong>Apply the residue test:</strong> A tool used well leaves a residue of skill in you. A tool used as a crutch leaves a hole. Same tool, different relationship. Before reaching for an AI skill, try the thing yourself first generate your own questions, your own design then compare. The gap is the curriculum.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*WLkzYH9b890iR87F" /></figure><p>A note on what feels like total loss: when an engineer who’s drifted starts practicing again, it feels humbling. They blank on things they once knew cold. That’s not regression. The knowledge wasn’t deleted, it was de-indexed retrieval got slow from disuse, and the slowness feels like absence. It comes back far faster than it left.</p><h4>How Engineers Will Actually Be Evaluated</h4><p>The old proxy DSA, LeetCode, can-you-produce is becoming a worse and worse measure of value, but it persists because nothing legible has replaced it yet.</p><p>What replaces it is likely this: <strong>can you take an ambiguous problem, decide what’s worth building, orchestrate AI to build it, and catch the AI when it’s confidently wrong?</strong> That last clause is the entire job. And it’s a pure judgment skill one that quietly decays if you never touch the metal.</p><p>So the engineers who win the next few years won’t be the ones who refuse AI, and they won’t be the ones who offload everything to it. They’ll be the ones who let AI do the generation while deliberately keeping their own judgment sharp enough to lead it.</p><h4>The Reframe Worth Holding Onto</h4><p>If an engineer is anxious about this — if they can feel the decay and it bothers them — that anxiety is evidence their judgment still works. The one who should actually be worried is the one who feels no friction at all, who’s riding the productivity high without noticing the floor moved.</p><p>The floor did move. Noticing it is the prerequisite for finding new footing.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a11ef5379ee5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Rise of AI Skills: Why Prompting is No Longer Enough]]></title>
            <link>https://medium.com/@ashwindevelops/the-rise-of-ai-skills-why-prompting-is-no-longer-enough-44643db5ed48?source=rss-3a90c1246405------2</link>
            <guid isPermaLink="false">https://medium.com/p/44643db5ed48</guid>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <dc:creator><![CDATA[Ashwindevelops]]></dc:creator>
            <pubDate>Sat, 09 May 2026 11:13:02 GMT</pubDate>
            <atom:updated>2026-05-09T11:13:02.143Z</atom:updated>
            <content:encoded><![CDATA[<p>I used to think the secret to better AI output was a better prompt. So I’d spend an hour refining a single sentence hoping that if I just got the wording right, the AI would finally give me what I needed.</p><p>Sometimes it worked. More often, it didn’t. And even when it did, it worked exactly once. The next conversation, I was back at zero. All that effort, gone. The knowledge didn’t persist.</p><p>That’s the fundamental problem with prompts: they are <strong>disposable</strong>. You write them, use them once, and they disappear. You cannot build on them, share them, or improve them over time.</p><p>We are in the middle of a much bigger shift. We’ve gone from <strong>Prompt Engineering</strong> (2023) to <strong>AI Agents</strong> (2025), and now we’re entering the era of <strong>AI Skills Systems</strong>. The real evolution isn’t about writing smarter prompts it’s about building <strong>reusable operational intelligence</strong>.</p><h4><strong>What Exactly is an AI Skill?</strong></h4><p>An AI skill is a <strong>named, reusable instruction set</strong> — stored as a file, triggered by a slash command, and available every single time you need it.</p><p>Think of it this way:</p><ul><li><strong>Prompt:</strong> “Review this code for security.” — you type this, once, into a chat. It is gone when the conversation ends.</li><li><strong>Skill:</strong> /code-review — a file that triggers a 7-step workflow: scans the auth layer, checks for secret leakage, validates input sanitization, and produces a severity table. <em>Every time. Consistently.</em></li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Cdhd_P3SczqhDnhS" /></figure><h4><strong>But Isn’t It Just a Prompt?</strong></h4><p>Here is the question every honest reader is already thinking: if a skill is just a text file with instructions, isn’t that just a better-written prompt?</p><p><strong>Honest answer: yes.</strong> At its core, a skill is text. The AI reads it and follows it. There is no separate execution engine, no magic runtime.</p><p>But this framing misses the point. A shell script is also “just text.” A library function is also “just code.” The difference isn’t <em>what they are made of</em> — it’s <em>how they are packaged, named, and used</em>.</p><p>A prompt is a command you type in a terminal window. It disappears when you close it. A skill is a function in your codebase — it has a name, it lives in a file, you can version it with git, share it with your team, and improve it over time.</p><p>The leap from “a prompt” to “a skill” is the same leap as from “a one-off bash command” to “a reusable function.” The unit is similar. The <strong>system</strong> is not.</p><h4><strong>Why Skills are Better than Prompts?</strong></h4><p>The biggest advantage of skills isn’t that they sound smarter. It’s that they <strong>persist</strong>. When you encode a workflow into a skill, you’re capturing expertise once and deploying it indefinitely.</p><p>This gives you three real advantages:</p><ul><li><strong>Higher Quality Output:</strong> The AI follows a defined process rather than guessing at what you want each time. The workflow exists before the conversation starts.</li><li><strong>Consistency:</strong> You get the same structured result every time you trigger the skill not just “similar,” but genuinely consistent because the instructions are identical every time.</li><li><strong>Compounding Value:</strong> A prompt helps you once. A skill helps you — and everyone on your team for as long as you use it. You improve it, it gets better. Knowledge stops living in your head and starts living in your system.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*h1t-enJu0zrPKZCs" /></figure><h4><strong>The “Grill-Me” Pattern: Clarify Before You Code</strong></h4><p>One of my favorite examples of a high-leverage skill is the “<strong>Grill-Me</strong>” pattern.</p><p>Most people tell an AI: “Build me a landing page for a coffee shop.” The AI then guesses what you want and gives you a generic result.</p><p>With the <strong>Grill-Me</strong> skill, the conversation changes:</p><ul><li><strong>Human</strong>: “I want to build a landing page for a coffee shop.”</li><li><strong>AI (Grill-Me Skill)</strong>: “Wait! Before I write a single line of code, I need to grill you on the details. What’s the vibe? Who is the target audience? Do you have a color palette? What’s the primary call to action?”</li></ul><p>The AI relentlessly interviews you until the requirements are crystal clear. Only then does it start building. This “clarification loop” ensures the final output is exactly what you need, not just a generic guess.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*sT1SyIqO1xvT6qsV" /></figure><h4><strong>Real-World Proof: Skills at Scale</strong></h4><p>This isn’t theoretical. Real people are already shipping serious work with skill systems.</p><p>Garry Tan, President &amp; CEO of Y Combinator, published <strong>gstack</strong> — a library of 23 skills that turns Claude into a virtual engineering team. Each skill plays a specific role: one acts as a CEO who challenges your product assumptions, one acts as an engineering manager who locks in architecture, one acts as a QA lead who opens a real browser and tests your app.</p><p>His results: <strong>10,000–20,000 lines of production code per day</strong>, part-time, while running YC full-time. In one week: 140,751 lines added, 362 commits.</p><p>What makes this work isn’t any single skill. It’s the <strong>composability</strong>. His sprint chains together:</p><blockquote>/office-hours → /plan-ceo-review → /plan-eng-review → /review → /ship.</blockquote><p>Each skill passes context to the next. That’s something you simply cannot do with ad-hoc prompts because prompts don’t persist between sessions.</p><p>Matt Pocock, one of the most respected TypeScript educators, has published his own engineering skills a focused set of workflows for code review, architecture decisions, and TypeScript-specific patterns.</p><p>This is the emerging pattern: experts encoding their knowledge into reusable, shareable skill libraries that anyone can install and use.</p><h4><strong>The New Architecture: Skills, Agents, and Tools</strong></h4><ul><li><strong>Tools (The Hands):</strong> External capabilities a browser, a database, a GitHub connection.</li><li><strong>Skills (The Expertise):</strong> Behavioral workflows that define <em>how</em> work gets done — a PR review skill, a QA skill, a security audit skill.</li><li><strong>Agents (The Coordinator):</strong> The decision-making layer that decides <em>which</em> skills and tools to use.</li><li><strong>MCP (The Nervous System):</strong> The Model Context Protocol that connects the AI to all of these external systems.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Mrkr91WNxEDgaJ0o" /></figure><h4><strong>The Bottleneck Has Shifted</strong></h4><p>The most important skills right now aren’t coding skills they are <strong>thinking skills</strong>.</p><p>Why? Because AI coding is already strong. The bottleneck has moved upward:</p><ul><li><strong>Old Bottleneck:</strong> Writing the code.</li><li><strong>New Bottleneck:</strong> Thinking clearly, challenging assumptions, and decomposing systems.</li></ul><p>The highest-leverage engineers in 2026 aren’t the fastest coders. They are the <strong>workflow architects </strong>the people who design the constraints and quality gates that AI needs to execute perfectly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*OzL_QAffncDrBAH3" /></figure><h4><strong>How to Start Using Skills Today</strong></h4><p>If you want to get better output from AI, stop writing long, messy prompts. Start building a <strong>Skill Library</strong>.</p><ol><li><strong>Identify Repetitive Tasks:</strong> What do you ask AI to do every day?</li><li><strong>Define the Workflow:</strong> Break that task down into 5–10 logical steps.</li><li><strong>Add Quality Gates:</strong> What does a “perfect” result look like? Tell the AI to check for those things explicitly.</li><li><strong>Use Activation Keywords:</strong> Give your skill a name (like /grill-me or /code-review) so you can trigger it instantly.</li></ol><h4><strong>Final Takeaway</strong></h4><p>The era of “Prompting” is over. We are now in the era of <strong>Software Engineering for Cognition</strong>.</p><p>A skill is a prompt but a prompt that has been <strong>named, packaged, versioned, and shared</strong>. That’s not a small difference. That’s the difference between knowledge that lives in your head and knowledge that lives in your system.</p><p>The future belongs to the builders who can turn raw intelligence into operational expertise.</p><p>Stop building things that talk. Start building things that <strong>work</strong>.</p><p>Happy building.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=44643db5ed48" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agentic AI: The Shift from Chatbots to Autonomous Workers]]></title>
            <link>https://medium.com/@ashwindevelops/agentic-ai-the-shift-from-chatbots-to-autonomous-workers-8219492c0bae?source=rss-3a90c1246405------2</link>
            <guid isPermaLink="false">https://medium.com/p/8219492c0bae</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[software-engineering]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Ashwindevelops]]></dc:creator>
            <pubDate>Sun, 12 Apr 2026 10:27:30 GMT</pubDate>
            <atom:updated>2026-04-12T10:27:30.474Z</atom:updated>
            <content:encoded><![CDATA[<p>I used to think that the peak of AI was a chatbot that could write a decent email or explain a complex concept.</p><p>I was wrong.</p><p>We are moving away from <strong>“Generative AI”</strong> (where the AI just <em>talks</em>) and toward <strong>“Agentic AI”</strong> (where the AI actually <em>does</em> things).</p><p>This isn’t a small upgrade. It’s a completely different way of building software.</p><p>Here’s a simple way to understand the difference:</p><p>A <strong>chatbot</strong> answers your question. An <strong>agent</strong> goes and gets the answer, checks if it’s right, fixes mistakes along the way, and comes back with the result — all on its own.</p><p>In 2026, the most valuable systems aren’t the ones that give you the best answer. They are the ones that can take a high-level goal, break it down into steps, and execute those steps — without you micromanaging every move.</p><h4>The Agentic Stack: What’s Under the Hood?</h4><p>To build an autonomous agent, you need more than just a prompt. You need a four-layer cognitive architecture:</p><ol><li><strong>Profiling</strong>: Defining the agent’s role (e.g., “You are a Senior DevOps Engineer”).</li><li><strong>Memory</strong>: Short-term context (what just happened) and long-term RAG (what happened last month).</li><li><strong>Tools</strong>: The APIs, browser access, and desktop controls the agent can use.</li><li><strong>Reasoning Loop</strong>: The “brain” that connects everything.</li></ol><p>Think of it like a new employee. They have a job title (Profiling), they remember past conversations (Memory), they have software access (Tools), and they can figure out what to do next on their own (Reasoning Loop).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LNCbaddfDEgPS8Ha" /><figcaption>The four layers every production agent needs</figcaption></figure><h4>Technical Implementation: The State Machine</h4><p>Most people build agents like a straight line: input goes in, output comes out. That’s called a<strong> linear pipeline</strong>. And it works great — until something breaks.</p><p>An API returns an error. A website changes its layout. A search comes back empty.</p><p>Linear pipelines crash. They don’t recover.</p><p>The fix? Build a <strong>State Machine </strong>instead.</p><p>Using frameworks like <strong>LangGraph</strong>, you define your agent as a graph where each “state” is a step — Planning, Executing, Validating. The agent moves between states based on what actually happens. If something fails, it doesn’t crash. It loops back to Planning and tries a different approach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*b0wZqpN26skVUl_n" /><figcaption>When validation fails, the agent doesn’t crash — it replans</figcaption></figure><h4>Tool-Use Protocol: Don’t Trust the LLM</h4><p>Here’s a mistake a lot of builders make early on: they give an LLM direct access to an API and hope for the best.</p><p>Don’t do that.</p><p>LLMs are probabilistic. They make things up. They’ll call your API with wrong parameters and do it confidently. In a production system, that’s a real problem.</p><p>You need a strict <strong>Tool-Use Protocol:</strong></p><ol><li><strong>Input Validation:</strong> Before the tool is even called, validate the LLM’s suggested parameters against a JSON schema. Wrong type? Stop it right there.</li><li><strong>Execution:</strong> Call the API inside a sandboxed environment. Never give an agent direct production access.</li><li><strong>Error Handling:</strong> If the API fails, feed the error back to the LLM as context. The agent reads the error and self-corrects — instead of silently failing.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*W_zgQKU1Gab-q2Xp" /><figcaption>Never let an LLM call an API without a guardrail in between</figcaption></figure><h4>The Self-Correction Loop: A Technical Example</h4><p>Let’s look at how an agent handles a real-world failure. Imagine an agent trying to book a flight:</p><ul><li><strong>Step 1 (Plan):</strong> Agent decides to call the get_flights API.</li><li><strong>Step 2 (Execute):</strong> The API returns an error: Invalid date format. Expected YYYY-MM-DD.</li><li><strong>Step 3 (Observe):</strong> The agent receives this error message.</li><li><strong>Step 4 (Self-Correct):</strong> The agent realizes its mistake, reformats the date, and calls the API again.</li></ul><p>A chatbot would have stopped at Step 2. An agent keeps going.</p><p>This Self-Correction Loop is what separates a brittle demo from a system you can actually put in production.</p><h4>Multi-Agent Systems: One Agent Is Not Enough</h4><p>Here’s something most people don’t talk about: for big, complex tasks, a single agent hits a ceiling.</p><p>The answer? <strong>Multiple specialized agents working together.</strong></p><p>Think of it like a team at a company:</p><ul><li>A <strong>Planning Agent </strong>breaks the big goal into smaller tasks.</li><li><strong>Specialist Agents </strong>each handle one thing — one for web research, one for writing, one for data analysis.</li><li>An <strong>Orchestrator Agent</strong> keeps track of everything and puts it all together.</li></ul><p>This is exactly how frameworks like <strong>AutoGen</strong> (by Microsoft) and <strong>CrewAI</strong> work. Agents talk to each other, share results, and divide the work — just like a real team.</p><h4>Large Action Models (LAMs): The New Frontier</h4><p>We’ve all heard of Large <em>Language</em> Models. Meet their next evolution: <strong>Large Action Models (LAMs).</strong></p><p>The difference is simple:</p><ul><li><strong>LLMs</strong> are trained on text. They understand language.</li><li><strong>LAMs</strong> are trained on <em>actions</em>. They understand how software works and how to use it.</li></ul><p>A LAM doesn’t need a custom API integration to use your internal dashboard. It can <em>see</em> the screen, understand the interface, and click buttons — just like a human would.</p><p>This means the gap between “has an API” and “can be automated” is shrinking fast. Almost any software tool becomes something an agent can use.</p><h3>The Brutal Truth for Builders</h3><p>If you’re still building “wrappers” around LLMs, you’re already behind. The real value is in the <strong>Orchestration Layer.</strong></p><p>The elite engineers of 2026 are focusing on:</p><ul><li><strong>State Management:</strong> Keeping track of complex, multi-step workflows without losing context. This is a distributed systems problem dressed up in AI clothing.</li><li><strong>Reliability:</strong> Retries, fallbacks, and self-correction loops that hold up in production — not just in demos.</li><li><strong>Security</strong>: Sandboxed environments, strict permissions, and audit logs. An agent with unchecked access is a disaster waiting to happen.</li></ul><h4>5 Things to Walk Away With</h4><ol><li><strong>Agents aren’t smarter chatbots</strong>. They’re a completely different architecture — state machines, memory layers, tool protocols, and reasoning loops.</li><li><strong>Self-correction is the whole game</strong>. Build the Observe → Reflect → Replan loop from day one. Without it, you don’t have an agent. You have a fragile script.</li><li><strong>Always put a guardrail between the LLM and your tools.</strong> Validate every parameter. Sandbox every execution. Feed errors back as observations.</li><li><strong>Multi-agent systems scale better. </strong>A team of specialized agents beats a single generalist for complex tasks — just like in real teams.</li><li><strong>Human-in-the-loop is a feature.</strong> Full autonomy is the goal — not the starting point. Design your oversight layer on purpose.</li></ol><h4>Final Takeaway</h4><p>The era of the “Chatbot” is over. We are now in the era of the “Autonomous Worker.”</p><p>The shift from simple generative models to agentic systems is the biggest opportunity in tech right now. It’s not about making AI smarter; it’s about making it more capable of acting in the real world.</p><p>Stop building things that talk. Start building things that work.</p><p>Happy building.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8219492c0bae" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>