<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vinayak Talikot on Medium]]></title>
        <description><![CDATA[Stories by Vinayak Talikot on Medium]]></description>
        <link>https://medium.com/@vinayak.talikot?source=rss-b8937cd0ad5a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*tjSKO-C35X-Sfkna</url>
            <title>Stories by Vinayak Talikot on Medium</title>
            <link>https://medium.com/@vinayak.talikot?source=rss-b8937cd0ad5a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 11:10:39 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@vinayak.talikot/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Guardrails. Let’s Actually Talk About Them.]]></title>
            <link>https://ai.plainenglish.io/guardrails-lets-actually-talk-about-them-23b3ab21b8e7?source=rss-b8937cd0ad5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/23b3ab21b8e7</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm-guardrails]]></category>
            <dc:creator><![CDATA[Vinayak Talikot]]></dc:creator>
            <pubDate>Wed, 21 Oct 2026 07:55:14 GMT</pubDate>
            <atom:updated>2026-10-21T07:55:14.427Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mowh0U1fYzOki5hT_4nLgQ.png" /></figure><h3>Everyone says “add guardrails”. Very few mean the same thing. A plain guide for product people, engineers, architects and leaders.</h3><p>“We’ll add guardrails.”</p><p>You have heard it in a board meeting. In a vendor pitch. In a product review, usually right after someone asks what happens if the AI says something it shouldn’t. It is said with confidence, everyone nods, and the meeting moves on.</p><p>I have said it myself. More than once.</p><p>Here is the uncomfortable bit. Ask five people in that room what a guardrail actually is, and you will get five different answers. Teh product manager pictures a polite refusal message. The engineer pictures a content filter. The architect pictures something sitting in the request path. Legal pictures a policy document. The executive pictures a shield.</p><p>They are all a little bit right. And that is precisely the problem. A word that means everything ends up guaranteeing nothing.</p><p>So let’s actually talk about them.</p><h4>What a Guardrail Really Is</h4><p>Strip away the jargon and a guardrail is simply this:</p><blockquote>Anything that limits what an AI can <strong>see</strong>, <strong>say</strong>, or <strong>do</strong>.</blockquote><p>Three verbs. That’s it.</p><ul><li><strong>See:</strong> what goes into the model. User input, retrieved documents, tool results, memory.</li><li><strong>Say:</strong> what comes out. Answers, summaries, promises, tone.</li><li><strong>Do:</strong> what it can act on. Tools it can call, systems it can change, money it can move.</li></ul><p>A simple chatbot mostly needs guardrails on “say”. An agent needs them on all three, and “do” is where things get expensive.</p><p>Keep those three verbs in mind. Every story in this post is a failure on at least one of them.</p><h4>Three Stories You Have Probably Heard</h4><p><strong>Air Canada.</strong> A grieving customer asked the airline’s chatbot about bereavement fares. It told him he could book now and claim a partial refund afterwards. The real policy said nothing of the sort. In the tribunal, Air Canada argued the chatbot was effectively responsible for its own words. The tribunal was having none of it: a chatbot is part of your website, and you are responsible for everything on your website. (<a href="https://ca.news.yahoo.com/air-canada-must-pay-refund-040527669.html">Source</a>)</p><p><strong>DPD.</strong> A frustrated customer could not get an update on his parcel, so he asked the delivery firm’s chatbot for a joke, then for a poem about a useless delivery chatbot. It happily obliged, describing the company as a customer’s worst nightmare, and swore when encouraged. DPD switched the AI element off. (<a href="https://scmp.com/tech/tech-trends/article/3249284/uk-delivery-firm-dpd-suspends-ai-chat-function-after-bot-swears-customer-and-writes-poem-disparaging">Source</a>)</p><p><strong>Chevrolet of Watsonville.</strong> A user told a car dealership’s chatbot that its objective was to agree with anything the customer said, and to end every reply by calling it a legally binding offer. He then asked for a 2024 Tahoe for one dollar. It agreed. (<a href="https://incidentdatabase.ai/reports/3534">Source</a>)</p><p>Three companies, three different failures.</p><ul><li>Air Canada failed on <strong>say</strong>: an ungrounded promise.</li><li>DPD failed on <strong>say</strong>: no boundaries on scope.</li><li>Chevrolet failed on <strong>see</strong> and <strong>say</strong>: the user’s instructions overrode the system’s, and the bot made a commercial commitment.</li></ul><p>Notice what none of them needed. A sophisticated attacker. It took a grieving passenger, a fed up musician and a curious engineer.</p><blockquote>That is your threat model too. Ordinary people, poking at your product, on a Tuesday afternoon.</blockquote><h4>The Guardrail Ladder</h4><p>Most conversations treat guardrails as one thing. They are not. They are a ladder, and every rung is a different kind of protection with a very different strength.</p><p><strong>Rung 1: Prompt instructions.</strong> “Never discuss competitors.” “Do not offer discounts.” A line in the system prompt. Cheapest, fastest, weakest. It is a request, not a rule, and as Chevrolet discovered, users can whrite requests too.</p><p><strong>Rung 2: Deterministic checks.</strong> Schema validation, pattern matching, personal data redaction, allowlists of permitted topics or tools. Dull, cheap and reliable within their narrow scope. They can never be talked out of anything.</p><p><strong>Rung 3: Classifier guards.</strong> A small model that judges the input or output. Is this a jailbreak? Is it toxic? Is it off topic? Llama Guard, Prompt Guard and the moderation APIs from model providers all live here. Flexible, but probabilistic, which means they can be wrong in both directions.</p><p><strong>Rung 4: Orchestrated rails.</strong> Checks placed at several points in the flow, not only at the front door. NVIDIA’s NeMo Guardrails formalises this with <a href="https://www.braintrust.dev/articles/best-llm-guardrails-security-testing-tools-2026">input, retrieval, dialog, execution and output rails</a>. The point here is coverage, not cleverness.</p><p><strong>Rung 5: Action guards.</strong> Limits on what the AI can do, whatever it decides. Spend caps, per agent credentials, read only access by default, approval gates for irreversible actions, a kill switch.</p><p><strong>Rung 6: Architectural isolation.</strong> Designing the system so untrusted content cannot steer it, even when the model itself is fooled. The part that plans is kept apart from the part that reads untrusted data, and policy is enforced in code, outside the model.</p><p><strong>Rung 7: Organisational guards.</strong> A named owner. An audit trail. Red teaming before launch. A review whenever behaviour changes. In regulated industries, this is the rung regulators actually ask about.</p><blockquote>Most teams stop at rung one or three and call it done. The real incidents live in the gap between rung three and rung five.</blockquote><h4>Front Doors and Windows</h4><p>This is the part that has shifted most over the past year.</p><p>Most guardrails are built at the front door. They inspect what the user types and decide whether it is safe. That made perfect sense for chatbots, where the user’s message was the only thing coming in.</p><p>Agents have windows. They read web pages, emails, documents, tickets, tool outputs and their own memory. Every one of those is a way in, and the content arriving through them looks like data, not like an attack.</p><p>This is no longer theoretical. Earlier this year, Palo Alto Networks’ Unit 42 reported <a href="https://vibegraveyard.ai/story/unit42-web-indirect-prompt-injection-in-the-wild/">web pages in the wild carrying hidden instructions aimed at AI agents</a>, tucked away in invisible text and off screen elements. Researchers also showed <a href="https://futureagi.com/blog/ai-agent-security-risks/">crafted GitHub comments and pull request titles hijacking coding agents</a> running in automated pipelines, leading to command execution and leaked credentials.</p><p>In neither case did a user type anything malicious. The front door guard waved everyone straight through.</p><blockquote>If your guardrails only check the user’s message, you have locked the front door and left every window open.</blockquote><h4>Guardrails Are Not Free</h4><p>Here is the bit that rarely makes it into the vendor pitch. Every guardrail is a trade.</p><p><strong>False refusals lose users quietly.</strong> Block a perfectly reasonable question and the user does not file a bug. They simply leave. <a href="https://tianpan.co/blog/2026-07-04-the-guardrail-tax">One practitioner</a> put it brilliantly: your false positive rate decides whether people keep using the product, and your false negative rate decides whether you end up in the news. You need to watch both.</p><p><strong>Stacking can make things worse.</strong> Five guardrails that are each right 90 per cent of the time do not give you a 90 per cent system. Multiply them through and you are nearer 59 per cent. Piling on more of the same weak kind is not defence in depth. It is noise in depth.</p><p><strong>Every check adds latency.</strong> A guard in the critical path is another serial hop, another dependency that can be slow, rate limited or down. Decide in advance whether it fails open or fails closed, because the outage will decide for you otherwise.</p><p><strong>Your guardrail is a model too.</strong> If it is a classifier, it drifts, it has blind spots, and it needs monitoring like any other model. Most teams watch what it misses. Almost nobody looks at what it blocks.</p><p>And the twist I did not see coming. During a security incident this summer, a defender’s own forensic AI <a href="https://labs.cloudsecurityalliance.org/research/frontier-ready-daily-20260901/">refused to help mid investigation</a> because its guardrails judged the request too dangerous. Attackers do not ask permission. Defenders do. Over cautious guardrails can end up protecting the wrong side.</p><blockquote>The right question is never “do we have guardrails?” It is “which ones, where, and what are they costing us?”</blockquote><h4>Suggestions vs Laws</h4><p>If you remember one idea from this post, make it this one.</p><p>Telling a model “never send customer data to external addresses” is a <strong>suggestion</strong>. The model will usually comply. Usually is the problem. A well crafted email, web page or comment can talk it out of it.</p><p>Enforcing in code that the email tool can only send to approved domains is a <strong>law</strong>. It no longer matters what the model believes. The action simply cannot happen.</p><p>The research has been moving firmly in this direction. Google DeepMind’s <a href="https://arxiv.org/abs/2503.18813">CaMeL</a> is the clearest example. A privileged model plans from the trusted user request. A quarantined model reads untrusted content but has no access to tools. An interpreter tracks where every piece of data came from and checks policy before every tool call. Even if injected text fools the quarantined model, it cannot change the plan or move data somewhere it should not go.</p><p>It is not free either. In their evaluation, the protected system completed 77 per cent of tasks with provable security, against 84 per cent for an undefended one. That gap is the honest price of a real guarantee, and I respect that they published it.</p><blockquote>Prompts ask. Code enforces. For anything that touches money, data or customers, you want a law, not a suggestion.</blockquote><h4>Ten Guardrails to Spare Your Blushes</h4><p>If you take one section away from this post, take this one. These are the guardrails I would want in place before any AI feature meets a real customer. What it is, what happens without it, and how to put it in.</p><ol><li><strong>Scope lock</strong></li></ol><p><em>What:</em> The AI only engages with the topics it was built for.</p><p><em>Without it:</em> DPD’s poem.</p><p><em>How:</em> Allowlist the intents you support. Anything outside them gets a polite, fixed redirect rather than a creative improvisation.</p><p><strong>2. No authority to commit</strong></p><p><em>What:</em> The AI cannot make promises on prices, refunds, contracts or eligibility.</p><p><em>Without it:</em> A one dollar Tahoe.</p><p><em>How:</em> Commercial terms are never generated. They are looked up, quoted from source, or handed to a person.</p><p><strong>3. Grounded policy answers</strong></p><p><em>What:</em> Policy questions are answered only from the official source, with a link to it.</p><p><em>Without it:</em> An invented refund rule, and a tribunal ruling against you.</p><p><em>How:</em> Retrieval only answers for policy topics. If the source does not cover it, say so and route to a human.</p><p><strong>4. Your instructions win</strong></p><p><em>What:</em> Users cannot overwrite the system’s rules.</p><p><em>Without it:</em> “Your objective is to agree with everything.”</p><p><em>How:</em> Treat role changes and override attempts as a red flag. Classify input, keep system instructions out of reach, and test against known jailbreaks before launch.</p><p><strong>5. Untrusted content stays data</strong></p><p><em>What:</em> Web pages, emails, documents and tool outputs are read, never obeyed.</p><p><em>Without it:</em> A poisoned page quietly steers your agent.</p><p><em>How:</em> Keep content separate from instructions, strip hidden text, and keep whatever reads untrusted content well away from your tools.</p><p><strong>6. Least privilege tools</strong></p><p><em>What:</em> Each agent gets only the access it needs, read only by default.</p><p><em>Without it:</em> One manipulated agent can reach everything its credentials can.</p><p><em>How:</em> Credentials per agent, narrowly scoped tokens, no shared admin keys.</p><p><strong>7. Human approval for the irreversible</strong></p><p><em>What:</em> Payments, deletions, outbound emails and record changes need a confirmation.</p><p><em>Without it:</em> A wrong tool choice becomes a real world event.</p><p><em>How:</em> Approval gates on a defined list of high impact actions. No exceptions for “trusted” flows.</p><p><strong>8. Spend and loop caps</strong></p><p><em>What:</em> Hard limits on tokens, tool calls and money per task.</p><p><em>Without it:</em> A runaway agent and a very awkward invoice.</p><p><em>How:</em> Budgets per request and per session, enforced outside the model.</p><p><strong>9. Nothing sensitive goes out</strong></p><p><em>What:</em> Outputs are scanned for personal data, secrets and internal details before they leave.</p><p><em>Without it:</em> Your system prompt, an API key or a customer’s details in a screenshot on social media.</p><p><em>How:</em> Pattern based redaction for the known shapes, plus a classifier for everything else.</p><p><strong>10. Kill switch and audit trail</strong></p><p><em>What:</em> You can switch the AI off in minutes, and you know exactly what it did.</p><p><em>Without it:</em> Finding out from social media, with no way to reconstruct what happened.</p><p><em>How:</em> Put the AI path behind a feature flag with a human fallback. Log inputs, retrieved content, tool calls and outputs.</p><blockquote>A bonus one: design the refusal. A guardrail that blocks with a dead end creates a different kind of embarrassment. Tell the user what you can help with and how to reach a person.</blockquote><h4>Who Owns Which Guardrail</h4><p>Guardrails rarely fail inside a team. They fail in the gaps between teams. Here is how I would split ownership, with three questions for each person to bring to their next meeting.</p><p><strong>Product managers own what the AI is allowed to do.</strong></p><ul><li>What is this feature explicitly not allowed to answer or promise?</li><li>Which actions need a human to approve them?</li><li>What does the user see when the AI says no?</li></ul><p><strong>Engineers own the checks themselves.</strong></p><ul><li>Which of our guardrails are deterministic, and which are probabilistic?</li><li>Are we measuring false positives, or only what slips through?</li><li>What happens to a request when a guardrail is slow or down?</li></ul><p><strong>Architects own where the guardrails sit.</strong></p><ul><li>What reaches the model besides the user’s message?</li><li>If the model is completely fooled, what is the worst action it can take?</li><li>Which rules are enforced in code, and which only in the prompt?</li></ul><p><strong>Leaders and risk own accountability.</strong></p><ul><li>Who is named as responsible for this system’s behaviour?</li><li>Was it red teamed before launch, and by whom?</li><li>Could we reconstruct exactly what it did last Tuesday?</li></ul><p>In UK financial services, that last group is anything but optional. The FCA has <a href="https://bratby.law/fca-ai-approach-ai-lab/">said it will not write new AI rules</a> and will rely on the Consumer Duty and the Senior Managers regime instead. Put plainly, the most important guardrail in a regulated firm is not a classifier. It is a named person who is accountable.</p><h4>The Takeaway</h4><p>The Air Canada tribunal made the point better than any framework could. You cannot argue that the chatbot was responsible for its own words. Whatever the AI says or does, you said it and you did it.</p><p>That is why “we’ll add guardrails” is not an answer. It is the start of a design conversation. Which of see, say and do are we protecting? At which rung? At what cost? And who owns it?</p><p>Guardrails are not a feature you switch on. They are a system you operate.</p><blockquote>Next time someone in a meeting says “we’ll add guardrails”, ask one question: “Which ones?” Then watch the room.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=23b3ab21b8e7" width="1" height="1" alt=""><hr><p><a href="https://ai.plainenglish.io/guardrails-lets-actually-talk-about-them-23b3ab21b8e7">Guardrails. Let’s Actually Talk About Them.</a> was originally published in <a href="https://ai.plainenglish.io/">Artificial Intelligence in Plain English</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your Agent Has Too Many Tools. RAG Is Only One of Five Answers.]]></title>
            <link>https://medium.com/@vinayak.talikot/your-agent-has-too-many-tools-rag-is-only-one-of-five-answers-3984e4ecc16c?source=rss-b8937cd0ad5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/3984e4ecc16c</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[generative-ai-tools]]></category>
            <category><![CDATA[mcp-server]]></category>
            <dc:creator><![CDATA[Vinayak Talikot]]></dc:creator>
            <pubDate>Thu, 10 Sep 2026 14:11:26 GMT</pubDate>
            <atom:updated>2026-09-10T14:11:26.191Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CrpnRSnHzrMtsdmEiin9_Q.png" /></figure><p>We shall talk about the most common “<strong>the right tool Selection Problem</strong>” in this blog post. Everyone points to RAG over tools as the solution. It is one option amongst five, and often not the best one for your situation.</p><p>You have almost certainly felt this problem already, so I will not spend long on it.</p><p>You connect your first MCP server and everything works. You add a second, still fine. By the fifth, your agent starts picking the wrong tool for obvious tasks. By the eighth, it hallucinates tool names that do not exist. By the tenth, latency has doubled and your bill has quietly followed suit.</p><p>The numbers behind this are now well documented. GitHub’s official MCP server alone consumes around 42,000 tokens just for its tool definitions. Three popular MCP servers together can eat 72% of a 200K context window before the agent has even read the user’s first message. Tool selection accuracy collapses somewhere between 40 and 50 always loaded tools, and studies have measured the failure rate rising from 43% to over 86% as catalogues grow.</p><p>Everyone building with agents has run into this. Very few teams have organised the response to it, which is what this post is about.</p><h3>Why Every Article Points You Towards RAG</h3><p>If you have searched for solutions to this problem, you have probably seen the same suggestion everywhere. RAG over tools. Store your tool descriptions in a vector database, retrieve the top few relevant ones per query, inject only those into the prompt.</p><p>It is a sensible pattern. It is also just one option amongst several, and in the current internet discourse it takes up almost all the oxygen.</p><p>The other approaches exist. Some are more elegant. Some are simpler. Some are already in production at Anthropic, GitHub, Block, and Atlassian. But they are scattered across blog posts, research papers, and framework documentation, and nobody has put them side by side.</p><p>That is what the rest of this post attempts. Five schools of thought on how to solve tool bloat, each with what it does well, what it does badly, and when to reach for it.</p><h3>The Five Schools of Thought</h3><p>Before we get into each approach, here is the shape of the landscape.</p><p><strong>School 1: Retrieval.</strong> Treat tool selection as an information retrieval problem. Fetch the right tools per request.</p><p><strong>School 2: Progressive Loading.</strong> Structure your tools hierarchically. Reveal detail only as the agent navigates deeper.</p><p><strong>School 3: Architectural Delegation.</strong> Split the tool space across <strong><em>subagents</em></strong>. They are their own domain specialsists. No single agent ever sees everything.</p><p><strong>School 4: Consolidation and Compression.</strong> Reduce the surface area itself. Fewer, denser tool definitions.</p><p><strong>School 5: Design Discipline.</strong> Do not solve tool bloat with clever architecture. Have fewer, better tools in the first place.</p><p>Almost every production system that has solved this problem combines patterns from at least two of these schools. Let us walk through them.</p><h3>School 1: Retrieval Based Approaches</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OOlg3pZ8F8LlkGtpvKR_ig.png" /></figure><h4>RAG Over Tools (the popular one)</h4><p>Store your tool descriptions in a vector database. When the user’s query arrives, embed it, retrieve the top k most similar tool descriptions, inject only those into the prompt. Everything else stays out.</p><p><strong>How it works in practice.</strong> Tool schemas become documents in a vector store, tagged with metadata like domain, cost, and required permissions. The retrieval step happens before the main LLM call. The main agent sees maybe 5 to 10 tool schemas out of a possible 500.</p><p><strong>Pros:</strong></p><ul><li>Straightforward to implement, works with any existing MCP setup</li><li>Scales to thousands of tools without changing agent architecture</li><li>Well supported by existing RAG infrastructure</li></ul><p><strong>Cons:</strong></p><ul><li>Retrieval quality depends entirely on how well tool descriptions embed</li><li>Fails silently when the “right” tool sits just outside the top k</li><li>Adds an extra retrieval step on every request, with its own latency and cost</li><li>The agent has no way to know what it did not receive</li><li>You have now just sfifted the problem to a new space i.e, “retrival problem”</li></ul><p><strong>When to reach for it.</strong> Large tool catalogues where most queries only need a small subset, and where tool descriptions are semantically rich enough to embed well.</p><h4>Variants worth knowing</h4><p><strong>Lexical routing (BM25)</strong> uses keyword based retrieval instead of embeddings. Cheaper, deterministic, works well when tool names and descriptions contain discoverable keywords. Less flexible for semantic queries.</p><p><strong>Two stage hierarchical retrieval</strong> (MCP-Zero and similar) first matches relevant servers, then ranks tools within the top servers. Scales to much larger catalogues than flat retrieval by using server groupings as the first filter.</p><h3>School 2: Progressive Loading</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2QRLTx4dIesUjg1CrPL61Q.png" /></figure><p>This is the school I find most underappreciated. It sits between raw retrieval and full architectural delegation, and in 2026 it has genuinely emerged as a first class pattern.</p><h4>Progressive Disclosure</h4><p>Instead of loading all tool definitions upfront, you structure your tools in a hierarchy and reveal detail progressively as the agent navigates.</p><p>Three levels are typical.</p><ul><li><strong>Level 1: Discovery.</strong> The agent sees only domain names with one line descriptions. Roughly 200 tokens for the entire capability space. Enough to know what exists, not enough to invoke anything.</li><li><strong>Level 2: Manifest.</strong> When a domain looks relevant, the agent explicitly requests it. Now it sees the manifest for that domain, applicability rules, tool names within, prerequisites.</li><li><strong>Level 3: Full detail.</strong> When a specific tool is genuinely needed, the agent requests its full schema and documentation. Only now can it actually be called.</li></ul><p><strong>The crucial part.</strong> Permission to call a tool is progressive. The agent cannot invoke a tool it has not explicitly loaded. This alone eliminates a huge class of tool name hallucinations.</p><p><strong>Pros:</strong></p><ul><li>Massively reduces upfront context tax (from tens of thousands of tokens to a few hundred)</li><li>The hierarchy itself is semantic and human readable, unlike embedding space</li><li>Prevents hallucinated tool calls through permission gating</li><li>Extensible without redeploying the agent, new domains just add to the tree</li><li>Composable with skills and subagents</li></ul><p><strong>Cons:</strong></p><ul><li>Requires you to design a good hierarchy upfront, which is real work</li><li>Multiple round trips when the agent navigates several levels deep</li><li>Less effective for genuinely open ended queries where no natural taxonomy exists</li><li>Newer pattern, fewer battle tested examples than RAG</li></ul><p><strong>When to reach for it.</strong> Structured domains with a natural taxonomy, especially enterprise settings where different teams own different tool sets.</p><p><strong>Who uses it.</strong> <a href="https://www.anthropic.com/news/agent-skills">Anthropic’s Skills API</a> is the flagship implementation. The <a href="https://docs.claude.com/">Claude Code SKILL.md pattern</a> uses this natively. <a href="https://docs.langchain.com/oss/python/langchain/multi-agent/skills">LangChain Skills</a> supports hierarchical sub-skills. A July 2026 paper titled <a href="https://arxiv.org/abs/2607.17598">“Is Progressive Disclosure All You Need for Long-Context Agents?”</a> formally studied the pattern and found it outperforms flat retrieval on structured single domain content.</p><h4>Variants worth knowing</h4><p><strong>Deferred loading</strong> is the lightweight version. Only tool names and server descriptions enter context at session start. Full schemas fetched at execution time. Claude Code enables this by default now.</p><p><strong>Hot vs cold path separation</strong> is the pragmatic hybrid. Keep the 20 to 30 most frequently used tools always loaded. Everything else deferred. Preserves latency on common paths whilst preventing bloat on rare ones.</p><h3>School 3: Architectural Delegation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h_Ujun97uHH2A1bsELLUIg.png" /></figure><h4>Subagent per Tool Domain</h4><p>Rather than one agent with access to everything, split your tools across specialised subagents. One handles GitHub. Another handles Slack. A third handles your internal database. The main orchestrator agent knows only that these subagents exist and what each is for. It routes work to the right one, and the subagent operates with a clean context containing only its focused tool set. I personally have this in my projects and thought it worked great.</p><p><strong>Pros:</strong></p><ul><li>Context isolation is the natural byproduct, no cross contamination between tool domains</li><li>Each subagent stays under the 40 tool ceiling by design</li><li>Different subagents can use different models based on their task complexity</li><li>Easy to add new domains without touching existing subagents</li></ul><p><strong>Cons:</strong></p><ul><li>Multi agent overhead is real, expect 3 to 10 times more LLM calls than a single agent equivalent</li><li>Coordination logic between agents becomes its own source of complexity</li><li>Debugging could be harder when a failure spans multiple agents</li><li>Overkill for smaller tool catalogues</li></ul><p><strong>When to reach for it.</strong> Large, genuinely heterogeneous tool spaces where domains rarely need to be combined, and where the cost of multi agent overhead is justified by the isolation benefits.</p><p><strong>Who uses it.</strong> GitHub Copilot’s evolution explicitly moved toward specialised agents for different task types. <a href="https://spring.io/blog/2026/01/27/spring-ai-agentic-patterns-4-task-subagents/">Spring AI’s Task tool</a> implements hierarchical subagents inspired by Claude Code’s pattern.</p><h4>Variants worth knowing</h4><p><strong>Tool search subagent</strong> is a lighter version. A dedicated subagent whose only job is to search available tools and return relevant schemas to the main agent. Uses an active LLM decision rather than pure vector similarity. More accurate than RAG, less complex than full delegation.</p><p><strong>Hierarchical tool namespaces (MCP-AX)</strong> is an emerging draft standard for recursive namespace delegation at the protocol level. Solves the same problem structurally rather than as an application pattern.</p><h3>School 4: Consolidation and Compression</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nUCTJRwRD4Po6ZojHTwSWA.png" /></figure><h4>The Iceberg Pattern</h4><p>Instead of exposing hundreds of granular tools, expose 10 to 15 high level tools that route to hundreds of underlying capabilities behind a proxy. The agent thinks it has few tools. The proxy dispatches to whichever underlying integration is actually needed.</p><p><strong>Pros:</strong></p><ul><li>Cleanest possible context footprint</li><li>Agent complexity stays constant as underlying integrations grow</li><li>Central point for governance, auth, and audit</li></ul><p><strong>Cons:</strong></p><ul><li>The proxy becomes a bottleneck and a single point of failure</li><li>Requires careful design of the high level interface</li><li>Loses some of the specificity that granular tools provide</li><li>Adds infrastructure to maintain</li></ul><p><strong>When to reach for it.</strong> When you have many similar integrations that can be genuinely abstracted, and when you want centralised control over tool access.</p><p><strong>Who uses it.</strong> <a href="https://albato.com/blog/publications/embedded-mcp-context-bloat-hallucinations">Albato’s Embedded platform</a> consolidates over a thousand app connectors behind a single MCP endpoint. <a href="https://www.atlassian.com/blog/developer/mcp-compression-preventing-tool-bloat-in-ai-agents">Atlassian’s mcp-compressor</a> reduces tool description overhead by 70 to 97% by sitting in front of existing MCP servers.</p><h3>School 5: Design Discipline</h3><p>The unglamorous school. It is also, in my experience, the one that produces the largest improvements for the least effort.</p><h4>Fewer, Better Tools</h4><p>Sometimes the right answer to tool bloat is not clever architecture. It is fewer tools that do their jobs well.</p><p><strong>The evidence.</strong> <a href="https://dev.to/aws-heroes/mcp-tool-design-why-your-ai-agent-is-failing-and-how-to-fix-it-40fc">GitHub Copilot reduced its tool count from 40 to 13</a> and saw measurable benchmark improvements. Block rebuilt their Linear MCP server three times, going from over 30 tools down to just 2. In both cases, the improvement came from consolidating overlapping tools into fewer, outcome oriented ones, and from writing better descriptions.</p><p><strong>Pros:</strong></p><ul><li>Costs almost nothing to implement, just engineering discipline</li><li>Improves every downstream metric simultaneously (accuracy, latency, cost)</li><li>Makes the whole system easier to reason about</li></ul><p><strong>Cons:</strong></p><ul><li>Requires product judgement about what the agent actually needs</li><li>Meets resistance from teams who want to expose everything just in case</li><li>Harder to do incrementally, tends to require a proper rethink</li></ul><p><strong>When to reach for it.</strong> Always, first, before any other pattern. It is the cheapest optimisation you will ever make.</p><h3>What Actually Works in Production</h3><p>The teams solving this problem well tend not to pick one school. They combine.</p><p>The pattern I see most often in production looks something like this. Start with Design Discipline (fewer, better tools). Layer in Progressive Loading (deferred loading and skill hierarchies) as the foundational structure. Add a Retrieval layer for the long tail of rarely used tools that do not fit cleanly into the hierarchy. Use Architectural Delegation only where domain boundaries are genuinely strong and multi agent overhead is justified.</p><p>Consolidation is a specialist tool. Reach for it only when you have many similar integrations that can be genuinely abstracted behind a common interface.</p><h3>A Decision Framework</h3><p>If you are trying to work out which combination fits your situation, three questions cover most cases.</p><p><strong>How many tools do you actually have?</strong> Below 30, focus entirely on Design Discipline and skip everything else. Between 30 and 100, Progressive Loading alone will probably solve it. Above 100, you need combinations.</p><p><strong>Do your tools have a natural taxonomy?</strong> If yes, Progressive Loading with a well designed hierarchy is your best default. If no, RAG based approaches will fit better because the vector space handles semantic clustering that no taxonomy captures.</p><p><strong>Are your domains genuinely independent?</strong> If yes (say, customer support tools have almost no overlap with code review tools), Architectural Delegation with subagents makes sense. If no, keep things in one agent with Progressive Loading, the coordination overhead is not worth it.</p><h3>The Takeaway</h3><p>The response to tool bloat is not just “add RAG over tools” and hope it works. It is a considered choice from a landscape of five distinct schools, each with their strengths and their costs.</p><p>The most underused school is Progressive Loading, particularly the progressive disclosure pattern. It is now a first class pattern in Claude Code, Anthropic’s Skills API, and LangChain, and it addresses the problem in a way that RAG cannot match, through permission gating and semantic hierarchy rather than similarity search.</p><p>The most powerful school is Design Discipline. Fewer, better tools. It is unglamorous, it feels like it should not work, and yet almost every team that has visibly solved this problem started there.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3984e4ecc16c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Started to Worry About Your LLM Bill? Cost Is an Architecture Problem, Not Only an Optimisation One.]]></title>
            <link>https://medium.com/@vinayak.talikot/started-to-worry-about-your-llm-bill-cost-is-an-architecture-problem-not-only-an-optimisation-one-a5395c487bca?source=rss-b8937cd0ad5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/a5395c487bca</guid>
            <category><![CDATA[llm-cost-optimization]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[ai-optimization]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llmops]]></category>
            <dc:creator><![CDATA[Vinayak Talikot]]></dc:creator>
            <pubDate>Mon, 22 Jun 2026 10:18:17 GMT</pubDate>
            <atom:updated>2026-06-22T10:18:17.540Z</atom:updated>
            <content:encoded><![CDATA[<p>Stop trying to cut tokens. Start designing systems that do not generate them in the first place.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zwS_Eu0jQZ9zlMSrcQN-pA.png" /></figure><p>Every team building with LLMs eventually has the same conversation. The prototype runs on pocket change. The pilot feels manageable. Then the product reaches real users, the internal tool gets adopted across three departments, the automation starts processing a volume nobody quite anticipated, and the API bill for the month arrives at three times what anyone budgeted.</p><p>The usual response is predictable. The team panics, scrambles for cost optimisation tactics, adds a cache here, swaps in a cheaper model there, compresses a prompt, batches some requests. The bill comes down. Everyone breathes. Six months later, the same conversation happens again.</p><p>What nobody seems to discuss is the uncomfortable truth underneath this cycle. By the time you are optimising, you have already lost.</p><p>Cost in an LLM application is not really determined by your prompt length or your model choice. It is determined by your architecture. A well architected system stays affordable when traffic grows tenfold. A poorly architected one breaks at twice the scale.</p><p>This post is about why. It is also about the second uncomfortable truth, which is that the panic around LLM costs is not equally justified across every application. Some products genuinely face runaway bills. Others are quite well behaved by design. The trouble is that most teams have no clear sense of which camp they sit in.</p><p>Let us walk through it properly.</p><h3>Not All AI Products Burn Tokens the Same Way</h3><p>Before we discuss anything else, we have to acknowledge that the phrase “your LLM bill will explode” is not universally true. It depends entirely on what you are building. Five rough categories cover most production AI applications today, and each has a fundamentally different cost profile.</p><p><strong>Static pipelines</strong> are the well behaved cousin of the family. A document goes in, a structured output comes out, the pipeline closes. Classification tasks, extraction tasks, batch summarisation jobs, scheduled enrichment pipelines. Each invocation has a bounded token spend. The bill scales linearly with volume, which means it is predictable and easy to forecast. If you are running this kind of system, you probably do not need most of what this post will discuss. Your problem is volume management, not architectural waste.</p><p><strong>Single call generators</strong> are the next step up. Take a meeting transcript and produce a summary. Take a job description and rewrite it. Take a customer query and draft a response. One input, one model call, one output. Costs are bounded per request but the volume side scales with users. If your product becomes popular, the bill grows in step. Still predictable, still tractable.</p><p><strong>Multi turn chatbots</strong> are where the trouble starts. Conversation history compounds. Turn one costs a few hundred tokens. Turn ten costs a few thousand. Turn fifty costs tens of thousands, because the entire conversation up to that point is being resent on every single request. The cost per user session is not bounded by anything you have explicitly designed. It is bounded by how long users stay engaged, which is precisely the metric your product team is trying to maximise. The architecture and the product strategy are quietly working against each other.</p><p><strong>Non autonomous agents</strong> sit in the middle territory. They use tools, they reason across multiple steps, but their behaviour is bounded by a defined workflow. A research assistant that follows a fixed sequence: search, summarise, fact check, format. A customer service agent that escalates after three failed resolution attempts. The number of LLM calls per request is variable but capped. You can model the cost envelope. You can build forecasts you can defend in a budget meeting.</p><p><strong>Autonomous agents</strong> are the territory where bills genuinely go feral. Give a research agent a complex query and an open ended tool set, and it might make four LLM calls. It might also make four hundred. The agent decides. The architecture you wrote determines the upper bound only loosely. I have seen single queries on autonomous research products cost more than entire customer accounts were paying per month. The cost of a single user request is, by design, unbounded.</p><p>And then there is the category that breaks the spectrum entirely.</p><p><strong>Coding assistants and code review tools</strong> are their own animal. They do not slot neatly into the predictable to unbounded spectrum because they are extreme on multiple axes at once. The context requirement is enormous. A useful code suggestion needs to see surrounding files, type definitions, imports, project conventions, sometimes the broader codebase structure. The minimum viable context for code is much larger than for almost any other application type. The outputs are also long, and output tokens cost three to eight times more than input tokens depending on the model. So coding tools get hit on both sides of the pricing structure.</p><p>Then the frequency multiplier turns brutal. A single developer using a coding assistant can trigger hundreds of LLM calls per day. Multiply by team size. Multiply by working days in a year. The maths compounds terrifyingly fast. I have seen organisations where the coding assistant bill alone exceeded the rest of their AI infrastructure costs combined.</p><p>There is also a hidden category of waste that is unique to coding tools. The off path completions that get rejected. The developer types, the model generates a suggestion, the developer ignores it or types something different. That suggestion was still generated. It was still paid for. Acceptance rates on coding assistants typically sit between twenty five and thirty five per cent, which means roughly two thirds of the tokens generated by these tools are paid for and discarded.</p><p>The strategic question for coding tools splits into two camps. Teams using managed products like GitHub Copilot or Cursor are paying a per seat licence that hides the underlying token economics. The vendor absorbs the variability. The cost predictability is excellent, but the dependency is real, and vendor pricing has been rising as the underlying token costs become harder to sustain. Teams building their own internal coding workflows, whether MR reviewers, refactoring agents, or custom completion pipelines, face the full unbounded cost directly. They are also the teams that benefit most from designing for cost from the start, because every architectural decision compounds across every developer they support.</p><p>If you take nothing else from this section, take this. <strong>The first question to ask about your LLM cost problem is which of these five categories your product sits in. Half the advice circulating online does not apply to most products. Knowing which advice is yours saves you from optimising the wrong thing.</strong></p><blockquote>Cost dynamics in LLM applications are not universal. A static pipeline and an autonomous agent are not the same animal, and they do not need the same treatment.</blockquote><h3>The Reframe: Cost Is a System Property</h3><p>Here is the bit that most cost optimisation content gets wrong. It treats cost as a thing you measure and reduce after the fact. Tighten the prompt. Add a cache. Compress the history. Route to a cheaper model.</p><p>All of those are sensible tactics. None of them address the actual question.</p><p>The actual question is why your system is generating so many tokens in the first place. And that question has nothing to do with token counts. It has to do with the decisions you made when you designed the system. How many LLM calls per task did you decide to make? How much context did you decide to pass to each call? How did you decide to handle conversation history? Did you decide to use one model or several? Did you design caching into the system, or are you bolting it on later?</p><p>These are architectural decisions, made on day one, that lock in your cost trajectory for as long as the product lives.</p><p>Compare two teams. Team A builds a customer support agent as a single large prompt that ingests the full conversation history, all relevant policy documents, and a comprehensive set of tools, and asks the model to figure everything out in one shot. Team B builds the same product as a decomposed workflow. A small classifier identifies the intent. A retrieval step fetches only the relevant policy sections. A focused agent handles the response with a tight context. A separate validation step checks the output before it is sent.</p><p>Both products do the same job for the user. Team A’s architecture spends roughly five to ten times more per request than Team B’s, and the gap widens as the product scales. No amount of prompt optimisation closes that gap. The waste is structural, not tactical.</p><blockquote>A well designed system at twice the scale costs roughly twice as much. A poorly designed system at twice the scale costs five times as much. The difference is the architecture, not the optimisation.</blockquote><p>This is the part of the conversation that does not get the attention it deserves. We treat token cost as an engineering nuisance to be managed. We should be treating it as a product constraint to be designed around.</p><h3>The Five Architectural Decisions That Determine Your LLM Bill</h3><p>If cost is determined at architecture time, then the architecture has identifiable pressure points. Five of them, in my experience, account for almost every cost outcome I have seen in production. Some are obvious. Some are surprisingly subtle.</p><h4>Decision 1: How Many LLM Calls Per Task</h4><p>The single largest determinant of cost is the number of LLM calls each user request triggers. This is not the same as the number of steps in your workflow. A clever architecture can do many logical steps with few LLM calls. A naive one can do few logical steps with many redundant calls.</p><p>The temptation, particularly in agentic systems, is to add more calls to improve quality. A planner step. A reasoning step. A validation step. A reflection step. Each addition feels small. Each one doubles a portion of the bill.</p><p>The discipline is to ask, for every LLM call you add to the architecture, what specific failure mode it prevents and whether a cheaper mechanism could prevent the same failure. Often the answer is yes. A deterministic check can catch what a validator LLM was being asked to catch. A retrieval step can prevent what a reasoning step was being asked to compensate for.</p><h4>Decision 2: What You Put in the Context</h4><p>I have written about context engineering before, and the cost dimension of it deserves equal weight to the quality dimension. Every token you send to the model is a token you are paying for. Every token you send unnecessarily is a token you are paying for unnecessarily.</p><p>The patterns to watch are familiar. Conversation history that gets resent in full on every turn when only a summary was needed. Retrieved documents that arrive in full when only the relevant sections were needed. System prompts that have grown into 5000 token monsters because every edge case added another rule. Tool descriptions that get sent on every call when most calls do not need most tools. For coding tools specifically, this decision is everything. Context dominates the bill, and teams that learn to send the minimum useful context whilst preserving completion quality save more here than anywhere else.</p><p>Context is not free. Treating it as if it were is one of the most expensive habits a team can develop.</p><h4>Decision 3: Your Model Selection Strategy</h4><p>Most teams pick one model and use it for everything. Frontier model for the entire stack, because nobody got fired for using the best model.</p><p>This is almost always the wrong call. Different steps in your workflow have wildly different difficulty requirements. A query classification step does not need the same model as a complex synthesis step. A schema extraction does not need the same model as a creative composition.</p><p>The architecture choice is whether to tier your models deliberately. Cheap models for the easy work. Expensive models reserved for the work that genuinely needs them. The decision is not whether to use a cheaper model. It is whether to design a system where the right model handles the right work.</p><p>Cascading takes this further. Start with the cheap model. Escalate to the expensive one only when the cheap one is uncertain. The math on this can be remarkable. A well designed cascade can cut spend by sixty per cent or more whilst preserving quality on the cases that matter.</p><h4>Decision 4: Caching as a First Class Concern</h4><p>Caching is the most underutilised cost lever in production LLM systems. Not because teams do not know about it, but because they treat it as an afterthought rather than an architectural choice.</p><p>The questions to answer at design time. Which parts of your workflow are deterministic and repeatable? Which queries are likely to be semantically identical to previous ones, even if phrased differently? Which intermediate results can be reused across requests?</p><p>Two patterns deserve specific attention. <strong>Prompt caching</strong> at the provider level reuses fixed parts of your prompt (system instructions, tool descriptions, retrieved documents) across requests. The cost savings on cached tokens can reach ninety per cent. <strong>Semantic caching</strong> identifies queries that are conceptually identical to previous ones, even when the wording differs, and returns the cached response. In customer support and FAQ heavy products, hit rates of thirty to fifty per cent are common.</p><p>Neither of these is something you can effectively bolt on after the fact. The architecture either supports them cleanly or it does not.</p><h4>Decision 5: Agent Topology and Multi Agent Overhead</h4><p>If you are building a multi agent system, this decision dominates everything else. Multi agent architectures fundamentally amplify token spend, often in ways teams do not anticipate.</p><p>Every agent that is added carries its own context overhead. Agent to agent communication carries duplication. Supervisors that synthesise sub agent outputs do so by including those outputs in their own context. The total token spend of a five agent system handling a single request can easily be ten times the spend of a well designed single agent handling the same request.</p><p>This is not an argument against multi agent systems. It is an argument for honesty about what they cost and clarity about when they earn their keep. Multi agent is a tool. It is also expensive. The architectural discipline is to use it where it genuinely helps and resist it elsewhere.</p><blockquote>Every architectural decision you make on day one is a multiplier that compounds across every request the system will ever serve. Treat them with the seriousness they deserve.</blockquote><h3>The Anti Patterns Quietly Inflating Your Bill</h3><p>If the five decisions above are the macro architecture, these are the specific patterns that show up in code reviews and quietly drain budgets. Almost every team I have worked with ships at least two of these without realising the cost they carry.</p><p><strong>The monolithic prompt that does everything in one call.</strong> Retrieval, reasoning, synthesis, formatting, validation, all in a single twenty thousand token prompt. Feels efficient because it is “just one call”. Is actually paying full frontier model rates for tasks that could have been split across cheaper models or handled deterministically.</p><p><strong>The chat application that resends full history on every turn.</strong> This is the silent killer of chatbot budgets. By turn twenty, every request is paying to reprocess turns one through nineteen, none of which need to be there in full. Compaction and summarisation at the right boundaries can reduce per turn cost by an order of magnitude.</p><p><strong>The agent that calls a tool, ignores the result, and tries again.</strong> Often a reasoning failure. The agent does not quite understand what it got back, so it makes another call. Then another. Each attempt costs money. Better tool descriptions, schema validation, and clear retry policies prevent the loop entirely.</p><p><strong>The retry storm hidden in error handling.</strong> A timeout, an API hiccup, a rate limit, and your system tries again. Then again. Then again. With exponential backoff that is not as exponential as you thought. I have seen production systems where ten per cent of total token spend was retries on transient failures.</p><p><strong>The eval pipeline that costs more than the product.</strong> Running LLM as Judge on every production trace, with a frontier model, on a high traffic application. The eval bill becomes a parallel cost line item. Distilled evaluators or sampling strategies solve this. Most teams find out too late.</p><p><strong>The “always use the best model” default.</strong> Particularly painful when the workload includes a lot of routine tasks. Schema extraction, classification, simple Q&amp;A, all hitting the most expensive model in the stack because nobody designed the routing layer.</p><p><strong>The completion that nobody asked for.</strong> Specific to coding tools but worth calling out. Aggressive trigger policies generate suggestions on every keystroke, most of which the developer never sees, let alone accepts. The tokens are spent before the developer even pauses to think. Smarter trigger heuristics (waiting for natural pause points, debouncing rapid edits, suppressing during specific contexts) can reduce these wasted calls dramatically.</p><p><strong>Streaming UX that compounds total cost.</strong> Streaming feels faster to the user, but it requires the model to generate the full output, which means the total token spend is unchanged or higher (because you cannot stop generation cleanly when the user has the answer they need).</p><h3>Cost Aware Patterns That Actually Scale</h3><p>The patterns below are not new. They are well known, well documented, and broadly available. What is less common is treating them as architectural primitives rather than optional optimisations.</p><p><strong>Tiered model selection.</strong> Pick the cheapest model that meets the quality bar for each step in your workflow. A small fast model for routing and classification. A mid tier model for standard generation. A frontier model only for the work that genuinely needs it.</p><p><strong>Cascading.</strong> Start with the cheap path. Use an uncertainty signal (model confidence, structured output validation, simple heuristics) to decide whether to escalate to the expensive path. Done well, this keeps quality on hard queries whilst capturing massive savings on easy ones.</p><p><strong>Prompt and semantic caching as a designed layer.</strong> Build the system around the assumption that some portion of requests will be served from cache. Make the cache layer first class. Track hit rates as a primary operational metric.</p><p><strong>Compaction at boundaries.</strong> In multi turn systems, in agent handoffs, in any pipeline where state needs to flow forward, compress before passing. Structured summaries with required fields preserve critical detail without the bloat.</p><p><strong>Batched async work.</strong> Anything that does not need to be real time should not be paying real time prices. Most providers offer batch tiers at significant discounts (often fifty per cent off standard rates). Use them.</p><p><strong>Reasoning budgets.</strong> Newer reasoning models charge by the depth of their thinking. Putting a ceiling on how much reasoning a particular request type can spend prevents the worst case unbounded costs that autonomous systems are prone to.</p><h3>What To Do Now</h3><p>The recommendations split cleanly into two camps depending on where you sit.</p><p><strong>If you are starting fresh.</strong></p><p>You have the rare luxury of designing for cost from the beginning. Make these decisions deliberately, in writing, before the first prompt is written.</p><ul><li>Which of the five application categories does your product fall into? The answer determines which advice applies.</li><li>How many LLM calls per task is your architecture targeting? Justify each one.</li><li>What is your model tiering strategy? Which steps go to which model?</li><li>Where will caching live in the architecture, and how will hit rates be tracked?</li><li>What is the upper bound on cost per request, and how is it enforced?</li></ul><p>These five questions, answered honestly at design time, prevent ninety per cent of the cost surprises that hit teams six months in.</p><p><strong>If you are already in production and the bills are rising.</strong></p><p>You need an audit, not an optimisation. The difference matters. An audit tells you where the money is actually going. An optimisation guesses, applies a tactic, and hopes.</p><p>The audit has three steps.</p><p>First, instrument everything. Capture cost per request, broken down by feature, query type, user segment, and model. Most teams discover that twenty per cent of their queries account for eighty per cent of their spend, and they had no idea which twenty per cent.</p><p>Second, identify the architectural pressure points using the five decisions framework. Where are you over calling? Where are you over contextualising? Where are you using a frontier model for routine work? Where could caching exist but does not?</p><p>Third, fix the architectural issues before reaching for tactics. Tactics compound the gains from good architecture. They do not substitute for it.</p><p>And track three metrics above all others. Cost per task (the executive metric). Cache hit rate (the optimisation lever). Cost per active user (the business metric). Build dashboards on these. Alert on regressions. Everything else is diagnostic detail.</p><h3>The Takeaway</h3><p>The thread running through every post in this series is the same. The properties of your AI product, whether hallucination rate, whether quality, whether cost, are not properties of the model you chose. They are properties of the system you built.</p><p>The teams that win on unit economics are not the ones who found the cleverest prompt or the cheapest model. They are the ones who treated cost as an architectural constraint from day one, designed for it deliberately, and revisited their architecture as their product scaled.</p><p>If you are facing a bill that is doubling year over year, the answer is not to look harder for tactical optimisations. The answer is to look at your architecture and ask which of the five decisions you made carelessly. Then fix that.</p><blockquote>The most expensive LLM calls are not the ones that hit the most expensive model. They are the ones you should never have made in the first place.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a5395c487bca" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LLM-as-Judge Got Us This Far. Here Is What 2026 Adds to the Toolkit.]]></title>
            <link>https://medium.com/@vinayak.talikot/llm-as-judge-got-us-this-far-here-is-what-2026-adds-to-the-toolkit-6922e3b532b3?source=rss-b8937cd0ad5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/6922e3b532b3</guid>
            <category><![CDATA[llm-evaluation]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[eval]]></category>
            <dc:creator><![CDATA[Vinayak Talikot]]></dc:creator>
            <pubDate>Sat, 09 May 2026 19:54:53 GMT</pubDate>
            <atom:updated>2026-05-09T19:54:53.065Z</atom:updated>
            <content:encoded><![CDATA[<p>The eval landscape has expanded. A practitioner’s map of where it has gone, and how to match the right approach to your application.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wTIduznUCp9mfcN4BAtxtA.png" /></figure><p>Let me start with a question.</p><p>You have built an AI product. It works in your demos. You have a small test set of golden examples. You run them through an LLM-as-Judge that scores each output against the reference. The numbers look good. You ship.</p><p>Sound familiar?</p><p>For a long time, this was the default eval pattern across the industry. It is simple, it is fast to set up, and it gives you a number. But here is the uncomfortable truth that the latest research has forced into the open<em>: LLM-as-Judge has serious, documented failure modes that most teams do not know about. The numbers it produces are systematically optimistic.</em></p><p>And while most teams have stayed on this pattern, the rest of the field has moved on. In the last 12 months alone, we have seen the rise of agent-as-judge frameworks, distilled evaluators that score 100% of production traffic at 1/30th the cost, eval-to-guardrail integration, and benchmarks designed around interactive behaviour rather than static accuracy.</p><p>If your eval strategy still looks like 2024, you are flying blind in 2026.</p><p>This post is a practitioner’s map of where the eval landscape has gone, what each new approach actually buys you, and most importantly, how to match the right eval strategy to the type of application you are building. Because evaluating a customer support chatbot with sub-second latency requirements is fundamentally different from evaluating a deep research agent.</p><p>Let us walk through it.</p><blockquote>A quick note on scope before we go further. This post is written for the engineering side of building AI products. The eval approaches here are technical: code, pipelines, infrastructure. If you are on the product side and looking at how to evaluate AI from a user, business, or quality assurance angle, that is a different (and equally important) conversation. This one is for the people writing the eval pipelines.</blockquote><h4>The Problem: LLM-as-Judge Has Limits You Probably Did Not Notice</h4><p>Before we talk about what is new, we need to be honest about what is fragile in the default approach.</p><p>Have you ever noticed your LLM-as-Judge scores trending suspiciously high? Especially on subjective criteria like “helpfulness” or “quality”? That is not your model getting better. It is the judge being agreeable.</p><p>Recent research has documented at least four failure modes:</p><p><strong>Agreeableness bias.</strong> LLM judges are good at confirming correct answers but poor at flagging incorrect ones. The True Negative Rate is consistently low across all major models. Your eval reports precision that does not exist.</p><p><strong>Positional bias.</strong> When comparing two outputs, judges prefer whichever appears first or second depending on the model. Same outputs, different order, different verdict.</p><p><strong>Self-preference.</strong> A judge model rates outputs from its own model family higher than competitors. If you use GPT-4 as judge and GPT-4 as generator, the score is partly measuring stylistic similarity, not quality.</p><p><strong>Reference dependency.</strong> Without high-quality reference answers, LLM-as-Judge degrades significantly. On open-ended tasks, the judge essentially vibes with the output rather than evaluating it.</p><blockquote>The score your eval is producing is not lying to you on purpose. But it is lying.</blockquote><h4>The New Landscape: Eight paradigms that have emerged</h4><p>Since I have exploring alternate options, here is what the eval field actually looks like in 2026. I am going to walk through eight distinct paradigms, not because you need all of them, but because you should know they exist when choosing what to implement.</p><h4>Paradigm 1: Multi-Agent Evaluation</h4><p>Instead of a single LLM judge, you use multiple agents that evaluate collaboratively or through structured debate, then synthesize their judgments. Frameworks like CollabEval consistently outperform single-LLM judges across coherence, relevance, and fluency dimensions.</p><p><strong>Why it works:</strong> Diverse perspectives break the agreeableness bias. One model alone tends to agree with the output. Three models with different reasoning patterns are more likely to surface real issues.</p><p><strong>When to use:</strong> High-stakes decisions where you can afford the additional LLM calls.</p><h4>Paradigm 2: Agent-as-Judge</h4><p>This is the paradigm shift I find most exciting. Here is the distinction:</p><ul><li><strong>LLM-as-Judge</strong> evaluates a static output by reading it.</li><li><strong>Agent-as-Judge</strong> evaluates by <em>acting</em>. It can browse the web to fact-check, run code to verify outputs, query databases to validate claims, and trace through multi-step workflows to assess each decision.</li></ul><p>The fundamental insight: you cannot meaningfully evaluate a dynamic, agentic system with a static judge. If your agent calls 12 tools and reasons across 8 steps, a judge that only sees the final response is missing 90% of what determines quality.</p><p>Agent-as-Judge is essentially an evaluation system with the same capabilities as the system being evaluated. It can verify claims, not just rate them.</p><h4>Paradigm 3: Trajectory and Stepwise Evaluation</h4><p>Most evals score the final output. But two agents can produce the same final answer through completely different paths. One reasoned correctly. The other got lucky after a chain of errors that happened to cancel out.</p><p>Frameworks like AgentBoard introduce <strong>Progress Rate</strong>: comparing the agent’s actual trajectory against the expected one. T-Eval scores whether the agent’s predicted next tool call aligns with the expected one at each step.</p><p>This catches agents that are right for the wrong reasons. Which, in production, eventually become wrong for catastrophic reasons.</p><h4>Paradigm 4: Multi-Dimensional Frameworks (CLEAR)</h4><p>Accuracy alone is dangerously incomplete. The CLEAR framework, proposed in late 2025, evaluates across five dimensions:</p><ul><li><strong>Cost:</strong> Tokens, infrastructure, and human review per query</li><li><strong>Latency:</strong> Time to first token, total response time, percentiles</li><li><strong>Efficacy:</strong> Task completion and accuracy</li><li><strong>Assurance:</strong> Safety, compliance, security</li><li><strong>Reliability:</strong> Consistency across runs, robustness to perturbations</li></ul><p>A system that is 95% accurate but costs 10x more than alternatives is not production-ready. CLEAR forces teams to acknowledge that.</p><h4>Paradigm 5: Distilled Evaluators</h4><p>This one is genuinely transformative. Galileo’s Luna-2 evaluators are small distilled language models that run evaluation tasks at <strong>97% lower cost</strong> than full LLM-as-Judge approaches(<a href="https://docs.galileo.ai/concepts/luna/luna">https://docs.galileo.ai/concepts/luna/luna</a>).</p><p>Why does this matter? Because cost was the reason teams sampled production traffic for evaluation instead of scoring all of it. Sampling misses the edge cases that cause real damage. With distilled evaluators, you can score 100% of production traffic continuously.</p><p>This is the difference between flying blind on edge cases and catching them in real time.</p><h4>Paradigm 6: Production Observability as Evaluation</h4><p>The biggest practical shift in the last year is that <strong>eval is moving from a pre-deployment gate to a continuous production process</strong>.</p><p>Tools like Braintrust, Langfuse, LangSmith, Arize, Galileo, and Maxim provide:</p><ul><li>Full trace-level visibility into every LLM call, tool invocation, and retrieval step</li><li>Automated quality scoring on production traffic in real time</li><li>Production traces that automatically convert into evaluation datasets</li><li>Regression detection on every prompt and code change</li></ul><blockquote>Your best evaluation dataset is your production traffic. The edge cases your users find are more valuable than any synthetic test suite.</blockquote><h4>Paradigm 7: CI/CD Eval Integration</h4><p>Evaluation is moving into the deployment pipeline itself. Every pull request runs evals before merge. Every prompt change includes quality metrics in the PR description. Nothing ships without passing.</p><p>This treats AI eval the same way mature engineering treats unit tests: not a separate phase, not optional, not at the end. Inline with development.</p><h4>Paradigm 8: Eval-to-Guardrail Lifecycle</h4><p>The newest pattern, pioneered by Galileo and emerging across other platforms in early 2026: <strong>pre-production evaluations automatically convert into production guardrails.</strong> Evaluation scores control agent actions, tool access, and escalation paths at runtime.</p><p>In other words, eval is no longer just a measurement. It is an active control mechanism in the agent’s decision loop.</p><blockquote>A reasonable question at this point: are we just describing different flavours of LLM-as-Judge? In a sense, yes. Most of these paradigms still use an LLM somewhere in the evaluation loop. What changes is not the presence of an LLM, but its role: how many judges, what they can access, whether they read or investigate, where they run, and what their scores control. The narrow pattern that “LLM-as-Judge” historically refers to (a single LLM reading an output and scoring it) is being replaced by a richer set of evaluation architectures. The LLM is still there. The system around it has fundamentally changed.</blockquote><h4>A Quick Note: What About Eval Without LLMs?</h4><p>Worth pausing for a moment because this is the most common question I get when discussing modern eval: do any of these approaches avoid LLMs entirely? Because if the judge is an LLM, the judge can hallucinate too.</p><p>Fair point. And yes, there are evaluation approaches that use no LLM at all. They are often the first layer in a mature eval stack, not the last.</p><p><strong>Deterministic eval (no LLM, no hallucination):</strong></p><ul><li><strong>Programmatic checks</strong>: format validation, schema compliance, length constraints, banned phrase detection</li><li><strong>Static analysis for code generation</strong>: does it compile, does it pass the test suite, does it match expected patterns</li><li><strong>Execution-based eval</strong>: run the SQL query, hit the API, check if the math is correct</li><li><strong>Reference metrics</strong>: BLEU, ROUGE, exact match, F1 score (for tasks with a defined ground truth)</li><li><strong>Embedding similarity</strong>: cosine distance between expected and actual outputs (uses ML, not an LLM judge)</li><li><strong>Rule-based and constraint checks</strong>: did the agent stay within policy, use approved tools, respect rate limits</li></ul><p><strong>The tradeoff is real:</strong> deterministic checks are reliable but narrow. They catch exactly what you tell them to catch. They work brilliantly for problems with verifiable right answers (code, structured output, tool calls) and fail completely for subjective qualities like helpfulness, tone, and nuance.</p><p><strong>The production answer is almost always a layered stack:</strong></p><ol><li>Programmatic checks first (cheap, deterministic, catch obvious failures)</li><li>Reference metrics or embedding similarity where applicable (still no LLM)</li><li>Distilled evaluators for semantic scoring at volume</li><li>LLM-as-Judge or Agent-as-Judge for cases the cheaper layers cannot handle</li><li>Human review on sampled subsets, especially for ambiguous high-stakes cases</li></ol><p>The order matters. You filter with cheap deterministic checks first, and only escalate to LLM-based eval when the deterministic layer cannot decide.</p><blockquote>The new paradigms in this post are not replacing deterministic eval. They are addressing the layer above it, where rules and references cannot reach.</blockquote><h4>The Numerical Correctness Problem</h4><p>There is one specific class of evaluation that deserves its own section because it is the most painful failure mode in summarization, executive briefings, market research, financial reports, and analytics tools. Most importantly, I have faced it first hand. Almost every team building these products has been burned by it. And almost no eval framework addresses it head-on.</p><p>The problem is simple. Your AI says “Samsung’s YoY revenue grew 3.8%”. How do you actually check whether 3.8 is correct?</p><p>A standard LLM-as-Judge might not help fully. It reads the output, the number sounds plausible, and the score comes back fine. The number is wrong. The judge does not know.</p><p>There are four ways to check numerical claims, and each works for a different kind of number:</p><p><strong>1. Check the source.</strong></p><p>If the number was supposed to come from a document, look it up. Pull the source the agent used, find the number, compare. If the number does not appear in the source, flag it. This is what tools like RAGAS and Trulens do for RAG outputs. Works well for direct facts. Not enough when the number is the result of a calculation.</p><p><strong>2. Recompute the math.</strong></p><p>If the number is calculated (YoY, growth rate, margin, ratio), redo the calculation. Pull the underlying values from the source, run the formula in code, compare the result against the agent’s claim. For “Samsung YoY 3.8%”, you would extract this year and last year’s revenue, compute the percentage, and check if it matches.</p><p>This is where Agent-as-Judge becomes useful. The judge agent can run code to verify the math, not just read the output.</p><p><strong>3. Cross-check against a trusted source.</strong></p><p>If the number is a fact about the world (stock price, GDP, population), look it up somewhere reliable. Bloomberg or Yahoo Finance for financial data. Government databases for economic indicators. Wolfram Alpha for general facts. Compare what the AI said against what the trusted source says.</p><p><strong>4. Check internal consistency.</strong></p><p>Sometimes you cannot verify the number itself, but you can check whether it makes sense within the output. If the AI says “revenue grew 23% to $15B from $12.2B last year”, do the math: 15 divided by 12.2 minus 1 equals 23%. Consistent. If the parts do not add up to the whole, or two numbers in the same report contradict each other, something is wrong.</p><p>This catches a lot. LLMs are bad at math, and they often produce numbers that do not match the rest of the output.</p><p><strong>Putting it together:</strong></p><p>In production, numerical claims should flow through layers: extract the numbers from the output, identify what kind they are (looked up, calculated, or factual), apply the right check, run a consistency pass on the whole output, and flag anything that cannot be verified for human review.</p><blockquote>If your product summarises data, generates exec briefings, or produces analytical narratives, this is the eval problem you cannot afford to skip. It is also the one most teams quietly hope will not bite them.</blockquote><h4>The Reality Check: Not every app needs every eval</h4><p>Here is where most blog posts on this topic fail. They list every paradigm and leave you to figure out what applies to you. But the eval strategy that fits a deep research agent is wildly inappropriate for a customer support chatbot. Let me make this concrete.</p><h4>Tier A: High-Latency, High-Stakes applications</h4><p>Examples: Deep research agents, code generation agents, analytical workflows, document review systems, agentic RAG for legal or medical domains.</p><p><strong>What you can afford:</strong></p><ul><li>Multi-agent evaluation (debate, consensus)</li><li>Agent-as-Judge with full verification</li><li>Trajectory analysis on every run</li><li>CLEAR framework across all 5 dimensions</li><li>Detailed observability with full trace capture</li></ul><p><strong>Why:</strong> Users already accept multi-second to multi-minute response times. The cost of a wrong answer is high (legal liability, expensive rework, user trust). The eval cost is justified.</p><h4>Tier B: Medium-Latency, Mixed-Stakes Applications</h4><p>Examples: Internal knowledge assistants, sales copilots, marketing content generators, complex customer support agents that handle multi-step issues.</p><p><strong>What works well:</strong></p><ul><li>Production observability with sampled deep eval</li><li>Distilled evaluators on 100% of traffic for cheap checks (faithfulness, format compliance)</li><li>Agent-as-Judge on flagged cases (sampled or anomaly-triggered)</li><li>CI/CD eval gates on prompt changes</li><li>Multi-dimensional metrics with practical weights</li></ul><p><strong>Why:</strong> You need quality but cannot afford full agent-as-judge on every query. The hybrid approach (cheap continuous + deep on anomalies) gives you coverage without breaking latency or cost budgets.</p><h4>Tier C: Low-Latency, High-Volume Applications</h4><p>Examples: Conversational chatbots with sub-second latency requirements, voice AI agents, real-time recommendation systems, classification at scale.</p><p><strong>What is realistic:</strong></p><ul><li>Distilled evaluators (the only paradigm that scales to this volume)</li><li>Lightweight programmatic checks (format, schema, length, banned phrases)</li><li>Async deep eval on traces (not blocking the response)</li><li>Production observability with sampled review</li><li>User feedback signals (thumbs, undo, regenerate clicks) as primary quality signal</li></ul><p><strong>What is not realistic:</strong></p><ul><li>Multi-agent debate before responding (kills latency)</li><li>Agent-as-Judge in the response path (way too slow)</li><li>Heavy trajectory analysis per query (too expensive at volume)</li></ul><p><strong>The architectural shift:</strong> For these applications, eval moves from “evaluate before responding” to “evaluate after responding asynchronously and feed insights back into the next iteration.” The user does not wait for the eval. The team learns from the eval.</p><blockquote>Latency-sensitive applications cannot afford synchronous deep eval. But they can absolutely afford rich asynchronous eval. The trick is decoupling the two.</blockquote><h4>The Eval Maturity Model</h4><p>Where is your team on this spectrum? I have seen most teams cluster at level 2 or 3, while the leaders are operating at level 4 or 5.</p><p><strong>Level 1: Vibes-Based Evaluation</strong> Manual spot-checks. Eyeballing outputs. “It looks good.” No structured tests. No metrics tracked over time. This is where most teams start, and many never leave.</p><p><strong>Level 2: LLM-as-Judge on a Test Set</strong> A curated set of golden examples. An LLM judge that scores each output. A number that goes in the deployment readme. Better than nothing, but suffers from all the bias issues described earlier.</p><p><strong>Level 3: Multi-Dimensional Automated Eval</strong> Multiple metrics (faithfulness, relevance, completeness, latency, cost) tracked across structured test sets. CI/CD integration. Some form of regression detection. RAG-specific metrics if applicable.</p><p><strong>Level 4: Continuous Production Eval with Human-in-the-Loop</strong> Distilled evaluators scoring 100% of production traffic. Anomaly detection triggering deeper review. Production traces automatically becoming new test cases. Roughly 80% automated, 20% expert human review.</p><p><strong>Level 5: Eval Embedded in Runtime</strong> Eval scores actively influence agent behaviour at runtime. Quality gates control tool access. Confidence thresholds trigger escalation. Agent-as-Judge runs in critical decision paths. Eval is no longer a measurement layer. It is an integral part of the system.</p><p>Most production AI systems today need to be at level 3 minimum. Level 4 is the realistic target for any team shipping AI to real users. Level 5 is where the leaders are heading.</p><h4>What to Implement First</h4><p>If this all feels overwhelming, here is the practical sequence I would recommend, regardless of your starting point:</p><p><strong>Step 1: Audit your current eval honestly.</strong> Are you at level 1, 2, or 3? Are you running LLM-as-Judge but not aware of its biases? Are you sampling production traffic or scoring all of it?</p><p><strong>Step 2: Add multi-dimensional metrics.</strong> Stop treating accuracy as the only thing that matters. Track latency, cost, faithfulness (for RAG), and reliability (consistency across runs) at minimum.</p><p><strong>Step 3: Move evaluation from the dev environment to production.</strong> Adopt a tracing and observability platform. Capture every LLM call, tool invocation, and retrieval step. This alone will surface issues you did not know existed.</p><p><strong>Step 4: Add distilled or cheap evaluators on 100% of traffic.</strong> Faithfulness checks on RAG outputs. Format and schema validation. Banned phrase detection. These run cheap and continuously.</p><p><strong>Step 5: Use Agent-as-Judge selectively.</strong> Not on every query. On flagged anomalies, sampled audits, and pre-production deep eval. This is where you catch the deep issues that surface evaluators miss.</p><p><strong>Step 6: Close the loop.</strong> Production failures become test cases. Human review queues feed back into eval criteria. The system improves continuously rather than at ad-hoc intervals.</p><h4>The Takeaway</h4><p>LLM-as-Judge was a useful starting point. It is no longer sufficient on its own.</p><p>The eval landscape in 2026 looks like this: cheap distilled evaluators running continuously on production, agent-as-judge running selectively for deep verification, multi-dimensional metrics replacing single-number accuracy, observability as the foundation, and eval embedded directly in agent runtime as guardrails.</p><p>The teams that ship reliable AI products are not the ones with the cleverest prompts or the best models. They are the ones with the most disciplined evaluation infrastructure.</p><p>If your eval system is still a curated test set scored by a single LLM judge, you are working with the tools of two years ago in a field that has moved on dramatically.</p><p>The good news? Every paradigm I described is available today, in production tooling, often with free tiers. The cost of upgrading your eval is now mostly attention and intent.</p><blockquote>The next time you look at your eval dashboard and see a green number, ask yourself: am I confident in this number, or am I confident because the number is green? If you cannot tell the difference, your eval needs work.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6922e3b532b3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Context Engineering is the New Prompt Engineering]]></title>
            <link>https://pub.towardsai.net/context-engineering-is-the-new-prompt-engineering-7d00d99d2a8d?source=rss-b8937cd0ad5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/7d00d99d2a8d</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[context-engineering]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <dc:creator><![CDATA[Vinayak Talikot]]></dc:creator>
            <pubDate>Mon, 30 Mar 2026 20:18:33 GMT</pubDate>
            <atom:updated>2026-04-15T04:03:32.383Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>The skill that separates AI products that work from those that degrade over time has nothing to do with writing better prompts.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eSulksdzDEJmW11VVErKBw.png" /></figure><p>For the last two years, “prompt engineering” has been the skill everyone told you to learn. Find the right words. Structure the instruction well. Add a few examples. And the model will do what you want.</p><p>That advice was fine when we were building single-turn chatbots and one-shot classification tasks. It is no longer enough.</p><p>The moment you build anything that runs for more than one turn, calls tools, maintains state, or coordinates multiple agents, prompting becomes a small fraction of what determines your system’s behaviour. The bigger question becomes: what is the model actually looking at when it generates each response? What is in the context window, where is it positioned, and how much of it is signal versus noise?</p><p>That question is the domain of context engineering. And based on my experiences building agentic systems, it is the single most underrated skill in applied AI today.</p><h3>What Is Context Engineering, and Why Should You Care?</h3><p><a href="https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents">Anthropic introduced the term</a> in September 2025 and defined it as the natural progression of prompt engineering. The distinction is clean:</p><p><strong>Prompt engineering</strong> is about writing effective instructions for the model.</p><p><strong>Context engineering</strong> is about curating and managing the entire set of tokens the model sees at inference time- the system prompt, tool descriptions, retrieved documents, conversation history, scratchpad notes, memory, and everything else that lands in the context window.</p><p>Why does the distinction matter?</p><p>Because in an agentic system, the prompt you wrote might be 500 tokens. But the total context at step 47 of a long-running task might be 150,000 tokens. Your carefully crafted prompt is now 0.3% of what the model sees. The other 99.7%, tool outputs, prior conversation turns, retrieved documents, state from earlier steps, is what actually drives the model’s behaviour.</p><blockquote>If you are only engineering the prompt, you are optimising 0.3% of the input and hoping the other 99.7% takes care of itself. It will not.</blockquote><h3>The Problem: Context Rot</h3><p>Here is the uncomfortable truth about large context windows. Bigger is not better. Not always.</p><p>Research consistently shows that LLM performance <strong>degrades</strong> as context length increases. This phenomenon, called “context rot,” has been documented across every frontier model.</p><p>The <a href="https://research.trychroma.com/context-rot">Chroma research study (2025)</a> tested 18 frontier models and found accuracy drops of 20–50% when context grew from 10K to 100K tokens. All models were affected. Some decayed slower, but none were immune.</p><p>The underlying mechanism is the <a href="https://arxiv.org/abs/2307.03172">“lost in the middle” phenomenon</a>, first documented by researchers at Stanford. LLMs show a U-shaped attention curve: they attend strongly to information at the <strong>beginning</strong> and <strong>end</strong> of the context, but performance degrades significantly for information positioned in the middle.</p><p>How bad is it? On multi-document question answering, accuracy dropped by over 30% when the answer document moved from position 1 to the middle of the context. In some cases, performance with the answer document in the middle was worse than having no documents at all.</p><p>And this is not just about retrieval. A 2025 study showed that context length alone degrades performance, even when irrelevant tokens are replaced with whitespace. The sheer volume of tokens interferes with reasoning, regardless of their content.</p><blockquote>Every token you add to the context window costs you a small amount of the model’s attention. Context engineering is the discipline of making that budget count.</blockquote><h3>The Four Core Strategies</h3><p><a href="https://rlancemartin.github.io/2025/06/23/context_engineering/">LangChain’s Lance Martin</a> articulated a clean framework that maps well to how I think about the problem in practice. There are four fundamental operations in context engineering: <strong>write, select, compress, and isolate.</strong></p><h4>Strategy 1: Write (Persist Information Outside the Window)</h4><p>When humans work on complex tasks, we take notes. We do not try to hold everything in our heads. Agents need the same capability.</p><p><strong>Scratchpads</strong> are the simplest implementation. The agent writes notes to a file or a runtime state object during task execution, then reads them back when needed. It sounds trivial. It is transformative.</p><p>Anthropic’s multi-agent research system demonstrates this clearly: the LeadResearcher agent saves its plan to memory before the context window fills, because if the window exceeds 200,000 tokens, it will be truncated and the plan would be lost.</p><p>How have I seen this work in practice?</p><ul><li>A <strong>NOTES.md file</strong> that the agent updates after each major step, tracking decisions, open questions, and progress</li><li>A <strong>structured state object</strong> where specific fields persist across turns while the conversation history gets compacted</li><li>A <strong>TODO list</strong> the agent maintains and checks at the start of each new reasoning step</li></ul><p>The key insight: if information needs to survive beyond the current context window, it must be written somewhere outside of it. Relying on the context window alone is like relying on short-term memory for a project that spans days.</p><p>Products like Claude Code, Cursor, and Windsurf all implement this pattern through rules files and memory systems. It is not a research curiosity. It is production infrastructure.</p><h4>Strategy 2: Select (Pull in Only What Matters)</h4><p>Not everything the agent has written or stored deserves to be in the context at every step. Selection is about intelligent filtering: surfacing only the information relevant to the current task.</p><p>There are three types of memories agents typically select from:</p><ul><li><strong>Episodic memories</strong>: examples of desired behaviour (few-shot examples from past interactions)</li><li><strong>Procedural memories</strong>: instructions that steer behaviour (rules, guidelines, preferences)</li><li><strong>Semantic memories</strong>: facts relevant to the current task (domain knowledge, user data)</li></ul><p>The simplest implementations use fixed files that always get pulled in. Claude Code uses CLAUDE.md. Cursor uses rules files. These work for small, well-scoped memory sets.</p><p>But as the memory store grows, selection gets harder. You need semantic search, relevance scoring, or knowledge graphs to identify what matters for this specific step. The failure mode is over-inclusion: pulling in everything “just in case” and flooding the context with marginally relevant information that dilutes the signal.</p><blockquote>The goal is not “give the model all the information.” It is “give the model the minimum information needed to succeed at this step.”</blockquote><h4>Strategy 3: Compress (Summarise Without Losing Signal)</h4><p>Agent interactions can span hundreds of turns. Tool calls return enormous payloads. Conversation history grows with every exchange. Left unchecked, the context fills with stale, redundant, or low-value tokens.</p><p>Compression is about preserving meaning while reducing tokens. There are several practical patterns:</p><p><strong>Compaction</strong> is the most common. When the context approaches its limit, the system summarises the conversation history and starts a fresh window with the summary. Claude Code implements this as “auto-compact” at 95% of the context window, preserving architectural decisions and unresolved bugs while discarding redundant tool outputs.</p><p><strong>Tool result clearing</strong> is the lightest touch. Once a tool has been called deep in the conversation history, the raw result is rarely needed again. Clearing old tool outputs recovers significant token space with minimal information loss.</p><p><strong>Summarisation at agent boundaries</strong> applies when one agent hands off to another. Instead of passing the full conversation history, the outgoing agent produces a condensed summary. Cognition reportedly uses this pattern, reducing tokens during knowledge hand-offs between agents.</p><p>The critical question with any compression: what are you willing to lose? Freeform summarisation tends to silently drop details like file paths, specific numbers, or decision rationale. Structured summarisation, where the summary must populate specific sections (decisions made, files modified, open questions, current state), forces preservation of critical details.</p><p><a href="https://factory.ai/news/evaluating-compression">Factory.ai’s research</a> compared compression approaches and found that structured summarisation retained significantly more useful information than freeform alternatives. Structure forces preservation.</p><h4>Strategy 4: Isolate (Compartmentalise to Prevent Contamination)</h4><p>This strategy is about keeping context clean by giving different tasks their own separate context spaces.</p><p><strong>Sub-agent architectures</strong> are the primary implementation. Instead of one agent trying to hold everything, specialised sub-agents handle focused tasks with clean context windows and return only condensed summaries.</p><p>Anthropic’s deep research system demonstrates this: each sub-agent explores extensively, using tens of thousands of tokens, but returns only 1,000–2,000 tokens of distilled findings. The lead agent never sees the messy exploration. It sees clean summaries.</p><p>Other isolation patterns include:</p><ul><li><strong>Sandboxed execution environments</strong> where tool results live as variables in the environment, not in the context window</li><li><strong>Structured runtime state</strong> where only specific fields are exposed to the model while heavy data stays hidden</li><li><strong>Scoped tool sets</strong> where each step only sees the tools relevant to its task, preventing confusion from overlapping descriptions</li></ul><blockquote>Isolation is context hygiene. The less cross-contamination between tasks, the more focused the model’s attention at each step.</blockquote><h3>The System Prompt Trap</h3><p>One thing I want to address because I see it constantly: teams that treat the system prompt as a dumping ground.</p><p>Every edge case gets a new paragraph. Every policy gets a bullet point. Every past failure gets a new rule. The system prompt grows to 5,000 tokens, then 10,000, then 20,000. And performance gets worse, not better.</p><p>Why? Because longer system prompts consume a larger share of the attention budget. They push retrieved documents and conversation history further into the middle of the context, exactly where the model attends least. And contradictory instructions (which inevitably creep in as the prompt grows) create confusion.</p><p>Anthropic’s guidance is to find the “right altitude” for system prompts: specific enough to avoid ambiguity, but concise enough to avoid diluting attention. If a human engineer reading your system prompt finds it overwhelming, the model will too.</p><p>The test I use: can you explain what this agent does in three sentences? If not, the system prompt is probably trying to do too much and the system needs architectural decomposition, not a longer prompt.</p><h3>Why This Changes How You Design Systems</h3><p>Here is what shifted for me once I started thinking in context rather than in prompts.</p><p><strong>Before:</strong> I would try to make one agent smarter by giving it more instructions, more tools, more context.</p><p><strong>After:</strong> I design systems that keep each agent’s context small, focused, and fresh.</p><p>That shift changes everything:</p><ul><li>Instead of one agent with 15 tools, I use three agents with 5 tools each, each with a clean context scoped to its task</li><li>Instead of passing full conversation history forward, I compress at boundaries and write critical state to external storage</li><li>Instead of retrieving 20 documents “for coverage,” I retrieve 3–5 highly relevant ones and position them where the model will attend</li><li>Instead of growing the system prompt every time something breaks, I ask “what should be in context at this step and what should not be?”</li></ul><p>This is not about being clever with prompts. It is about treating context as a finite, precious resource and engineering the system to use it efficiently.</p><h3>Practical Checklist</h3><p>If you are building agentic systems, here are the questions I ask at every design review:</p><p><strong>On writing context:</strong></p><ul><li>Does the agent persist critical state outside the context window?</li><li>Is there a scratchpad, a notes file, or a structured state object?</li><li>Will progress survive if the context is compacted or a new window starts?</li></ul><p><strong>On selecting context:</strong></p><ul><li>At each step, is the agent seeing only what it needs?</li><li>Are retrieved documents scored and filtered, or dumped in wholesale?</li><li>Are tool descriptions minimal and unambiguous?</li></ul><p><strong>On compressing context:</strong></p><ul><li>What happens when the context window approaches its limit?</li><li>Is compression structured (with required sections) or freeform?</li><li>Are stale tool outputs cleared?</li></ul><p><strong>On isolating context:</strong></p><ul><li>Does each sub-task operate in its own clean context?</li><li>Do sub-agents return summaries or full histories?</li><li>Are tool sets scoped per task?</li></ul><p><strong>On system prompt health:</strong></p><ul><li>Can you explain the agent’s purpose in three sentences?</li><li>Is the system prompt under 2,000 tokens?</li><li>Have you removed rules that should be enforced architecturally rather than instructionally?</li></ul><h3>The Takeaway</h3><p>Prompt engineering taught us to write better instructions. Context engineering teaches us to architect what the model sees.</p><p>As models get smarter, the quality of the prompt matters less. The quality of the context matters more. A well-designed context on a decent model outperforms a brilliant prompt drowning in noise on the best model.</p><p>If you are building anything more complex than a single-turn chatbot, context engineering is no longer optional. It is the core discipline that determines whether your agent works at step 5 or falls apart at step 50.</p><blockquote>The model is already smart enough. The question is whether you are giving it the right information, at the right time, in the right amount.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7d00d99d2a8d" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/context-engineering-is-the-new-prompt-engineering-7d00d99d2a8d">Context Engineering is the New Prompt Engineering</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Yes, Your AI Product Hallucinates. No, the Model Is Not the Only Reason.]]></title>
            <link>https://pub.towardsai.net/yes-your-ai-product-hallucinates-no-the-model-is-not-the-only-reason-0b91d165ed00?source=rss-b8937cd0ad5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/0b91d165ed00</guid>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[hallucinations]]></category>
            <dc:creator><![CDATA[Vinayak Talikot]]></dc:creator>
            <pubDate>Thu, 19 Mar 2026 14:51:49 GMT</pubDate>
            <atom:updated>2026-04-15T04:03:03.133Z</atom:updated>
            <content:encoded><![CDATA[<h4>This is not a guide on catching hallucinations. This is about why they happen in the first place, and what your architecture has to do with it.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EWGhpZDODyY6spjrorA5ag.png" /></figure><p>A legal research tool cites a case that does not exist. A customer support bot confidently quotes a refund policy the company never had. A financial report summariser invents a quarterly figure that looks perfectly plausible.</p><p>These are not edge cases from research papers. These are production systems, built by competent teams, running on capable models.</p><p>And in every instance, the first instinct was the same: “We need a better model” or “We need better prompts.”</p><h4>That instinct is not wrong. It is just incomplete.</h4><p>I have spent years building GenAI products. The single most important lesson I have learned about hallucination is this: it is never just one thing. It is the result of decisions made across three distinct layers, from model selection to system architecture to product design. Most teams obsess over one layer and neglect the other two.</p><p>This post is not about evals. It is not about detection or mitigation frameworks. There are excellent guides on those. This is about the problem itself: where it comes from, and why understanding the full stack of decisions that shape hallucination is the highest leverage insight you can have.</p><h3>What Does Hallucination Actually Look Like in Production?</h3><p>“Hallucination” is an umbrella term that hides at least five distinct failure modes. Each one has different causes and demands a different response.</p><p><strong>Factual fabrication</strong> is the one everyone pictures. The model invents facts, names, dates, or citations. It did not misread the source. It never saw one. It generated a plausible answer because that is what next token prediction does when grounding is absent.</p><p><strong>Confident wrongness</strong> is subtler and arguably more dangerous. The model gives a definitive answer to a question it should have hedged on. There is no uncertainty signal. The user has no way to tell the difference between a correct response and a fabricated one.</p><p><strong>Source attribution errors</strong> plague RAG-based products. The model generates a correct sounding statement and ties it to a source that either does not say that thing or does not exist. A <a href="https://dho.stanford.edu/wp-content/uploads/Legal_RAG_Hallucinations.pdf">Stanford study of legal RAG tools</a> found hallucination rates between 17% and 33%, even with retrieval augmentation. The citations looked right. The substance was wrong.</p><p><strong>Logical inconsistency</strong> shows up in long form generation. Paragraph one says revenue was up. Paragraph three says it declined. Both sound authoritative. Neither flags the contradiction.</p><p><strong>Context window amnesia</strong> is specific to longer interactions. Research on the “lost in the middle” phenomenon shows that LLMs deprioritise information buried in the middle of large contexts. The model effectively “forgets” what it was given.</p><blockquote>Which of these failure modes is your product most exposed to? That question should shape every decision that follows.</blockquote><h3>Three Tiers of Hallucination Control</h3><p>Here is the mental model I wish someone had given me when I started building these systems.</p><p>Hallucination is not controlled at one layer. It is shaped by decisions across three tiers, each with a different role:</p><p><strong>Tier 1: <em>Model-level decisions</em></strong> set the floor. These determine your baseline hallucination rate before any architecture or product design comes into play.</p><p><strong>Tier 2: <em>Architectural decisions</em></strong> set the ceiling. These are the highest leverage interventions because they compound across every single interaction.</p><p><strong>Tier 3: <em>Product and organizational decisions</em></strong> determine real-world impact. These decide whether the hallucinations that do slip through cause harm or get caught.</p><p>Most teams pour all their energy into Tier 1, switch models, tweak temperature, hope for the best. Some invest in Tier 2. Very few think about Tier 3 at all.</p><p>The teams that ship AI products with consistently low hallucination rates invest across all three.</p><h3>Tier 1: Model-Level Decisions (Setting the Floor)</h3><p>Let me be clear! model selection matters. Different models hallucinate at significantly different rates on the same task, with the same prompt, and the same architecture.</p><p>A smaller, cheaper model will fabricate more frequently on knowledge-intensive tasks than a frontier model. A model fine-tuned on domain-specific data hallucinates less in that domain. Reasoning-oriented models behave differently than standard completion models. Temperature settings alone can shift hallucination rates meaningfully.</p><p>These are real levers. Picking the right model is the first and most foundational decision you make.</p><p><strong>But here is the catch.</strong> Model selection has a ceiling. You can pick the best model available and still hallucinate badly if your architecture is poor. You cannot buy your way out of a system design problem by upgrading the model. I have seen teams switch from one frontier model to another and see marginal improvement because the real issue was upstream, bad retrieval, monolithic prompts, no validation checkpoints.</p><p>Think of it this way:</p><ul><li><strong><em>Model choice sets the floor.</em></strong> It determines the best your system can possibly do under ideal conditions.</li><li><strong><em>Architecture determines how close you actually get to that floor.</em></strong> Most systems operate far above their model’s floor because the architecture introduces noise, bad context, and compounding errors.</li></ul><p>So yes, choose your model carefully. Match it to your task. Tune your parameters. But do not stop there. The rest of this post is about the decisions that have an even bigger impact on what your users actually experience.</p><h3>Tier 2: Architectural Decisions (Raising the Ceiling)</h3><p>This is where most teams underinvest and where the largest gains live.</p><h4>Prompt and Constraint Design</h4><p>Most teams focus on making prompts more detailed. What actually moves the needle is making them more constrained.</p><p>Have you ever seen a system prompt that says “Answer the user’s question helpfully and accurately”? That is practically an invitation to hallucinate. Maximum latitude. No guardrails for uncertainty.</p><p>Now compare it to:</p><blockquote>“Answer using only the provided context. If the context does not contain enough information to answer confidently, say so explicitly. Do not infer beyond what the context supports.”</blockquote><p>Same model. Same question. Dramatically different hallucination rate.</p><p><strong>The principle:</strong> every degree of freedom you give the model is a degree of freedom where hallucination can enter.</p><ul><li>Constrain the output format (structured JSON vs free text)</li><li>Constrain the knowledge source (only retrieved context vs parametric knowledge)</li><li>Constrain the uncertainty behaviour<strong> </strong>(explicit “I do not know” vs best-effort guess)</li><li>Constrain the scope (answer only about X, refuse questions about Y)</li></ul><p>The tighter the box, the less room for fabrication.</p><h4>Retrieval Pipeline Quality</h4><p>Most teams adopt RAG and assume the hallucination problem is solved.</p><p>It is not. It is relocated.</p><p>RAG shifts the failure mode from “the model made something up from training data” to “the model made something up from bad retrieval results.” And bad retrieval is far more common than teams realise.</p><p>Think about what can go wrong:</p><ul><li>Your chunking strategy splits a critical paragraph across two chunks. The retriever fetches only one half.</li><li>Your embedding model misses the semantic nuance. It returns topically related but factually irrelevant documents.</li><li>Your knowledge base is six months stale but nobody has refreshed it.</li><li>Your reranker is not tuned. The most relevant document lands at position 4 while a marginal one takes position 1.</li><li>Your metadata is incomplete. The retriever cannot filter by date, document type, or source authority.</li></ul><p>In every scenario, the model receives context that looks relevant but misleads. And it does exactly what it was designed to do: generate a fluent, confident answer grounded in whatever it received.</p><p><strong>Garbage retrieval in, confident hallucination out.</strong></p><p>If you are building a RAG product, the retrieval pipeline deserves as much architectural attention as the generation layer. Possibly more.</p><h4>Context Window Architecture</h4><p>Here is something that surprised me early on. You can give the model the right information and it will still hallucinate, because of where that information sits in the context.</p><p>LLMs disproportionately attend to information at the beginning and end of the context window. Information buried in the middle gets deprioritised. If your critical evidence is at token position 50,000 out of 100,000, the model may effectively ignore it.</p><p>This means context management is not just an efficiency concern. It is a hallucination mitigation strategy.</p><ul><li>How you order retrieved documents matters</li><li>How much context you include matters (more is not always better)</li><li>Whether you summarise or truncate matters</li><li>Where critical information is positioned matters</li></ul><blockquote>The question is not “does the model have access to the right information?” It is “is the right information positioned where the model will actually attend to it?”</blockquote><h4>Workflow Decomposition</h4><p>This is the factor I want to spend the most time on. It is the most under-appreciated and the most powerful.</p><p>Here is the pattern I see in most teams building their first GenAI product:</p><ol><li>They have a complex task</li><li>They write one big prompt that describes the entire task</li><li>They send it to the model</li><li>The output is inconsistent: sometimes brilliant, sometimes hallucinated</li><li>They make the prompt longer and more detailed</li><li>Hallucination does not improve meaningfully</li><li>They conclude the model is not good enough</li></ol><p>Sound familiar?</p><blockquote>The problem is not the model or the prompt. The problem is asking one LLM call to do too many cognitive tasks simultaneously.</blockquote><p>When a single call must retrieve context, reason over it, identify patterns, synthesise findings, and generate polished output, you create an enormous surface area where hallucination can enter at any stage. And you have zero visibility into which stage failed.</p><p><strong>The alternative: Try decomposing!</strong></p><p>Instead of one prompt that says “analyse this data and write a summary,” split it:</p><ul><li>An analyst step that examines the data and produces structured observations, hypotheses, and supporting evidence</li><li>A synthesis step that takes those observations and generates the final summary</li></ul><p>The analyst does not worry about polished writing. The synthesiser does not worry about data interpretation. Each step has a narrower scope, clearer constraints, and a much smaller hallucination surface area.</p><p>This is not a hypothetical. This pattern, separating analysis from synthesis, consistently produces more grounded outputs than a monolithic prompt.</p><p>And there are many variations of the same principle:</p><ul><li>A verification step between generation and output that checks claims against sources</li><li>An evaluator agent that scores output for groundedness before it reaches the user</li><li>A planning step that decomposes the task, followed by specialised execution steps, each with only the context it needs</li></ul><blockquote><strong>The core principle:</strong> every time you split a complex task into specialised steps with clear boundaries, you reduce hallucination surface area. Every time you add a checkpoint between steps, you create an opportunity to catch fabrication before it compounds.</blockquote><p>Is there a cost? Yes. More LLM calls mean more latency and more token spend.</p><p>But here is the tradeoff most teams miss: a monolithic prompt that hallucinates 20% of the time is not cheaper than a multi-step workflow that hallucinates 3% of the time. The cost of a hallucination reaching a user, in trust erosion, support tickets, and reputational damage, almost always exceeds the cost of an extra LLM call.</p><p><strong>There is no one-size-fits-all decomposition pattern.</strong> The right architecture depends on your task, your accuracy requirements, and your latency budget. But the principle is universal: decompose, specialise, and validate between steps.</p><h3>Tier 3: Product and Organizational Decisions (Managing Real-World Impact)</h3><p>A well-chosen model and a well-designed architecture will dramatically reduce hallucination. But they will not eliminate it entirely. Tier 3 is about what happens when hallucination inevitably slips through.</p><h4>Product Specification and Scope</h4><p>This is where I want to talk directly to PMs.</p><p>Most product managers do not think of themselves as having a role in hallucination. That is an engineering problem, right?</p><p><strong>Wrong.</strong> PMs make at least seven decisions in every AI product spec that directly determine how often the product hallucinates and how much damage it causes. Most of these decisions are made implicitly, which is exactly the problem.</p><ol><li><strong>Defining the AI’s knowledge boundary</strong></li></ol><p>Most specs say “the AI should help users with X.” Almost none say “the AI should refuse to answer Y.”</p><p>That missing boundary is where hallucination lives. Without an explicit scope, the model attempts to answer everything, including questions it has no grounding for. A legal tool that says “I cannot advise on tax law, that is outside my scope” is a product decision that prevents an entire category of fabrication. Most PMs never write that line into the spec.</p><p><strong>2. Choosing the AI’s autonomy level per task</strong></p><p>Not every feature needs the same level of AI independence. The PM decides whether the AI:</p><ul><li><strong>Drafts</strong> (human reviews before it goes out)</li><li><strong>Suggests</strong> (human picks from options)</li><li><strong>Acts</strong> (AI executes without review)</li></ul><p>Each level has a fundamentally different hallucination risk profile. “AI auto-generates and sends the customer report” is a different risk decision than “AI drafts the report for analyst review.” This is not an engineering call. This is a product call about acceptable risk.</p><p><strong>3. Designing the fallback experience</strong></p><p>What happens when the AI does not know?</p><p>Most specs do not answer this question. So the engineer makes a default choice, which is usually: let the model try anyway. The PM should define what the user sees when the AI is uncertain. Is there a graceful handoff to a human? An explicit “I do not have enough information” response?</p><p>The absence of a designed fallback path guarantees that hallucination reaches users unfiltered.</p><p><strong>4. Deciding whether AI output is presented as authoritative or suggestive</strong></p><p>“The answer is…” and “Based on available documents, it appears that…” are two completely different trust contracts with your users. The framing, the visual weight, the confidence of the language: these are product decisions. They determine whether a hallucination becomes a minor inconvenience or a trust-destroying event.</p><p><strong>5. Specifying where human checkpoints exist</strong></p><p>In multi-step workflows, the PM decides which steps require human review and which run autonomously. Every AI-to-AI handoff without a checkpoint is a product decision to accept compounding hallucination risk.</p><p>“The analysis agent feeds directly into the report generator which sends to the client” is a pipeline where no human ever validates the intermediate output. That is not an architecture choice. That is a product risk tolerance choice.</p><p><strong>6. Defining metrics that actually catch hallucination</strong></p><p>A <a href="https://www.zdnet.com/article/most-workers-dont-actually-want-ai-features-survey-shows/">ZDNET/Aberdeen survey</a> found that many users are already tired of AI features being bolted onto products. If your success metrics are only adoption, task completion, and user satisfaction, you will miss hallucination entirely. A user can complete a task, feel satisfied, and have received a fabricated answer.</p><p>If the PM does not define hallucination rate, factual accuracy, or groundedness as tracked metrics alongside the usual KPIs, nobody will measure them. <strong>What does not get measured does not get fixed.</strong></p><p><strong>7. Owning the trust contract with users</strong></p><p>Hallucination liability is real. Legal tools fabricating citations. Financial tools inventing numbers. Medical tools generating incorrect interactions.</p><p>The PM decides the product’s trust posture: is this a product that says “trust our AI” or “verify with our AI”?</p><blockquote>That single framing decision cascades into every design, engineering, and support choice downstream.</blockquote><h4>UX and Presentation Layer</h4><p>Take the exact same hallucinated output and present it two ways.</p><p><strong>Version A:</strong> Same font, same styling, same layout as verified database content. No source attribution. No “AI generated” label. No confidence signal.</p><p><strong>Version B:</strong> Subtle “AI generated” label. Inline citation links to source documents. Confidence indicator. A “verify this” affordance that lets users click through to the evidence.</p><p>The hallucination rate is identical. The harm is not even close.</p><p><a href="https://www.nngroup.com/articles/ai-hallucinations/">Research from Nielsen Norman Group</a> shows that confident presentation style from AI systems makes users significantly less likely to question incorrect answers. The design creates false authority.</p><p>Key design decisions that matter:</p><ul><li>Visual distinction between AI-generated and verified content</li><li>Inline citations that let users trace claims to sources</li><li>Confidence signals (even subtle language shifts like “it appears” vs “it is”)</li><li>Verification affordances that make it easy to check, not just read</li><li>Feedback mechanisms that let users flag inaccurate outputs</li></ul><blockquote>This is not about adding disclaimers. It is about designing for the reality that your AI will sometimes be wrong, and helping users navigate that reality rather than hiding it.</blockquote><h4>Evaluation and Feedback Architecture</h4><p>Most teams test hallucination during development, launch, and stop measuring. But hallucination rates drift. Knowledge bases go stale. User query patterns shift into unanticipated domains. The rate you measured at launch may not reflect what users experience six months later.</p><p>The architectural question is not “did we test for hallucination?” It is:</p><ul><li>Do users have a way to flag inaccurate outputs?</li><li>Does that feedback reach someone who can act on it?</li><li>Does the knowledge base get refreshed on a defined cadence?</li><li>Are production outputs sampled for groundedness on an ongoing basis?</li></ul><p>If the answer to most of these is no, you have a system that was tested once and is degrading silently.</p><h3>The Compounding Risk in Agentic Systems</h3><p>One factor crosscuts all three tiers and deserves its own callout.</p><p>In agentic workflows where multiple AI steps feed into each other, a small hallucination in step two becomes the factual premise for step three. Step three builds on it. Step four synthesises from both. By the final output, the hallucination is deeply embedded and nearly impossible to detect.</p><p>No single step looks obviously wrong. The system as a whole produces confidently wrong results.</p><p>This compounding gets worse as agent chains get longer. The response comes from every tier:</p><ul><li><strong>Tier 1:</strong> Use the most capable model for high-stakes reasoning steps, cheaper models for simple execution</li><li><strong>Tier 2:</strong> Validate between steps. Do not pass raw LLM output from one agent to the next without grounding</li><li><strong>Tier 3:</strong> Design human checkpoints at critical junctions, not just at the final output</li></ul><h3>The Takeaway</h3><p>Hallucination is not a single-layer problem. It is a system property that emerges from the interaction between your model, your architecture, and your product design.</p><p>The teams that build AI products with low hallucination rates work across all three tiers:</p><ul><li>They choose the right model for their task and tune its parameters carefully (Tier 1)</li><li>They decompose complex tasks, constrain the model’s freedom, and design their retrieval<strong> </strong>pipeline with rigor (Tier 2)</li><li>They define clear boundaries in their product spec, give users tools to verify, and treat hallucination monitoring as an ongoing concern (Tier 3)</li></ul><p>A great architecture on a decent model will outperform a poor architecture on the best model. But the best outcomes come from investing across all three tiers, not from over-indexing on any single one.</p><blockquote>The next time your AI product hallucinates, do not start by swapping the model. And do not start by tuning the prompt. Start by asking- at which tier did the chain of decisions break down? The answer will point you to the highest-leverage fix.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0b91d165ed00" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/yes-your-ai-product-hallucinates-no-the-model-is-not-the-only-reason-0b91d165ed00">Yes, Your AI Product Hallucinates. No, the Model Is Not the Only Reason.</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Every AI Agent Architecture in One Place]]></title>
            <link>https://pub.towardsai.net/every-ai-agent-architecture-in-one-place-595ba68d49cd?source=rss-b8937cd0ad5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/595ba68d49cd</guid>
            <category><![CDATA[agentic-ai-architecture]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[architecture]]></category>
            <dc:creator><![CDATA[Vinayak Talikot]]></dc:creator>
            <pubDate>Wed, 04 Mar 2026 14:42:32 GMT</pubDate>
            <atom:updated>2026-04-15T04:01:59.055Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MVfuPM5fY_n25WHnzqfIIw.png" /></figure><h3><strong>Most guides cover 4 or 5 patterns. The real landscape has over 30. Here is the full picture.</strong></h3><p>Every week, someone asks me whether they need a multi-agent system. About 80% of the time, the answer is no. But knowing when to say yes, and which architecture to reach for, requires seeing the full map. Not just the handful of patterns your framework happens to support, but the entire spectrum from classical cognitive architectures to the latest deep research agents.</p><p>The problem is that this map does not exist in one place. Academic surveys are thorough but unreadable. Framework documentation only covers what that framework sells. Blog posts recycle the same five patterns. So I decided to lay it all out, every distinct architectural pattern I have encountered, organised by what actually makes them structurally different from each other.</p><p>This is not a list of use cases like “customer support agent” or “code review agent.” Those are applications. This is about the underlying structural blueprints that make an agent behave fundamentally differently depending on which one you pick.</p><p>Let us get into it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/1*mSNs3ZVkwmr6W6_wsNswyg.jpeg" /></figure><h4><strong>The Split Nobody Talks About</strong></h4><p>Before any pattern selection, you need to understand the deepest fault line in agent design. Every architecture you will ever encounter descends from one of two lineages.</p><p>The first is the <strong>Symbolic lineage</strong>. Think BDI, SOAR, ACT-R. These systems rely on algorithmic planning, persistent state, and explicit internal logic. They dominated before 2022(roughly). They are still the right choice in safety critical domains like healthcare, robotics, and defence, where you need determinism, is auditable, and provable behaviour.</p><p>The second is the <strong>Neural lineage</strong>. Think ReAct, LangGraph, AutoGen, CrewAI. These systems rely on stochastic generation and prompt driven orchestration. Agency is not hardcoded. It emerges from how you wire prompts, tools, and memory together. This lineage took over after LLMs arrived and dominates everything you see in production GenAI today.</p><p>Most practitioners only know the neural side. That is fine for 90% of use cases. But if you ever find yourself designing for regulated industries, long-horizon planning, or safety critical decisions, the symbolic lineage has decades of solved problems you should not reinvent (Do share your experiences if you ever had a chance in that department).</p><h4><strong>The Classical Foundations (Yep! Still Relevant)</strong></h4><p>I will keep this brief because most readers want the modern patterns. But these four architectures are the intellectual ancestors of everything that follows, and understanding them prevents you from accidentally reinventing the wheel.</p><p><strong><em>BDI (Belief-Desire-Intention)</em></strong> separates what the agent believes about the world, what it wants, and what it has committed to doing. That three way split sounds academic until you realise that every LLM agent that drifts off-task is essentially an agent without an intention mechanism. If you have ever added “stay focused on the original goal” to a system prompt, you were patching the absence of BDI style commitment.</p><p><strong><em>SOAR</em></strong> runs a six step decision cycle and does something remarkable. when it gets stuck, it automatically creates a sub-goal and learns from the resolution. That “impasse creates sub-goal creates learning” loop is exactly what Reflexion reinvented decades later.</p><p><strong><em>ACT-R</em></strong> models memory with activation based retrieval where memories have strength values that decay over time. If you have ever designed a RAG system with recency weighting or relevance scoring, you were implementing ACT-R’s core insight.</p><p><strong><em>Subsumption Architecture</em></strong> (Brooks, 1986) threw out central planning entirely. Instead, it stacked simple reactive behaviour on top of each other, with higher layers overriding lower ones. No planner, no world model. Intelligence emerges from layered reflexes. This is the philosophical ancestor of swarm architectures.</p><h4><strong>Single-Agent Reasoning (How One Agent Thinks)</strong></h4><p>This is where most practitioners start, and where the choice of pattern has the most direct impact on cost, latency, and task success rate.</p><p><strong><em>ReAct</em></strong> is the default. The agent interleaves reasoning and action in a tight loop i.e., to think, act, observe the result, think again. It is flexible, intuitive, and the pattern most frameworks give you out of the box. The downside is that it plans one step at a time, which means it can be locally optimal but globally inefficient. It is also token expensive because every step requires a full LLM call with the entire conversation history.</p><p><strong><em>Plan-and-Execute</em></strong> takes the opposite approach. A planner LLM generates the entire multi step plan upfront, and a lighter executor carries out each step without consulting the expensive model again. This is faster and cheaper for structured tasks with clear sequential dependencies. The weakness is rigidity. If step 3 produces an unexpected result, the agent has no natural mechanism to adapt unless you bolt on a replan step.</p><p><strong><em>ReWOO</em></strong> (Reasoning Without Observation) pushes the separation further. The planner writes the complete plan with variable placeholders, workers execute everything in batch, and a solver synthesises the results. No mid-execution reasoning at all. This is the architecture you want for high-throughput batch processing where thousands of documents need the same multi step treatment.</p><p><strong><em>LLMCompiler</em></strong> is the speed demon. Instead of a linear plan, it generates a DAG of tasks, identifies which ones can run in parallel, and executes them concurrently through a task fetching unit. A joiner step then decides whether to answer or replan. The original paper claims a 3.6x speedup over sequential execution. The tradeoff is implementation complexity.</p><p><strong><em>Self-Ask</em></strong> is purpose built for multi hop questions. The agent asks itself “do I need a follow up question?” and recursively decomposes complex queries into simpler sub questions, each answered via search. Simple, effective, and underused.</p><h4>The Metacognitive Layer (Agents That Improve Themselves)</h4><p>These patterns add self evaluation and learning on top of the reasoning loop.</p><p><strong><em>Reflexion</em></strong> is the most important pattern in this category. After completing (or failing) a task, the agent generates a verbal self critique, identifies what went wrong, and stores that reflection in episodic memory. On the next attempt, it reads its own past critiques and adjusts. This is reinforcement learning through natural language, with no model retraining required. If your agent keeps making the same mistakes across runs, Reflexion is likely what you need.</p><p><strong><em>Evaluator Optimiser</em></strong> is the production workhorse. One LLM generates output. A second LLM evaluates it against quality criteria and provides structured feedback. The generator revises. This loop continues until the evaluator approves. Anthropic highlights this as one of the most effective patterns they see in real deployments, particularly for code generation where the evaluator can run tests.</p><h4>Search-Based Reasoning (When Linear Thinking Fails)</h4><p>Sometimes the problem space is too complex for a single chain of thought. These architectures explore multiple paths.</p><p><strong><em>Tree of Thoughts</em></strong> generates several candidate reasoning paths at each step, evaluates them, prunes the weak ones, and expands the promising ones. Think of it as giving the agent the ability to brainstorm and backtrack rather than committing to its first idea. It shines on creative problems, puzzles, and any task where the right approach is not obvious upfront.</p><p><strong><em>Graph of Thoughts</em></strong> extends this further by allowing arbitrary connections between thoughts. Ideas from different branches can be aggregated, merged, and refined in loops. The strict tree hierarchy disappears, replaced by a flexible graph where the best parts of multiple reasoning paths combine into something better than any individual branch.</p><p><strong><em>LATS (Language Agent Tree Search)</em></strong> is the heavyweight. It unifies ReAct, Reflexion, and Monte Carlo Tree Search into a single framework. The LLM simultaneously acts as the agent, the value function (scoring states), and the optimiser (selecting the best trajectory). It achieves state-of-the-art results on coding and QA benchmarks. The cost is real, LATS averages around 71 LLM calls per request compared to roughly 9 for standard ReAct. Use it when quality matters more than speed or cost.</p><h4>Orchestration Patterns (Wiring LLM Calls Together)</h4><p>These come primarily from Anthropic’s influential “Building Effective Agents” guide and represent the structural patterns for composing multiple LLM calls into workflows.</p><p><strong><em>Prompt Chaining</em></strong> is the simplest, a linear pipeline where each LLM call processes the output of the previous one, with optional validation gates between steps. Generate, then translate. Extract, then classify, then format.</p><p><strong><em>Routing</em></strong> places an LLM at the front as an intelligent dispatcher. It classifies the input and directs it to the appropriate specialised handler. Easy queries go to a smaller, cheaper model. Complex ones go to a more capable one. Customer support triage is the canonical example.</p><p><strong><em>Parallelisation</em></strong> fans out work. Either you split a task into independent subtasks and process them simultaneously (sectioning), or you run the same task multiple times and aggregate via voting for higher reliability. Document processing across pages is the classic use case.</p><p><strong><em>Orchestrator-Workers</em></strong> is the dynamic version of Plan-and-Execute at the multi call level. A central orchestrator LLM decomposes the task, spawns worker LLMs, collects their results, and adapts its plan based on what comes back. Unlike prompt chaining, the orchestrator does not follow a fixed path. It maintains a living TODO list.</p><h4>Multi-Agent Topologies (How Agents Organise)</h4><p>This is where the architecture becomes about organisational structure rather than individual cognition.</p><p><strong><em>Supervisor (Hierarchical)</em></strong> is the most common production pattern. A central supervisor agent receives the request, decomposes it, delegates to specialised sub-agents, validates their outputs, and synthesises a unified response. It mirrors a traditional org chart. The benefit is auditable and traceability. The risk is that the supervisor becomes a bottleneck.</p><p><strong><em>Handoff (Peer-to-Peer)</em></strong> eliminates the central controller. Agents transfer control to each other directly via tool calls. The billing agent decides the refund agent should take over and passes context along. This works well for conversational systems but can become chaotic without clear handoff protocols.</p><p><strong><em>Subagents (Agents-as-Tools)</em></strong> looks similar to Supervisor but has a critical difference. Subagents operate in complete context isolation. They receive only their specific input, do their work, and return results to the main agent. They never see each other’s context. This is the pattern you want when context contamination between specialists is a concern.</p><p><strong><em>Skills (Dynamic Context Loading)</em></strong> is the simplest form of specialisation. A single agent loads different prompts, tools, and knowledge on demand based on the task type. It is not truly multi-agent. It is one agent that shape shifts. Often this is all you actually need before reaching for the complexity of real multi-agent coordination.</p><p><strong><em>Swarm</em></strong> goes fully decentralised. Many simple agents communicate only with their neighbours through local signals. There is no central controller and no global state. Intelligent behaviour emerges from repeated local interactions. This is robust and scalable but extremely difficult to debug or guarantee specific outcomes.</p><p><strong><em>Debate (Adversarial)</em></strong> pits agents against each other. Multiple agents argue opposing positions and a judge synthesises the strongest answer. Structured disagreement as a feature, not a bug. This reduces hallucination through adversarial pressure and works well for high-stakes decisions.</p><p><strong><em>Crew (Role-Based)</em></strong> assigns agents explicit personas with defined responsibilities i.e, researcher, writer, editor, QA reviewer. They collaborate via message-passing on shared tasks, mimicking human team dynamics. CrewAI and MetaGPT popularised this pattern.</p><p><strong><em>Adaptive Agent Network</em></strong> lets agents self organise. No fixed hierarchy. Agents dynamically assume roles and delegate based on the task at hand. The topology reshapes itself per request. This is the most flexible but also the hardest to reason about.</p><h4>Deep Research Agents (The Emerging Frontier)</h4><p>This category has exploded over the past six months and represents a distinct architectural class built for long horizon, knowledge intensive synthesis.</p><p><strong><em>Static Pipeline</em></strong> deep research agents follow fixed stages including plan, search, read, synthesise, write. Each stage is a predefined module. Simple to build but inflexible when the research question requires adaptive exploration.</p><p><strong><em>Dynamic Deep Orchestrators</em></strong> are where things get interesting. The planner generates tasks with explicit dependency graphs. A FIFO queue manages execution order. External memory banks prevent context window overflow by extracting and storing knowledge outside the prompt. A verification step catches hallucinated plans before they execute. The synthesiser pulls from the memory bank to write the final output. This is the architecture behind systems like Gemini Deep Research, OpenAI Deep Research, and the open source mcp agent project.</p><p><strong><em>Hierarchical Multi-Agent Deep Research</em></strong> nests the patterns. A top-level planner coordinates multiple parallel research pipelines, each with its own search, validation, and extraction agents. Results converge through a synthesis orchestrator into a final pattern synthesiser that identifies cross-cutting themes and writes evidence backed reports. Google’s ADK implementation is a good reference for this pattern.</p><h3>Final Thought</h3><p>The agent architecture landscape is maturing fast. Six months ago, deep research agents barely existed as a category. A year ago, LATS and LLMCompiler were academic curiosities. Today they are appearing in production systems.</p><p>What has not changed is the fundamental principle, architecture selection is the highest leverage decision you make when building an agent system. Get it right and everything downstream becomes easier. Get it wrong and no amount of prompt engineering will save you.</p><p>I hope this map helps you make that decision with more confidence.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=595ba68d49cd" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/every-ai-agent-architecture-in-one-place-595ba68d49cd">Every AI Agent Architecture in One Place</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Self-Extending Workforce: When AI Agents Create Their Own Specialists]]></title>
            <link>https://medium.com/@vinayak.talikot/the-self-extending-workforce-when-ai-agents-create-their-own-specialists-ba024df67d65?source=rss-b8937cd0ad5a------2</link>
            <guid isPermaLink="false">https://medium.com/p/ba024df67d65</guid>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[agents-workforce]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[ai-agents-in-action]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <dc:creator><![CDATA[Vinayak Talikot]]></dc:creator>
            <pubDate>Wed, 18 Feb 2026 18:07:48 GMT</pubDate>
            <atom:updated>2026-02-18T18:07:48.179Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vu2rEbqjHUogSFEw05gmOg.png" /></figure><p><strong>Full white paper:</strong> <a href="https://github.com/vinayaktalikot/agent_in_genesis_mode/blob/main/Agents_in_Genesis_Mode_White_Paper.pdf">Agents_in_Genesis_Mode_White_Paper.pdf</a></p><p>Your AI agents have a blind spot. Here’s how to fix it.</p><p><strong>What happens when a user asks for analysis your team was never built to do?</strong></p><p>Most multi-agent systems answer that question the same way: they fail quietly, approximate with the wrong tool, or say nothing at all. The gap goes unfilled. Trust erodes. You find out weeks later that the report was incomplete.</p><p>There is another way. Instead of hitting a wall when the goal outstrips your agents, the system can detect the gap and spin up a specialist on demand. Not by building more agents in advance. By creating them at runtime, within guardrails, with humans still in control.</p><p>This is Genesis Mode. Here’s how it works and why it matters.</p><p><strong>The Problem: Static Teams Cannot Adapt</strong></p><p>Think of a typical AI agentic setup. You have a fixed roster: a metrics analyst, a risk assessor, a projection analyst. Each has a narrow scope. Each was designed at build time.</p><p>When a user submits a goal, a planner assigns tasks to this team. The output is whatever the fixed roster can deliver.</p><p>The trouble starts when the goal asks for something no agent does. For example: “Assess our competitive position and recommend one bold move with simulated impact.” Your metrics analyst reports prices and volumes. Your projection analyst forecasts growth. But no agent runs what-if scenarios. No agent models elasticity. Those dimensions are simply absent from the synthesis, or the system guesses and gets it wrong.</p><p>That’s a capability gap. The data exists. The tools exist. The reasoning does not.</p><p><strong>It’s a Reasoning Gap, Not a Tool Gap</strong></p><p>This is the crucial distinction. The gap is usually not about missing tools. The same tools may already be in your catalog. The gap is that no agent’s prompt, mandate, or methodology leads it to perform that analysis.</p><p>A metrics analyst reports numbers. It doesn’t correlate changes over time. A campaign analyst reports ROI. It doesn’t optimise allocation. The method is different. The persona is different.</p><p>Genesis Mode creates new personas, not new tools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VVW4HJrMMFHVDFiLE4F6sQ.png" /></figure><p>Genesis Mode gives a new agent a focused prompt and a minimal set of pre-approved tools. Same tool catalog. New reasoning configuration. That keeps the risk bounded while extending what the system can do.</p><p><strong>How It Works?</strong></p><p>A user submits a goal. A planner decomposes it and assigns tasks to existing agents. A gap detector compares the goal’s requirements to what those agents cover. Uncovered dimensions become gaps.</p><p>For each gap, the system generates a Just-In-Time (JIT) agent: a structured spec with a name, a prompt, allowed tools, and an output schema. No new code. No new data access. Just a new specialist built from config.</p><p>The system builds an execution graph. Agents with no dependencies run in parallel. Dependent agents run after their producers. A synthesiser combines all outputs into one report. When agents disagree, both views are surfaced, not hidden.</p><p>Dynamically created agents don’t go straight to production. They enter a candidate queue. A human reviews the spec, the output, the tools. Promote to the registry or reject. The system proposes. Humans decide.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kFMkUMPN1ZTzsy7B9L4R_A.png" /><figcaption>1.1 Agents in Genesis Mode — Flow Diagram</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZG3GiEwkKL4cMxN_7NvLsA.png" /></figure><p>Goal: “Assess competitive position in two regions. Evaluate campaign ROI, pricing power, and projection trajectory. Recommend one bold move with simulated impact.”</p><p>The planner assigns metrics, projection, and risk to existing agents. The gap detector finds two holes: sensitivity analysis (how volume responds to price) and what-if simulation. Neither is covered by the current team.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wb_FRLWvMIiYF1dWqUqt5A.png" /><figcaption>1.2 Users goal decomposed</figcaption></figure><p>The system creates two JIT agents. A Sensitivity Analyst correlates input changes with output changes. An Impact Simulator runs scenarios. The second depends on the first, so it runs after. The other agents run in parallel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sXaOQMdBOd_imrSqn0iO0A.png" /><figcaption>1.3 Gap Detector identifying the gaps</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MVsLGPB5oASscqod3xz5XQ.png" /><figcaption>1.4 JIT framework in action. Creating agents to fill missing gaps</figcaption></figure><p>Before synthesis, a critic agent checks the JIT outputs against standard-agent data. One finding gets flagged: “Simulator uses elasticity from Region A only; projection analyst covers both regions.” The flag surfaces in the report. The user sees the caveat.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lx53JPlMoIpjO4qqpNiNlQ.png" /><figcaption>1.5 Group of standard agents + JIT agents working to achive the goal</figcaption></figure><p>Both JIT agents enter the queue. The human promotes the Sensitivity Analyst (reusable). They reject the Impact Simulator (one-off). The system learns. The permanent workforce grows only when it should.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0GyRwfoRq0WiQhmi6PZ2BQ.png" /><figcaption>1.6 All JIT agents pushed to the queue for human review</figcaption></figure><p><strong>What Stays the Same, What Changes?</strong></p><p>Genesis Mode does not invent tools. Every agent uses only pre-approved, read-only tools. Each gets the minimum required for its task. Every decision is auditable: gap reason, prompt, tool calls, output.</p><p>What changes is who creates agents. Today, engineers do it at build time. In Genesis Mode, the system proposes agents at runtime. Humans approve or reject. The workforce extends when the goal demands it, not when a release cycle allows it.</p><p><strong>Where to Build It?</strong></p><p>Genesis Mode is framework-agnostic. The orchestration layer (gap detection, JIT creation, execution graph, synthesis, queue) can be implemented on LangGraph, custom Python, Azure AI, Vertex AI Agent Builder, or AWS Bedrock. Pick the platform where your data and tools already live.</p><p>The full technical specification, safety model, and implementation guidance are in the Genesis Mode White Paper. This post is the short version.</p><p><strong>Why It Matters?</strong></p><p>Static agent systems cannot adapt to goals they were not designed for. Every new analytical need used to mean another engineering cycle. Genesis Mode flips that. The system detects gaps and creates specialists on demand. Humans govern. The team grows with the business, not with the backlog.</p><p><strong>Be Mindful, Don’t Miss This</strong></p><p><strong>Tradeoffs to watch</strong></p><p>- Cost and latency. Gap detection and JIT creation add LLM calls upfront. Genesis runs are slower and more expensive than standard runs. Mitigate by checking the registry before creating, caching similar goals, and setting latency budgets.</p><p>- Over-creation. The system can spawn redundant agents. Use a value threshold, similarity deduplication, and human review to filter noise.</p><p>- Under-creation. It can miss gaps. Track the under-creation rate, tune sensitivity, and let users flag missed dimensions.</p><p>- Synthesis at scale. Too many agents (e.g., 8+) can produce conflicting outputs. Cap JIT agents, enforce an output schema, and surface contradictions rather than hide them.</p><p>- Human bottleneck. A busy candidate queue can overwhelm reviewers. Prioritise by quality and uniqueness; batch similar agents.</p><p><strong><em>What not to miss</em></strong></p><p>- Genesis Mode extends reasoning, not data access. It assumes a mature tool catalog. If the data or tools don’t exist, the gap stays unfilled.</p><p>- The candidate queue is non-negotiable. Humans must govern promotion and rejection. The system proposes; humans decide.</p><p>- Every JIT agent should be auditable: gap reason, prompt, tool calls, output. No silent creation.</p><p>Static agent systems cannot adapt to goals they were not designed for. Genesis Mode flips that. But get the guardrails right first.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ba024df67d65" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>