<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Saurabh Sarkar on Medium]]></title>
        <description><![CDATA[Stories by Saurabh Sarkar on Medium]]></description>
        <link>https://medium.com/@dreams-smoke?source=rss-798029bc6edb------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*T6jPePwv1CS387rvviweww.jpeg</url>
            <title>Stories by Saurabh Sarkar on Medium</title>
            <link>https://medium.com/@dreams-smoke?source=rss-798029bc6edb------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 13:49:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dreams-smoke/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How Running an AI Company Changed the Way I Think About Engineering Leadership]]></title>
            <link>https://medium.com/@dreams-smoke/how-running-an-ai-company-changed-the-way-i-think-about-engineering-leadership-182d37600042?source=rss-798029bc6edb------2</link>
            <guid isPermaLink="false">https://medium.com/p/182d37600042</guid>
            <category><![CDATA[teamwork]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[applied-ai]]></category>
            <category><![CDATA[ai-learning]]></category>
            <category><![CDATA[engineering-leadership]]></category>
            <dc:creator><![CDATA[Saurabh Sarkar]]></dc:creator>
            <pubDate>Mon, 02 Nov 2026 23:36:37 GMT</pubDate>
            <atom:updated>2026-11-02T23:36:37.426Z</atom:updated>
            <content:encoded><![CDATA[<p><em>What building an AI company taught me about uncertainty, evaluation, and leading teams that deliver real outcomes.</em></p><p>For three years, I built and led my own AI company. Before that, I spent much of my career in large technology organizations, working on infrastructure, platforms, and machine learning systems. Today, I’m back in big tech as an Applied AI Architect at AWS.</p><p>That journey has changed how I think about engineering leadership.</p><p>As an engineer, you’re often focused on how to build something. Running a company forces you to ask a different question: should we build it at all? Every feature competes for limited engineering capacity. Every hire is a commitment. Every architectural decision has consequences, and eventually the technology has to create value for someone or it doesn’t survive.</p><p>Now, returning to a large technology organization, I find myself looking at Applied AI through both lenses: the engineering discipline needed to build systems at scale and the startup discipline of questioning assumptions, prioritizing outcomes, and learning quickly.</p><p>But building an AI company taught me something else, and it’s what I want to write about here.</p><p><strong>Applied AI is not traditional software engineering with an LLM bolted onto the stack.</strong></p><p>Traditional software has plenty of uncertainty. But we can usually say what correct behavior looks like, even when building it is hard.</p><p>In Applied AI, we have to discover both what the system can reliably do and how to tell whether it did the right thing. We’re building products on top of components whose behavior is probabilistic, whose capabilities keep shifting, and whose failures aren’t always visible from the code.</p><p>A system works beautifully on twenty examples and struggles with the twenty-first. An agent completes a workflow several times in a row and then fails because the context changed, a tool returned something unexpected, or the model made a bad call.</p><p>And the model may have been working exactly as designed the whole time.</p><p>That’s the difference. We’re not just implementing functionality. We’re continuously discovering what the system can reliably do, under what conditions, at what cost, and with how much human oversight.</p><p>I think that changes how Applied AI teams should be built and led.</p><h3>1. The First Job Is to Reduce Uncertainty, Not Write Code</h3><p>One of the more expensive lessons I’ve learned is that starting implementation early can cost you a lot.</p><p>A customer describes a problem. The team sees an opening for an agent. Within a day we’re talking about tools, orchestration, memory, retrieval, and architecture. Before long we’re building.</p><p>And several questions never got answered. Can the model actually do this task consistently? Does the customer have the data to make it work? What does a correct result look like? What happens when the system isn’t sure? How much reliability does the workflow really need?</p><p>These aren’t implementation details. They decide whether the product is viable.</p><p>So I’ve started treating early Applied AI engineering as a process of reducing uncertainty. Before building the full solution, I’d rather run a small experiment that tells us whether the core capability is even achievable.</p><p>Take a representative dataset. Define success. Try a few approaches. Study the failure patterns. If the system isn’t reliable enough, figure out whether the limit is in the model, the context, the tools, or how we framed the problem.</p><p>Only then decide how much engineering to put behind it.</p><p>This changes what I count as progress. An engineer who spends a week proving an approach won’t work may have created more value than one who spends a month building a sophisticated prototype on an assumption nobody validated.</p><p>That doesn’t mean experimentation goes on forever. The point is to reduce uncertainty enough to make a decision.</p><p><strong>Evidence should drive architecture, not the other way around.</strong></p><h3>2. The Hardest Engineering Is Often Around the Model</h3><p>Model capabilities get most of the attention, and understandably so. Better reasoning, bigger context windows, improved tool use, and multimodal input are making things practical that weren’t a year ago.</p><p>But in much of the enterprise Applied AI work I’ve done, the model is only part of the problem.</p><p>Take an agent working with enterprise data. The model might be perfectly capable of reasoning about the task. But does it have the right context? Does it understand how the data relates? Can it tell information it should trust from information it shouldn’t? Does it have the right tools? Can we trace how it got to its answer?</p><p>And what happens when one of those assumptions breaks?</p><p>I’ve started thinking about Applied AI systems in terms of their <strong>reliability boundaries</strong>. Not whether a model can perform a task, but under what conditions the whole system can be trusted to perform it.</p><p>What context does it need? Which tools can it use safely? What happens when the input falls outside the cases we’ve tested? When should it stop, ask for clarification, or hand control back to a person? Which failures can it recover from without making things worse?</p><p>A large part of Applied AI engineering is finding those boundaries and designing the system to stay inside them.</p><p>This is where context engineering, retrieval, tool design, evaluation, observability, and failure recovery stop being nice-to-haves. We underestimate how much of the reliability problem lives outside the model. A more capable model helps, but it doesn’t fix missing context, bad data, wrong tool permissions, or a flaky workflow. Adding more agents doesn’t fix those either. Often it just makes the failure harder to understand.</p><p>So I’d want the team thinking about the whole system, not model performance in isolation. The question isn’t whether the model can produce an impressive answer. It’s whether the complete system can deliver the expected outcome repeatedly, show its work when something goes wrong, and recover sensibly.</p><p>That’s a much higher bar.</p><h3>3. Evaluation Should Drive Development, Not Follow It</h3><p>One habit I’d establish early in any Applied AI team: define success before investing heavily in implementation.</p><p>In traditional software we write tests against behavior we can specify. In Applied AI, defining correctness is itself a big part of the work. A result can be factually right but incomplete. An agent can finish every requested step and still violate a business constraint. An answer can look reasonable and be useless in the customer’s real workflow.</p><p>So evaluation has to start with the customer, not the engineering team. What does success mean for the person using this? Which mistakes are tolerable and which aren’t? How do we measure consistency? When should the system stop and ask instead of pressing on?</p><p>From there you build the representative datasets, the deterministic checks, the model-based graders, the human review loop, and the production monitoring.</p><p>But evaluation can’t be a one-time gate before launch. It has to live inside the development loop. As the system meets new data, new customer behavior, and new failure patterns, those observations should shape how the evals improve.</p><p>That doesn’t mean dumping every production failure into the eval set. The goal is to spot the failure patterns that matter, turn them into representative tests, and keep some evaluation data held out so we aren’t just optimizing for yesterday’s mistakes. Skip that step and you get a system that performs beautifully on everything it’s already seen and no more reliable in the real world.</p><p>A good evaluation process lets us test new models, prompts, context strategies, tools, and architectures, and catches regressions along the way. Over time the team learns not just whether the system works, but why it works and where it doesn’t.</p><p>I’d also measure more than answer quality. Did the agent finish the task? How often did a human have to step in? What was the latency? What did a successful outcome cost? When something went wrong, could we reconstruct the run?</p><p>Those aren’t secondary operational metrics. They’re part of whether the product is useful.</p><p><strong>Evaluation isn’t just how we test an Applied AI product. It’s how we decide what to build next.</strong></p><h3>4. Small Teams Should Own Customer Outcomes, Not AI Components</h3><p>Running a small engineering organization made me appreciate how much team structure affects how fast you learn.</p><p>With a handful of engineers and several customer commitments, you can’t create a separate function for every layer of the AI stack. You need engineers who can connect customer requirements, data, models, infrastructure, and production behavior.</p><p>Not everyone has to know everything. But I’d much rather have one small team responsible for a customer problem end to end than several teams each owning one technical component.</p><p>There’s a big difference between asking someone to build an agent that performs five steps and asking them to reduce the time a customer spends reviewing results without hurting accuracy. The first is an implementation task. The second is an outcome.</p><p>When engineers own the outcome, they can question whether an agent is even necessary. Maybe a deterministic validation step solves half the problem. Maybe better context removes the need for a complicated orchestration layer. Maybe the right answer is assistance rather than full automation.</p><p>That freedom matters.</p><p>It shapes hiring too. I’d take engineering fundamentals, curiosity, judgment, and comfort with ambiguity over expertise in any particular AI framework. Frameworks turn over fast. The ability to understand an unfamiliar problem, design an experiment, read the results, and make a sensible engineering decision is far more durable.</p><p>And I want engineers who challenge assumptions, including mine. The technology is moving too quickly for anyone to get attached to a particular architecture. The goal isn’t to prove the original idea was right. It’s to find what works.</p><h3>5. Managers Need to Run Discovery and Delivery Differently</h3><p>This is probably the biggest change I’d make to how Applied AI teams are managed.</p><p>Engineering managers have always dealt with uncertainty. Applied AI adds a specific kind: you may not know whether a model can reliably do a task until you’ve tested it. That becomes a problem when every initiative gets managed like a conventional delivery project.</p><p>How do you estimate a feature when the underlying capability hasn’t been validated? How do you commit to a date when eval results might force an architecture change? How do you assess an engineer who spent a week on an approach and concluded it wasn’t viable?</p><p>The answer, I think, is to separate discovery from delivery.</p><p>For discovery work, I want a clear hypothesis, a representative test, a time budget, and a decision we expect the evidence to inform.</p><p>For delivery work, I want predictable execution, engineering quality, operational readiness, and accountability.</p><p>The two need different expectations. A discovery experiment that disproves its hypothesis can be a success. A delivery project that keeps missing commitments because nobody validated the assumptions is a different story, and managers have to be able to tell them apart.</p><p>There’s a second question managers have to answer, though.</p><p><strong>When is discovery actually finished?</strong></p><p>Not when we have a promising demo. It’s finished when we have enough evidence to make a responsible commitment: we understand the important failure modes, we’ve measured against representative cases, we know which assumptions have held up, and we’re explicit about what still has to be learned in production.</p><p>We don’t need to eliminate every uncertainty. That’s rarely possible. But we should understand enough of the system’s reliability boundaries to know what we’re comfortable promising. And if the evidence isn’t there yet, I’d rather say so than turn an experiment into a delivery commitment and hope the remaining problems sort themselves out.</p><p>Managers should also care less about visible activity and more about the quality of the decisions their teams are making. Pull requests merged, tickets closed, agents deployed: none of that tells you whether value got created. Sometimes the most valuable thing an engineer does all quarter is stop the team from spending three more months on the wrong approach.</p><p>And managers need to create enough clarity that engineers can decide things on their own. What problem are we solving? Which constraints matter? What risks are acceptable? What evidence would change our direction? People who understand those things need far less supervision.</p><p>The principle I keep returning to is that <strong>a manager shouldn’t become the team’s most important dependency</strong>.</p><p>If every architectural decision, customer conversation, or prioritization call needs the manager in the room, the team moves at the speed of one person. I’d rather build a team that makes good decisions when I’m not there. That takes trust, technical mentorship, and being okay with people choosing an approach I wouldn’t have.</p><p>It’s not easy when you’re the one accountable for the outcome. But that’s the shift from solving problems yourself to building a team that can.</p><h3>6. Autonomy Is a Tradeoff, Not the Goal</h3><p>With all the excitement around agents, it’s easy to equate progress with autonomy. More tools, more reasoning steps, more actions completed without a human.</p><p>I get the appeal. But autonomy isn’t a useful measure of product value on its own.</p><p>If an AI system takes a four-hour workflow down to fifteen minutes, that’s already a big win. Does it matter that a human still approves the result? In many enterprise workflows, that approval is exactly what makes the system trustworthy.</p><p>I’d treat autonomy as something the system earns. Start with assistance. Automate the well-understood steps. Measure reliability. Learn the failure modes. Expand autonomy where the risk and the results justify it. And design the system so it knows when to stop, ask, or hand control back.</p><p>This connects directly to reliability boundaries. The goal isn’t an agent that can act unsupervised in every situation. It’s knowing where autonomy creates value, where it introduces risk we can’t accept, and how to keep the system inside the conditions where it operates reliably.</p><p>The same discipline applies to complexity. Not every workflow needs multiple agents. Not every problem needs the most capable model. Not every task gets better with more reasoning steps. Sometimes a smaller model with better context is enough. Sometimes deterministic software is more reliable. Often the best architecture combines both.</p><p>And it applies to cost. Every model call, tool call, retry, and reasoning step costs something. I want teams to know not just how many tokens they’re using but what those tokens are buying. What’s the cost per completed task? How much human effort is actually saved? How often does someone have to intervene? Does the outcome justify the operating cost?</p><p>These belong in the architecture discussion from the start. Because the customer doesn’t care how many agents collaborated behind the scenes. They care whether the problem got solved.</p><h3>7. Scale the Evidence Before Scaling the Organization</h3><p>Running a company with limited resources taught me that adding people doesn’t automatically solve problems. Sometimes it helps. Sometimes it adds coordination overhead, creates new dependencies, or lets an organization put off a hard prioritization call.</p><p>Before expanding an Applied AI team, I’d want evidence that we’ve found something repeatable. Do we know which customer problems we’re good at? Can we evaluate solutions consistently? Do we know what a successful outcome costs to deliver? Can we deploy and operate it reliably? Will another engineer or team actually improve what we ship?</p><p>If the answers are clear, scale. If not, I’d rather put the money into stronger evals, better tooling, clearer priorities, or removing bottlenecks.</p><p>That’s not an argument for keeping teams artificially small. It’s an argument for growing because the work justifies it, not because the budget allows it.</p><p>Large companies have something to offer here too. Startups move fast, but that speed often comes with technical debt, operational shortcuts, and too much riding on a few individuals. I wouldn’t want to recreate those problems inside big tech. The valuable part of startup thinking is the discipline around focus, ownership, experimentation, and customer outcomes. Large organizations can pair that with mature infrastructure, security, operational practice, and the ability to invest over a longer horizon.</p><p>That’s a strong combination, as long as the organization keeps its ability to make decisions quickly.</p><h3>One Last Thought</h3><p>Three years of running my own company haven’t given me a formula for building Applied AI teams. If anything, they’ve made me more aware of how much depends on the problem, the customer, the technology, and the people in the room.</p><p>But they have changed how I think about engineering leadership. Earlier in my career I spent most of my time thinking about how to build systems that scale. Now I spend just as much time thinking about how to build teams that make good decisions when the answers aren’t obvious.</p><p>That matters more than usual in Applied AI. The models will change. The frameworks will change. Some of the architecture we choose today will look unnecessary in six months. Trying to predict every shift is a losing game. Building a team that can evaluate new capabilities, understand their limits, and adapt quickly is a much better investment.</p><p>I don’t think the best Applied AI organizations will be the ones with the biggest model budgets, the most elaborate agent architectures, or the largest teams. They’ll be the ones that can consistently turn uncertain AI capabilities into reliable, measurable outcomes customers actually value.</p><p>And whether that team sits inside a startup or a company of hundreds of thousands, the leadership challenge looks remarkably similar.</p><p><strong>Reduce uncertainty before you scale execution. Build systems around outcomes, not models. And build teams that keep making good decisions as the technology changes.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=182d37600042" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What a code-history wiki actually buys a coding agent]]></title>
            <link>https://medium.com/@dreams-smoke/what-a-code-history-wiki-actually-buys-a-coding-agent-2942a02cab39?source=rss-798029bc6edb------2</link>
            <guid isPermaLink="false">https://medium.com/p/2942a02cab39</guid>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[optimization]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Saurabh Sarkar]]></dc:creator>
            <pubDate>Wed, 23 Sep 2026 00:01:25 GMT</pubDate>
            <atom:updated>2026-09-23T00:01:25.365Z</atom:updated>
            <content:encoded><![CDATA[<p><em>We asked how much a codebase’s story changes the code an agent writes, and on which dimensions. On a well-run library the answer turned out to be narrow, specific, and cheaper than we expected.</em></p><p>Here is the question we started with. A coding agent reading a repository at HEAD is a very fast new hire who has read all the code and none of the history. It sees what the code <em>is</em>. It doesn’t see that the team spent six months migrating off a pattern that still has three hundred call sites, and that adding a three-hundred-and-first would be a small act of vandalism. That knowledge lives in the git log, in pull-request discussions, in the heads of people who were there.</p><p>So we built a generator that recovers it. Point it at a git repository and it produces a multi-page wiki (module pages, architecture, glossary) plus the page we cared about, a journey.md that distils the commit history into prose: epochs, convention migrations, &quot;this used to be called X, it&#39;s called Y now, here&#39;s the PR, here&#39;s why.&quot;</p><p>It’s open source, and it’s the thing being measured in everything below: <a href="https://github.com/aws-samples/sample-code-journey-wiki-generator">aws-samples/sample-code-journey-wiki-generator </a>. There’s no service to run. It ships as agent instructions plus page templates, packaged for Claude Code (as a /wiki skill), Kiro IDE (as a spec), and Kiro CLI (as an agent). The agent does the git archaeology and writes the pages.</p><p>The question this post answers is the obvious next one: <strong>how much does that help, and on which dimensions?</strong> We set out to measure lift and break it down across convention adherence, architectural fit, historical awareness, and completeness. What we found is that on a mature, self-documenting library, most of those dimensions have no room to move, and the honest answer collapses onto a single dimension where the effect is total. The rest of this post is that answer, and the reason it’s shaped the way it is.</p><p><strong>What we set out to measure, and what we could.</strong> The plan was a clean lift-decomposition: run the same task with and without the wiki, score all four dimensions, report where the journey page helped and by how much. The corpus had other plans. Four of the five things we tried to measure were already answered by the repository itself, so the wiki had no room to move them, and no experiment on this repo could show a lift that isn’t there. The study that survived contact with the data is narrower and sharper: <em>which single dimension still has headroom, and how completely does the journey page move it?</em> Everything below is organized around that question. The answer first, then why the other four dimensions dropped out.</p><h3>The design</h3><p>Five arms, layered on top of full read access to the same repository:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/922/1*5jtODxbbceqi2qeIjN1FcQ.png" /></figure><p>The comparison we cared about was D minus C: same wiki, one page different. If the journey page does anything, it shows up there. E minus A asks the mirror question. Does the journey page do it <em>alone</em>?</p><p>The task was a <em>convention trap</em>. Take a real library (Powertools for AWS Lambda for Python, pinned to a commit) and find a place where a v2-era API was renamed in v3. Drop in a fresh file that needs the functionality. A model whose training data is thick with v2 code has a prior pulling it toward the old name. Does the context save it? Fifteen trials: three tasks, three kinds of change (a function rename, an API-shape migration, an accessor deprecation with a semantic twist), five arms.</p><p>Everything was controlled the way you’d hope. Identical checkout per arm, in-repo answer keys pruned identically across all five, seed file md5-identical, prompt <em>byte</em>-identical across arms via a masked digest, so only the files on disk differed. We re-assembled every workspace afterwards and diffed it to confirm no agent had edited the library source. Fifteen out of fifteen clean.</p><h3>The answer, up front</h3><p>Of everything we tried to measure, <strong>one dimension moved, and it moved completely.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NtQ6rBumvrt0J2zmaZyJxQ.png" /></figure><p>So the answer to “how much, and where”:</p><blockquote><strong><em>Where:</em></strong><em> exactly one dimension. The agent’s ability to ground its choice in the repository’s actual history (the dated PR, the documented reason), not just name the right API. </em><strong><em>How much:</em></strong><em> on that dimension the effect is complete. Arms with the journey page scored 3/3; every arm without it scored 0/3. D−C = E−A = </em><strong><em>+100 percentage points</em></strong><em>, consistent across all three tasks. And the journey page delivered it at </em><strong><em>19.6% of the full wiki’s injected context</em></strong><em> (2,793 vs 14,219 tokens).</em></blockquote><p>That is a smaller claim than “the wiki makes agents write better code,” and the gap between the two is the whole story. Every other dimension was already maxed out. Not because the wiki is weak, but because the repository answers those questions itself. To see why, look at what kept hitting the ceiling.</p><h3>Why only one dimension could move</h3><p>We didn’t plan to write about measurement. We planned to measure lift. But four times in a row, the instrument that was supposed to show lift turned out to have no headroom, and understanding <em>why</em> is what tells you which dimension is real.</p><h3>The trap kept defeating itself</h3><p>First smoke run: 5/5 trap avoidance in every arm, floor included. The correct API was demonstrated in the repo’s own examples/ directory; any agent that grepped found it. No history required.</p><p>We hardened it. Picked a rename where the deprecated function is still present, exported and callable, so there’s no ImportError to give the game away, and pruned every example and doc that showed the new form. Second smoke run: 5/5 again. This time the cause was fundamental. The deprecated function carried its own antidote:</p><pre>@deprecated(&quot;`extract_data_from_envelope` is deprecated; use `query` instead.&quot;, category=None)<br>def extract_data_from_envelope(data, envelope, jmespath_options=None):<br>  warnings.warn(<br>      &quot;The extract_data_from_envelope method is deprecated in V3 &quot;<br>      &quot;and will be removed in the next major version. Use query instead.&quot;,<br>      ...<br>  )</pre><p>Every agent, in every arm, opened the definition of the function it was about to call, read that line, and corrected itself. The floor arm didn’t need history. It needed to read one decorator. And you can’t prune that: it’s in the body of the code under test.</p><h3>So we asked whether <em>any</em> trap could discriminate</h3><p>Instead of hardening a third trap, we asked the prior question: on this repository, can a convention trap discriminate <em>at all</em>? We classified every migration the journey page documented on two axes. <strong>Lure</strong>: is the old form still present and callable at HEAD? <strong>Oracle</strong>: does the working tree itself name the successor, via a @deprecated annotation, a warning string, a docstring?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*48htwzdyMv-6RqlT" /></figure><p>Nine documented migrations. Five had a live lure. All five landed in Q1, tempted then corrected by the tree. Q2, the only quadrant with any code-level headroom, was empty.</p><p>This generalises well beyond our project: <strong>a library disciplined enough to ship deprecation annotations has, by that very discipline, published the answer next to the question.</strong> The better maintained the codebase, the less room any external context (your wiki, your CLAUDE.md, your RAG pipeline) has to improve code-level correctness on convention tasks. If you&#39;re evaluating a context system and you get a null result, check this first. You may not have learned that your context is useless. You may have learned that you picked a well-run repository.</p><p>We ran the full pilot anyway, so the ceiling would stand as a pre-registered prediction rather than a post-hoc excuse. Trap avoidance: 15/15, every arm, exactly as predicted. In all fifteen trials the agent found and cited the in-source deprecation marker, unprompted.</p><p>One thing to be plain about: nothing in that pilot was executed. “Trap avoided” was a regex over the diff and “correct API” was an ast.parse. No test suite was run. We fixed that later, and it&#39;s the fourth ceiling below.</p><h3>The LLM judge maxed out too</h3><p>We had a second track: an LLM-as-judge scoring each blinded submission 1 to 5 on four dimensions, one being historical awareness. Blinding was strict. A script rewrote every mention of a context file to “a project documentation file” and <em>refused to emit</em> a bundle that still leaked (it caught two real leaks on the first run). Three passes, randomised order, median per dimension.</p><p>Historical awareness came back <strong>5.00 out of 5 for all fifteen submissions, floor included.</strong> The judge wasn’t broken. Here it is explaining a perfect score for the no-context arm:</p><blockquote><em>“Names the rename explicitly, cites the </em><em>@deprecated marker and </em><em>PowertoolsDeprecationWarning, and chooses </em><em>query because it is the current name, not merely by accident.&quot;</em></blockquote><p>That is a <em>correct</em> reading of our rubric, whose score-5 anchor was “could cite why: the project moved from X to Y.” The agent cited why. It got the fact from a decorator instead of from history, and the rubric couldn’t tell the difference. The lesson, which is not about us:</p><blockquote><strong><em>An LLM-judge rubric for a context claim can be quietly satisfied by the very context you’re trying to control for.</em></strong></blockquote><p>There’s a coda worth keeping, because it’s the difference between honest and overclaiming. That 5.00 is a median of three. One pass, blind to arm, docked exactly two trials on this dimension, and <em>both were the floor arm</em>, reasoning that they “argued only from the @deprecated decorator in source, with no reference to the project&#39;s version trajectory.&quot; That is our whole hypothesis, rediscovered from the anchors alone by a rater we gave no hint to. Then the median of three erased it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BI8jFNhIn2FWd6dZ" /></figure><p>So the rubric was ambiguous, not unusable. One rater in three read the boundary the way we meant it, and median-of-three, which we picked for robustness, is exactly the wrong aggregator when the discriminating rater is the minority. Never look only at the aggregate. Report per-rater means per condition.</p><h3>And when we finally executed the code, the network answered</h3><p>A metric nobody executed is a metric nobody has tested. So we rebuilt the outcome on SWE-bench Verified, where the measure is the project’s own test suite running in a container and the grader is somebody else’s harness, on psf__requests-1766: quote the qop directive in Digest auth, per RFC 2617. (To avoid handing the agent the answer key, the wiki was generated from a clone whose history <em>ends</em> at the base commit, verified by grepping the generated pages for any post-cut identifier and refusing to run if one appeared.)</p><p>All five arms produced <em>byte-identical</em> patches, one pair of quotes, and all six graded tests passed in all five. That’s the fourth ceiling, and again we called it before the run: a one-token RFC fix has no room for context to move an executed pass/fail.</p><p>Except the harness reported four resolved and one not. The floor arm, same bytes as the others, was marked unresolved because one <em>unrelated</em> test posted to the live public httpbin service and got a 502. resolved is a single boolean AND over &quot;fixed the bug&quot; and &quot;broke nothing,&quot; and a stranger&#39;s server flipped it. Identical bytes, different verdict, and the difference was the weather. An executed metric feels more authoritative than a judge; at k=1 it was noisier. The point that survives: <strong>an executed metric is not automatically a trustworthy one.</strong> Record <em>which</em> tests failed, not just the boolean.</p><h3>The dimension that did move</h3><p>Every ceiling above is a dimension the repository can satisfy on its own. To measure the one it <em>can’t</em>, we built a metric the repository can’t fake.</p><p>The first attempt failed instructively. We keyed on version tokens: does the agent say “v3”? But the in-source warning text itself says “deprecated in V3,” so the token proves nothing about where the knowledge came from. So we went through the migration facts and kept only the ones grep confirmed were <strong>absent from the pruned tree</strong>: release dates (2024-09-23), pull-request identifiers (#5028), and the documented <em>reason</em> (&quot;too generic,&quot; &quot;per RFC 9110&quot;). Zero occurrences each. An agent that produces one of those did not get it from the code; there&#39;s nowhere else to get it. We kept the discredited version-token check in the results, labelled as a non-discriminator, precisely so the contrast stays visible.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hZzaeasVqX_mcJdEMEpgLw.png" /></figure><p>Read the two middle columns side by side. That contrast is the whole argument. “Version-aware” (does the agent name the major version) is at ceiling everywhere because the tree hands it over. “History-grounded” asks for a fact the tree does not contain, and it splits perfectly on whether the journey page was present.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*Pg3OBaAlyuXdeN4u" /></figure><p>One honest boundary on that column: it detects a <em>citation</em>. An agent that repeats a release date it was handed scores the same as one that reasoned from it. So this measures whether provenance <em>reached</em> the agent, not whether the agent <em>understood</em> it. That’s the difference between the claim we can defend and the one we wanted.</p><p>Two details make it more than a tautology.</p><p><strong>Arm C scored 0/3 despite a confound in its favour.</strong> The generated wiki <em>leaks</em>: for two of three tasks, the glossary and module pages state the migration outright. We chose not to redact them, because gutting the module pages would have manufactured the effect, and reported the leak instead. Arm C still scored zero, because those pages say <em>what</em> changed without saying <em>when, in which PR, or why</em>. You cannot ablate history out of a generated wiki by deleting one page; only the journey page carried provenance.</p><p><strong>The smallest artifact won.</strong> Arm E matched the full wiki on the only discriminating metric at 2,793 tokens against 14,219, which is 19.6% of the injected context, and cited <em>fewer</em> files than the floor arm (6.3 vs 7.0). The other 80% of the wiki bought exploration, not grounding: C minus A on history grounding is exactly zero.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*p73HQVX48DhdyFLH" /></figure><p>The page-by-page breakdown carries a practical lesson. journey.md is the single largest page in the wiki at 19.6% of its tokens, and no module page exceeds 4.6%. So the obvious optimisation, trimming the longest page, deletes the only page that worked and recovers a fifth of the injected context. The 80% you can safely drop is spread across nineteen small files. Cost and value were inversely distributed here, and you&#39;d have no way to know that without measuring the pages separately.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*zGxBm7cnkslWVird" /></figure><p>And in the executed run, the journey-only arm didn’t just cite history. It used a real pre-cut commit to <em>reject</em> a plausible wrong fix (echoing back every qop value the server offered), on the grounds that the project had already handled that case and forwarding the list would claim a protection level the client doesn&#39;t implement. The arms without the journey page rejected the same alternative from the source alone. Which is the honest shape of the whole result: <strong>the history changed the argument, not the code.</strong></p><h3>What we’ll claim, and what we won’t</h3><blockquote><em>On a self-documenting library, the journey page doesn’t change </em><strong><em>what</em></strong><em> a strong agent writes. It changes whether the agent can say </em><strong><em>why</em></strong><em> what it wrote is current. It is the only artifact we tested that transmits provenance, and it does so at a fifth of the injected context of the full wiki.</em></blockquote><p>What that <em>isn’t</em>: fifteen statically scored trials on one repo and one model, where “trap avoided” means the correct call appears in the source, not that the code runs, plus five executed trials on a single SWE-bench instance. k=1 throughout, no statistical claims. Exploration-cost numbers are self-reported by the agents (kernel-level read tracking didn’t work on our filesystem), and the token figures measure the artifact injected, not what a trial consumed.</p><p>Most importantly: we showed provenance is <em>transmitted</em>, not that it is <em>used</em>. Whether knowing <em>why</em> changes what an agent writes needs a different corpus, one with live lures and <strong>no</strong> in-source oracle (the empty Q2 quadrant). On a well-documented library like Powertools, more trials would only buy tighter error bars around a ceiling. That’s the next experiment, and it’s a corpus-recruitment problem, not a sample-size one.</p><h3>Try it on your own repository</h3><p>The most useful thing you can do with this is not take our tables on trust. Run the generator on a codebase you know and read the journey.md it produces. That page is where the measured value was concentrated, and on your own repo you&#39;ll know right away whether its account of the history rings true.</p><blockquote><a href="https://github.com/aws-samples/sample-code-journey-wiki-generator"><strong><em>aws-samples/sample-code-journey-wiki-generator</em></strong><em> </em></a><em>Point it at a git repo; it works as a Claude Code </em><em>/wiki skill, a Kiro IDE spec, or a Kiro CLI agent. No service, no dependency to install into your project: it&#39;s agent instructions plus page templates.</em></blockquote><p>If you’re evaluating whether context helps <em>your</em> agent, three things from this study are worth stealing regardless of what tool you use:</p><ol><li><strong>Audit your corpus for headroom before you run anything.</strong> The lure × oracle taxonomy took an afternoon and would have saved two experimental rounds. “Can this benchmark possibly show my effect?” is a cheaper question than “did it?”</li><li><strong>Name a fact your context uniquely supplies, and measure only that.</strong> Our valid metric came from grepping the tree for facts available <em>exclusively</em> from history. If you can’t name one, you don’t have a measurable hypothesis yet. You have a preference.</li><li><strong>Distrust instruments that flatter you.</strong> Run your no-context baseline through the judge first, look past the median at per-rater scores, and when you execute, record which tests failed rather than just the boolean. Every ceiling in this post was an instrument agreeing with us for the wrong reason.</li></ol><p>The result we wanted would have been a nicer chart. What we got is a narrow claim we’d defend in review, and a tool worth pointing at your own history. We’ll take the trade. The harness, every trial artifact, the judge bundles, and the scripts that failed are kept alongside the write-up, so you can ask for them rather than trust the tables.</p><p>Refer to previous work on TCE: <a href="https://medium.com/@dreams-smoke/curing-version-drift-with-a-delta-the-model-reads-first-317866483498?sharedUserId=dreams-smoke">https://medium.com/@dreams-smoke/curing-version-drift-with-a-delta-the-model-reads-first-317866483498</a></p><p>Code Artifact: <a href="https://github.com/aws-samples/sample-code-journey-wiki-generator">https://github.com/aws-samples/sample-code-journey-wiki-generator</a></p><p># ai #llm #ai-agents #coding #ai-for-developers #gen-ai</p><blockquote>Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2942a02cab39" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A token is not a unit of anything]]></title>
            <link>https://medium.com/@dreams-smoke/a-token-is-not-a-unit-of-anything-30886cfcd5a0?source=rss-798029bc6edb------2</link>
            <guid isPermaLink="false">https://medium.com/p/30886cfcd5a0</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[token]]></category>
            <category><![CDATA[ai-cost-optimization]]></category>
            <category><![CDATA[ai]]></category>
            <dc:creator><![CDATA[Saurabh Sarkar]]></dc:creator>
            <pubDate>Fri, 11 Sep 2026 02:53:04 GMT</pubDate>
            <atom:updated>2026-09-12T18:19:21.373Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vdN2Rs4HxqBYVxIr5yDSeA.png" /></figure><p>Every AI budget in the world is denominated in a unit that no two vendors define the same way. Until we fix that, nothing we call “optimization” is really measurement.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/1*-G-kRcj4DFg-Vd73goxNPA.png" /><figcaption>Same two words. Same meaning. Three different invoices. Splits are illustrative; measured spreads across the major providers run 1.06–1.57× on prose and up to 2.65× on tool definitions.</figcaption></figure><p>If you run AI workloads at any scale, you have a spreadsheet somewhere that multiplies a token count by a dollar figure. That spreadsheet is wrong, and not by a rounding error. It is wrong in the way a shipping quote is wrong when one carrier prices by the box and the other prices by the pallet, and nobody wrote down how big a box is.</p><p>We have spent three years getting very good at driving down the price of a token while quietly ignoring that the token itself is not a standardized quantity. Two things need normalizing, and they are different problems: <strong>the cost unit</strong>, so a dollar figure means the same thing across models; and <strong>the definition</strong>, so a token count means the same thing across models. Get both, and efficiency work becomes engineering. Without them, it stays folklore.</p><h3>The price collapsed. The bill didn’t.</h3><p>The headline trend is genuinely remarkable. GPT-4 arrived in March 2023 at thirty dollars per million input tokens. Within sixteen months, GPT-4-class quality was available for fifteen cents. Today the cheapest capable models sit around a dime per million, and the flagship tier lists at roughly a twelfth of where it started.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-U06WejgPsX24D2CyKM5vQ.png" /><figcaption>Assembled from published list prices (TokenCost’s price index: GPT-4 at $30 in March 2023, GPT-4o mini at $0.15 in July 2024, current flagships around $2.50). BenchLM’s broader frontier index, which tracks a cohort median rather than the single cheapest flagship, sat 84% below its March 2023 base as of September 2026 — a gentler six-fold decline. The projection assumes the 3–5× annual decline some analysts expect through 2027, tapering after.</figcaption></figure><p>And yet almost every team I talk to is spending more this year than last. That is not mismanagement; it is arithmetic. Agentic workloads multiply tokens per task faster than unit prices fall. One autonomous agent can burn through more context in an afternoon than a human user would in a month. Gartner projects a further 90%+ drop in per-inference cost by 2030 — and, in the same forecast, expects agentic models to consume 5–30× more tokens per task than a chatbot, so that “as token consumption rises faster than token costs fall, overall inference costs are expected to increase.”</p><blockquote><em>When your unit price is falling and your bill is rising, the one thing you cannot afford is a unit you can’t compare.</em></blockquote><h3>Problem one: the unit isn’t a unit</h3><p>Tokenization was never standardized. Each provider ships its own encoder, trained on its own corpus, optimized for its own vocabulary. The same paragraph fed to three models produces three different counts. Reported comparisons put a prompt at roughly 140 tokens under one vendor’s encoder and above 180 under another, and for some content types — tool definitions especially — the spread reaches two to three times.</p><p>Which means the number on the pricing page is not a price. It’s a price per box, and nobody told you the box size.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PeCMB1s8n9wPT2q0CrnpaA.png" /><figcaption>Illustrative, using token-count spreads reported across provider tokenizers. The vendor with the lowest advertised rate is the most expensive for this content — a result the pricing page cannot show you.</figcaption></figure><h3>Problem two: nobody agrees what a token is</h3><p>This one is subtler and, I think, more damaging. Even if every vendor used an identical encoder, they would still be counting different things. The token has quietly fragmented into a family of token types, each with its own price and its own visibility rules.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VIG2hZ2bRjl4NCgVKIaXeQ.png" /></figure><p>The reasoning row deserves special attention. You are billed for tokens you cannot see, generated by a process you cannot inspect, at output rates. Academic work has already flagged the obvious governance gap here: a 2025 framework called CoIn proposes a trusted third-party auditor for hidden reasoning tokens precisely because users have no way to verify counts they’re being charged for. When the largest line item in your bill is unobservable, “optimization” is a guess with a dashboard attached.</p><h3>What the missing standard actually costs you</h3><ul><li><strong>Model selection becomes theatre.</strong> You cannot rank models by cost if the denominators differ. Teams routinely pick the cheaper price sheet and pay more.</li><li><strong>Forecasts drift.</strong> A vendor can ship a new tokenizer and the same prompt costs more overnight: Anthropic’s own guidance is that its 4.7-generation models produce roughly 30% more tokens than their predecessors, with code and JSON closer to 35%. Budgets built on last quarter’s ratio quietly decouple.</li><li><strong>Attribution fails.</strong> In multi-tenant products, cost per customer is derived from token counts. If those counts aren’t comparable across the models you route between, your unit economics are fiction — and so is your pricing.</li><li><strong>Optimization can’t be validated.</strong> Migrate a workload and save 30%? Some of that is your engineering. Some is a friendlier tokenizer. You cannot separate the two, so you cannot repeat the win deliberately.</li><li><strong>Internal models get a free pass.</strong> Self-hosted inference gets compared on GPU-hours while API models get compared on tokens. The build-versus-buy decision is made across two incompatible units.</li></ul><h3>Where this ends if nothing changes</h3><p>Extrapolate the current trajectory and the divergence widens rather than closing. Per-token prices keep falling. Token consumption per task keeps rising as agents get longer horizons. Token <em>types</em> keep proliferating — every new capability arrives with a new meter. The gap between the number on the pricing page and the number on the invoice becomes the dominant term in AI budgeting.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hznSZ60jmI2ObzV7kR1n-g.png" /><figcaption>The paradox that defines AI cost management right now: unit prices in freefall, aggregate bills climbing, and no shared unit in which to explain the difference to a CFO.</figcaption></figure><h3>What normalization looks like in practice</h3><p>This is not a call for vendors to adopt a common tokenizer. They won’t, and they shouldn’t — tokenizer design is a real part of model quality. The fix belongs in the measurement layer, and it has two halves.</p><h3>A normalized cost unit</h3><p>Define a reference unit — call it a normalized token unit — computed by applying one canonical encoder to the semantic payload of a request. Vendor tokens remain how you’re billed. The reference unit becomes how you compare. Effective cost is then simply what you paid divided by the normalized units you actually transacted, and suddenly two models sit on the same axis.</p><pre>effective_rate = (vendor_tokens × vendor_price) / reference_units</pre><pre># both models, same denominator, finally comparable<br>model_a: (140 × $3.00/M) / 100 NTU = $4.20 per kNTU<br>model_b: (265 × $2.50/M) / 100 NTU = $6.63 per kNTU</pre><p>This is not a novel idea and that’s a point in its favour. Antarctica’s One-Token Model proposes exactly this kind of provider-independent normalization layer — a reference token sitting above vendor-specific tokens — on the grounds that “a token in one system is not the same as a token in another.” The finance side is moving the same direction: the Linux Foundation, in partnership with the FinOps Foundation, announced the Tokenomics Foundation ahead of FinOps X 2026 and formally launched it in August with thirty founding members to set open standards for AI billing, and native token tracking is scoped into FOCUS 1.5, the next revision of the cloud cost specification. The plumbing is arriving. What’s missing is adoption inside engineering orgs.</p><h3>A normalized token definition</h3><p>The second half is a schema. Every token event your system emits should be tagged along a few orthogonal axes, so that “we used 4 million tokens” becomes a statement with actual content:</p><pre>token_event {<br>  phase      input | output<br>  cache      miss | write | read<br>  visible    emitted | internal<br>  origin     user | system | tool | retrieval | reasoning<br>  modality   text | image | audio | document<br>  vendor_n   &lt;billed count&gt;<br>  ref_n      &lt;normalized count&gt;<br>}</pre><p>Capture this at your own gateway, not from vendor invoices. Invoices arrive late, aggregate aggressively, and use each vendor’s taxonomy. A metering layer in front of every model — internal and external alike — is the only place the counting is under your control.</p><blockquote><em>The organizational version of this matters as much as the technical one. Internal models are usually costed in GPU-hours by the infra team while external models are costed in tokens by whoever owns the API key. Until both land in the same unit, in the same ledger, you don’t have an AI budget — you have two budgets that happen to fund the same product.</em></blockquote><h3>What you get back</h3><p>Standardization sounds like hygiene. It isn’t. It’s the precondition for every efficiency lever worth pulling:</p><ul><li><strong>Routing becomes calculable.</strong> Send each request class to the genuinely cheapest model for that content, rather than the one with the friendliest marketing.</li><li><strong>Prompt and context work gets a measurable return.</strong> You can tell caching wins from tokenizer luck, and repeat the ones that were real.</li><li><strong>Reasoning budgets become a design parameter.</strong> Once internal tokens are first-class in your schema, “how much thinking should this task get” turns into a tunable number instead of a surprise.</li><li><strong>Cost per outcome becomes reachable.</strong> The metric FinOps practitioners keep asking for — cost per verified outcome, per successful task — is the one almost nobody can calculate today, because it requires a stable denominator underneath. Normalization is that denominator.</li><li><strong>Build-versus-buy gets an honest answer.</strong> Self-hosted and API inference finally compare on one axis.</li></ul><p>The industry spent three years optimizing the price of a token. The next few years belong to whoever can say, with a straight face and a defensible number, what a token actually was.</p><h3>References</h3><ol><li>Sun et al. <a href="https://arxiv.org/abs/2505.13778">CoIn: Counting the Invisible Reasoning Tokens in Commercial Opaque LLM APIs</a> (2025) — a verification framework for auditing hidden reasoning-token billing.</li><li><a href="https://arxiv.org/abs/2606.24616">AI Tokenomics: The Economics of Tokens, Computation, and Pricing in Foundation Models</a> — on token accounting as a model-agnostic unit and the variation that undermines it.</li><li><a href="https://antarctica.substack.com/p/the-one-token-model">The One-Token Model</a>, Antarctica — proposes a normalized reference token independent of provider tokenizers, spanning cost, energy and emissions.</li><li><a href="https://www.tensorzero.com/blog/stop-comparing-price-per-million-tokens-the-hidden-llm-api-costs/">Stop comparing price per million tokens</a>, TensorZero — identical inputs measured through each provider’s own token counter; 1.06–2.65× spreads by content type.</li><li><a href="https://www.digitalapplied.com/blog/llm-tokenizer-cost-variance-reference">Tokenizer Variance: Why Identical Prices Cost More</a>, Digital Applied — including Anthropic’s ~30% token-count increase on 4.7-generation models.</li><li><a href="https://www.finops.org/insights/token-economics-the-atomic-unit-of-ai-value/">Token Economics: The Atomic Unit of AI Value</a>, FinOps Foundation.</li><li><a href="https://www.nops.io/blog/finops-x-2026-day-1-keynote/">FinOps X 2026 Day 1 keynote recap</a>, nOps — the token bill as one of nine cost layers, and cost per verified outcome as “the metric everyone wants but almost no one can calculate.”</li><li><a href="https://www.finops.org/insights/introducing-focus-1-4/">Introducing FOCUS 1.4</a>, FinOps Foundation — ratified June 2026; scopes native AI token support into FOCUS 1.5.</li><li><a href="https://www.linuxfoundation.org/press/linux-foundation-launches-the-tokenomics-foundation-to-define-the-economics-and-roi-of-ai-value">Linux Foundation Launches the Tokenomics Foundation</a> (August 4, 2026) — formal launch, founding members, and the plan to deliver token cost telemetry in FOCUS.</li><li><a href="https://www.hpcwire.com/aiwire/2026/03/25/gartner-forecasts-90-drop-in-llm-inference-costs-by-2030/">Gartner Forecasts 90% Drop in LLM Inference Costs by 2030</a>, HPCwire — including the 5–30× tokens-per-task figure for agentic models.</li><li><a href="https://benchlm.ai/llm-pricing-trends">LLM API Pricing Trends</a>, BenchLM — the frontier token price index cited in the first chart.</li><li><a href="https://tokencost.app/blog/ai-price-index">AI Price Index 2023–2026</a>, TokenCost — timeline of major list-price changes across providers.</li><li><a href="https://www.silicondata.com/blog/llm-cost-per-token">Understanding LLM Cost Per Token</a>, Silicon Data — on cross-provider token count variance and output/input price ratios.</li></ol><blockquote>Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=30886cfcd5a0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Curing version drift with a delta the model reads first | Tool-Calling Efficiency (TCE)]]></title>
            <link>https://medium.com/@dreams-smoke/curing-version-drift-with-a-delta-the-model-reads-first-317866483498?source=rss-798029bc6edb------2</link>
            <guid isPermaLink="false">https://medium.com/p/317866483498</guid>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Saurabh Sarkar]]></dc:creator>
            <pubDate>Mon, 27 Jul 2026 23:43:03 GMT</pubDate>
            <atom:updated>2026-08-31T20:08:32.320Z</atom:updated>
            <content:encoded><![CDATA[<p>If a coding agent fails because its knowledge of a tool is frozen at training time, you shouldn’t need to retrain it. Hand it the current form of the command at inference time. We tested whether that works, and what it saves.</p><blockquote>Series 2/2. 29 models . Two experiments . Bedrock · July 2026</blockquote><h3>First, what “version drift” is</h3><p>Every model has a knowledge cutoff, the date its training data stops. Command-line tools keep changing after that date. So a model can be perfectly smart and still type a command that was correct when it was trained but is wrong today. The textbook case: docker-compose up (a hyphen, the old v1 binary) became docker compose up (a space, a subcommand) years ago, yet models trained on older data still emit the hyphen form. The command fails not because the model reasoned badly, but because its <em>factual knowledge of the tool</em> is stale. We call this version drift.</p><p>In an <a href="https://medium.com/@dreams-smoke/when-the-models-memory-goes-stale-measuring-cli-version-drift-tool-calling-efficiency-tce-c0eb53588a11?sharedUserId=dreams-smoke">earlier post</a> we measured how common this is across 29 models on 30 version-sensitive tasks, and found that the staleness of a model’s training cutoff predicts how often it drifts, at Pearson r = −0.88 (the older the cutoff, the more failures). That post diagnosed the disease. This post is about the fix, and about being honest where the fix is still a projection rather than a live measurement. You don’t need to have read the first post to follow this one.</p><h3>The thesis</h3><p>If drift is caused by knowledge frozen at training time, the fix should not require retraining: hand the model a delta, “the current form of this command is X”, at inference time. The question is whether that injection (a) raises command correctness and (b) net-saves tokens once its own context cost is paid.</p><p>Those are two separate claims, and we test them with two separate experiments, because a raw model and a real coding agent fail differently. A raw model gets one shot: wrong command → FAIL. A coding agent <em>runs</em> the command, sees the error, and retries, so drift shows up there not as failure but as waste, the failed attempt plus the error plus the retry. The first experiment measures correctness; the second measures wasted tokens.</p><h3>Experiment one, does the delta fix correctness?</h3><p>The first experiment is a single-response test across many models: give each of the 29 models a version-sensitive task, capture the one command it emits, and grade it with an executable oracle, the same setup as the earlier diagnosis. Now we add three ways of helping the model: Control (no help), arm A (prepend the relevant delta to the system prompt, so it’s there before the model acts), and arm C (retrieve the delta and retry only <em>after</em> a first-attempt failure). A “delta” here is just a short note of the form <em>“the current form of this command is X.”</em> We ran each two ways: an <em>oracle</em> config that feeds the exactly-right delta (the ceiling of what’s possible) and an <em>exact-match</em> config that must <em>retrieve</em> the right delta from a shared store on its own, the realistic pipeline.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z7uKlPYYycJ1C-B5YB_gzQ.png" /><figcaption>Each stem is one model. Grey dot = control pass rate; colored dot = with the delta injected. Colored by era (red = 2023 cutoff, amber = 2024, green = 2025+). The delta pulls almost every model up to the ~95% band.</figcaption></figure><p>ConfigControlA · system-promptC · retry-on-failOracle (ceiling)58.5%95.3% (+36.8)96.1% (+37.6)Exact-match (realistic)58.6%95.1% (+36.4)95.7% (+37.1)</p><p>The delta lifts first-attempt command correctness from ~59% to ~95%, a +37-point swing. And the realistic exact-match retriever nearly matches the oracle ceiling, so the gain isn’t an artifact of a hand-fed answer key, the store actually <em>finds</em> the right delta on its own. (Arm A and arm C land in the same place here because a single-response test has no real retry loop yet; the two arms diverge in the second experiment, where retries cost real tokens.)</p><h3>The number that closes the loop</h3><p><strong>−0.97</strong></p><p>Plot each model’s baseline correctness against how much the delta helps it, and the points fall on a line: Pearson r = −0.97. The stalest models gain most, Llama 3.1 8B and Nova Micro climb from ~35% to nearly 100%, a +60-point rescue, while the newest models barely move (Opus 4.7: 90% → 97%) because they had almost nothing to cure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MpFg_PrZVlVFs28FL6AMEA.png" /><figcaption>Dose-response. The worse a model does on its own (left), the more the delta buys it (up). Fit line r = −0.97, n = 29.</figcaption></figure><p>The earlier diagnosis found staleness <em>predicts failure</em> at r = −0.88. The fix’s mirror: the delta’s <em>benefit</em> is predicted by that same staleness at r = −0.97. The fix undoes precisely the deficit the drift created.</p><h3>Experiment two, does it save money in a real agent?</h3><p>Correctness is the single-response story. The second experiment is a real coding-agent test: instead of grading one command, we let an actual agent work a task end to end. In a real agent the fix is supposed to erase the <em>retry loop</em>, the wrong command, the error, the correction, so what matters is tokens, not just pass/fail. We put a real coding agent (Claude Code) inside a container where every CLI is at its current version, wired the injection into a hook that fires just before each command runs, and measured turns and tokens to success with injection off versus on.</p><p>Here we have to be honest about what the first live run found, which is a null result, and an informative one. On a current-cutoff model (Claude Haiku 4.5), across 77 shell commands the agent actually ran, zero carried stale syntax. The agent already knows the current CLIs and self-corrects, so there was no retry loop to eliminate, and injecting anyway just added context tokens. That is exactly what the −0.97 curve predicts for a model with a recent cutoff, the models that barely drift have almost nothing to cure. The harness works end to end and the injection channel is verified live; what it lacks is a <em>drift-exhibiting agent</em> to measure against.</p><p>Building it turned up one more instance of the phenomenon eating its own tail: Claude Code’s hook API had itself drifted. Our injector used the old {&quot;decision&quot;:&quot;approve&quot;} output schema and was a silent no-op on the current version until we caught it and switched to hookSpecificOutput.additionalContext. Version drift, all the way down.</p><h3>So we projected the savings, honestly</h3><p>The real-agent test is Claude-only by design (the agent <em>is</em> Claude Code), and every current Claude model is recent-cutoff, so no live run can manufacture drift that isn’t there. Rather than fake it, we project the token savings from measured inputs. The model is one line:</p><pre>net_gain(task) = P(drift) · retry_loop_cost_saved − injection_overhead</pre><p>Every term is measured except one:</p><ul><li>P(drift) per model = 1 − control pass rate, straight from the single-response test (13% for the newest model, up to 67% for the stalest).</li><li>injection overhead = the extra input tokens each arm costs, measured in the single-response test (~110 tokens for arm A, paid always; less for arm C, paid only on failure).</li><li>per-turn agent cost = what one real Claude Code turn burns, measured from the real-agent control runs (~$0.005/turn).</li><li>R = retry turns per drift, the one thing the live run couldn’t measure (Haiku never drifted). So we don’t invent a point estimate, we <em>sweep</em> it over {1, 2, 3} and report the whole band.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JccYDEmpqOfvTE_99YAlaw.png" /><figcaption>Projected net $/task saved (arm C), by model cutoff. Three bands = three assumptions about how many turns a drift burns (R = 1, 2, 3). Every band stays above zero, and rises toward the stalest models.</figcaption></figure><p>Retry cost RMean net (arm A)Mean net (arm C)Range across models1 extra turn+$0.0020+$0.0021+$0.0006 → +$0.00332 extra turns+$0.0041+$0.0041+$0.0013 → +$0.00673 extra turns+$0.0061+$0.0062+$0.0019 → +$0.0100</p><p>Three things fall out, and all three match the correctness story:</p><ul><li>Net-positive everywhere. Even the newest model at the cheapest retry assumption saves. One skipped retry turn (~$0.005) dwarfs the ~110-token injection cost, so overhead never exceeds the loop it prevents.</li><li>Arm C ≥ arm A everywhere. C pays nothing on tasks the model already gets right, so it wins outright where drift is rare (the newest models) and ties A where almost everything drifts.</li><li>Savings scale with staleness, the token-side mirror of the −0.97 correctness curve. The stalest models save 3–5× what the newest do.</li></ul><p>The live null and the projection are the <em>same result</em> seen from two ends. Current Claude sits at the far-right, low-drift edge of this curve, where projected saving is smallest, which is exactly where the live run measured ~zero. The cure isn’t unnecessary; it’s unnecessary <em>yet</em>, for the frontier model, on the tasks it already knows.</p><h3>What the projection is, and isn’t</h3><p>It is a projection, not a live agent measurement. The load-bearing assumptions, R and a constant per-turn cost, come from a 13-run, single-model, single-repeat real-agent test. Treat the <em>shape</em> (positive, staleness-scaled, C ≥ A) as the finding and the absolute dollars as order-of-magnitude. Measuring it live needs an agent that actually drifts, an older-cutoff model in the loop, or a synthetic-drift harness that strips a current model’s version knowledge. That’s the next build.</p><h3>How the fix actually works</h3><p>There’s no retraining and no model surgery. A PreToolUse hook sees the command the agent is about to run, checks it against a small delta store (regex patterns for known stale forms → the current form), and on a match hands the agent a note before the command executes:</p><pre>⚠️ this command may use STALE CLI syntax (class-8 drift):<br>  • [docker-compose-v1-to-v2] `docker-compose` (hyphen) is the v1<br>    binary; v2 is the `docker compose` subcommand.<br>    suggested: docker compose up -d</pre><p>The store is deliberately dependency-free and self-refreshing: the PostToolUse hook harvests <em>stale → correct</em> command pairs whenever the agent fails and then succeeds at a fingerprint-matched command, so real sessions grow the store for free. The whole mechanism is packaged to drop into any Claude Code project, see the repository below.</p><h3>What’s next</h3><ul><li>Get the measured dollar number. Put a drift-exhibiting agent in the loop (older model or synthetic-drift harness) and measure net tokens saved live, to confirm the projected shape.</li><li>Expand past the conservative suite. Today’s 30 tasks skew to the lowest-drift class and lack a dedicated non-drift control stratum, so “without hurting outcomes” is only lightly tested. Add high-drift renames plus a clean control set.</li><li>Embedding retrieval as a second stage. Exact-match can only understate the fix (a miss injects nothing); an embedding retriever trades higher recall for mis-fire risk, a separate question that doesn’t gate this result.</li></ul><p>Method notes: single-response test, 29 models on AWS Bedrock, 30 version-sensitive CLI tasks, 3 arms × 2 retrieval configs, 960 graded rows each, executable-oracle graded, correlation is Pearson r over per-model averages (n = 29). Real-agent test, real Claude Code (2.1.212) headless in a current-CLI container, Haiku 4.5, 13 tasks × 2 arms. Savings are projected from the single-response drift rates and the real-agent per-turn costs with retry depth swept over {1,2,3}; see simulate_savings.py and TRACK_A_SIM_FINDINGS.md. Charts generated numerically from the result JSON via make_cure_charts.py.</p><blockquote>Series 1/2: <a href="https://medium.com/@dreams-smoke/when-the-models-memory-goes-stale-measuring-cli-version-drift-tool-calling-efficiency-tce-c0eb53588a11?sharedUserId=dreams-smoke">https://medium.com/@dreams-smoke/when-the-models-memory-goes-stale-measuring-cli-version-drift-tool-calling-efficiency-tce-c0eb53588a11</a></blockquote><p>Code Artifact: <a href="https://github.com/aws-samples/sample-improve-ai-tool-call-efficiency">https://github.com/aws-samples/sample-improve-ai-tool-call-efficiency</a></p><p># ai #llm #ai-agents #coding #ai-for-developers #gen-ai</p><blockquote><em>Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=317866483498" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Token Standard: Why AI Tokens May Become the New Unit of Work]]></title>
            <link>https://medium.com/@dreams-smoke/the-token-standard-why-ai-tokens-may-become-the-new-unit-of-work-df0c4a9ef5c5?source=rss-798029bc6edb------2</link>
            <guid isPermaLink="false">https://medium.com/p/df0c4a9ef5c5</guid>
            <category><![CDATA[tokenomics]]></category>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[agents]]></category>
            <dc:creator><![CDATA[Saurabh Sarkar]]></dc:creator>
            <pubDate>Fri, 10 Jul 2026 23:13:39 GMT</pubDate>
            <atom:updated>2026-07-10T23:13:39.249Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*CibdAnbQt1yZOGcT" /></figure><p>We are watching a new unit of account being born, and most people haven’t noticed yet.</p><p>It isn’t a cryptocurrency. It isn’t a central bank experiment. It’s the AI token, the unit of computation that powers models like Claude, GPT, and Gemini. Today it’s a line item on an engineering budget. My thesis: tokens won’t become money. They will become something arguably more important, the standard metering layer for AI-augmented work, the unit in which cognitive labor gets budgeted, allocated, and measured. And the companies that learn to govern token-to-output ROI first will outperform everyone else.</p><p>Here’s the data.</p><h3>Tokens are already showing up in compensation</h3><p>This stopped being a thought experiment in March 2026, when Nvidia CEO Jensen Huang proposed at GTC that engineers should receive roughly half their base salary again, in tokens <a href="https://techcrunch.com/2026/03/21/are-ai-tokens-the-new-signing-bonus-or-just-a-cost-of-doing-business/">TechCrunch</a>. He wasn’t ahead of the market; he was describing it. Some reports claim over 40% of tech companies now include some form of AI credit allocation in employee benefits, up from under 5% eighteen months earlier, a figure circulating in secondary coverage that I haven’t traced to a primary survey, so treat it as directional. What’s better documented is the behavior underneath it.</p><p>The compensation math is already material. Venture capitalist Tomasz Tunguz calls AI inference the <em>fourth component</em> of engineering compensation, alongside salary, bonus, and equity: with 75th-percentile engineer salaries at $375,000 and annual inference costs of $100,000, tokens now represent over 20% of fully loaded cost <a href="https://zenvanriel.com/ai-engineer-blog/token-budgets-ai-engineers-compensation-model/)">Zen van Riel’s analysis</a>. Microsoft’s Charles Lamanna described a job candidate who made a daily token budget an explicit condition of accepting the offer, and argued that $100K/year in tokens for a $500K engineer is a great trade if it triples their output <a href="https://thelettertwo.com/2026/03/26/ai-token-budgets-employee-compensation/">The Letter Two / GeekWire</a>.</p><p>And people are behaving the way they behave around any scarce, valuable resource. The New York Times reported on “<em>tokenmaxxing</em>”, engineers at Meta and OpenAI competing on internal leaderboards that track token consumption. Stripe CEO Patrick Collison has warned about token pilfering. Teradata’s CEO says some lead developers burn their entire monthly allocation in three hours <a href="https://fortune.com/2026/05/20/ceos-ai-tokens-paychecks-justify-spend/">Fortune, May 2026</a>. When a resource is hoarded, gamed, stolen, and status-tracked, it has become a real unit of allocation, whether or not it ever becomes money.</p><h3>Why tokens are the natural unit for cognitive labor</h3><p>A useful unit of work allocation needs three properties: it must map to something universally needed, it must be scarce enough to force real allocation decisions, and it must be measurable. Tokens are becoming all 3, and the macro numbers prove it.</p><p><strong>The demand is enormous and compounding.</strong> Global AI spending is forecast to reach roughly $2.5 trillion in 2026, up 44% year over year <a href="https://www.finout.io/blog/predicting-ai-spending-in-2026-what-the-data-actually-tells-us">Finout</a>. Average enterprise AI budgets have grown from $1.2M in 2024 to $7M in 2026, with some Fortune 500 companies reporting monthly inference bills in the tens of millions <a href="https://oplexa.com/ai-inference-cost-crisis-2026/">FinOps Foundation data via Oplexa</a>.</p><p><strong>Scarcity persists even as unit prices fall — the Jevons paradox.</strong> Per-token costs have collapsed roughly 280x in two years, from ~$60 per million output tokens at GPT-4’s launch to fractions of a cent <a href="https://tech-insider.org/big-tech-ai-infrastructure-spending-2026/">Tech Insider</a>. Yet total spend keeps rising, because agentic workflows consume 100–1,000x more tokens per task than a simple chat query <a href="https://authoritytech.io/curated/inference-economics-changed-enterprise-ai-buying">AuthorityTech, citing VentureBeat</a>. Gartner projects inference costs falling over 90% by 2030, while explicitly warning that consumption will rise faster than costs fall, so total inference spend <em>increases</em> <a href="https://www.gartner.com/en/newsroom/press-releases/2026-03-25-gartner-predicts-that-by-2030-performing-inference-on-an-llm-with-1-trillion-parameters-will-cost-genai-providers-over-90-percent-less-than-in-2025">Gartner, March 2026</a>. Cheap units still force allocation decisions when everyone needs exponentially more of them.</p><p><strong>It’s measurable in a way labor never was.</strong> Every token maps to compute consumed and work produced. Google’s internal Antigravity platform went from half a trillion tokens per day in March 2026 to over three trillion by mid-May <a href="http://investing.com/">Investing.com</a>, citing <a href="https://www.investing.com/analysis/the-ai-token-pricing-crisis-behind-openai-and-anthropics-revenue-race-200680777">VentureBeat</a>. You cannot meter “productivity.” You can meter tokens perfectly.</p><p>In short: tokens are becoming the new budget line for cognitive labor. That framing is defensible today, not a prediction.</p><h3>What’s happening now and what’s still hypothesis</h3><p>It’s worth being precise about which parts of this shift are documented and which are extrapolation.</p><p><strong>Documented: software is repricing itself in token-linked units.</strong> GitHub moved Copilot from flat-rate subscriptions to usage-based AI Credits tied directly to token consumption in June 2026, with one developer’s projected monthly cost jumping from roughly €67 to €966 under the new model <a href="https://www.investing.com/analysis/the-ai-token-pricing-crisis-behind-openai-and-anthropics-revenue-race-200680777">Investing.com</a>. To be clear, this is usage-based billing, not vendor settlement in tokens, but when the world’s largest developer platform bills in token-linked units, the <em>metering</em> layer has already migrated.</p><p><strong>Documented: CFOs are allocating in this unit.</strong> Teradata’s CEO frames capital allocation as a live choice between agentic spend, tokens, and human spend, deploying wherever return is highest. One analysis puts current token costs at roughly 4x compensation on a usage basis against an expected 10x productivity return <a href="https://fortune.com/2026/05/20/ceos-ai-tokens-paychecks-justify-spend/">Fortune</a>. Humans and tokens are being priced on the same ledger line. That’s the ROI-governance discipline this whole thesis turns on.</p><p><strong>Hypothesis: vendor and contractor relationships denominated in tokens.</strong> This has not happened yet, and I want to label it clearly as a prediction. But the logic is straightforward: a design shop, a law firm, a data-labeling vendor, each is increasingly a thin human layer wrapped around token consumption. One Ericsson engineer reportedly already spends more on Claude than he earns in salary, with his employer covering the difference. If the dominant input cost of a service is tokens, quoting and budgeting that service in tokens is a small step. Watch for the first agency that prices a project as “N billion tokens plus a human premium.”</p><p><strong>Explicitly not my claim: tokens replacing the dollar.</strong> Currency requires transferability, fungibility, store of value, legal and tax treatment, and broad acceptance. AI tokens currently fail most of those tests: they’re issued by a handful of providers, aren’t interchangeable across ecosystems, don’t vest or appreciate, and deflate as compute cheapens. Skeptics are right about all of this <a href="https://techcrunch.com/2026/03/21/are-ai-tokens-the-new-signing-bonus-or-just-a-cost-of-doing-business/">TechCrunch</a>, and average cost per million tokens falling from roughly $10 to $2.50 in a single year, per Ramp’s enterprise spending data, would make tokens a bizarre store of value.</p><p>But notice: none of those failures prevent tokens from being a unit of <em>work allocation</em>. Hours aren’t transferable or fungible either, and they denominated labor for a century. The bar for a metering standard is much lower than the bar for money, and tokens have already cleared it inside thousands of companies.</p><h3>The real thesis</h3><p>Tokens will not become money. They may become the standard metering layer for AI-augmented work, the unit in which cognitive labor is budgeted, compensated at the margin, and measured for return. If that happens, the durable advantage won’t belong to whoever hoards the most tokens. It will belong to the organizations that build the discipline of token-to-output ROI: knowing, per team and per workflow, what a token buys and whether it was worth spending.</p><p>The measurement frameworks are already emerging, boards asking for cost per resolved ticket instead of raw token spend, and Stanford’s Cost-of-Pass research arguing the true metric is cost per correct solution, not cost per token <a href="https://authoritytech.io/curated/inference-economics-changed-enterprise-ai-buying">AuthorityTech</a>. The companies that operationalize this first will compound an efficiency advantage the same way early adopters of unit economics did in cloud.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jMxgVATJYzTBEHxb" /></figure><h3>What to watch</h3><p>Five signals that will confirm or kill this thesis over the next 24 months:</p><p>1. <strong>Token grants spreading beyond engineering</strong> into sales, ops, legal, and non-tech industries.</p><p>2. <strong>Token-to-output ROI metrics</strong> appearing in board decks and earnings calls, cost per resolved ticket, cost per shipped feature, replacing raw spend charts.</p><p>3. <strong>The first vendor contract quoted in tokens</strong> rather than dollars, following GitHub’s usage-based repricing. (This is the hypothesis section’s proving ground.)</p><p>4. <strong>A cross-provider metering standard</strong> that normalizes tokens across models, making budgets portable.</p><p>5. <strong>Compensation surveys</strong> listing token budgets as a standard line item alongside equity, Tunguz’s “fourth component” formalized.</p><p>When those dominoes fall, we won’t say companies “adopted AI.” We’ll say work got a new unit of measure, and the ledger of business shifted from hours to intelligence.</p><p>The dollar isn’t going anywhere. But the unit of <em>work</em> is changing. The companies and people who learn to govern it first will set the benchmark for everyone else.</p><h3>Sources</h3><p>- TechCrunch — <a href="https://techcrunch.com/2026/03/21/are-ai-tokens-the-new-signing-bonus-or-just-a-cost-of-doing-business/">Are AI tokens the new signing bonus?</a> (Mar 2026)</p><p>- Fortune — <a href="https://fortune.com/2026/05/20/ceos-ai-tokens-paychecks-justify-spend/">CEOs are handing out AI tokens like paychecks</a> (May 2026)</p><p>- Gartner — <a href="https://www.gartner.com/en/newsroom/press-releases/2026-03-25-gartner-predicts-that-by-2030-performing-inference-on-an-llm-with-1-trillion-parameters-will-cost-genai-providers-over-90-percent-less-than-in-2025">Inference costs to fall 90%+ by 2030 while consumption outpaces</a> (Mar 2026)</p><p>- Zen van Riel — <a href="https://zenvanriel.com/ai-engineer-blog/token-budgets-ai-engineers-compensation-model/">Token budgets and the fourth component of compensation</a></p><p>- The Letter Two — <a href="https://thelettertwo.com/2026/03/26/ai-token-budgets-employee-compensation/">AI token budgets: the new employee benefit</a> (Mar 2026)</p><p>- Tech Insider — <a href="https://tech-insider.org/big-tech-ai-infrastructure-spending-2026/">Big Tech’s ~$700B AI infrastructure buildout</a></p><p>- <a href="http://investing.com/">Investing.com</a> — <a href="https://www.investing.com/analysis/the-ai-token-pricing-crisis-behind-openai-and-anthropics-revenue-race-200680777">The token pricing crisis behind OpenAI and Anthropic’s revenue race</a> (May 2026)</p><p>- Finout — <a href="https://www.finout.io/blog/predicting-ai-spending-in-2026-what-the-data-actually-tells-us">Global AI spending forecast, 2026</a></p><p>- Oplexa — <a href="https://oplexa.com/ai-inference-cost-crisis-2026/">The AI inference cost crisis of 2026</a></p><p>- AuthorityTech — <a href="https://authoritytech.io/curated/inference-economics-changed-enterprise-ai-buying">How inference economics changed enterprise AI buying</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=df0c4a9ef5c5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[When the Model’s Memory Goes Stale, Measuring CLI Version Drift | Tool-Calling Efficiency (TCE)]]></title>
            <link>https://medium.com/@dreams-smoke/when-the-models-memory-goes-stale-measuring-cli-version-drift-tool-calling-efficiency-tce-c0eb53588a11?source=rss-798029bc6edb------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0eb53588a11</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[ai]]></category>
            <category><![CDATA[genai]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[coding]]></category>
            <dc:creator><![CDATA[Saurabh Sarkar]]></dc:creator>
            <pubDate>Fri, 10 Jul 2026 23:07:48 GMT</pubDate>
            <atom:updated>2026-08-31T20:08:02.845Z</atom:updated>
            <content:encoded><![CDATA[<p>Coding agents don’t only fail because they reason badly. Sometimes they fail because their knowledge of a tool is frozen at training time, and the tool moved on. We measured it across 29 models.</p><blockquote>Series 1/2. 29 models . 30 version-sensitive tasks . 3 measurement modes. Bedrock · July 2026</blockquote><p>A coding agent writes docker-compose up -d. It fails. The reflex is to blame the model&#39;s reasoning. But look closer: the model isn&#39;t confused about what you asked, it&#39;s using a command that was <em>correct in 2022</em> and quietly renamed to docker compose (no hyphen) in 2023. The model&#39;s mental model of the toolchain is simply out of date.</p><p>We call this version drift. It’s a distinct, under-studied failure class, and unlike a reasoning error, it has a cheap fix that doesn’t involve retraining anything. This post is about proving it’s real, measuring how bad it gets, and where it bites hardest.</p><h3>The hypothesis</h3><p>The claim we set out to test, precisely:</p><p>A measurable fraction of tool-call failures are caused by the model’s knowledge being pinned to training-era tool versions, separately attributable (it’s stale knowledge, not bad reasoning) and separately fixable without retraining.</p><p>We’d have been wrong two ways: if drift turned out to be ~0%, or if the newest models failed as often as the oldest. Spoiler, neither happened.</p><h3>Where this started: a baseline on real sessions</h3><p>We didn’t begin with version drift, we backed into it. This work lives inside a broader project on tool-call efficiency, and the first step was purely observational: ingest 84 real Claude Code sessions, 4,046 tool calls, and ask where the token budget actually goes.</p><p>The answer reproduced the industry finding on our own traces: read and search operations dominate, accounting for ~39% of tool-call tokens, squarely in line with Microsoft’s FastContext, which put read/search at 46.5% of an agent’s tokens. So far, so expected.</p><p>But sitting inside the ~4% error rate was a failure pattern that didn’t look like bad reasoning at all, commands that failed because the <em>tool itself had changed</em> since the model was trained. That observation is what turned into this experiment: take the phenomenon out of noisy real logs and into a controlled setting where we could measure it cleanly.</p><p>The baseline told us where tokens go. It also surfaced a failure class nobody isolates, which is what the rest of this post measures.</p><p>For context, we track eight classes of tool-call waste, redundant calls, retries, hallucinated parameters, over-scoped reads, poor sequencing, wrong-tool, speculative calls, and stale-schema. The first seven adapt an existing credit taxonomy (TACO’s <em>useful / redundant / misleading</em> spine). The eighth, stale-schema, our name for version drift, is the one nobody else isolates, because its symptom looks identical to an ordinary invalid-parameter error. The difference is the <em>cause</em>: not a model mistake, but context that’s gone out of date. Same symptom, different fix, and that’s the whole reason it’s worth separating out.</p><h3>What would make this real</h3><p>Before running anything, we wrote down what would count as a genuine result rather than a fishing expedition, three bars:</p><ul><li>Measurable. Drift has to be a non-trivial, quantifiable fraction of failures, not statistical noise.</li><li>Ordered. Failures should track knowledge cutoff: older models drift more. A dose-response, not a coin flip.</li><li>Attributable. We must be able to separate stale-schema failures from genuine reasoning errors, objectively, not by vibes.</li></ul><p>And one bonus bar that shaped the whole design: an objective oracle. A command either works on the current tool or it doesn’t. No model judging another model, no human hand-labeling a thousand transcripts. As you’ll see, hitting that bar is what most of the engineering went into.</p><h3>A field guide to version drift</h3><p>Before the numbers, it helps to see the shape of the thing. Version drift isn’t one bug, it’s a family, and each member has a different flavor. Here are the ones we built tasks around, drawn from real changelogs:</p><p>ChangeThe stale answerThe current answerTypeCompose v1 → v2docker-compose updocker compose uprenamed subcommandkubectl dry-run--dry-run (bare)--dry-run=clientdeprecated flaggit branchinggit checkout -bgit switch -coutdated defaultuv self-versionuv versionuv self versionrenamed subcommandTerragrunt runterragrunt run-allterragrunt run --allrenamed subcommandAWS SSO loginaws sso configure <em>(never existed)</em>aws configure ssohallucinatedCargo add depedit Cargo.toml by handcargo addunknown new tooluv package mgr<em>&quot;what is uv?&quot;</em>uv pip installunknown new tool</p><p>Notice the range. Some are a single character (the hyphen in docker-compose). Some are a whole tool the model has never heard of (uv, first released February 2024). And one, aws sso configure, is a command that <em>has never existed in any version</em>; the model reversed the word order of the real aws configure sso and stated it with total confidence. That last category matters: a stale model doesn&#39;t just fall behind, it fabricates a plausible-sounding present.</p><p>Every task was authored from an official changelog or migration guide and pinned to a real, dated version change, then double-checked against current documentation, because (as we’ll admit later) our own first drafts were riddled with the very errors we were trying to measure.</p><h3>How this compares to what’s out there</h3><p>Version drift isn’t unknown to the field. What’s new is the <em>angle</em>. Four families of prior work, and where each stops short:</p><p>ApproachWhat it doesWhat it doesn’tWhere ours helpsFastContext / TACOProve tool-call waste exists &amp; costs tokensDon’t isolate <em>why</em> a call failsWe name a specific, fixable cause: version driftGitChameleon / VersiCodeMeasure version-conditioned code generationScope is <em>Python library</em> APIs, not CLIs; string-match gradingWe cover shell CLIs, graded by real executionToolBench / StableToolBenchLarge-scale tool-calling capability eval<em>Hide</em> API drift with cached/virtual APIs; LLM-judge, no exit codeWe treat drift as the signal &amp; grade by executionTerminal-Bench / InterCodeReal command-execution harnessVersion-agnostic, no knowledge-cutoff dimensionWe add the cutoff axis &amp; a direct version probe</p><p>No existing method has both an executable CLI oracle and version-sensitivity tied to model cutoff. That intersection is the gap we set out to fill.</p><p>The most instructive of these is ToolBench. They ran into version drift the hard way, their real-world APIs kept getting deprecated and breaking evaluation, and their solution was to <em>engineer it away</em> with cached, virtual APIs. That’s the standard instinct: drift is noise to eliminate. We do the opposite. Drift is the measurement.</p><p>There’s a deeper point in the GitChameleon comparison worth dwelling on. Version-conditioned generation for <em>library APIs</em> is genuinely hard, GitChameleon reports that even strong models land around 48–51% on their benchmark, rising to ~59% with retrieval augmentation. That’s encouraging for us in two ways. First, it’s independent evidence that the drift phenomenon is large and real, not a quirk of our task set. Second, their RAG result is a preview of our proposed fix: giving the model the relevant version information at inference time measurably helps. The open question we take on is whether the same holds for <em>CLIs</em>, where the “API” is a command line, the failure is an exit code, and the fix is a changelog delta rather than a doc snippet.</p><p>And a note on grading, because it’s where methods quietly diverge. String-matching (VersiCode) is cheap but brittle, it can’t tell a working command from a plausible-looking broken one. LLM-as-judge (ToolBench) is flexible but circular, you’re using one model’s possibly-stale knowledge to grade another’s. We wanted neither. An exit code is not an opinion.</p><h3>How we measured it, three modes</h3><p>We look at the same phenomenon through three lenses of increasing fidelity:</p><ul><li>Mode 1, Version knowledge. Ask the model directly: “what’s the latest version of X you know of?” Compare to the real latest, fetched live from official sources. This measures staleness <em>directly</em>.</li><li>Mode 2, Command (regex). “Write the command to do X.” Grade the answer by pattern: modern form vs. deprecated form. Fast, but blunt.</li><li>Mode 3, Command (execution). Take that same answer and <em>run it</em> in a container with every tool at its latest version.</li></ul><h3>The reframing that made execution tractable</h3><p>Running commands sounds like it needs a Kubernetes cluster, a Docker daemon, cloud credentials, infrastructure nightmares. The unlock was a simple question: we don’t need the command to <em>succeed</em>. We only need to know whether the current CLI recognizes the syntax.</p><p>So a usage error, a command-not-found, or a deprecation warning counts as drift. A <em>runtime</em> failure, “cannot connect to daemon,” “no credentials”, means the syntax was fine; it just couldn’t finish. That distinction let us test docker, kubectl, and aws with no daemon, no cluster, and no credentials. It also catches drift a regex misses: git init succeeds cleanly but silently creates a master branch, the executable check sees the wrong branch name; a pattern match wouldn&#39;t.</p><h3>The oracle, worked through</h3><p>Concretely, the whole harness is a single “toolbox” container with every target CLI installed at its latest version, git, node/npm, uv, cargo, kubectl, terraform, docker, aws, terragrunt. To grade a model&#39;s answer, we drop it into a fresh, isolated working directory in that container and read what the CLI says back. Three outcomes:</p><ul><li>docker compose up -d → the daemon isn&#39;t running, so it errors, but with a <em>connection</em> error, not a usage error. Syntax valid → PASS.</li><li>docker-compose up -d → command not found (v1 isn&#39;t installed anymore). Drift → FAIL.</li><li>kubectl apply --dry-run → runs, but prints --dry-run is deprecated, use --dry-run=client. A deprecation warning is drift. FAIL.</li></ul><p>That third case is the subtle one, and it’s why we don’t grade on exit code alone: the deprecated form still <em>works</em> today, so a naive “did it exit zero” check would wave it through. We treat a deprecation warning as drift because it’s a command living on borrowed time, exactly the thing a current-knowledge model wouldn’t write.</p><p>We keep the regex grader too, and run both. Not redundancy, cross-examination. When they disagree, one of them is wrong, and the disagreement tells us which. More on that below.</p><h3>The headline result</h3><p><strong>−0.88</strong></p><p>Across 29 models, the correlation between a model’s version-knowledge staleness and its command correctness is −0.88. Strong, and negative: the more out-of-date the knowledge, the more the commands break.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kxfCAvcWLMxaIcH4t0yiqA.png" /></figure><p>An honest boundary: this shows the cause (stale knowledge → broken commands). It does <em>not</em> yet show the cure, that injecting current versions fixes it. That’s the next experiment.</p><h3>It’s a gradient, not a cliff</h3><p>If the effect were real, we’d expect a smooth dose-response by knowledge cutoff, not a random split. That’s exactly what we see. Grouping models by cutoff era, command pass rate climbs steadily:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W_Sc7R6DClR8JA6_aXRaqQ.png" /></figure><p>The most charming detail is in Mode 1. Only the 2023-era models <em>hallucinated</em> versions that don’t exist yet, Claude 3 Haiku told us the latest uv was 1.0.0; it&#39;s actually at 0.11. Old models don&#39;t just lag reality, they confidently invent a plausible future.</p><h3>Where drift bites hardest</h3><p>Not all version changes are equal traps. We tagged each task by the <em>type</em> of change, and the spread is 4×:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EcGE2UaVgxSPIv9lTFHGEA.png" /></figure><p>A renamed subcommand is the worst offender, docker-compose → docker compose drifts 40% of the time, four times the rate of a genuinely new tool. That makes intuitive sense: a rename is a trap. The model holds a strong, confident, <em>wrong</em> prior, it &quot;knows&quot; the old form cold, so it never pauses. A brand-new tool it&#39;s never heard of, by contrast, it might at least hedge on, ask about, or approach cautiously. Confidence is the hazard, not ignorance.</p><p>A few numbers on the machinery behind these results. The executable oracle ran on 627 commands and passed 73% of them, the rest either drifted or hit a genuine problem. Getting there took some plumbing discipline: our first full sweep had a 63% infrastructure error rate (wrong model IDs, unsupported invocation modes) that we drove down to 9% before trusting any numbers, a reminder that a clean-looking result usually sits on top of a pile of quietly-fixed harness bugs. And running the two oracles in parallel flagged 8 disagreements, every one of which turned out to be a real weakness in one grader or the other.</p><p>There’s a practical lesson buried in this ranking. If you’re building a compatibility layer, you don’t need to cover the entire changelog of every tool, you get most of the value by targeting the <em>renames and moved subcommands</em>, because that’s where confident-and-wrong lives. The long tail of brand-new tools mostly takes care of itself: a model that’s never seen a tool tends to fail loudly rather than silently.</p><h3>Two tracks: the lab and the field</h3><p>Everything above is the static track, a controlled experiment, clean conditions, causal claims. But a lab result you can’t observe in the wild is only half a finding, so we built a second, dynamic track alongside it: a pair of hooks that sit inside a live Claude Code session.</p><p>A PreToolUse hook sees each shell command before it runs; a PostToolUse hook sees the outcome. Together they do three things. They record every command and its exit status, building a real-world corpus. They flag version-drift candidates on the fly, a command that failed with a usage-error signature. And they auto-harvest &quot;stale → correct&quot; pairs: when a command fails and a near-identical one succeeds moments later, that pair is a free, labeled example of a drift and its fix.</p><p>The two tracks share one artifact, the delta store, the running list of “what changed.” The lab validates entries; the field discovers new ones. It’s a flywheel: usage surfaces drift, drift populates the store, the store feeds the fix.</p><h3>What we learned building it</h3><p>The findings above are the tidy version. The build taught us some things the charts don’t show.</p><ul><li>Version knowledge is the cleaner signal than command drift. Command errors are noisy, even current models trip 10–20% of the time on genuinely hard or ambiguous tasks that have nothing to do with staleness. The direct “what version do you know?” probe is a purer measure of the underlying thing, which is why the −0.88 correlation runs <em>from</em> version knowledge <em>to</em> command correctness, not the other way.</li><li>Two oracles beat one, because their disagreements are where the truth hides. Running regex and execution in parallel surfaced 8 cases where they split. Each was informative. In one, a model answered kubectl get events; the regex flagged it as drift (the newer form is kubectl events), but execution passed it, because kubectl get events still works fine. The regex was too strict. In another, a valid aws bedrock-runtime invoke-model was failed by the oracle because, run with no arguments, it printed a usage message that tripped our drift signature. The oracle was too strict. Both got fixed, and we&#39;d never have found either without the second opinion.</li><li>The trap is self-referential, we were the bug we were studying. Our very first draft of this experiment shipped with stale Bedrock model IDs, CLI flags that had been renamed, and version numbers we’d simply made up from memory. We were writing an experiment about frozen knowledge <em>using</em> frozen knowledge. The only cure was the same one we’re proposing for the models: stop trusting memory, fetch the current truth from live sources. Every model ID, every version, every pass-pattern in the final run was verified against a live source or official doc.</li><li>“Latest everything” is a maintenance commitment, not a snapshot. The toolbox container is only a valid oracle for as long as it’s actually current. The moment it goes stale, it starts mismarking correct answers as drift, the oracle itself drifts. Rebuilding it is part of the method, not a one-time setup.</li></ul><h3>The fix: a delta store the model reads at runtime</h3><p>If staleness is the cause, the fix isn’t a smarter model, it’s a continuously-updated cheat sheet handed to the model right before it acts. Detect the local tool version, pull the relevant changelog deltas, inject them into context. The model’s weights never change; its <em>context</em> gets a patch.</p><p>The important property is that this layer sits <em>outside</em> the model. It’s model-agnostic, it works for any LLM regardless of cutoff, even a frozen or obsolete one. Tools change weekly; model cutoffs land quarterly at best. That gap is permanent and structural, and a runtime layer closes it for every model at once, retroactively.</p><p>There are four ways to deliver the delta, and we plan to A/B all of them:</p><ul><li>System-prompt injection. Prepend the relevant deltas before the task. Simplest; tests prevention against the cost of extra context.</li><li>Pre-call interception. Check the command the model is about to run against the delta store, and inject a correction hint only when drift is detected. Reactive, low context overhead, this is what the PreToolUse hook already does.</li><li>Post-failure retrieval. On the first failure, retrieve relevant deltas and retry. Tests whether the delta helps recovery versus a blind retry.</li><li>Fine-tuned adapter. A small LoRA trained on delta-annotated examples. The stretch goal, is this learnable cheaply, or is context injection always the better lever?</li></ul><p>The model doesn’t need to be smarter. It needs to know what changed.</p><h3>Reproduce it</h3><p>The whole experiment is deliberately cheap to re-run, because the ground truth moves and the result should be re-checked against it. The moving parts:</p><ul><li>Models: 29 on AWS Bedrock via the Converse API, spanning cutoffs from early 2023 to early 2026, Anthropic Claude (3 through 4.x), Meta Llama (3–4), Mistral, Amazon Nova, DeepSeek, and OpenAI GPT-OSS.</li><li>Tasks: 30 version-sensitive prompts, each pinned to a dated change and validated against current docs.</li><li>Ground truth: latest versions fetched live at run time from GitHub releases, dl.k8s.io, and nodejs.org, never hardcoded.</li><li>Oracle: one “latest-everything” container; grade on syntax recognition, cross-checked with regex.</li></ul><p>Re-run it in six months against newer models and the expectation is clear: the newest cohort’s dots march further toward the top-left as their cutoffs catch up, but the <em>slope</em> stays negative, because there will always be a frontier of tool changes newer than any model’s training. The gap doesn’t close. It moves.</p><h3>What’s next</h3><ul><li>Scale the field instrument. Measure real-world drift prevalence and its token cost in live coding sessions.</li><li>A/B the fix. Does delta injection actually cut drift without hurting outcomes? Map the staleness-decay curve.</li><li>Self-refreshing store. Auto-populate from harvested stale→correct pairs plus nightly changelog syncs.</li></ul><h3>References &amp; sources</h3><h3>Prior work we build on</h3><p>The methods and framing above draw on these; where we cite a specific number (e.g. FastContext’s 46.5%, GitChameleon’s 48–51%) it comes from the corresponding source.</p><ol><li>FastContext — read/search operations as the dominant cost center in coding-agent loops; our token-cost anchor. <a href="https://arxiv.org/abs/2606.14066">arXiv:2606.14066</a> <em>(preprint; verify current status)</em></li><li>TACO — the useful / redundant / misleading credit taxonomy our waste-class spine adapts. <a href="https://arxiv.org/abs/2606.30251">arXiv:2606.30251</a></li><li>GitChameleon — version-conditioned code generation for Python libraries; the closest methodological cousin and our blueprint for pinning versions in a container. <a href="https://arxiv.org/abs/2507.12367">arXiv:2507.12367</a></li><li>VersiCode — version-specific code completion and a taxonomy of API lifespan (addition / deprecation). <a href="https://arxiv.org/abs/2406.07411">arXiv:2406.07411</a></li><li>ToolBench / ToolLLM — large-scale tool-calling capability eval; the cautionary example that <em>engineered drift away</em> with virtual APIs. <a href="https://arxiv.org/abs/2307.16789">arXiv:2307.16789</a></li><li>Terminal-Bench — real terminal-command execution harness (version-agnostic). <a href="https://www.tbench.ai/">tbench.ai</a></li><li>InterCode — interactive code environments with execution feedback. <a href="https://arxiv.org/abs/2306.14898">arXiv:2306.14898</a></li><li>τ-bench / τ²-bench — pass^k reliability metric for tool-use agents. <a href="https://arxiv.org/abs/2406.12045">arXiv:2406.12045</a></li></ol><h3>CLI changelogs &amp; docs the tasks were sourced from</h3><p>Every version-sensitive task was authored from an official changelog or migration guide, and each ground-truth “latest version” was fetched live at run time from the tool’s release source.</p><ol><li>Docker Compose V2 migration — docker-compose → docker compose. <a href="https://docs.docker.com/compose/migrate/">docs.docker.com/compose/migrate</a></li><li>kubectl — dry-run, events, debug (ephemeral containers), create-token. <a href="https://kubernetes.io/docs/reference/kubectl/">kubernetes.io/docs/reference/kubectl</a> · release notes at <a href="https://kubernetes.io/releases/">kubernetes.io/releases</a></li><li>git — git switch / git restore (2.23), init -b (2.28). <a href="https://git-scm.com/docs/git-switch">git-scm.com/docs</a></li><li>AWS CLI v2 — pager behavior, aws configure sso, bedrock-runtime. <a href="https://docs.aws.amazon.com/cli/latest/userguide/">docs.aws.amazon.com/cli</a></li><li>Terraform — native test, import block config generation, removed block. <a href="https://developer.hashicorp.com/terraform">developer.hashicorp.com/terraform</a></li><li>uv (Astral) — uv run, uv self version, --system-certs. <a href="https://docs.astral.sh/uv/">docs.astral.sh/uv</a> · <a href="https://github.com/astral-sh/uv/blob/main/CHANGELOG.md">changelog</a></li><li>Cargo (Rust) — cargo add (1.62), cargo remove (1.66). <a href="https://doc.rust-lang.org/cargo/">doc.rust-lang.org/cargo</a></li><li>Node.js — stable --watch, npm pkg, corepack status. <a href="https://nodejs.org/docs/latest/api/cli.html">nodejs.org/docs</a></li><li>Terragrunt — CLI redesign (run --all). <a href="https://terragrunt.gruntwork.io/docs/migrate/cli-redesign/">terragrunt.gruntwork.io</a></li></ol><p>Method notes: 29 models on AWS Bedrock (Anthropic, Meta, Mistral, Amazon Nova, DeepSeek, OpenAI GPT-OSS), 30 version-sensitive CLI tasks across docker, kubectl, git, aws, terraform, uv, cargo, node, terragrunt. Ground-truth versions fetched live from official release sources. All pass patterns validated against current documentation. Executable oracle runs a single “latest-everything” container and grades on syntax recognition, not full execution. 3 models excluded for access errors. Correlation is Pearson r over per-model averages (n=29).</p><p>Reference IDs are provided for traceability; arXiv identifiers marked <em>preprint</em> should be verified for current publication status before formal citation.</p><p>Refer to the basics: <a href="https://medium.com/@dreams-smoke/the-token-standard-why-ai-tokens-may-become-the-new-unit-of-work-df0c4a9ef5c5">https://medium.com/@dreams-smoke/the-token-standard-why-ai-tokens-may-become-the-new-unit-of-work-df0c4a9ef5c5</a></p><blockquote>Series 2/2: <a href="https://medium.com/@dreams-smoke/curing-version-drift-with-a-delta-the-model-reads-first-317866483498">https://medium.com/@dreams-smoke/curing-version-drift-with-a-delta-the-model-reads-first-317866483498</a></blockquote><p>Code Artifact: <a href="https://github.com/aws-samples/sample-improve-ai-tool-call-efficiency">https://github.com/aws-samples/sample-improve-ai-tool-call-efficiency</a></p><p># ai #llm #ai-agents #coding #ai-for-developers #gen-ai</p><blockquote>Any opinions in this article are those of the individual author and may not reflect the opinions of AWS.</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0eb53588a11" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[       .]]></title>
            <link>https://medium.com/@dreams-smoke/-2dd42c09b685?source=rss-798029bc6edb------2</link>
            <guid isPermaLink="false">https://medium.com/p/2dd42c09b685</guid>
            <category><![CDATA[rabbit-r1-to-the-rescue]]></category>
            <dc:creator><![CDATA[Saurabh Sarkar]]></dc:creator>
            <pubDate>Thu, 02 Jul 2026 03:12:39 GMT</pubDate>
            <atom:updated>2026-07-02T03:12:39.567Z</atom:updated>
            <content:encoded><![CDATA[<h3>𝗠𝘆 𝗥𝗮𝗯𝗯𝗶𝘁 𝗥𝟭 𝗵𝗮𝗱 𝗯𝗲𝗰𝗼𝗺𝗲 𝗮 𝗱𝗿𝗮𝘄𝗲𝗿 𝗼𝗿𝗻𝗮𝗺𝗲𝗻𝘁. 𝗧𝗵𝗶𝘀 𝘄𝗲𝗲𝗸𝗲𝗻𝗱 𝗜 𝗴𝗮𝘃𝗲 𝗶𝘁 𝗮 𝗿𝗲𝗮𝗹 𝗷𝗼𝗯</h3><p><a href="https://www.linkedin.com/company/rabbitinc/">rabbit inc.</a> recently shipped a way to connect the R1 to OpenClaw, so the device can talk to an agent gateway instead of just the stock assistant:<br><a href="https://www.rabbit.tech/blog/connect-your-r1-to-openclaw">https://www.rabbit.tech/blog/connect-your-r1-to-openclaw</a></p><p>Cool idea — but two things got in my way:</p><p>1. The R1 only speaks the OpenClaw “clawdbot-gateway” protocol.<br> 2. That path leans on provider API keys. I don’t live in API keys — I live in Claude Code (<a href="https://www.linkedin.com/company/anthropicresearch/">Anthropic</a>) and Codex (<a href="https://www.linkedin.com/company/openai/">OpenAI</a>) every day, signed in with my normal subscriptions (OAuth/login, e.g. OpenAI), not raw keys.</p><p>So instead of fighting it, I leaned into it.</p><p>I wrote a small WebSocket shim that impersonates an OpenClaw gateway. The R1 thinks it’s talking to <a href="https://www.linkedin.com/company/openclawai/">OpenClaw</a> — but every push-to-talk turn actually gets routed to whatever agent is running on my server (an Apple Mac Mini M4 sitting on my desk):</p><p>R1 → Wi-Fi → shim on the Mac Mini → local Claude Code / Codex CLI → answer streamed back to the R1 screen.</p><p>The nice part: it reuses the CLI’s own auth, so it just works with my Claude and Codex logins. No API key juggling. The R1 becomes a dedicated hardware push-to-talk remote for the coding agents I already use.</p><p>Bonus gotcha for anyone trying the official path: current OpenClaw floors its gateway at protocol v4, but shipped R1 firmware only speaks v3 — so a fresh gateway rejects the very device it’s meant to serve (“gateway unreachable”). My shim negotiates down to v3, and I documented the downgrade fix for the official route too.</p><p>Open-sourced it (MIT), Claude Code + Codex backends included:<br> <a href="https://github.com/sarkarsaurabh27/rabbit-r1-hooks">https://github.com/sarkarsaurabh27/rabbit-r1-hooks</a> 🤖</p><p>Repurposing hardware you already own &gt; buying more hardware. Finally using my Rabbit R1 for something. 🐇</p><p><strong>#rabbitr1</strong> <strong>#claudecode</strong> <strong>#codex</strong> <strong>#aitools</strong> <strong>#opensource</strong> <strong>#buildinpublic #hackathon</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*npVfAdbI6y-PNWBu" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*O5bOxOrB9AwY1X3X" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*c3eplGy_dzAPzGQI" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2dd42c09b685" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Gardening Taught Me About Parenting]]></title>
            <link>https://medium.com/@dreams-smoke/some-lessons-about-growth-culture-identity-and-letting-go-130e6dd1f34a?source=rss-798029bc6edb------2</link>
            <guid isPermaLink="false">https://medium.com/p/130e6dd1f34a</guid>
            <dc:creator><![CDATA[Saurabh Sarkar]]></dc:creator>
            <pubDate>Wed, 01 Jul 2026 23:37:55 GMT</pubDate>
            <atom:updated>2026-07-01T23:38:27.475Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FV_8QBbR_ioRsVGz" /><figcaption>My Angel ❤</figcaption></figure><p><em>Some lessons about growth, culture, identity, and letting go.</em></p><p>A few weeks ago, I was standing in my backyard looking at a small garden bed.</p><p>Some plants were thriving. Others were struggling. A few had grown in directions I never expected.</p><p>Meanwhile, my four-year-old son was enthusiastically helping in his own way. Digging where I didn’t plan to dig. Watering plants that didn’t need water. Asking an endless stream of “why?” questions. Like most amateur gardeners, I started asking myself the usual questions:</p><ul><li>Did I water enough?</li><li>Too much?</li><li>Did the soil need something different?</li><li>Should I move the plant to a sunnier spot?</li></ul><p>As I stood there thinking about these things, a thought crossed my mind:</p><blockquote><strong><em>Parenting is remarkably similar to gardening.</em></strong></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*HLw8ztZXivZgsAs5" /></figure><p>As parents, we spend countless hours trying to create the right environment.</p><p>We provide:</p><ul><li>Love</li><li>Education</li><li>Encouragement</li><li>Boundaries</li><li>Experiences</li><li>Values</li></ul><p>We try to protect our children from storms while exposing them to enough challenges that they grow stronger. We carefully choose the seeds we hope to plant:</p><ul><li>Kindness</li><li>Curiosity</li><li>Resilience</li><li>Culture</li><li>Family traditions</li></ul><pre>The lessons we learned from our own parents. The lessons we learned the hard way. The lessons we wish someone had taught us earlier.</pre><p>Yet, despite all our efforts, we eventually discover something every gardener already knows.</p><blockquote><strong><em>We can prepare the soil. We cannot control how the plant grows.</em></strong></blockquote><p>And perhaps we shouldn’t.</p><h3>The Desire to Pass Something Forward</h3><p>As human beings, especially those raising children between cultures, we often feel responsible for preserving something meaningful.</p><ul><li>Where they came from</li><li>The stories of previous generations</li><li>Their language and traditions</li><li>The values that shaped us</li></ul><p>There is something beautiful about that desire.</p><p><em>It comes from love. It comes from gratitude. It comes from wanting continuity in a world that changes rapidly.</em></p><p>Yet the older I get, the more I realize that every generation inherits a world different from the one their parents knew. But recently I’ve found myself asking:</p><blockquote><em>Are we trying to pass on our answers, or are we trying to teach our children how to ask better questions?</em></blockquote><h3>From Answers to Questions</h3><p>When I was younger, I thought wisdom meant accumulating answers. The older I get, the more I think wisdom may be about asking better questions. Many of the beliefs I hold today were inherited from someone else.</p><pre>Some proved valuable. <br>Some proved incomplete. <br>Some I had to unlearn. <br>Some I continue to question.</pre><p>If every generation simply accepts what the previous generation believed, progress becomes impossible. At the same time, if every generation ignores everything that came before it, wisdom is lost. The challenge is finding balance. Perhaps our role is not to hand our children a finished map. Perhaps our role is to teach them how to navigate. Not to tell them exactly what to think. But to teach them how to think. Not to give them all the answers. But to help them ask thoughtful questions. Not to define their identity. But to help them discover it.</p><h3>The Garden Doesn’t Belong to the Gardener</h3><p>One of the humbling realities of gardening is that</p><blockquote><em>the garden does not exist for the gardener.</em></blockquote><p>The gardener may prepare the conditions</p><p>But the garden develops according to its own nature.</p><p>A tomato plant will never become a rose because the gardener wants it to.</p><p>And a rose should never be judged for not becoming a tomato plant.</p><p>Children are no different.</p><pre>Every parent carries hopes and dreams. We imagine what success looks like. We imagine what values matter most. We imagine what paths our children might take. Yet eventually they become their own people.</pre><p>They will have:</p><ul><li>Different experiences</li><li>Different challenges</li><li>Different opportunities</li><li>Different dreams</li></ul><p>Their world will not be our world.</p><p><strong><em>And perhaps that is exactly how it should be.</em></strong></p><h3>What We Leave Behind</h3><p>Today, I think less about whether my son will believe exactly what I believe.</p><p>If I’m honest, part of me hopes he values many of the same things I do. The traditions I grew up with. The stories that shaped me. The lessons that took me years to learn.</p><p>But another part of me realizes that many of my strongest beliefs came from questioning what I inherited.</p><p>If I want him to grow, I have to allow him that same freedom. Instead, I hope he inherits something more durable:</p><ul><li>Curiosity</li><li>Humility</li><li>Empathy</li><li>Courage to question assumptions</li><li>Openness to new ideas</li><li>Respect for tradition</li><li>Willingness to evolve</li></ul><p>If he carries those forward, the specific answers may matter less.</p><h3>A Gardener’s Job</h3><p>The more I reflect on parenting, the more I believe a gardener’s job is surprisingly simple:</p><p>✓ Prepare the soil</p><p>✓ Provide nourishment</p><p>✓ Protect when necessary</p><p>✓ Step back when necessary</p><p>✓ Be patient</p><p>✓ Accept uncertainty</p><p>✓ Trust the process</p><p>The goal is not to create a copy of ourselves. The goal is to help another human being become fully themselves.</p><p>The older I get, the less I think parenting is about shaping a child into who we want them to become. And the more I think it is about creating the conditions for them to discover who they already are.</p><p>Just like a garden, that journey is messy, unpredictable, beautiful, and humbling.</p><blockquote><em>And maybe that’s exactly why it is worth doing.</em></blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/843/0*6MgyhkFzCKZSHsj2" /></figure><p><strong>As parents, we plant seeds whose full bloom we may never see.</strong></p><p><strong>But we plant them anyway.</strong></p><h3>Question for readers</h3><p>What values are you hoping to pass on to the next generation, and which ones do you hope they challenge?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=130e6dd1f34a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[9 Things I Learned Reading Claude Code’s Source Code]]></title>
            <link>https://medium.com/@dreams-smoke/9-things-i-learned-reading-claude-codes-source-code-62c369a1d290?source=rss-798029bc6edb------2</link>
            <guid isPermaLink="false">https://medium.com/p/62c369a1d290</guid>
            <category><![CDATA[productivity]]></category>
            <category><![CDATA[best-practices]]></category>
            <category><![CDATA[claude-code]]></category>
            <category><![CDATA[agents]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Saurabh Sarkar]]></dc:creator>
            <pubDate>Tue, 07 Apr 2026 00:40:04 GMT</pubDate>
            <atom:updated>2026-04-12T21:12:19.301Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A deep dive into how Anthropic’s production agent actually works — with the real code, the real design decisions, and the research that explains why each decision was made</em></p><p>Most “agent best practices” posts give you generic advice. This one is different.</p><p>Claude Code is one of the few production-grade AI coding agents with readable source code. I went through it. What follows are 9 concrete things the Anthropic engineering team did — with the actual TypeScript — matched against the benchmark research that explains <em>why</em> each decision matters.</p><p>Each section ends with what you should change in your own agent today.</p><h2>Benchmarks at a Glance</h2><figure><img alt="" src="https://cdn-images-1.medium.com/max/932/1*rz7nHz5lOogY3ccaokYcHg.png" /></figure><h2>1. CLAUDE.md Is Not in the System Prompt</h2><p>This is the most counterintuitive thing in Claude Code’s architecture, and the one most teams building similar tools get wrong.</p><p>Every Claude Code user knows about CLAUDE.md — the file where you put project-specific instructions. The natural assumption is that it gets injected into the system prompt. <strong>It does not.</strong> It goes into the user turn.</p><pre>// src/context.ts<br>export const getUserContext = memoize(<br>  async (): Promise&lt;{ [k: string]: string }&gt; =&gt; {<br>    const claudeMd = shouldDisableClaudeMd<br>      ? null<br>      : getClaudeMds(filterInjectedMemoryFiles(await getMemoryFiles()))    <br>    return {<br>      ...(claudeMd &amp;&amp; { claudeMd }),<br>      currentDate: `Today&#39;s date is ${getLocalISODate()}.`,<br>    }<br>  },<br>)</pre><p>And here’s where it gets injected:</p><pre>// src/utils/api.ts<br>export function prependUserContext(<br>  messages: Message[],<br>  context: { [k: string]: string },<br>): Message[] {<br>  return [<br>    createUserMessage({<br>      content: `&lt;system-reminder&gt;\nAs you answer the user&#39;s questions, you can use the following context:\n${Object.entries(<br>        context,<br>      )<br>        .map(([key, value]) =&gt; `# ${key}\n${value}`)<br>        .join(&#39;\n&#39;)}<br><br>      IMPORTANT: this context may or may not be relevant to your tasks. You should not respond to this context unless it is highly relevant to your task.\n&lt;/system-reminder&gt;\n`,<br>      isMeta: true,<br>    }),<br>    ...messages,<br>  ]<br>}</pre><p>getUserContext() returns claudeMd and currentDate. Both are wrapped in a &lt;system-reminder&gt; block inside a <strong>user message</strong> — isMeta: true marks it as infrastructure, invisible to the user — and prepended to the conversation. Contrast this with getSystemContext(), which returns gitStatus and is appended to the actual system prompt via appendSystemContext().</p><p><strong>Why this is the right call — and the research behind it:</strong></p><p>Chroma’s 2025 study measured 18 frontier LLMs and found a universal result: context degradation begins at <strong>50K tokens even in models with 1M-token windows</strong>, and information in the middle of context is recalled with ~30% lower accuracy than content at the start or end (Liu et al., Stanford 2024). The system prompt is where the model’s core identity and instructions live — it needs to be compact, stable, and reliably attended to.</p><p>CLAUDE.md is project-specific. If it were in the system prompt, two things would go wrong: every project change would bust the prompt cache (70–90% of the system prompt is otherwise stably cacheable), and project context would compete with core instructions for the model’s attention budget. By moving it to the first user turn, the system prompt stays stable and cacheable at org scope, and CLAUDE.md is treated as contextual information the model reads — not as part of its identity.</p><p><strong>What to change today:</strong> If your agent’s project-specific context is in the system prompt, move it to the first user turn wrapped in a &lt;system-reminder&gt; block. Keep your system prompt stable. Let project context be contextual.</p><h2>2. Cache Boundaries Are Named and Enforced in Code</h2><p>Token usage explains <strong>80% of BrowseComp performance variance</strong> (Google Research, 2025). The ceiling on your agent’s performance is often not the model — it’s how efficiently you’re spending the token budget you have. Cache misses are the biggest unforced error in production agents.</p><p>Claude Code handles this by making cache boundary violations visible at the code level. Two functions, explicitly named to signal cost:</p><pre>// src/constants/systemPromptSections.ts<br><br>/**<br> * Create a memoized system prompt section.<br> * Computed once, cached until /clear or /compact.<br> */<br>export function systemPromptSection(<br>  name: string,<br>  compute: ComputeFn,<br>): SystemPromptSection {<br>  return { name, compute, cacheBreak: false }<br>}<br><br>/**<br> * Create a volatile system prompt section that recomputes every turn.<br> * This WILL break the prompt cache when the value changes.<br> * Requires a reason explaining why cache-breaking is necessary.<br> */<br>export function DANGEROUS_uncachedSystemPromptSection(<br>  name: string,<br>  compute: ComputeFn,<br>  _reason: string,  // ← you must write why<br>): SystemPromptSection {<br>  return { name, compute, cacheBreak: true }<br>}</pre><p>DANGEROUS_uncachedSystemPromptSection takes a _reason parameter. The underscore means it&#39;s not used at runtime — it&#39;s documentation-as-code. Every engineer who calls this function has to write a reason why this section should recompute every turn and break the cache. The naming makes the cost visible in code review.</p><p>The system prompt has a literal boundary marker between globally-cacheable and dynamic content:</p><pre>// src/constants/prompts.ts<br>export const SYSTEM_PROMPT_DYNAMIC_BOUNDARY =<br>  &#39;__SYSTEM_PROMPT_DYNAMIC_BOUNDARY__&#39;</pre><p>Feature flags follow the same explicit-cost convention:</p><pre>// src/coordinator/coordinatorMode.ts<br>checkStatsigFeatureGate_CACHED_MAY_BE_STALE(&#39;tengu_scratch&#39;)</pre><p>_CACHED_MAY_BE_STALE is the name — not a comment. When you call this function you are declaring you accept a stale value. The alternative (initializeFeatureFlags()) makes a network call. Every call site makes the tradeoff explicit.</p><p><strong>What to change today:</strong> Name your volatile prompt sections explicitly. Use VOLATILE_ or UNCACHED_ as a prefix for anything that recomputes each turn. If 70–90% of your system prompt is stable, treat that stability as an asset worth protecting.</p><h2>3. The Verification Agent Cannot Edit Files — By Design</h2><p>73% of enterprise AI agents fail in their first year (Cleanlab, 2025). The most common reason isn’t model capability — it’s that self-reported success isn’t verified. Claude Code ships a built-in verification agent that addresses this architecturally:</p><pre>// src/tools/AgentTool/built-in/verificationAgent.ts<br><br>export const VERIFICATION_AGENT: BuiltInAgentDefinition = {<br>  agentType: &#39;verification&#39;,<br>  disallowedTools: [<br>    AGENT_TOOL_NAME,<br>    EXIT_PLAN_MODE_TOOL_NAME,<br>    FILE_EDIT_TOOL_NAME,    // cannot edit files<br>    FILE_WRITE_TOOL_NAME,   // cannot write files<br>    NOTEBOOK_EDIT_TOOL_NAME,<br>  ],<br>  source: &#39;built-in&#39;,<br>  background: true,<br>  model: &#39;inherit&#39;,<br>  getSystemPrompt: () =&gt; VERIFICATION_SYSTEM_PROMPT,<br>  criticalSystemReminder_EXPERIMENTAL:<br>    &#39;CRITICAL: This is a VERIFICATION-ONLY task. You CANNOT edit, write, or create files IN THE PROJECT DIRECTORY. You MUST end with VERDICT: PASS, VERDICT: FAIL, or VERDICT: PARTIAL.&#39;,<br>}</pre><p>FILE_EDIT_TOOL_NAME and FILE_WRITE_TOOL_NAME are in disallowedTools. This is not a prompt instruction — it&#39;s an architectural constraint. The agent literally cannot edit files. Not because it was told not to, but because the tool isn&#39;t in its available set.</p><p><strong>Why the separation matters:</strong> A 2025 refinement study (1,000 problems, 11 domains) found that unguided self-reflection at the frontier adds just <strong>+1.8 pp</strong> over 5 iterations. The same model given guided external feedback with structured mechanisms gained <strong>+80% within 5 turns</strong>. The difference is not model capability — it’s structure. A separate verification agent with its own tool constraints, its own system prompt, and a required verdict format is structured external feedback. Asking your implementation agent to “verify its own work” is unguided self-reflection.</p><p>The criticalSystemReminder_EXPERIMENTAL field is also revealing — it&#39;s a separate injection mechanism that provides a persistent reminder at a location designed to survive context rot, independent of the main system prompt. Even a good system prompt can be &quot;forgotten&quot; as context grows. The critical reminder is a second anchor.</p><p><strong>What to change today:</strong> Separate your verification agent architecturally. Use disallowedTools at the framework level, not just a prompt instruction. A verification agent that can edit files is not a verification agent — it&#39;s just another implementation agent with a different name.</p><h2>4. The Denial Circuit Breaker Is Exact and Enforced</h2><p>OWASP’s 2025 audit found prompt injection in <strong>73% of production agentic deployments</strong>. Standard injection success rates run 50–84%. The most capable adaptive attacks succeed over 85% of the time against undefended systems. Yet only 34.7% of organizations have dedicated injection defenses.</p><p>Claude Code’s permission system includes a circuit breaker with specific, hardcoded thresholds from production incident data:</p><pre>// src/utils/permissions/denialTracking.ts<br><br>export const DENIAL_LIMITS = {<br>  maxConsecutive: 3,<br>  maxTotal: 20,<br>} as const<br><br>export function shouldFallbackToPrompting(<br>  state: DenialTrackingState,<br>): boolean {<br>  return (<br>    state.consecutiveDenials &gt;= DENIAL_LIMITS.maxConsecutive ||<br>    state.totalDenials &gt;= DENIAL_LIMITS.maxTotal<br>  )<br>}</pre><p>3 consecutive denials, or 20 total: fall back to prompting the user. Two distinct failure modes:</p><ol><li><strong>Consecutive</strong> (3 max): stuck in a loop on the same blocked action — injection or runaway behavior</li><li><strong>Total</strong> (20 max): operating broadly outside permitted scope — misconfiguration or session drift</li></ol><p>State is tracked immutably. A success resets the consecutive counter but not the total:</p><pre>export function recordDenial(state: DenialTrackingState): DenialTrackingState {<br>  return {<br>    ...state,<br>    consecutiveDenials: state.consecutiveDenials + 1,<br>    totalDenials: state.totalDenials + 1,<br>  }<br>}<br><br>export function recordSuccess(state: DenialTrackingState): DenialTrackingState {<br>  if (state.consecutiveDenials === 0) return state<br>  return { ...state, consecutiveDenials: 0 }<br>}</pre><p>The immutable update pattern is intentional — denial state cannot be mutated in place, which prevents a compromised tool from silently resetting its own counter. For async subagents whose setAppState is a no-op, a localDenialTracking field is passed through the agent context so the counter accumulates correctly even when the agent can&#39;t reach the main state store.</p><p>Layered defense reduces injection attack success from <strong>73.2% → 8.7%</strong> (layered defense research, 2025). The denial circuit breaker is one layer. The immutability of state is another.</p><p><strong>What to change today:</strong> Add a denial circuit breaker. Track consecutive and total separately. When either threshold triggers, surface to the user — do not retry. Those exact numbers (3, 20) came from production incidents; they’re a reasonable starting point.</p><h2>5. Tools Default to Unsafe</h2><p>40% of agent frameworks have exploitable tool-execution flaws (OWASP / security research, 2025). Most of these aren’t in the tool logic — they’re in the defaults. A tool that doesn’t declare its access scope, mutability, or concurrency behavior gets assumed into a posture by the framework.</p><p>Claude Code’s posture is fail-closed:</p><pre>// src/Tool.ts<br><br>const TOOL_DEFAULTS = {<br>  isEnabled: () =&gt; true,<br>  isConcurrencySafe: (_input?) =&gt; false,   // assume NOT safe<br>  isReadOnly: (_input?) =&gt; false,           // assume WRITES<br>  isDestructive: (_input?) =&gt; false,<br>  checkPermissions: (input, _ctx?): Promise&lt;PermissionResult&gt; =&gt;<br>    Promise.resolve({ behavior: &#39;allow&#39;, updatedInput: input }),<br>  toAutoClassifierInput: (_input?) =&gt; &#39;&#39;,   // skip security classifier<br>  userFacingName: (_input?) =&gt; &#39;&#39;,<br>}<br><br>export function buildTool&lt;D extends AnyToolDef&gt;(def: D): BuiltTool&lt;D&gt; {<br>  return {<br>    ...TOOL_DEFAULTS,<br>    userFacingName: () =&gt; def.name,<br>    ...def,<br>  } as BuiltTool&lt;D&gt;<br>}</pre><ul><li>isConcurrencySafe: false — assume cannot run in parallel</li><li>isReadOnly: false — assume writes</li><li>toAutoClassifierInput: () =&gt; &#39;&#39; — invisible to the security classifier</li></ul><p>A new tool that doesn’t declare itself read-only is treated as a writer. A new tool that doesn’t declare concurrency safety is serialized. A new tool without a classifier representation is invisible to auto-mode security analysis — which is safer than a wrong representation.</p><p>The OWASP Top 10 for Agentic Applications (2026) lists “insecure tool/plugin design” as #2. The tool interface also enforces interruptBehavior:</p><pre>/**<br> * - &#39;cancel&#39; — stop the tool and discard its result<br> * - &#39;block&#39;  — keep running; the new message waits<br> * Defaults to &#39;block&#39; when not implemented.<br> */<br>interruptBehavior?(): &#39;cancel&#39; | &#39;block&#39;</pre><p>A read-only search tool should cancel when interrupted — no point finishing a search the user has moved past. A write operation should block — abandoning a half-written edit is worse than making the user wait.</p><p><strong>What to change today:</strong> Explicitly declare isReadOnly, isConcurrencySafe, and isDestructive on every tool. Fail-closed defaults prevent the silent privilege creep that causes 40% of framework-level exploits.</p><h2>6. Agents Are Typed and Lifecycle-Managed</h2><p>A 20-step agent at 95% per-step reliability has only a <strong>36% chance of end-to-end success</strong> — pure compounding. Every premature termination, failed cleanup, or undetected hung agent eats directly into that already-thin margin.</p><p>Claude Code’s task system distinguishes seven agent types and five lifecycle states:</p><pre>// src/Task.ts<br><br>export type TaskType =<br>  | &#39;local_bash&#39;<br>  | &#39;local_agent&#39;<br>  | &#39;remote_agent&#39;<br>  | &#39;in_process_teammate&#39;<br>  | &#39;local_workflow&#39;<br>  | &#39;monitor_mcp&#39;<br>  | &#39;dream&#39;<br><br>export type TaskStatus =<br>  | &#39;pending&#39;<br>  | &#39;running&#39;<br>  | &#39;completed&#39;<br>  | &#39;failed&#39;<br>  | &#39;killed&#39;<br><br>export function isTerminalTaskStatus(status: TaskStatus): boolean {<br>  return status === &#39;completed&#39; || status === &#39;failed&#39; || status === &#39;killed&#39;<br>}</pre><p>&#39;killed&#39; is distinct from &#39;failed&#39;. A failed task encountered an error; a killed task was deliberately stopped. This matters for retry logic — a killed task should not be retried, a failed task might be. isTerminalTaskStatus is the single source of truth for &quot;will this task transition again?&quot; — all cleanup logic gates on this one function.</p><p>Task IDs are typed by agent type with a security-conscious alphabet:</p><pre>// 36^8 ≈ 2.8 trillion combinations, sufficient to resist brute-force symlink attacks.<br>const TASK_ID_ALPHABET = &#39;0123456789abcdefghijklmnopqrstuvwxyz&#39;<br><br>const TASK_ID_PREFIXES: Record&lt;string, string&gt; = {<br>  local_bash:          &#39;b&#39;,<br>  local_agent:         &#39;a&#39;,<br>  remote_agent:        &#39;r&#39;,<br>  in_process_teammate: &#39;t&#39;,<br>  local_workflow:      &#39;w&#39;,<br>  monitor_mcp:         &#39;m&#39;,<br>  dream:               &#39;d&#39;,<br>}</pre><p>The prefix makes the agent type instantly readable from any task ID in logs. The design comment — “resist brute-force symlink attacks” — shows someone thought about what an attacker could do with predictable IDs. &#39;dream&#39; is an internal designation for speculative agent modes that explore possibilities without committing.</p><p><strong>What to change today:</strong> Add explicit lifecycle states. Distinguish killed from failed. Gate all cleanup logic on a single isTerminalStatus() function. The 36% end-to-end success ceiling gets worse fast when your cleanup paths are inconsistent.</p><h2>7. The Verification Agent’s System Prompt Is the Product</h2><p>Scaffold design — not model capability — accounts for <strong>+8 percentage points on SWE-bench</strong> (62.3% → 70.3%, same model, SWE-bench Feb 2025). That’s more than many model upgrades deliver. The verification agent’s system prompt is where most of that gain lives.</p><p>The structure is worth studying as a template:</p><p><strong>1. Named failure modes first.</strong> The prompt opens by naming the two ways the verification agent fails:</p><blockquote>“You have two documented failure patterns. First, verification avoidance: when faced with a check, you find reasons not to run it — you read code, narrate what you would test, write ‘PASS,’ and move on. Second, being seduced by the first 80%: you see a polished UI or a passing test suite and feel inclined to pass it, not noticing half the buttons do nothing…”</blockquote><p><strong>2. Hard prohibitions in ALL CAPS.</strong> === CRITICAL: DO NOT MODIFY THE PROJECT ===. Frontier models respond to visual prominence. Important prohibitions get headers.</p><p><strong>3. Type-specific strategies.</strong> The prompt provides distinct verification paths for frontend, backend/API, CLI, infrastructure, library, bug fix, mobile, data/ML, migrations, and refactoring. One-size-fits-all verification is how agents confirm the happy path and call it done.</p><p><strong>4. Required output format.</strong> Every check: Command run → Output observed → Result. A PASS without a command is a skip. The caller can spot-check by re-running the command.</p><p><strong>5. Pre-emptive rationalization blocking:</strong></p><blockquote>“- ‘The code looks correct based on my reading’ — reading is not verification. Run it.</blockquote><blockquote>* ‘The implementer’s tests already pass’ — the implementer is an LLM. Verify independently.</blockquote><blockquote>* ‘This is probably fine’ — probably is not verified. Run it.”</blockquote><p><strong>6. Mandatory adversarial probe.</strong> Before PASS, the agent must run at least one concurrency, boundary, idempotency, or orphan-operation probe. This is the guard against confirming the happy path.</p><pre>const VERIFICATION_WHEN_TO_USE =<br>  &#39;Invoke after non-trivial tasks (3+ file edits, backend/API changes, infrastructure changes).&#39;</pre><p>The “3+ file edits” threshold is the point at which self-verification becomes unreliable enough to warrant an independent agent. Below that, the self-reporting risk is acceptable. Above it, the +80% gain from structured external feedback over +1.8pp unguided self-reflection justifies the overhead.</p><p><strong>What to change today:</strong> Write your verification agent’s system prompt around its failure modes, not its goals. Require evidence, not narration. Require a parseable verdict. Pre-write the rationalizations it will reach for — then block them.</p><h2>8. Swarm Initialization Distinguishes Resumed From Fresh</h2><p>Multi-agent systems reduce error amplification from 17.2× (unstructured) to <strong>4.4× with centralized orchestration</strong> (Google DeepMind, Dec 2025) — but only when the agents correctly reconstruct their state on startup. A resumed agent that re-initializes as a fresh agent loses its team identity, its permission hooks, and its place in the workflow.</p><p>Claude Code’s swarm initialization explicitly handles both paths:</p><pre>// src/hooks/useSwarmInitialization.ts<br><br>export function useSwarmInitialization(<br>  setAppState: SetAppState,<br>  initialMessages: Message[] | undefined,<br>): void {<br>  useEffect(() =&gt; {<br>    if (isAgentSwarmsEnabled()) {<br>      const firstMessage = initialMessages?.[0]<br>      const teamName = firstMessage?.teamName<br>      const agentName = firstMessage?.agentName<br><br>      if (teamName &amp;&amp; agentName) {<br>        // RESUMED — identity stored in transcript<br>        initializeTeammateContextFromSession(setAppState, teamName, agentName)<br>        const teamFile = readTeamFile(teamName)<br>        const member = teamFile?.members.find(m =&gt; m.name === agentName)<br>        if (member) {<br>          initializeTeammateHooks(setAppState, getSessionId(), {<br>            teamName, agentId: member.agentId, agentName,<br>          })<br>        }<br>      } else {<br>        // FRESH spawn — identity from environment<br>        const context = getDynamicTeamContext?.()<br>        if (context?.teamName &amp;&amp; context?.agentId &amp;&amp; context?.agentName) {<br>          initializeTeammateHooks(setAppState, getSessionId(), context)<br>        }<br>      }<br>    }<br>  }, [setAppState, initialMessages])<br>}</pre><p>A resumed agent reads teamName and agentName from firstMessage — the transcript is the source of truth. A fresh agent reads from the environment. Team membership is stored in a teamFile on disk; each member has an agentId that survives resume. The permission hooks are initialized from this persisted state.</p><p><strong>Why this distinction matters at scale:</strong> Princeton NLP’s 2025 study found single agents win <strong>64% of sequential tasks</strong> — multi-agent coordination only pays on parallelizable work. Gains plateau above 4 concurrent agents. Coordination overhead for resumed sessions that incorrectly re-initialize as fresh agents is exactly the kind of waste that drives teams below that threshold.</p><p><strong>What to change today:</strong> Explicitly handle the fresh vs resumed startup case for every stateful agent. Store agent identity in the transcript, not just the environment. On resume, reconstruct from transcript state. A resumed agent that re-initializes fresh is indistinguishable from a new agent — it loses everything that makes coordination work.</p><h2>9. The Buffer Gate for Message Ordering</h2><p>Single-agent latency is 2–4 seconds per task; multi-agent is 8–15 seconds (multi-agent framework benchmarks, 2025). Interleaved messages during session flush are one of the fastest ways to turn a 2-second response into a broken one that requires a full restart.</p><p>When a Claude Code session starts, it flushes historical messages to the server in a single HTTP POST. New messages that arrive during this flush must be queued — not sent — or they interleave with the historical batch and produce a broken conversation state.</p><pre>// src/bridge/flushGate.ts<br><br>export class FlushGate&lt;T&gt; {<br>  private _active = false<br>  private _pending: T[] = []<br><br>start(): void { this._active = true }<br><br>end(): T[] {<br>    this._active = false<br>    return this._pending.splice(0)   // atomic: drain and clear in one operation<br>  }<br><br>enqueue(...items: T[]): boolean {<br>    if (!this._active) return false<br>    this._pending.push(...items)<br>    return true<br>  }<br><br>drop(): number {<br>    this._active = false<br>    const count = this._pending.length<br>    this._pending.length = 0<br>    return count<br>  }<br><br>deactivate(): void {<br>    this._active = false<br>    // Does NOT drop items — new transport will drain them<br>  }<br>}</pre><p>deactivate() clears the active flag without dropping items — for transport replacement. drop() discards everything — for permanent close. end() uses splice(0), not this._pending then this._pending = [], because splice(0) is atomic: returns all items and empties the array in a single operation with no window for new items to slip in.</p><p>For capacity management, the poll loop uses two intervals:</p><pre>// src/bridge/pollConfigDefaults.ts<br><br>const POLL_INTERVAL_MS_NOT_AT_CAPACITY = 2000     // actively seeking work<br>const POLL_INTERVAL_MS_AT_CAPACITY = 600_000       // 10 minutes — liveness only</pre><p>Not-at-capacity: poll every 2 seconds. At-capacity: poll every 10 minutes — not to pick up work (the WebSocket transport handles that), but as a liveness signal to keep the Redis key alive (TTL = 4 hours; 10-minute poll gives 24× headroom).</p><p><strong>What to change today:</strong> If your agent handles streaming messages or session bridging, implement a flush gate before you need it. The bugs from interleaved messages are hard to reproduce and hard to debug. The gate takes 30 minutes to add proactively and days to diagnose after the fact.</p><h2>What These 9 Patterns Have in Common</h2><p><strong>1. Architecture enforces constraints that prompts cannot.</strong> The verification agent can’t edit files because the tool isn’t available. Denial limits are hardcoded constants, not configurable at runtime. Permission defaults are fail-closed. Instructions drift; architecture holds.</p><p><strong>2. Naming makes costs visible.</strong> DANGEROUS_uncachedSystemPromptSection, checkStatsigFeatureGate_CACHED_MAY_BE_STALE, _reason as a required-but-unused parameter. The naming convention is the documentation. Cache-breaking decisions can&#39;t be made accidentally.</p><p><strong>3. Failure modes are written down.</strong> The verification agent’s system prompt names its own two failure patterns. The denial circuit breaker has exact thresholds from production incidents. The flush gate has a deactivate() path specifically for transport replacement. These aren&#39;t defensive programming — they&#39;re the result of things going wrong in production, being understood, and being fixed precisely.</p><p><strong>4. Fresh and resumed are always distinguished.</strong> Every stateful component handles both startup paths. This is the difference between a system that works in demos (always fresh) and one that works in production (frequently resumed).</p><p><strong>5. Context placement is intentional.</strong> CLAUDE.md in the user turn, git status in the system prompt, critical reminders in a separate injection field. The middle of context is where information goes to be forgotten — ~30% accuracy drop (Liu et al., Stanford 2024). Every piece of information has a deliberate placement relative to that constraint.</p><p>The 73% first-year failure rate isn’t a model problem. The models are good enough. These patterns are what separates agents that work in production from agents that work in demos.</p><h2>Seeing It In Practice: A Real Review of OpenClaw</h2><p>To make this concrete, here’s what /review-agent produced when run against <a href="https://github.com/openclaw">OpenClaw</a> (<a href="https://github.com/openclaw/openclaw">https://github.com/openclaw/openclaw</a>), a production-grade open-source pi/subagent runtime. The command explored the repo automatically — no code was pasted.</p><p>A few things worth noting from this output:</p><p><strong>What scored well:</strong> OpenClaw had two ✅ dimensions — orchestration and security — which are the hardest to retrofit. The prompt-cache stability engineering (prompt-cache-stability.ts, system-prompt-cache-boundary.ts) is exactly the pattern from section 2 of this post: capability IDs normalized and sorted, explicit stable/dynamic boundary enforced in code. The defense-in-depth security stack (sandbox isolation, safe-bin allowlists, session write locks, audit logging) maps directly to the Zero Trust model in the best practices.</p><p><strong>Where the gaps were:</strong> Four ⚠️ dimensions. The most critical: no independent verification worker. Agents self-report pass/fail with no adversarial second pass — which is the unguided self-reflection problem (+1.8 pp vs +80% with structured external feedback). The second gap: working memory is excellent but semantic/procedural memory tiers don’t exist as first-class constructs. Every session starts cold.</p><p><strong>The top improvement:</strong> Spawn a fresh verification subagent after every implementation task. Give it only <em>what</em> changed, not <em>how</em> — and an adversarial brief. One architectural change, 17.2× → 4.4× error amplification.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Vx1ZZFhrSDXxck1kOQDEWg.png" /></figure><h2>The Reference Library</h2><p>All 9 best-practice docs with benchmark tables and primary source citations are open source, along with skills for Claude Code, Cursor, Codex CLI, Gemini CLI, Windsurf, Aider, and Continue.dev:</p><p><a href="https://github.com/sarkarsaurabh27/agent-loop-learning">https://github.com/sarkarsaurabh27/agent-loop-learning</a></p><pre>/review-agent    → full audit against 9 dimensions with ✅/⚠️/❌ scorecard<br>/improve-agent   → improvement cards for a specific component<br>/best-practices  → load a reference doc by topic keyword</pre><p><em>All code excerpts are from the Claude Code source (April 2026 codebase). All benchmark numbers include primary source citations in the reference library above.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62c369a1d290" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Startup 101 : Selecting the best deployment resource — EC2 vs EKS vs Lambda]]></title>
            <link>https://medium.com/@dreams-smoke/startup-101-selecting-the-best-deployment-resource-ec2-vs-eks-vs-lambda-3c664c7dbf6f?source=rss-798029bc6edb------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c664c7dbf6f</guid>
            <dc:creator><![CDATA[Saurabh Sarkar]]></dc:creator>
            <pubDate>Thu, 19 Sep 2024 19:50:26 GMT</pubDate>
            <atom:updated>2024-09-19T19:50:26.094Z</atom:updated>
            <content:encoded><![CDATA[<h2>Startup 101 : Selecting the best deployment resource — EC2 vs EKS vs Lambda</h2><p>When we started working on our new project idea for creating data pipelines for our customer, it required us deploying our service on a cloud provider. AWS was what came to our mind given we wanted to leverage some existing managed services/products for customers and of course the fact that AWS provides some initial credits to start with.</p><p>We wanted to develop an API service and of course in AWS’s ecosystem there are multiple ways to achieve it. You can go serverless, you can plan your capacity and make a data driven decision of choosing a particular instance to have more control, or you can outsource your resource management to certain orchestrators. For a developer, who wants to solve their particular use case, even after working hard to write the most optimal service, this becomes a blocker of how should I deploy it or what resource should I use. I wish I could simply say “it works on my machine” and ship my box, however, it’s not practical and that’s why we have all these cloud providers to “make our life easy”.</p><p>Here are the top 3 choices we evaluated:</p><p>Amazon EC2 (Elastic Compute Cloud):</p><p>* EC2 is Infrastructure as a Service (IaaS), which means you have more control over the underlying infrastructure.</p><p>* You provision and manage virtual machines (EC2 instances) where you install and configure your application and dependencies.</p><p>* EC2 offers a wide range of instance types with varying CPU, memory, and GPU configurations, so you can choose instances that fit your specific workload and budget.</p><p>* Cost depends on the instance type, uptime, and other factors, and it may require more upfront configuration and management.</p><p>* It provides full flexibility and control but might take longer to set up compared to serverless options like Lambda.</p><p>Amazon EKS (Elastic Kubernetes Service):</p><p>* EKS is a managed Kubernetes service that abstracts the underlying infrastructure, allowing you to focus on deploying and managing your containerized applications.</p><p>* It’s a good choice if you want to use containerization (e.g., Docker) to package your application, as it provides orchestration and scaling capabilities.</p><p>* EKS can scale your application automatically based on demand, which can be helpful for quick delivery.</p><p>* Cost includes both the EKS control plane and the EC2 instances used for your containers, but it can be more cost-effective than managing Kubernetes yourself.</p><p>AWS Lambda:</p><p>* Lambda is a serverless compute service where you write code (functions) and AWS handles the infrastructure provisioning, scaling, and management.</p><p>* It’s an excellent choice if your application can be broken down into small, event-driven functions and if you want to minimize operational overhead.</p><p>* Lambda can be extremely cost-effective for workloads with sporadic or variable usage because you only pay for the compute time your functions use.</p><p>* It may be less suitable for applications with long-running processes or specific runtime requirements.</p><p>To choose the right deployment resource, consider the following factors:</p><p>* Application Architecture: The nature of your application (monolithic, microservices, serverless) will influence your choice.</p><p>* Development Speed: If you need to deliver quickly, Lambda or EKS can be faster to set up and manage compared to EC2 because they abstract much of the underlying infrastructure.</p><p>* Cost: EC2 gives you more control over instance types, which can be budget-friendly if you choose cost-effective instances. Lambda can be cost-effective for sporadic workloads.</p><p>* Resource Requirements: Calculate the required compute resources for your application. EC2 and EKS let you choose specific instance types, while Lambda dynamically scales based on function invocations.</p><p>* Skillset: Consider your team’s expertise. EC2 and EKS require more knowledge of infrastructure management, whereas Lambda abstracts much of it.</p><p>This is exactly what we did, evaluated our needs based on the above factors. Let me provide some insight there:</p><ul><li>For a startup we need to produce as fast as possible. We don’t have change management documents or process for releasing a new feature. It happens over night. So we required some resource, which is one click away and doesn’t need too much intervention.</li><li>Initially we just required to deploy a REST API over an external endpoint which could expect an incoming traffic of max 5000 requests in a day or sometimes even less. The requirement grows over time as you scale with your customers. This is where we calculated the trade-off of capacity planning/management and going serverless. We chose the latter.</li><li>Cost is a very important factor for an early stage startup. Extravagant utilization is always never an option. Depending on our usage we evaluated per instance cost of EC2, EKS cluster and per request call for Lambda. Given that the usage is sporadic and not consistent; to reserve certain size of resources is not the most optimal choice.</li><li>Usually for a web service (micro service), with limited resources wrt developers, you want to spend your time &amp; focus on the business statement rather than figuring out the underlying architecture, which might be the focus in the future. Even if you have AWS expertise, spending unnecessary time on resource planing might not be the best of the ideas for a tech startup.</li></ul><p>Usually, it is completely subjective to known vs unknowns, resources to spare and of course cost to pay. We do have resources now in both EC2 and EKS as we have specific requirements from customers, with expected traffic and usage. But initially, we ended up finalizing Lambda service to start with.</p><p>For basic resource requirements, you can start by estimating your application’s CPU and memory needs based on usage patterns and then choose instance types or Lambda function configurations accordingly. AWS also provides cost calculators to help estimate expenses for each service. Remember to factor in potential future growth when making your decision.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c664c7dbf6f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>