<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ali Ibrahim on Medium]]></title>
        <description><![CDATA[Stories by Ali Ibrahim on Medium]]></description>
        <link>https://medium.com/@techwithibrahim?source=rss-fd16cb1da896------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*zUrydUUwoXHPljyRx2anbA.png</url>
            <title>Stories by Ali Ibrahim on Medium</title>
            <link>https://medium.com/@techwithibrahim?source=rss-fd16cb1da896------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 14:03:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@techwithibrahim/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Add Agent Skills Support to a Custom Agent Harness]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://techwithibrahim.medium.com/how-to-add-agent-skills-support-to-a-custom-agent-harness-5dd110ca4630?source=rss-fd16cb1da896------2"><img src="https://cdn-images-1.medium.com/max/1200/1*dGu-lj7whCe-4Kr_7MiuEQ.png" width="1200"></a></p><p class="medium-feed-snippet">Add Agent Skills support to your custom AI agent: SKILL.md discovery, progressive disclosure, tool loading, and evals, with TypeScript and&#x2026;</p><p class="medium-feed-link"><a href="https://techwithibrahim.medium.com/how-to-add-agent-skills-support-to-a-custom-agent-harness-5dd110ca4630?source=rss-fd16cb1da896------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://techwithibrahim.medium.com/how-to-add-agent-skills-support-to-a-custom-agent-harness-5dd110ca4630?source=rss-fd16cb1da896------2</link>
            <guid isPermaLink="false">https://medium.com/p/5dd110ca4630</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Ali Ibrahim]]></dc:creator>
            <pubDate>Mon, 02 Nov 2026 13:31:02 GMT</pubDate>
            <atom:updated>2026-11-02T13:31:02.127Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Before You Add WebMCP: the Security Problem Nobody’s Selling You]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://techwithibrahim.medium.com/before-you-add-webmcp-the-security-problem-nobodys-selling-you-a38da0f0e36b?source=rss-fd16cb1da896------2"><img src="https://cdn-images-1.medium.com/max/1200/1*RRrorwFo0E926OWb4Aaagg.png" width="1200"></a></p><p class="medium-feed-snippet">WebMCP lets a web page hand tools to AI agents, and most coverage sells it as your website becoming an API. The spec, WebKit, and even&#x2026;</p><p class="medium-feed-link"><a href="https://techwithibrahim.medium.com/before-you-add-webmcp-the-security-problem-nobodys-selling-you-a38da0f0e36b?source=rss-fd16cb1da896------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://techwithibrahim.medium.com/before-you-add-webmcp-the-security-problem-nobodys-selling-you-a38da0f0e36b?source=rss-fd16cb1da896------2</link>
            <guid isPermaLink="false">https://medium.com/p/a38da0f0e36b</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Ali Ibrahim]]></dc:creator>
            <pubDate>Sun, 25 Oct 2026 15:04:20 GMT</pubDate>
            <atom:updated>2026-10-25T15:04:20.209Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[We Tried Replacing Our LLM Judges With Jev]]></title>
            <link>https://techwithibrahim.medium.com/we-tried-replacing-our-llm-judges-with-jev-4de6492a594b?source=rss-fd16cb1da896------2</link>
            <guid isPermaLink="false">https://medium.com/p/4de6492a594b</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Ali Ibrahim]]></dc:creator>
            <pubDate>Fri, 02 Oct 2026 13:01:03 GMT</pubDate>
            <atom:updated>2026-10-02T13:01:03.337Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ItSojzX_Fti6k0AdnJF4gA.png" /></figure><p>Your agent needs checking. So you call another model to check it. Now every eval has a second bill, and a second model that can be wrong.</p><p>TypeSafe’s Jev offers a cheaper way to judge: give it an input and a typed question, get back a probability instead of a paragraph. On our eval inputs, it cost about <strong>180 times less than Claude Sonnet 4.6</strong>.</p><p>Could we replace our LLM judges with it? We tested that. <strong>Jev alone fell behind Sonnet. A Jev-first cascade reached comparable accuracy at about a sixth of the cost.</strong></p><p>This is the short version of <strong>Agentailor Report AR-001</strong>. <a href="https://blog.agentailor.com/reports/ar-001-can-jev-replace-llm-judges">Read the full report in HTML</a> or <a href="https://blog.agentailor.com/reports/ar-001-can-jev-replace-llm-judges#get-pdf">get the PDF</a>.</p><h3>Cheap is easy to measure. Right is harder.</h3><p>We took <strong>193 items</strong> from our production agent’s eval work: 175 historical judge inputs and 18 edited answers designed to expose rarer failures. Each item pairs an answer with a rubric, such as whether it disclosed the agent’s model or made unsupported claims.</p><p>We replayed Jev, Sonnet, Haiku and Flash-Lite five times each. The reference was <strong>human labels</strong>, with blind labeling and a review pass. Two items remained unsure, leaving 191 for accuracy scoring.</p><p>Here is what the majority verdicts got right:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vQbliVtLP0bed36yo-R_tg.png" /><figcaption>Accuracy against cost per 1,000 verdicts</figcaption></figure><p>These are costs on our inputs, at prices recorded on September 23, 2026.</p><p>Flash-Lite won on overall accuracy. It also <strong>missed 13.2% of real defects</strong>, against Sonnet’s 5.8%. A judge can score better overall while letting more broken answers through. For an eval that gates a merge, that distinction matters.</p><p>The rubric mattered too. On narrow yes/no checks, Jev scored <strong>96.3%</strong> and all three LLM judges scored <strong>98.2%</strong>. On nuanced decision tables, Jev dropped to <strong>80.6%</strong>. Cheap judgments get harder when the question contains several interacting rules.</p><h3>The useful part is knowing when to escalate</h3><p>Jev returns a probability with its verdict. That gave us something more useful than a cheaper yes or no: a way to route uncertain judgments.</p><p>On the <strong>125 items</strong> where its probability was above 0.9 or below 0.1, Jev made <strong>zero errors</strong>. Its mistakes clustered closer to 0.5.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NyXyW8f39J3r-MndT4zAWw.png" /><figcaption>Jev’s error rate by how far its probability is from 0.5</figcaption></figure><p>So we tested a cascade:</p><ol><li>Ask Jev first.</li><li>Keep its confident verdicts.</li><li>Send the uncertain ones to Sonnet.</li></ol><p>One tested setting sent <strong>16.2%</strong> of items to Sonnet. It reached <strong>93.7% accuracy</strong> at <strong>$2.22 per 1,000 verdicts</strong>, versus Sonnet alone at 93.2% and $13.21. It also caught all <strong>38 real defects</strong> in the labeled set.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NHTcpQkDJ-BaJ_EO8OXjOA.png" /><figcaption>Measured on the same labeled set used to assess the routing settings. Validate the threshold on fresh cases before deploying.</figcaption></figure><p>That is the promising result. It is also <strong>an in-sample result</strong>: we evaluated the routing settings on the same labels used to assess them. Zero missed defects here is no promise about the next batch. A deployment needs fresh labeled data to validate its threshold.</p><h3><strong>A better judge cannot see missing evidence</strong></h3><p>The experiment also caught a problem in our harness.</p><p>One rubric judged the build brief the agent produces after a session. But the judge could not see the earlier tool calls. Sonnet accused the brief of inventing sources because it believed the agent had never read them.</p><p>The agent had read them. <strong>The harness had hidden the evidence.</strong></p><p>Sonnet, Haiku and Jev all scored 0% on that rubric. Paying for a stronger judge would not fix the input it received.</p><p>Before changing judges, check what each rubric asks them to establish, and whether they can actually see it.</p><h3><strong>What I would change first</strong></h3><p>For simple checks, benchmark a cheap judge against your own hand labels. For long decision tables, keep the stronger judge and its written reasoning. For a confidence-based cascade, validate the routing on fresh cases and track missed defects alongside accuracy.</p><p>Our evidence comes from one agent and one labeler, who also owns it. The full report includes the label corrections, repeat-run results, rubric breakdowns and the cascade explorer. Its public repository lets you recompute every reported number; the underlying case texts are withheld.</p><p><a href="https://blog.agentailor.com/reports/ar-001-can-jev-replace-llm-judges"><strong>Read AR-001: Can Jev Replace Our LLM Judges?</strong></a></p><p>Want a copy to keep or share? <a href="https://blog.agentailor.com/reports/ar-001-can-jev-replace-llm-judges#get-pdf"><strong>Get the PDF</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4de6492a594b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Write Your First AI Agent Evals]]></title>
            <link>https://techwithibrahim.medium.com/how-to-write-your-first-ai-agent-evals-b9507c6f380d?source=rss-fd16cb1da896------2</link>
            <guid isPermaLink="false">https://medium.com/p/b9507c6f380d</guid>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Ali Ibrahim]]></dc:creator>
            <pubDate>Mon, 14 Sep 2026 18:56:09 GMT</pubDate>
            <atom:updated>2026-09-14T18:56:09.961Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QaBZAfv5YLjXG_NJ5cG_9g.png" /></figure><p>Where the first cases come from, which ones earn a place in the suite, why every case that pushes a behavior needs one that bounds it, how to grade without an LLM judge, and what the first honest run surfaced.</p><p>This is one chapter of an evolving case study. <a href="https://blog.agentailor.com/cameron"><strong>Catch up on the story so far</strong></a></p><h3>The starting point</h3><p><a href="https://blog.agentailor.com/cameron">Cameron</a> — the local-first personal finance agent I build in public, and whose <a href="https://github.com/agentailor/cameron">source is open</a> — had a unit test suite that was green, fast, and free. It also could not catch the failures I actually worried about.</p><p>A unit test proves truncated: true is <em>in</em> a tool&#39;s JSON payload. It cannot prove the agent <strong>noticed</strong> it. That gap is where evals live, and closing it takes two things the test suite did not need: somewhere to run the real agent, and a set of tasks worth running.</p><h3>You need a harness first, and it might be you</h3><p>You cannot write an eval case without something to run it on. That something is a harness: it drives the real agent, records what happened, and decides whether the result was good enough.</p><p>For a small agent, <strong>the harness can be you.</strong> Run the agent, read the answer, decide if it is right. That is a real eval — it has a case, a run, and a grader, and the grader is a human. It is the correct starting point, and it stays correct right up until the run count times the case count stops fitting in an afternoon.</p><p>Cameron passed that point. The suite below needs a wiped database between runs, a paused tool call read off a checkpoint, and eleven cases where some run three times — twenty-one agent invocations that nobody is going to drive by hand. So it got a real one: a harness that drives the agent against a sandbox Postgres and grades with plain functions. You have the same three options — grade by hand, <a href="https://blog.agentailor.com/blog/agent-eval-harness">build one</a>, or adopt a framework — and the article on building it walks through the four components so the choice is informed.</p><p>Either way, what you have at that point is a harness and an empty suite. These are the notes from filling it.</p><h3>What a case actually is</h3><p>A case is <strong>one task you give the agent, plus the graders that decide whether what came back was acceptable.</strong> That is the whole shape:</p><pre>{<br>  id: &#39;truncated-page-not-reported-as-total&#39;,<br>  prompt: &#39;How much did I spend on dining this year?&#39;,<br>  graders: [toolCalled(&#39;run_sql&#39;), statesAmount(FIXTURE.diningTotal)],<br>}</pre><p>A prompt, and assertions over what happened. Not an input and an expected output — there is no single correct answer string to compare against, which is why the right-hand side is a list of graders rather than a value.</p><p>Two things follow from that, and they shape everything below. A grader can assert on <strong>what the agent said</strong> or on <strong>what the agent did</strong> — which tools it called, what is in the database afterwards — and the second kind is usually the stronger one. And a case is not passed or failed by a person reading it, so whatever you want to be true has to be expressible as code.</p><h3>Where the first cases come from</h3><p>The hard part is not the format. It is knowing which eleven tasks are worth paying a model to run.</p><p>You cannot reason your way to that list from an empty page, and you do not have to. There are three places the first cases come from, and none of them require an eval suite to already exist:</p><ul><li><strong>You, using your own agent.</strong> Run it and pay attention to what makes you wince. This is where every case in this suite started.</li><li><strong>Traces, if the agent is in production.</strong> Real failures, real phrasings, no imagination required. It is the best source there is, and it is the honest reason tracing comes before evals: you cannot distill cases from traffic you never recorded.</li><li><strong>Someone who knows the domain.</strong> A business expert who has never seen the code can usually list the edge cases that matter within minutes, because they know which wrong answers are expensive.</li></ul><p>Cameron’s came from the first. Before any of this existed I sat down and probed the agent by hand, and it told me two things: how much had been spent on dining (<strong>$2,752.25</strong>, correct), and then, asked to list the transactions and add them up, <strong>$2,812.25</strong>. Sixty dollars off, stated with total confidence.</p><p>That produced a list of observed failures, and the next move was to write tests for them. Some landed cleanly: the underlying tool really was returning a capped page that looked complete, and <a href="https://blog.agentailor.com/blog/testing-tools-for-ai-agents">a unit test caught that in a millisecond</a>. Others would not reduce to a test at all. The payload was correct and the agent still added it up wrong — and only sometimes, which is the part no assertion over a return value can reach.</p><p>I could have fixed that one and probed it again by hand. But a fix you cannot re-verify is a fix nobody can prove holds, and re-verifying by hand does not survive the tenth time. So the failures that had no test-shaped home became the first eval cases instead.</p><p>That order is worth keeping. <strong>Observe a failure, try to push it down to the cheapest layer that can catch it, and let evals inherit only what will not fit.</strong> A case that a unit test could have caught is a case you will pay for on every run, forever.</p><h3>What earns a case</h3><p>Those three sources give you candidates, not a suite. Left unfiltered they sprawl — the temptation is to keep one case per failure you ever saw, or worse, one per feature, which produces something expensive, slow, and mostly redundant with tests you already have.</p><p>The rule that worked: <strong>a case earns its place by guarding a defect class a unit test provably cannot reach.</strong> For Cameron that meant five:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0EAvVfaBZ350h5cXBEucVg.png" /></figure><p>Eleven cases, grouped by the defect class they guard rather than by feature:</p><pre>analysis     total-via-sql-aggregate                      aggregate in SQL, don&#39;t sum rows by hand<br>             top-categories-ranking                       GROUP BY in SQL, don&#39;t page transactions<br>             listing-uses-query-transactions              the inverse guard: list with the list tool<br><br>truncation   truncated-page-not-reported-as-total         never report the capped page&#39;s 200 rows<br><br>approval     log-expense-pauses-for-approval              pause at the gate, then write exactly one row<br>             denied-expense-writes-nothing                a denied mutation leaves the ledger untouched<br><br>discipline   no-double-prompt-on-mutation                 the tool call IS the proposal, don&#39;t ask twice<br>             unknown-category-recovers                    an unknown category is not &quot;no spending yet&quot;<br><br>csv          csv-import-confirms-ambiguous-date-format    5 July, not 7 May: ask before importing<br>             csv-import-does-not-import-before-confirming inspect on turn one, never write<br>             csv-import-maps-accented-category-header     map `Catégorie`, don&#39;t translate it</pre><p>Small on purpose — the anti-pattern is believing you need a big suite before you start.</p><h4>Pair every case that pushes with one that bounds</h4><p>The first tool-selection case asserts the agent uses SQL for a total. On its own, that invites a prompt fix that over-rotates into <em>always</em> use SQL — breaking plain listings, with the suite still green.</p><p>So each case that pushes a behavior gets a partner that bounds it. “Show me my 5 most recent transactions” asserts the opposite: query_transactions, <strong>not</strong> SQL.</p><blockquote><strong>Note:</strong> If you seed your own fixture rather than running against production data, it is a design surface too: <a href="https://blog.agentailor.com/blog/agent-eval-harness">engineer it so a wrong answer stays visible</a>. One habit belongs with the cases themselves — <strong>derive expected values from the fixture, never hardcode them.</strong> Type a total in by hand and every case asserting it goes red the day you add a row, for a reason that has nothing to do with the agent.</blockquote><h3>The vacuous pass</h3><p>None of the eleven cases uses an LLM judge, which sounds like a stance and is really just a consequence: every one of them asserts a figure, a tool name, or a row count, and <strong>a number has one spelling.</strong> Reach for a judge when the target genuinely has many correct spellings, and not before — <a href="https://blog.agentailor.com/blog/agent-eval-harness">the harness guide</a> has that side, with a worked rubric.</p><p>Staying judge-free has a trap of its own, though, and it is the subtlest way to write a worthless case: <strong>a negative grader with nothing positive beside it.</strong></p><p>toolNotCalled(&quot;run_sql&quot;) passes when the agent errors, refuses, or answers from thin air. It passes hardest when the agent does <em>nothing at all</em>. Every negative assertion needs a positive one proving the work actually happened.</p><h3>Making the approval gate observable</h3><p>Cameron’s first hard rule is that it never mutates without explicit human approval — and it was the last capability to get a case, because the harness had been running with the approval middleware switched off. An agent built that way cannot pause at all, so any case asserting the rule would have gone green against an agent with no gate. (<a href="https://blog.agentailor.com/blog/agent-eval-harness">The harness guide</a> has why that happens, and what a runner has to capture so it cannot.)</p><p>Making the rule evaluable means the case declares its own decision — approval: &quot;allow&quot; | &quot;deny&quot; - and the runner answers the pause the way the UI would, reading the paused call off the checkpoint and resuming with it:</p><pre>// With `approval` set the middleware is live, so a mutating call pauses instead of running.<br>if (testCase.approval) {<br>  const actions = await pendingActions(agent, threadId)<br>  interrupts.push(...actions) // captured separately — see below<br><br>  if (actions.length &gt; 0) {<br>    const decision: Decision =<br>      testCase.approval === &#39;allow&#39;<br>        ? { type: &#39;approve&#39; }<br>        : { type: &#39;reject&#39;, message: &#39;The user denied this action.&#39; }<br>    // One decision per action request, positionally aligned — the middleware batches a<br>    // turn&#39;s approval-requiring calls into a single interrupt.<br>    result = await agent.invoke(<br>      new Command({ resume: { decisions: actions.map(() =&gt; decision) } }),<br>      config<br>    )<br>  }<br>}</pre><p>Note pendingActions returning an empty array when nothing is pending. For an approval case that is itself the finding: the mutation never reached the gate.</p><p>That empty array is also the only thing separating the two outcomes, because <strong>a paused tool call and an executed one look identical in the trajectory</strong> — a paused call is still a call the model made. The two cases come back from a real run byte-identical on everything except the row count:</p><pre>// log-expense-pauses-for-approval          // denied-expense-writes-nothing<br>{ &quot;trajectory&quot;: [&quot;log_expense&quot;],            { &quot;trajectory&quot;: [&quot;log_expense&quot;],<br>  &quot;interrupts&quot;: [&quot;log_expense&quot;] }             &quot;interrupts&quot;: [&quot;log_expense&quot;] }</pre><p>For a case author that has one practical consequence: <strong>you cannot assert the gate from the trajectory.</strong> The assertion has to be an allow/deny pair on the same prompt, separated only by the decision, and graded on what is in the database afterwards:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b-nGVb2IXtpd9bb0hRsBwQ.png" /></figure><p>A regression where a rejected mutation still writes is the worst bug this project could ship, and it now fails loudly.</p><blockquote><strong><em>Agent Briefings:</em></strong><em> Practical AI agent engineering — what works in production, and what’s just hype. </em><a href="https://blog.agentailor.com/#newsletter"><strong><em>Get the next one in your inbox.</em></strong></a></blockquote><h3>What the first honest run found</h3><p>Every case passed but one. The red one was unknown-category-recovers, and it was worth more than all the greens put together, because it turned out to be <strong>two</strong> problems wearing one coat.</p><p>The case asks about a category the fixture does not contain, and expects the agent to establish that rather than report an empty result as “no spending yet.” As first written:</p><pre>{<br>  id: &#39;unknown-category-recovers&#39;,<br>  prompt: &#39;How much did I spend on Entertainment?&#39;, // a category the fixture does not contain<br>  graders: [<br>    toolCalled(&#39;list_categories&#39;),<br>    categoriesListed(), // only checks the list_categories result<br>  ],<br>}</pre><p><strong>The case was broken.</strong> It asserted the agent would call list_categories. The agent instead reached for describe_finance_schema then run_sql - a different route to the same place, and an entirely legitimate one. The case was asserting a path when what it cared about was an outcome.</p><p>Worse, the second grader was conditioned on that same tool. When list_categories never ran, it had nothing to check and reported green. A vacuous pass, sitting in the one case that had gone red for an unrelated reason - which is how a grader that asserts nothing stays hidden.</p><p>The fix was to stop naming the route:</p><pre>graders: [<br>  // Route-agnostic: several tool paths establish which categories exist, and asserting<br>  // one would fail a run that got there another way.<br>  anyToolCalled(&#39;list_categories&#39;, &#39;query_transactions&#39;),<br>  // The recovery must name a category that really exists. Atomic — a category name has<br>  // one spelling — so this survives having no judge.<br>  statesAnyOf(FIXTURE.categoriesBySpend.map((c) =&gt; c.category)),<br>]</pre><p>Note the second grader reading from the fixture rather than a hardcoded list. Add a category to the seed and the case keeps asserting the right thing.</p><p><strong>And a real product defect.</strong> Asked about a category that does not exist, a bare SUM() returns SQL NULL, which is indistinguishable from a category that exists with no spending. The agent passed that ambiguity straight through:</p><pre>// The failing run: two tool calls, and a SQL NULL shown to the user.<br>{<br>  &quot;trajectory&quot;: [&quot;describe_finance_schema&quot;, &quot;run_sql&quot;],<br>  &quot;finalText&quot;: &quot;The result is **null**, which means you don&#39;t have any transactions<br>                categorized as **Entertainment** yet. Would you like me to:<br>                1. Check what categories you *do* have on record? ...&quot;<br>}</pre><p>Two things are wrong there. The user is being shown <strong>null</strong>, a SQL artifact that means nothing to them. And “yet” implies the category exists and is simply empty — when it does not exist at all, and the three real categories were one tool call away. The agent even offers to go check them, which is the right next step, framed as a question instead of taken.</p><p>Rerunning the same prompt produced a different trajectory:</p><pre>// The rerun: one more tool call, and a correct answer.<br>{<br>  &quot;trajectory&quot;: [&quot;describe_finance_schema&quot;, &quot;run_sql&quot;, &quot;list_categories&quot;],<br>  &quot;finalText&quot;: &quot;**You don&#39;t have an \&quot;Entertainment\&quot; category yet.** Your current<br>                categories are: - Dining - Groceries - Transport ...&quot;<br>}</pre><p>Same prompt, same NULL from the same query. The difference is the third tool call: this run went and checked instead of guessing, and the answer is correct because of it.</p><p>That is the finding, and it is one no single run produces. Not “the agent gets this wrong” but <strong>the recovery is not reliable</strong> — the correct path costs an extra round trip, and sometimes it is not taken. A bug report saying “sometimes answers from a bare NULL” is actionable in a way that either run alone is not.</p><p>The fix went into the tool, not the prompt. run_sql now recognizes the ambiguous payload and says what it means:</p><pre>// An aggregate over zero rows returns NULL, and that payload is identical whether the<br>// filtered name exists or not — so the agent has to guess unless the result says which.<br>if (values.length &gt; 0 &amp;&amp; values.every((v) =&gt; v === null)) {<br>  return (<br>    &#39;Every value is NULL, which is what an aggregate returns when NO rows matched — it is not a &#39; +<br>    &#39;zero balance. This does NOT confirm the filtered values exist; an unknown category/account &#39; +<br>    &#39;name gives the same NULL. Check the spelling against the data (e.g. `list_categories`) &#39; +<br>    &#39;before reporting an empty result to the user.&#39;<br>  )<br>}</pre><p>That is the cheapest layer the failure could be pushed to, and it is the same move as the <a href="https://blog.agentailor.com/blog/testing-tools-for-ai-agents">truncation fix</a>: the agent was reasoning correctly over a payload that was ambiguous, so the payload got fixed rather than the reasoning. The case has passed every run since. It stays in the suite anyway — it is what proves the hint changes behavior, and what will catch it if a future prompt change stops the agent reading the note.</p><p>Both fixes landed together, which is the honest sequence: the case had to be able to fail for the right reason before the defect underneath it was worth chasing.</p><h3>Repeats are not noise reduction</h3><p>The obvious reading of a flaky case is that you should run it a few times and take the majority, to stop noise from deciding the result. That is backwards. The flapping <em>is</em> the result.</p><p>Run unknown-category-recovers once and you learn &quot;broken&quot; or &quot;fine,&quot; depending on which of those two trajectories you caught. Run it several times and you learn the rate - it was close to a coin flip - and the rate is the thing you would actually put in a bug report. &quot;Sometimes answers from a bare NULL&quot; and &quot;always does&quot; are different bugs with different priorities, and a single run cannot tell them apart.</p><p>That is what pass@k is for, and it has two knobs answering different questions. <strong>n</strong> is how much evidence you buy. <strong>k</strong> is how much a failure costs: require unanimity when a wrong outcome is silent or unrecoverable, and a majority when the behavior legitimately varies but you want the common case right.</p><pre>single:   { n: 1, passK: 1 }  // the default: near-deterministic structural assertions<br>majority: { n: 3, passK: 2 }  // behavior known to vary run to run<br>strict:   { n: 3, passK: 3 }  // a wrong outcome that is silent, unrecoverable, or both</pre><p>unknown-category-recovers runs majority - three runs, two must pass - because the behavior it guards is exactly the kind that varies. While the tool fix was being verified it ran at <strong>n=5</strong> and came back 5/5, twice; in the suite it now reports 3/3. Those are different claims, and the report records both numbers rather than a verdict:</p><pre>{ <br>  &quot;id&quot;: &quot;unknown-category-recovers&quot;, <br>  &quot;passed&quot;: true, <br>  &quot;runsAttempted&quot;: 3, <br>  &quot;runsPassed&quot;: 3 <br>}</pre><p>Contrast denied-expense-writes-nothing, which is strict. A rejected mutation that still writes is silent and unrecoverable, so two out of three is not a pass - it is a bug that happens to be outvoted.</p><p>Repeats are opt-in because they multiply the bill, and the right default depends on what the suite is for. A suite that <strong>gates merges</strong> wants repeats everywhere, since an unrepeated green cannot be told apart from a lucky one. Cameron’s suite gates nothing — it exists to be read — so most cases run once and repeats are spent only where instability is the point.</p><h3>The case that had to be built differently</h3><p>Everything above grades a single question and answer. CSV import could not work that way, and it is the capability that mattered most: a wrong date format produces a clean, <em>successful</em> import with transactions on the wrong dates. No error, no truncation signal, nothing to alert on. Every other failure in this suite is loud; that one is silent.</p><p>Two things had to change.</p><p><strong>Multi-turn.</strong> The whole point of the import flow is that the agent <strong>stops and asks</strong> before writing, and a single turn cannot express the user’s answer. So prompt becomes an array, and the second turn is the user answering the question the first turn should have provoked:</p><pre>prompt: [<br>  `${ATTACHMENT}<br><br>Import these transactions into my checking account.`,<br>  // The user answers the question the prompt requires the agent to ask. If it already<br>  // imported on turn 1, the graders catch it — this turn cannot un-import anything.<br>  `Yes — those dates are ${csv.dateFormat} (so 05/07/2026 is 5 July). Go ahead.`,<br>]</pre><p>Turns replay on one thread id and the checkpointer carries the conversation — a much smaller change than it sounds, because the agent already had persistent memory. The runner just had to stop re-counting the whole thread on every turn.</p><p><strong>A fixture built to make the failure visible.</strong> The file is French, and every date has a day under 13:</p><pre>Date,Libellé,Montant,Catégorie,Revenu/dépense<br>05/07/2026,Café du matin,4.50,Dining,Dépense<br>06/07/2026,Boulangerie,12.00,Groceries,Dépense<br>07/07/2026,Ticket de métro,2.10,Transport,Dépense</pre><p>05/07/2026 is a real date under <em>both</em> readings - 5 July and 7 May. Guess wrong and the import succeeds with zero bad rows and no error. The only evidence is the month the rows landed on. The accented Catégorie header and the French type column do a second job: a translated mapping is rejected outright, so English defaults cannot paper over the problem.</p><p>Then grade the <strong>consequence, not the conversation</strong>. Not “did it ask about the ambiguous date?” — that is a claim with many spellings, and there is no judge here — but “did the rows land in July or in May?” One is an opinion. The other is a row in a database:</p><pre>export function importedInMonth(month: number, wrongMonth: number): Grader {<br>  return {<br>    id: `importedInMonth(${month})`,<br>    grade: async () =&gt; {<br>      const months = await importedMonths()<br>      if (months.length === 0) return result(id, false, &#39;nothing was imported&#39;)<br>      const wrong = months.filter((m) =&gt; m.month !== month)<br>      if (wrong.length === 0) return result(id, true)<br>      // Name the diagnosis, not just the mismatch.<br>      const misread = wrong.some((m) =&gt; m.month === wrongMonth)<br>      return result(<br>        id,<br>        false,<br>        `imported rows landed on month(s) ${wrong.map((m) =&gt; `${m.month} (${m.count})`).join(&#39;, &#39;)}` +<br>          (misread ? ` — month ${wrongMonth} means the date format was read backwards` : &#39;&#39;)<br>      )<br>    },<br>  }<br>}</pre><p>That grader takes the <strong>wrong</strong> month as well as the right one, which is what turns a red run into a diagnosis. “Expected July, got May” is a mismatch you still have to think about. “Month 5 means the date format was read backwards” is the bug report already written.</p><p>That is the shape I would reach for again: when the failure you fear is silent, stop asserting on what the agent said and go look at what it did.</p><h4>What the scripted array could not do</h4><p>That last paragraph still holds. The array does not.</p><p>A scripted turn is replayed <strong>regardless of what the agent said</strong> — turn two goes out whether or not turn one provoked the question it answers. That works exactly as long as every question the agent asks is one I predicted when writing the case. Weeks later I taught Cameron to establish the owner’s currency before writing to the ledger, it started asking about currency, and the script kept answering about dates. Three cases went red while the agent was behaving better than before.</p><p>The suite had been quietly encoding an assumption I never meant to make: <strong>it rewarded agents that guess and punished agents that ask.</strong></p><p>The replacement is a simulated user — the case supplies a fact sheet of what the owner knows, and an LLM answers the agent’s actual questions from it. The graders in this article did not change, because the simulator changes what the agent <em>hears</em>, never what is asserted. Why that is not the LLM judge this suite still refuses, and what it costs in attribution, is <a href="https://blog.agentailor.com/blog/simulated-users-for-agent-evals">devlog 4</a>.</p><h3>Running the suite</h3><p>Eleven cases, five of them repeating, is twenty-one agent invocations — about three minutes and <a href="https://blog.agentailor.com/blog/agent-eval-harness">roughly forty cents</a>. The console output is one line per case, a dot per run:</p><pre>  truncated-page-not-reported-as-total ••• PASS (3/3)<br>  log-expense-pauses-for-approval • PASS (1/1)<br>  denied-expense-writes-nothing ••• PASS (3/3)<br>  unknown-category-recovers ••• PASS (3/3)</pre><p>The dots are the pass@k policy made visible: PASS (3/3) and PASS (1/1) are different claims, and the run you are reading should say which one it is.</p><p>That scrolls away, though, and the run cost money. So every run also writes a standalone HTML page alongside the JSON:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hu0-NT0LKpTH3PMiuRDQ0Q.png" /></figure><p>Each case expands to its graders, its trajectory, and the agent’s final answer — which is what you actually want when a case goes red, because “which grader failed” is rarely the whole story. The importedInMonth message naming a backwards date format only helps if you can find it.</p><p><a href="https://blog.agentailor.com/static/reports/cameron-eval-report.html"><strong>Open the full report</strong></a> — the real artifact from the run described in this article, all eleven cases.</p><p>The suite itself lives in <a href="https://github.com/agentailor/cameron/tree/main/eval">eval/</a>, and running it needs the sandbox stack plus an API key:</p><pre>docker compose -f compose.eval.yaml up -d   # Postgres + MinIO, on eval ports<br>pnpm eval                                   # every case<br>pnpm eval approval                          # or filter by id or tag</pre><p>Why a sandbox rather than the real database, how the runner captures interrupts, and what to reach for instead of building this yourself are all in <a href="https://blog.agentailor.com/blog/agent-eval-harness">Building an Agent Eval Harness</a>.</p><h3>What I would keep</h3><ul><li><strong>Start from observed failures, not imagination.</strong> Your own use, production traces, or someone who knows the domain. Then push each one to the cheapest layer that can catch it.</li><li><strong>Group cases by defect class, not by feature.</strong> It keeps the suite small and every case justifiable.</li><li><strong>Pair pushing cases with bounding ones</strong>, or a prompt fix will satisfy the suite by overcorrecting.</li><li><strong>Stay judge-free while your assertions are atomic.</strong> Numbers and tool names do not need a model to grade them.</li><li><strong>Never let a negative grader stand alone.</strong> A run that did nothing should not score green.</li><li><strong>Derive expected values from your fixture</strong>, never hardcode them — a constant fails for reasons that have nothing to do with the agent.</li><li><strong>Write the report to disk.</strong> Console output scrolls away; a slow, paid run deserves an artifact you can diff and re-read.</li><li><strong>Spend repeats where behavior actually varies</strong>, not everywhere. Match the default to whether your suite gates anything.</li><li><strong>Expect red on the first honest run.</strong> A case that has never been observed failing is not known to test anything — and the one failure taught me more than every pass combined.</li></ul><h3>Resources</h3><ul><li><strong>Repo:</strong> <a href="https://github.com/agentailor/cameron">github.com/agentailor/cameron</a> — the cases are <a href="https://github.com/agentailor/cameron/tree/main/eval/cases">eval/cases/</a>, the graders <a href="https://github.com/agentailor/cameron/blob/main/eval/graders.mts">eval/graders.mts</a></li><li><strong>The report:</strong> <a href="https://blog.agentailor.com/static/reports/cameron-eval-report.html">every case from this run</a>, with trajectories and final answers</li><li>The harness underneath: <a href="https://blog.agentailor.com/blog/agent-eval-harness">Building an Agent Eval Harness</a></li><li>What replaced the scripted turns: <a href="https://blog.agentailor.com/blog/simulated-users-for-agent-evals">When Your Eval Script Answers a Question Nobody Asked</a></li><li>The layer below that: <a href="https://blog.agentailor.com/blog/testing-tools-for-ai-agents">Testing Tools for AI Agents</a></li><li>Cases from production traces: <a href="https://blog.agentailor.com/blog/how-to-write-ai-agent-evals">How to Write AI Agent Evals That Prove You Wrong</a></li><li>Follow the arc on the <a href="https://blog.agentailor.com/cameron">Cameron hub</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b9507c6f380d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Agent Plugins Explained: Skills and MCP in One Folder]]></title>
            <link>https://techwithibrahim.medium.com/agent-plugins-explained-skills-and-mcp-in-one-folder-9f6e49b517d7?source=rss-fd16cb1da896------2</link>
            <guid isPermaLink="false">https://medium.com/p/9f6e49b517d7</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Ali Ibrahim]]></dc:creator>
            <pubDate>Sun, 06 Sep 2026 13:01:03 GMT</pubDate>
            <atom:updated>2026-09-06T13:01:03.572Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cLMBnjKFuBz9rkDdaZcWIQ.png" /></figure><p>Agent Plugins is an open standard for packaging Agent Skills and MCP servers into one portable folder any client can load. What plugin.json and mcp.json actually define, which clients support it, and whether to package one yet. This article was originally published at <a href="https://blog.agentailor.com/posts/agent-plugins-explained"><strong>Agentailor blog.</strong></a></p><h3>Introduction</h3><p>On August 6, 2026, <strong>Agent Plugins 1.0.0</strong> was published: an open, vendor-neutral standard for packaging Agent Skills and MCP servers into a single portable directory. Vercel proposed it. Amazon, Anysphere (Cursor), GitHub, Microsoft, and OpenAI refined it. Google announced the same day that it was joining as a core maintainer and shipping plugin support in its own products.</p><p>Here is the surprising part: <strong>almost none of it is new.</strong></p><p>Agent Plugins does not invent a skill format. It does not invent a tool protocol. It defines a manifest with two required fields, then points at two specifications that already existed and already had cross-client adoption. The entire standard is a wrapper.</p><p>That restraint is the reason to pay attention. Most “new standard” announcements ask you to learn a new format. This one asks you to put things you may already have into two folders with fixed names.</p><p>If you have read our guides on <a href="https://blog.agentailor.com/blog/how-to-build-and-deploy-agent-skill-from-scratch?utm_source=agentailor_blog&amp;utm_medium=blog_post&amp;utm_campaign=agent_plugins_explained">building an Agent Skill from scratch</a> and <a href="https://blog.agentailor.com/blog/build-an-mcp-v2-server-typescript?utm_source=agentailor_blog&amp;utm_medium=blog_post&amp;utm_campaign=agent_plugins_explained">building an MCP server</a>, you already know both halves of a plugin. This article covers the wrapper around them.</p><p><strong>What you’ll learn:</strong></p><ul><li>What a plugin actually is on disk, field by field</li><li>Why building on existing standards is the design decision that matters</li><li>The failure-isolation rule worth stealing for your own systems</li><li>What v1 deliberately leaves out, and why that is a feature</li><li>Which clients support it today, and where Claude Code sits</li><li>Whether packaging one is worth your time today</li></ul><h3>What Agent Plugins Actually Is</h3><p>A plugin is a directory. That is the whole idea.</p><pre>my-plugin/<br>├── plugin.json          # Required manifest<br>├── skills/              # Optional: Agent Skills<br>│   └── summarize/<br>│       └── SKILL.md<br>├── mcp.json             # Optional: MCP server declarations<br>└── com.example.client/  # Optional: client-specific extensions</pre><p>The problem it solves is one that plugin authors have been quietly absorbing for a year. Every client grew its own packaging format, even when the things being packaged were identical. The same skill and the same MCP server had to be repackaged per client, each with a different manifest name, directory layout, and configuration shape.</p><p>The spec authors call the result <strong>fork and drift</strong>. You publish for one client, someone forks it for another, and the two copies diverge. Authors faced a choice between reaching every client and using what made each client valuable.</p><p>Agent Plugins removes the packaging decision from that equation. One directory, loadable by any conformant client.</p><h3>Why Almost None of It Is New</h3><p>The two component types were deliberate choices, not an arbitrary starting set.</p><p><strong>Agent Skills</strong> are folders containing a SKILL.md with YAML frontmatter and Markdown instructions. Two required fields, name and description, and a body the agent loads on demand.</p><p><strong>MCP</strong> connects agents to tools and data over a defined wire protocol, now stewarded by the Linux Foundation. Our <a href="https://blog.agentailor.com/blog/mcp-v2-whats-changing-and-why?utm_source=agentailor_blog&amp;utm_medium=blog_post&amp;utm_campaign=agent_plugins_explained">MCP v2 guide</a> covers where the protocol itself is heading.</p><p>The specification is explicit that this pairing is the point. v1 focuses on these two because both have “established specifications outside this project and meaningful cross-client adoption.” Other candidate component types, commands, hooks, agents, and LSP servers, are named and excluded because their formats have not converged yet.</p><p>This is a standard that waited for consensus instead of manufacturing it. Note also the division of labor: Agent Plugins defines the mcp.json <strong>configuration file</strong>, while the MCP specification still owns the wire protocol. The new standard claims only the packaging layer.</p><h3>Anatomy of a Plugin</h3><h4>plugin.json: two required fields</h4><p>The manifest is deliberately thin. Only $schema and name are required.</p><pre>{<br>  &quot;$schema&quot;: &quot;https://agent-plugins.org/schemas/1.0.0/plugin.schema.json&quot;,<br>  &quot;name&quot;: &quot;deploy-toolkit&quot;,<br>  &quot;version&quot;: &quot;1.2.0&quot;,<br>  &quot;description&quot;: &quot;Deployment runbook and its tool integration&quot;,<br>  &quot;author&quot;: { &quot;name&quot;: &quot;Your Team&quot;, &quot;url&quot;: &quot;https://example.com&quot; },<br>  &quot;license&quot;: &quot;MIT&quot;,<br>  &quot;keywords&quot;: [&quot;deployment&quot;, &quot;ci&quot;]<br>}</pre><p>name must be 1 to 64 characters, lowercase alphanumeric with hyphens and periods, and cannot start or end with a symbol or contain -- or ... Everything beyond $schema and name is optional metadata: version, description, author, homepage, repository, license, keywords, and extensions.</p><p>The schema is <strong>closed</strong>, meaning no fields beyond those are permitted. But there is a smart exception: a client encountering an unknown top-level field must report and ignore it, then keep loading the plugin. Strict validation, forgiving failure.</p><h4>skills/: a fixed location, not a new format</h4><p>Skills live in skills/, and that location is not configurable. Each immediate child directory containing a file named exactly SKILL.md is treated as one skill.</p><p>That is the entire integration. Your existing skills, unchanged, moved into a folder with a fixed name. They still follow the <a href="https://agentskills.io/specification">Agent Skills specification</a>, including the scripts/, references/, and assets/ conventions.</p><p>The rigidity is intentional. The spec offers no flexibility in where components live and no inline declarations, because every configurable path is a place where two clients could disagree.</p><h4>mcp.json: transports, and the two placeholders</h4><p>MCP servers are declared in a separate mcp.json at the plugin root, never inline in the manifest.</p><pre>{<br>  &quot;$schema&quot;: &quot;https://agent-plugins.org/schemas/1.0.0/mcp.schema.json&quot;,<br>  &quot;mcpServers&quot;: {<br>    &quot;local-tools&quot;: {<br>      &quot;type&quot;: &quot;stdio&quot;,<br>      &quot;command&quot;: &quot;node&quot;,<br>      &quot;args&quot;: [&quot;${PLUGIN_ROOT}/server/index.js&quot;],<br>      &quot;env&quot;: { &quot;CACHE_DIR&quot;: &quot;${PLUGIN_DATA}/cache&quot; }<br>    },<br>    &quot;remote-api&quot;: {<br>      &quot;type&quot;: &quot;streamable-http&quot;,<br>      &quot;url&quot;: &quot;https://api.example.com/mcp&quot;<br>    }<br>  }<br>}</pre><p>Three transports are defined: stdio, streamable-http, and sse (the legacy transport). Transport is always declared explicitly, with no negotiation and no fallback. A client supporting MCP must support at least one of stdio or streamable-http; sse support is optional.</p><p>Two placeholders exist, and only two:</p><ul><li><strong>${PLUGIN_ROOT}</strong> resolves to the plugin&#39;s own directory.</li><li><strong>${PLUGIN_DATA}</strong> resolves to a client-managed writable directory that <strong>survives plugin updates</strong>.</li></ul><p>PLUGIN_DATA is the more interesting one. It is where installed dependencies, node_modules, virtual environments, generated code, and caches belong. Without it, every plugin update would blow away installed state, and each client would invent its own answer.</p><p>Expansion is narrow by design: a single, non-recursive textual replacement, applied only to MCP args, env values, and cwd. Clients must not perform any other placeholder or environment-variable expansion. One caveat worth internalizing: values in env are visible package data, not a secrets mechanism. Do not put credentials there.</p><p>Paths are also contained. Plugin-relative paths must begin with ./ and must resolve inside the plugin root. A &quot;../bin/server&quot; that escapes the root is rejected outright.</p><h4>Client extensions: the escape hatch that keeps the core small</h4><p>Clients still have features the portable core does not cover. Rather than growing the spec, Agent Plugins gives them namespaced space using reverse-domain naming, in the manifest:</p><pre>{<br>  &quot;extensions&quot;: {<br>    &quot;com.example.client&quot;: { &quot;autoActivate&quot;: true }<br>  }<br>}</pre><p>and as a top-level directory:</p><pre>my-plugin/<br>└── com.example.client/<br>    └── hooks/<br>        └── hooks.json</pre><p>Clients must ignore namespaces they do not implement, without validating the contents. The spec assigns <strong>no</strong> portable meaning to anything inside these directories.</p><p>This is what lets the core stay small. Vendor-specific capabilities such as custom agents, commands, rules, and hooks get a home that does not require a spec revision, and the portable surface stays genuinely portable.</p><blockquote><strong>Agent Briefings:</strong> Practical AI agent engineering — what works in production, and what’s just hype. <a href="https://blog.agentailor.com/#newsletter"><strong>Get the next one in your inbox.</strong></a></blockquote><h3>The Design Decision Worth Stealing</h3><p>The best engineering idea in this specification has nothing to do with plugins.</p><p>Agent Plugins draws a hard line between failures that kill the package and failures that kill one component:</p><p><strong>Fatal, meaning the whole plugin is rejected:</strong></p><ul><li>plugin.json is not valid JSON</li><li>$schema or name is missing, or name breaks the naming rules</li><li>Any path in the package resolves outside the plugin root</li></ul><p><strong>Non-fatal, meaning skip that piece and keep going:</strong></p><ul><li>A skill violates the Agent Skills spec, so that skill is skipped</li><li>An MCP server entry is malformed, declares an unknown transport, or fails to connect</li><li>The client does not support the declared transport</li><li>An unknown top-level manifest field appears</li></ul><p>The governing rule is stated plainly: a failure isolated to a component type, entry, or process must not prevent the client from loading independently valid components.</p><p>The practical payoff is real. If your plugin ships a skill plus an MCP server, and the MCP server’s remote endpoint is down or the user never configured its credentials, <strong>the skill still works</strong>. The plugin degrades instead of disappearing.</p><p>Anyone who has watched one bad entry take down an entire config file knows why this matters. It is a good default for any plugin system you design.</p><h3>What v1 Deliberately Leaves Out</h3><p>The non-goals are as informative as the spec itself. v1 does <strong>not</strong> define archive formats (plugins are directories, not .zip files), registries or package managers, credential and OAuth configuration, sandboxing, a trust or permission model, provenance verification, plugin-to-plugin dependencies, auto-update mechanisms, or a test harness. The companion FUTURE_CONSIDERATIONS.md lists each as deferred rather than dismissed.</p><p>What that means in practice: <strong>portability stops at packaging.</strong> Discovery, installation, permissions, and trust remain per-client. A plugin format is not a marketplace, and it is not a security boundary. Installing a third-party plugin still means running third-party code with whatever privileges your client grants it.</p><h3>Who Supports It Today</h3><p>The spec is governed by a technical steering committee of Amazon, Anysphere (Cursor), Microsoft, OpenAI, and Vercel, with Google joining as a core maintainer on launch day. Notably absent is Anthropic, which created both of the standards being packaged.</p><p>At launch, Vercel listed ChatGPT and Codex, Cursor, GitHub Copilot, Kiro, and VS Code. On August 12, GitHub shipped support in VS Code, Copilot CLI, the Copilot SDK, and the Copilot app, installable through the Awesome Copilot marketplace.</p><p><strong>Claude Code is not among them</strong>, and it already ships its own plugin format: the manifest lives at .claude-plugin/plugin.json, with commands/, agents/, skills/, hooks/, and .mcp.json alongside it. Same word, different structure, not currently compatible. If you want to reach Claude Code plus the launch clients today, plan on two manifests. The expensive parts, the SKILL.md files and the MCP server itself, are shared. Only the wrapper differs.</p><blockquote><strong>Note:</strong> Client support for a standard this new moves fast. The rosters above are what primary sources confirmed in August 2026; check <a href="https://agent-plugins.org/compatible-clients">agent-plugins.org/compatible-clients</a> for the current list before making packaging decisions.</blockquote><h3>Should You Package One Yet?</h3><p>It depends on what you already have.</p><p><strong>If you maintain a skill or MCP server that people install today, yes.</strong> The migration cost is close to zero. Add a plugin.json with two required fields, move existing skills into skills/, and declare servers in mcp.json. You are not rewriting anything. You are renaming directories.</p><p><strong>If you are shipping a skill and a server that belong together, this is the strongest case.</strong> Bundling was the actual gap. A deployment runbook and the tool integration it depends on could not previously travel as one unit, and users had to install two things and hope the versions matched.</p><p><strong>If you are building for one client and have no distribution plans, not yet.</strong> The standard’s value is portability. Without a second target, you are adopting a manifest for its own sake, and your client’s native format probably exposes more of what makes it useful.</p><p><strong>If you need permissions, sandboxing, provenance, or secret management, wait.</strong> v1 explicitly does not cover those, and building around gaps the spec plans to fill is how you end up migrating twice.</p><p>The honest summary: Agent Plugins is not a leap forward in capability. It is a modest, well-scoped agreement that removes a repetitive packaging tax. Standards that succeed usually look like this, boring, narrow, and mostly made of parts that already worked.</p><p>We will follow up with a hands-on guide that builds and ships a real plugin end to end, once the client ecosystem settles enough for the walkthrough to stay accurate.</p><h3>Resources</h3><ul><li><a href="https://agent-plugins.org/specification">Agent Plugins Specification</a> — the full normative v1.0.0 contract</li><li><a href="https://agent-plugins.org/">Agent Plugins homepage</a> — author and client implementation guides</li><li><a href="https://agent-plugins.org/schemas/1.0.0/plugin.schema.json">JSON Schemas</a> — the canonical manifest schema</li><li><a href="https://github.com/agentplugins/agent-plugins-spec/blob/main/FUTURE_CONSIDERATIONS.md">Future Considerations</a> — what v1 defers and why</li><li><a href="https://agentskills.io/specification">Agent Skills Specification</a> — the SKILL.md format</li><li><a href="https://vercel.com/changelog/introducing-agent-plugins-1-0-0">Introducing Agent Plugins 1.0.0</a> — Vercel’s launch announcement</li><li><a href="https://github.blog/changelog/2026-08-12-agent-plugins-1-0-in-vs-code-copilot-cli-and-the-copilot-app/">Agent Plugins 1.0 in VS Code and Copilot</a> — GitHub’s rollout</li><li><a href="https://developers.googleblog.com/agent-plugins-package-your-skills-tools-and-more/">Agent plugins: package your skills, tools, and more</a> — Google’s announcement</li></ul><h3>Related Articles</h3><ul><li><a href="https://blog.agentailor.com/blog/how-to-build-and-deploy-agent-skill-from-scratch?utm_source=agentailor_blog&amp;utm_medium=blog_post&amp;utm_campaign=agent_plugins_explained">How to Build and Deploy an Agent Skill from Scratch</a> — build the skill half of a plugin</li><li><a href="https://blog.agentailor.com/blog/top-ai-agent-standards-2026?utm_source=agentailor_blog&amp;utm_medium=blog_post&amp;utm_campaign=agent_plugins_explained">Top AI Agent Standards to Know in 2026</a> — AGENTS.md, SKILL.md, and DESIGN.md in context</li><li><a href="https://blog.agentailor.com/blog/mcp-v2-whats-changing-and-why?utm_source=agentailor_blog&amp;utm_medium=blog_post&amp;utm_campaign=agent_plugins_explained">MCP v2: What’s Changing, What’s Deprecated, and Why</a> — where the protocol half is heading</li><li><a href="https://blog.agentailor.com/blog/build-an-mcp-v2-server-typescript?utm_source=agentailor_blog&amp;utm_medium=blog_post&amp;utm_campaign=agent_plugins_explained">Build an MCP v2 Server in TypeScript</a> — build the MCP half of a plugin</li><li><a href="https://blog.agentailor.com/blog/top-ai-agent-protocols-2026?utm_source=agentailor_blog&amp;utm_medium=blog_post&amp;utm_campaign=agent_plugins_explained">Top 5 AI Agent Protocols to Know in 2026</a> — how MCP fits among agent protocols</li></ul><h3>Stay measured as the field moves</h3><p>What actually matters for building and scaling AI agents in production — and what’s just hype. Straight from the work, no filler. <a href="https://blog.agentailor.com/#newsletter"><strong>Join Agent Briefings</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9f6e49b517d7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building an Agent Eval Harness: The Complete Guide]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://techwithibrahim.medium.com/building-an-agent-eval-harness-the-complete-guide-a9fbae3a5c27?source=rss-fd16cb1da896------2"><img src="https://cdn-images-1.medium.com/max/1200/1*dSLmPw1DgecR7aQpd_rkJw.png" width="1200"></a></p><p class="medium-feed-snippet">Originally published at https://blog.agentailor.com</p><p class="medium-feed-link"><a href="https://techwithibrahim.medium.com/building-an-agent-eval-harness-the-complete-guide-a9fbae3a5c27?source=rss-fd16cb1da896------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://techwithibrahim.medium.com/building-an-agent-eval-harness-the-complete-guide-a9fbae3a5c27?source=rss-fd16cb1da896------2</link>
            <guid isPermaLink="false">https://medium.com/p/a9fbae3a5c27</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Ali Ibrahim]]></dc:creator>
            <pubDate>Tue, 01 Sep 2026 00:00:15 GMT</pubDate>
            <atom:updated>2026-10-09T10:14:24.905Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Maintaining Prompts as Models Improve: Your System Prompt Has a Shelf Life]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://techwithibrahim.medium.com/maintaining-prompts-as-models-improve-your-system-prompt-has-a-shelf-life-59421365a2c7?source=rss-fd16cb1da896------2"><img src="https://cdn-images-1.medium.com/max/1086/1*BVhjc6NchU3x-_Lis_V5cQ.png" width="1086"></a></p><p class="medium-feed-snippet">Anthropic deleted over 80% of Claude Code system prompt for Claude 5 with no measurable loss on their coding evals. The instructions did&#x2026;</p><p class="medium-feed-link"><a href="https://techwithibrahim.medium.com/maintaining-prompts-as-models-improve-your-system-prompt-has-a-shelf-life-59421365a2c7?source=rss-fd16cb1da896------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://techwithibrahim.medium.com/maintaining-prompts-as-models-improve-your-system-prompt-has-a-shelf-life-59421365a2c7?source=rss-fd16cb1da896------2</link>
            <guid isPermaLink="false">https://medium.com/p/59421365a2c7</guid>
            <category><![CDATA[writing-prompts]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[ai-agent]]></category>
            <dc:creator><![CDATA[Ali Ibrahim]]></dc:creator>
            <pubDate>Sat, 22 Aug 2026 13:31:01 GMT</pubDate>
            <atom:updated>2026-08-22T13:31:01.399Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Testing Tools for AI Agents: catching the bugs your agent can’t tell you about]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://techwithibrahim.medium.com/testing-tools-for-ai-agents-catching-the-bugs-your-agent-cant-tell-you-about-44fa2f45b09e?source=rss-fd16cb1da896------2"><img src="https://cdn-images-1.medium.com/max/1200/1*GSGgIDzS6ETvrn0l_2IbOQ.png" width="1200"></a></p><p class="medium-feed-snippet">You designed the tool well. Nothing verifies it still does what its description promises. An audit of eight tools found four contract bugs&#x2026;</p><p class="medium-feed-link"><a href="https://techwithibrahim.medium.com/testing-tools-for-ai-agents-catching-the-bugs-your-agent-cant-tell-you-about-44fa2f45b09e?source=rss-fd16cb1da896------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://techwithibrahim.medium.com/testing-tools-for-ai-agents-catching-the-bugs-your-agent-cant-tell-you-about-44fa2f45b09e?source=rss-fd16cb1da896------2</link>
            <guid isPermaLink="false">https://medium.com/p/44fa2f45b09e</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Ali Ibrahim]]></dc:creator>
            <pubDate>Sun, 16 Aug 2026 17:34:18 GMT</pubDate>
            <atom:updated>2026-08-16T17:34:18.674Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[Is LangChain Still Worth It in 2026?]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://techwithibrahim.medium.com/is-langchain-still-worth-it-in-2026-e6c3410582ce?source=rss-fd16cb1da896------2"><img src="https://cdn-images-1.medium.com/max/1672/1*CL7jry_ylzC64rPp9w2G7A.png" width="1672"></a></p><p class="medium-feed-snippet">Is LangChain worth it in 2026? A practitioner&#x2019;s guide to the v1 stack: create_agent, middleware, deploying, observing, and MCP, with an&#x2026;</p><p class="medium-feed-link"><a href="https://techwithibrahim.medium.com/is-langchain-still-worth-it-in-2026-e6c3410582ce?source=rss-fd16cb1da896------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://techwithibrahim.medium.com/is-langchain-still-worth-it-in-2026-e6c3410582ce?source=rss-fd16cb1da896------2</link>
            <guid isPermaLink="false">https://medium.com/p/e6c3410582ce</guid>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Ali Ibrahim]]></dc:creator>
            <pubDate>Sat, 25 Jul 2026 19:53:48 GMT</pubDate>
            <atom:updated>2026-07-25T19:53:48.078Z</atom:updated>
        </item>
        <item>
            <title><![CDATA[MCP v2: What’s Changing, What’s Deprecated, and Why]]></title>
            <description><![CDATA[<div class="medium-feed-item"><p class="medium-feed-image"><a href="https://techwithibrahim.medium.com/mcp-v2-whats-changing-what-s-deprecated-and-why-736c6199c55a?source=rss-fd16cb1da896------2"><img src="https://cdn-images-1.medium.com/max/1672/1*wGP8ql8S7fsRcL3vNtJVQQ.png" width="1672"></a></p><p class="medium-feed-snippet">MCP v2 breaking changes explained: the protocol goes stateless and deprecates sampling, roots, and logging. What is changing across every&#x2026;</p><p class="medium-feed-link"><a href="https://techwithibrahim.medium.com/mcp-v2-whats-changing-what-s-deprecated-and-why-736c6199c55a?source=rss-fd16cb1da896------2">Continue reading on Medium »</a></p></div>]]></description>
            <link>https://techwithibrahim.medium.com/mcp-v2-whats-changing-what-s-deprecated-and-why-736c6199c55a?source=rss-fd16cb1da896------2</link>
            <guid isPermaLink="false">https://medium.com/p/736c6199c55a</guid>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[software-development]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[technology]]></category>
            <dc:creator><![CDATA[Ali Ibrahim]]></dc:creator>
            <pubDate>Fri, 17 Jul 2026 13:31:01 GMT</pubDate>
            <atom:updated>2026-07-17T13:31:01.699Z</atom:updated>
        </item>
    </channel>
</rss>