<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dhivakarnath on Medium]]></title>
        <description><![CDATA[Stories by Dhivakarnath on Medium]]></description>
        <link>https://medium.com/@dhivakarnath26?source=rss-6b41509fc2dd------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*2CTQpiCRAntEMn5r</url>
            <title>Stories by Dhivakarnath on Medium</title>
            <link>https://medium.com/@dhivakarnath26?source=rss-6b41509fc2dd------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 20:54:09 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dhivakarnath26/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[AI Agents Can Bargain. They Should Never Own the Price.]]></title>
            <link>https://medium.com/@dhivakarnath26/ai-agents-can-bargain-they-should-never-own-the-price-f0c62af53cb2?source=rss-6b41509fc2dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/f0c62af53cb2</guid>
            <category><![CDATA[multi-agent-systems]]></category>
            <category><![CDATA[ai-engineering]]></category>
            <category><![CDATA[agent-network-protocol]]></category>
            <category><![CDATA[forward-deployed-engineer]]></category>
            <category><![CDATA[trustworthy-ai]]></category>
            <dc:creator><![CDATA[Dhivakarnath]]></dc:creator>
            <pubDate>Thu, 19 Nov 2026 13:10:11 GMT</pubDate>
            <atom:updated>2026-11-19T13:10:11.450Z</atom:updated>
            <content:encoded><![CDATA[<p><em>A buyer agent, three suppliers, and a signed quote on Agent Network Protocol. The model speaks. The number stays in code.</em></p><p><em>The model speaks the reply. Code owns the price.</em></p><p>A supplier agent put <strong>$2.10</strong> in the message and dressed the rest up as a system instruction. The signed unit price did not move. I took that supplier out of the running. The award went to an agent whose number was actually signed.</p><p>If you buy for a company, that is the difference between a conversation and a commitment. If you build agents, that is the difference between a demo that sounds finished and a quote you can audit.</p><p>I wanted one rule to survive both rooms: a fluent reply cannot rewrite the price.</p><p><strong>ANP (Agent Network Protocol)</strong> is an open protocol for agents that do not share a company login. Each agent gets an identity — a DID, a Decentralized Identifier — plus a way to be discovered and a way to sign what it sends. A quote can be checked. It does not have to be trusted because the wording sounds right.</p><p>I used that protocol for one job: cross-company quote-to-order. One buyer agent, a marketplace, and three suppliers. They exchange signed envelopes. Code owns every number. Claude Haiku and Amazon Nova, on Amazon Bedrock, only speak the replies. This is a local Compose demo. I am not claiming a live multi-company market.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*opaD3VZFuzFLLkpb.png" /><figcaption><em>Figure 1. The desk opens like an exchange floor, not a chatbot: pick a scenario, describe a need, watch every party get its own lane.</em></figcaption></figure><h3>The gap</h3><p>B2B RFQs still die in PDFs and late inboxes. Agents make that worse when free text can rewrite the number.</p><p>The open question is not “can a model sound convincing?” It is:</p><ul><li><strong>Who is this counterparty’s agent?</strong> Identity, without a shared password vault.</li><li><strong>What can it do?</strong> Discoverable interfaces, not hard-coded URLs.</li><li><strong>Which numbers are binding?</strong> Signed fields, not free text.</li><li><strong>Who may commit the company?</strong> A person, before the order binds.</li></ul><p>That is the layer <a href="https://agent-network-protocol.com/">Agent Network Protocol (ANP)</a> is aimed at: DIDs (Decentralized Identifiers), discovery, signed requests, agent descriptions. I used it for quote-to-order.</p><h3>What I built</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7qnDUa6HIoFphxh1PgSQng.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*nmqswaMqt1TPT9NS.png" /><figcaption><em>Architecture. A person confirms and approves. The buyer searches the marketplace, then bargains with the suppliers that search returned. Redis and Postgres hold nonces and sessions. The buyer and the suppliers both call Amazon Bedrock. The price is already set before they speak. The Live ANP probe is separate: its greet and echo never enter ranking.</em></figcaption></figure><p>Stack: FastAPI roles, Redis for nonce replay, Postgres, a Vite board, Docker Compose, local *.test DIDs, Bedrock for the words. Every quote carries a <strong>quote hash</strong>. I verify the signature before a price can win. Local *.test DIDs show the pattern. They are not a production DID trust fabric.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*50NEw-bctWWenQj-.png" /><figcaption><em>Figure 2. Before anyone is contacted, the person sets the ask and the walk-away. The walk-away is never sent to suppliers.</em></figcaption></figure><p>Models never write these fields (illustrative signed quote shape):</p><pre>{<br>  &quot;kind&quot;: &quot;quote&quot;,<br>  &quot;unit_price&quot;: 0.29,<br>  &quot;lead_time_days&quot;: 5,<br>  &quot;price_basis&quot;: &quot;contract&quot;,<br>  &quot;final_offer&quot;: true,<br>  &quot;meets_date&quot;: true<br>}</pre><p>Free-text wording can mention dollars only if they already appear in that payload. Anything else is discarded or flagged.</p><h3>How the bargain works</h3><p>This is not “model, please negotiate.”</p><p><strong>Buyer policy</strong></p><ul><li>A target unit price and a walk-away, set per run</li><li>Up to <strong>4</strong> autonomous rounds</li><li>A counter on every live offer still above target</li><li>The <strong>lowest signed rival</strong>, cited as leverage</li><li>A stop when the best price is at or under target, or the rounds run out</li><li>The walk-away throws a quote out of ranking, and is never disclosed</li></ul><p><strong>Supplier defense</strong></p><ul><li>Opens at the best applicable book price: list, then contract, then volume</li><li>Concedes toward a <strong>private floor</strong> that is never disclosed</li><li>A <strong>token move</strong> when it is already the cheapest and no lower rival is cited</li><li>A larger concession when it is undercut</li><li><strong>Expedite</strong> when the standard lead misses the date, with its own floor</li><li><strong>Best and final</strong> when it cannot move further</li></ul><p><strong>Models</strong></p><ul><li>Haiku reads the request and explains the ranking</li><li>Nova speaks the supplier reply (“I can come down from $0.31 to $0.30…”)</li></ul><p>I could have let a model negotiate. That demo is shorter, and easier to overfit. A policy engine is more code. I took the extra code because I wanted an audit trail, and because I did not want a poisoned sentence to move the price.</p><h3>Bolts, 10 days: the contract wins the rush</h3><p><strong>Ask</strong>: 2,000 stainless hex bolts M8×40 within 10 days.</p><p>S1 opens on <strong>contract</strong> at <strong>$0.31</strong>, with a 5-day lead. S2 can only make the date by <strong>expediting</strong>, and opens near <strong>$0.47</strong>. The buyer counters at <strong>$0.27</strong> and cites the rival.</p><ul><li>S1: $0.31 → $0.30 → <strong>$0.29 best and final</strong></li><li>S2: $0.47 → <strong>$0.33 best and final</strong>, expedited</li></ul><p><strong>S1 wins at $0.29</strong>. On this toy catalog that is about <strong>$40</strong> against the best opening (2,000 × $0.02). Every quote was signature-checked.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5M0qOmd2ruMy5PXI.png" /><figcaption><em>Figure 3. Opening versus final, the price path, the expedite badge, verified signatures, and a locked quote hash for Approve.</em></figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lfLFJgsoyQE9Wbhh.png" /><figcaption><em>Figure 4. The bargain as numbered messages across Buyer, Marketplace, S1, and S2.</em></figcaption></figure><h3>Bolts, 30 days: the calendar flips the winner</h3><p>Same part. <strong>30 days</strong> to deliver. S2 no longer needs to expedite. It walks down toward its floor and wins <strong>near $0.28</strong>. S1 holds <strong>near $0.29</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1_XU7HVpVkiiuPOg.png" /><figcaption><em>Figure 5. Same agents, same books, different date, different winner.</em></figcaption></figure><h3>Shaft seal: name the compound, then defend the price</h3><p><strong>Ask</strong>: 300 shaft seals, 45mm, within 14 days.</p><p>All three ask for the <strong>compound</strong>. The buyer answers <strong>FKM</strong> from its spec sheet, then bargains. The leader gives ground slowly. The undercut supplier chases harder. In the runs I kept, <strong>S3</strong> finished near <strong>$3.38</strong>, still above its private floor. A person still approves the exact hashed quote.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*TtwxzL-vCfrDCKKG.png" /><figcaption><em>Figure 6. Clarify, then price, then defend, then rank.</em></figcaption></figure><h3>The $2.10 that could not win</h3><p>I compromised <strong>S3</strong> for one run. The message carried injected “system” instructions and a fake <strong>$2.10</strong>. The <strong>signed</strong> unit price stayed honest.</p><p>The buyer flagged the instruction-like text and the unsigned dollars, <strong>quarantined</strong> S3, and stopped bargaining with it. Ranking used signed fields only. The honest suppliers kept going. <strong>S1</strong> won on signed numbers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3TETEN0fH37ViF3w.png" /><figcaption><em>Figure 7. Free text is untrusted. The banner shows what was ignored. The board shows who can still win.</em></figcaption></figure><blockquote><em>Agent markets do not fail because a sentence is fluent. They fail when that sentence can rewrite the quote. I rank only what was signed.</em></blockquote><h3>Then I asked a harder question</h3><p>The desk shows the business pattern. It does not, by itself, prove I am speaking ANP rather than a private dialect.</p><p>So I pointed an isolated probe at the official protocol. It uses the <strong>anp SDK</strong>, pinned at <strong>0.4.4</strong> to match the Remote Test Agent on this path. I am not claiming that pin is the newest client on PyPI. The probe talks to the ANP team’s <strong>Remote Test Agent</strong> (<a href="https://github.com/agent-network-protocol/anp-agent-example">anp-agent-example</a>) on a private Docker network. No desk keys. No AWS. No shared Redis with the procurement run.</p><p>What came back: identity loaded, agent description fetched, greet and echo listed, a signed <strong>greet</strong> that returned my DID, and a signed <strong>echo</strong> that round-tripped.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*loqphY3k1TJhvpcP.png" /><figcaption><em>Figure 8. A separate mode from the desk. Connect, discover, sign a call, check the DID. Those replies never enter ranking or the models.</em></figcaption></figure><p>I also aimed the probe at public amap on agent-connect.ai. That path timed out from my network, and didhost.cc returned HTML instead of DID JSON. <strong>The client spoke the official example peer</strong>. I did not prove that every public host is healthy.</p><h3>What this pattern can carry</h3><p><strong>It can carry:</strong> DID identity, signed envelopes, nonce replay, quote hashes, marketplace discovery, policy for the bargain and the floor and the expedite and the quarantine, models as a layer I can swap, and a person at confirm and at approve.</p><p><strong>It does not yet prove:</strong> multi-tenant SaaS, SSO, an ERP write-back, that every public ANP host stays up, or a formal security audit.</p><p>I showed the contract between the agents. I did not show a global procurement network.</p><h3>What I will not hand to a model</h3><ol><li><strong>The number.</strong> Code owns it.</li><li><strong>The trust.</strong> A signature owns it.</li><li><strong>The strategy.</strong> Policy owns it.</li><li><strong>The commitment.</strong> A person owns it.</li><li><strong>The interop.</strong> The real protocol owns it.</li></ol><h3>If you want the desk</h3><p><strong>Source</strong>: <a href="https://github.com/Dhivakarnath/anp-procurement-desk">github.com/Dhivakarnath/anp-procurement-desk</a>.</p><p>Protocol entry points: <a href="https://agent-network-protocol.com/">ANP docs</a>, <a href="https://www.agent-network-protocol.com/specs/">specs</a>, <a href="https://agent-network-protocol.com/docs/anp-getting-started-guide">getting started</a>, and the <a href="https://github.com/agent-network-protocol/anp-agent-example">example agent</a>.</p><p>Would you let a model <em>propose</em> the counter, or only <em>say</em> a counter that code already computed?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f0c62af53cb2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Where the model decides, and where it must not]]></title>
            <link>https://medium.com/@dhivakarnath26/where-the-model-decides-and-where-it-must-not-346dfaf18943?source=rss-6b41509fc2dd------2</link>
            <guid isPermaLink="false">https://medium.com/p/346dfaf18943</guid>
            <category><![CDATA[agent-harness]]></category>
            <category><![CDATA[agentic-ai]]></category>
            <category><![CDATA[accounts-payable]]></category>
            <category><![CDATA[langgraph]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <dc:creator><![CDATA[Dhivakarnath]]></dc:creator>
            <pubDate>Tue, 29 Sep 2026 10:29:33 GMT</pubDate>
            <atom:updated>2026-09-29T10:29:33.679Z</atom:updated>
            <content:encoded><![CDATA[<h4><strong>Building a replayable AP exception agent with a small model, a deterministic supervisor, and an eval harness that can fail the build.</strong></h4><p>An invoice that should have been held can still look like a successful demo. The model reads the PDF, proposes a GL code, says “approve,” and the UI turns green. In accounts payable that is not a product. A controller has to replay <em>why</em> the invoice moved, against the policy that was in force that day, with a human as the approver of record when a ceiling is crossed. If the model owns matching, tolerances, or the write, the trail is a conversation, not an audit.</p><p>This is a write-up of production v1 of an AP exception layer I built to make that trail real. The claim is not that a frontier model is unnecessary in general. The claim is narrower: <strong>on this job, a small multimodal model is enough when control flow is not in the prompt.</strong> Amazon Nova Lite runs extraction and non-PO GL coding. Everything that makes the system employable — routing, three-way match, fourteen policy checks, segregation of duties, the pause for a human, the ERP write contract — is code.</p><p>Ingress is upload. ERP and notifications are MCP connectors with a QuickBooks Online–shaped ledger. The quality proof is an adversarial eval gate plus a small live Bedrock sample, labeled as such. The code lives in [ap-exception-harness](https://github.com/Dhivakarnath/ap-exception-harness). What follows is the decision record, the measurement, one failure I did not have to publish, and the connector work that comes next.</p><h3>What “done” means</h3><p>The system terminates at <strong>approved for payment</strong>. It does not pay. That ceiling is <a href="https://github.com/Dhivakarnath/ap-exception-harness/blob/main/docs/adr/ADR-011-no-payment-execution.md">ADR-011</a>, not a prompt instruction.</p><p>That boundary is the product. A payment tool would make a clever agent look more complete and would make every other guarantee harder to defend. The ERP connector may post a bill, place a hold, or raise an exception. There is no pay, disburse, or remit in the toolset; a test asserts the names are disjoint, and the MCP server does not expose a payment scope.</p><p><strong>An invoice is done when one of four routes is recorded, with an actor and a timestamp:</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UCtZX2-cvKplRuege6RC9Q.png" /></figure><p>Auto-approve from a human is illegal. An agent naming itself as approver on a human-approval route is illegal. Those constraints live in the domain model and again as database check constraints. Defense in depth is not a slogan here; it is the same rule in two layers so a bypass of one still fails.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*kLrraoH_7F2HC3v_.png" /><figcaption><strong><em>Route and status on recorded runs — not a success rate. Auto-approved, held, and needs-approval are ledger outcomes; awaiting review is a paused graph, not a finished demo. Cost tiles are inference only.</em></strong></figcaption></figure><p>Industry surveys put average invoice processing at <strong>8.2 days</strong> and all-in cost near <strong>$9.84</strong> (<a href="https://www.bottomline.com/cdn/1517/5157/1685/Ardent_Partners_-_State_of_ePayables_2025_-_Bottomline_-_FINAL.pdf">Ardent Partners, <em>State of ePayables 2025</em></a>). Exception resolution sits at a <strong>4–5 day</strong> median elapsed (<a href="https://www.apqc.org/resources/benchmarking/open-standards-benchmarking/measures/cycle-time-days-resolve-invoice-error">APQC, cycle time to resolve an invoice error</a>). Those numbers describe the <em>problem</em>. They are not this deployment’s STP rate, and they include mailroom, approver queues, and payment runs I do not run. I compare only the automation step, and I say so next to the number.</p><h3>Options I refused</h3><p><strong>The core decision was where the model is allowed to decide. Four shapes were on the table.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NucziB6nWvi1RJOUWiD7fQ.png" /></figure><p><strong>I took D</strong>. That choice, the options I refused, and the revisit rule are the <a href="https://github.com/Dhivakarnath/ap-exception-harness/blob/main/docs/adr/ADR-001-system-architecture.md">compiled architecture decision record</a> — written in docs/ as I built, not for this post.</p><p><a href="https://www.anthropic.com/engineering/building-effective-agents">Anthropic’s</a> distinction is the right vocabulary: a <em>workflow</em> has predefined code paths; an <em>agent</em> lets the model direct process and tools. AP matching is not an open-ended task. The step count is known. Latency and cost compound with every extra model call, and a finance stakeholder cannot accept a different tool sequence for the same invoice tomorrow.</p><p><a href="https://docs.langchain.com/oss/python/langgraph/overview">LangGraph</a> is in the stack because it lets a graph mix hand-coded steps with LLM-driven ones, persist a paused run, and interrupt for a human — not because “I built an agent.”</p><p>The residual risk of D is honest: two paths, three eval surfaces, three permission layers. If v2 has to change the supervisor’s routing or the policy engine’s check names to onboard a customer ERP, the v1 interface was wrong. That rule is in <a href="https://github.com/Dhivakarnath/ap-exception-harness/blob/main/docs/adr/ADR-001-system-architecture.md">ADR-001</a> and again in the <a href="https://github.com/Dhivakarnath/ap-exception-harness/blob/main/docs/v2-integration-roadmap.md">v2 roadmap</a> so I cannot quietly violate it later.</p><h3>Where it runs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-uVzHlRK2pBSEdxwkGFG5w.png" /><figcaption><strong>Production v1. Authenticated users submit invoices to application containers that apply policy, call Amazon Bedrock in us-east-1, persist state in PostgreSQL, and post decisions to ERP through MCP. AWS Cloud holds AWS IAM and Amazon Bedrock (Nova Lite and Titan Embeddings V2).</strong></figcaption></figure><h3>The line</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*io8J-UJzFmlecWvG.png" /><figcaption><strong>Who owns the decision. Amber boxes may call Nova Lite. Blue boxes are code. Green boxes are a human or an ERP write.</strong></figcaption></figure><p>A PDF arrives through the upload UI. An ingress gateway validates it, content-hash dedupes it, and persists a documents row before any model runs. Docling parses structure and escalates to OCR only when the structural yield is implausibly low. Nova Lite then fills a schema-constrained Invoice — fail loud on type coercion, with per-field confidence and a source region so the UI can highlight the page.</p><p>Routing is a property of that invoice, never a model output. A PO reference takes the PO path: the graph reads purchase order and goods receipt from ERP, runs three-way match, and <strong>inherits</strong> GL from the PO. No PO takes hybrid retrieval (BM25 + pgvector, rerank, mandatory citations) and a second Nova Lite call for a GL proposal. A faithfulness gate can refuse to auto-code. An ungrounded proposal becomes HITL with no account invented to look complete.</p><p>Fourteen named checks then run from a versioned YAML pack — manufacturing is PO-heavy, retail is non-PO with a $1,000 touchless ceiling, manufacturing $2,500. Verdicts are pass, fail, flag, or skip. The decide node reads the ledger. It does not re-litigate arithmetic in a prompt.</p><p>Writes are earned. Agent RBAC hides write tools from any model that should not see them. Policy middleware refuses post_erp_action when the ledger would block it. The MCP server checks scopes before the mock ledger moves. The same forbidden call is tested at the agent layer and again with that layer bypassed. A read-only token’s post_bill is refused at the server.</p><p>When the route is approval, the graph pauses on a Postgres checkpointer. The run, the invoice, the extraction provenance, and one pending review row are written <strong>before</strong> the interrupt, so a process restart does not erase the queue. A human approves, edits the GL, or rejects — from the run page or the Reviews queue. The human is the actor of record. The model is not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EhQ1oZeofhZcP5jK.png" /><figcaption><strong><em>HITL as a persisted row: the ceiling names the tier; Approve, Edit GL, and Reject are human actions of record. The queue copy is the restart invariant, not marketing.</em></strong></figcaption></figure><p><strong>Nova Lite</strong> is the worker on two nodes because those nodes are judgment: read a messy page, propose a GL when the PO does not already have one. I did not run a head-to-head against Sonnet or GPT-4o on this invoice set, so this post does not claim the small model “beat” a frontier one. What I can claim is eligibility. Matching, ceilings, and SoD are not in the prompt; a larger model would not make those checks more true.</p><p>Industry positioning, labeled as <strong>not this pilot</strong>: on the <a href="https://aws.amazon.com/blogs/machine-learning/document-intelligence-evolved-building-and-evaluating-kie-solutions-that-scale/">FATURA invoice KIE study</a>, Nova Lite text extraction reported F1 <strong>0.9222</strong> at <strong>$0.22 per 1,000 pages</strong> versus Nova Pro <strong>0.9793</strong> F1 at roughly <strong>$2.88–$4.27</strong> per 1,000 pages. The smaller model wins the cost/latency frontier; the larger one wins raw F1. That is a tradeoff table, which is how a model choice should be written. A separate <a href="https://blog.sema.cloud/blog/textract-vs-nova-lite">practitioner bakeoff on a 250-document JSON contract</a> reported Lite at <strong>$0.00036 / 6.6 s / 242 of 250</strong> against Textract at <strong>$0.01 / 41.9 s / 238 of 250</strong> — and the point of that write-up was that the extraction lane was swappable when Bedrock deprecated the previous model. I took the same lesson: the schema is the contract, the weights are a challenger.</p><h3>Cost, accuracy, latency — three numbers, three labels</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*fEhuatmofAXcDvIF.png" /><figcaption><strong>How I measure. A gated score may fail CI. A reported score is live and small-n. A cited score is someone else’s survey. They never share a headline.</strong></figcaption></figure><p><a href="https://hamel.dev/blog/posts/evals/">Hamel Husain’s</a> rule is the one I actually used: unsuccessful LLM products fail on evaluation, and <a href="https://hamel.dev/blog/posts/evals-faq/">switching the model is not the first lever</a> unless error analysis says the model is the problem. <a href="https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents">Anthropic</a> is blunter still: when you evaluate “an agent,” you evaluate <strong>harness and model together</strong>. Their CORE-Bench anecdote is the shape of my own incident later — Opus 4.5 moved from 42% to 95% after grader bugs and a less-constrained scaffold were fixed, not after a bigger model.</p><p>I split three surfaces on purpose.</p><h3>Gated (CI may go red)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/874/1*B6zUSeXC69hvJyl0poZp7Q.png" /><figcaption><strong>make eval on a 30-mode adversarial smoke set, </strong><a href="https://github.com/Dhivakarnath/ap-exception-harness/blob/main/backend/evals/results.json"><strong>results</strong></a></figcaption></figure><p>Routing consistency is predicted <strong>without peeking at the manifest label</strong>. That sentence exists because an earlier scorecard did peek, and 100% task completion meant nothing. The full quick dataset is 90 adversarial cases, also 100% policy adherence after I aligned manifest expectations with engine truth rather than lowering the gate. The mix is exception-heavy by design: 15 auto-approve, 42 route-for-approval, 27 hold, 6 reject. It is not a steady-state AP book.</p><p>Deterministic tool scoring on live traces grades selection and arguments separately. <a href="https://deepeval.com/docs/metrics-argument-correctness">DeepEval’s argument judge</a> is kept for task and RAG, where judgment is actually required. Averaging selection with arguments into one “tool use” tile is how a 67% hides a 100% and a 67%. I stopped doing that.</p><h3>Reported (not a ship gate)</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zmBqypKttEN-yXw_OJwoXQ.png" /><figcaption><strong>Live Bedrock sample, </strong><strong>make eval-live --smoke, same date, extraction n=5, agent n=2</strong></figcaption></figure><p>Live tiers need AWS credentials in the shell. They are expensive enough that they are not make check. A 50% on two argument cases is a sample-size confession, not a model indictment. <a href="https://eugeneyan.com/writing/product-evals/">Eugene Yan’s interval arithmetic</a> applies: a point estimate with n=2 does not get a gate. I still show it, because hiding it would recreate the dashboard I had to kill.</p><p>Per-upload judges run asynchronously after a terminal live run and land in run_evaluations. A paused HITL run is not scored as finished. The Evals UI labels manifest gate versus upload judges so a reader cannot mistake “in progress” for quality.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7mcKV9sTYQMuzan2.png" /><figcaption><strong><em>Same screen, two labels: the manifest CI gate at 100%, live argument correctness at 50% on n=2. The red number is the point — a metric that can look bad is the one worth showing.</em></strong></figcaption></figure><h3>Operational (this deployment, small n)</h3><p>make roi-metrics, 18 September 2026, document-processed runs with extraction provenance — methodology and the measured-vs-cited split are in the <a href="https://github.com/Dhivakarnath/ap-exception-harness/blob/main/docs/roi-framing.md">ROI framing</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DOy1Odgkd_U9FMpduPmGNg.png" /></figure><p>The scoped comparison I <em>will</em> make: <a href="https://www.apqc.org/resources/benchmarking/open-standards-benchmarking/measures/cycle-time-days-resolve-invoice-error">APQC’s median <strong>4 days</strong></a> elapsed to resolve an invoice error versus <strong>~10 s</strong> on a touchless automation path. That is the decision step, not invoice-to-pay, and HITL still waits on a person. <a href="https://www.bottomline.com/cdn/1517/5157/1685/Ardent_Partners_-_State_of_ePayables_2025_-_Bottomline_-_FINAL.pdf">Ardent’s <strong>$9.84</strong></a> all-in must not be netted against <strong>$0.0007</strong> inference without a labor model. Different cost bases in one subtraction is how a blog becomes a pitch.</p><p>Latency for a deterministic graph is a forecastable p95 once volume exists. An open tool loop’s step count is a random variable. I chose the graph in part so a latency budget is a property of the pipeline, not a hope about the model.</p><h3>The dashboard that could lie</h3><p>The failures that taught me the most were not “the model hallucinated a total.” They were <strong>green numbers that would not survive a second engineer reading the grader</strong>. The compiled write-up is the <a href="https://github.com/Dhivakarnath/ap-exception-harness/blob/main/docs/post-build-report.md">post-build report</a>; the raw trail is the <a href="https://github.com/Dhivakarnath/ap-exception-harness/blob/main/docs/incident-log.md">incident log</a>.</p><p>Early scorecard reported task completion at 100% while the route predictor read the manifest label it was supposed to be judged against. Tool and argument correctness were hard-coded to 1.0 in places, so DeepEval never saw a real ERP trace. Policy smoke sat at 83% until I fixed the <em>expectations</em>, not the engine, to match what the checks actually do. Live task prompts over-required PO tools on non-PO uploads and punished argument scores for calls the rubric should not have demanded.</p><p>That is the same family of bug <a href="https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents">Anthropic documents on CORE-Bench</a>: the agent was not 42% capable; the grader was wrong and the scaffold was too tight. <a href="https://hamel.dev/blog/posts/evals/">Hamel’s Lucy</a> plateaued on vibe-checks. My plateau was a CI that could not fail.</p><p>The invariant I added: <strong>grade outcomes and traces, not the sheet you already know.</strong> Routing labels are predicted from the invoice and the ledger. Tool scores come from tool_trace recorded at the ERP seam. Policy smoke is a bit-for-bit ledger comparison. Headline RAG is a <strong>minimum</strong> of the triad, never an average that hides a weak dimension. If a metric cannot go red, it does not belong on the scorecard.</p><p>Two cousins, same lesson. Expired AWS credentials surfaced as “the model did not return a valid extraction” until I split CredentialsError at the Bedrock seam. The UI listed runs from process memory, so a refresh showed “No runs yet” while Postgres already had the invoices — and a paused HITL run skipped persistence entirely, so a restart wiped the review queue. Two sources of truth is how a demo lies after lunch. The run, the document image, the field provenance, and the pending review now have to exist as rows before I tell a human the case is waiting.</p><p>I would still do three things earlier: define eval contracts before the Evals page, deploy a docker “prod mode” soon enough to catch the empty-queue bug, and put make eval in hosted CI on day one. The local gate is real; GitHub Actions is still a v2 item.</p><h3>What v1 will not pretend to be</h3><p>Upload is the only live ingress. Email, ERP webhook, and bucket/SFTP adapters exist as protocol-conforming stubs that raise rather than silently drop a document. That is deliberate: a stub that swallows is worse than an honest NotImplementedError.</p><p>ERP is a QuickBooks Online–shaped mock. The MCP tool names are the contract: get_purchase_order, get_goods_receipt, list_historical_bills, post_erp_action (approve / hold / flag). Canonical-to-ERP field mapping lives in the connector, not in the supervisor. Notifications are a console channel with the same notify_approver shape a Slack app would keep.</p><p>The adversarial dataset is synthetic diversity. It does not close the gap to a customer’s real book. Slice 15 — full E2E on a clean deploy, latency budgets, HITL on both channels, induced failure with no fallback — is still a checklist, not a trophy.</p><p>None of that is an apology for mocks. It is the fidelity label Fin and Anthropic put next to a number: this score is a frozen replay; that score is production. Mixing them in one table without labels is the amateur tell.</p><h3>What would change my mind</h3><p>v2 is connector swaps, not a rewrite. The interface list is the <a href="https://github.com/Dhivakarnath/ap-exception-harness/blob/main/docs/v2-integration-roadmap.md">v2 integration roadmap</a>.</p><p>Email, webhook, and bucket adapters should hit the same InvoiceReceived event the upload path already hits. A customer ERP — QuickBooks Online or ERPNext — should implement the same MCP tools, with OAuth, idempotent posts, and sandbox tests that record what the mock hid: partial PO lines, rate limits, stale GRNs, multi-currency quirks. Slack interactivity should resume the same HITL endpoint the in-app queue uses. If any of that requires editing supervisor.py routing or policy/engine.py checks, I fix the v1 interface first.</p><p>I would revisit Nova Lite on extraction if a held-out customer sample showed field F1 that a larger model recovered and the harness did not — after error analysis, not before. I would gate argument correctness only when n is large enough that a 50% is not a coin flip. I would publish p95 latency once make roi-metrics is no longer a two-row snapshot.</p><p>The test for this architecture is simple. Give it an invoice that looks auto-approvable and is not — missing GRN, duplicate, blocked vendor, over ceiling. The model may still read the page well. The ledger has to refuse. The review has to survive a restart. The scorecard has to be able to go red.</p><p>If those hold with a small model, the harness was the product. If they only hold after I put matching back in a prompt, I chose the wrong option in the table above, and I should say so.</p><h3>Sources</h3><p>Industry and craft citations are linked at first mention in the body.</p><p>The records below are not appendices written for this essay. They live in my <a href="https://github.com/Dhivakarnath/ap-exception-harness/tree/main/docs">Github docs</a> , next to the code, because they were the working papers of the build. Each link is the file on GitHub.</p><h3>Record</h3><p><a href="https://github.com/Dhivakarnath/ap-exception-harness/blob/main/blog/where-the-model-decides.md">This essay on GitHub</a> -&gt; Canonical post, with figures</p><p><a href="https://github.com/Dhivakarnath/ap-exception-harness/blob/main/docs/adr/ADR-001-system-architecture.md">ADR-001 — system architecture</a> -&gt; Options, the composite decision, consequences, revisit rule</p><p><a href="https://github.com/Dhivakarnath/ap-exception-harness/blob/main/docs/adr/ADR-011-no-payment-execution.md">ADR-011 — no payment execution</a> -&gt; Terminates at approved for payment</p><p><a href="https://github.com/Dhivakarnath/ap-exception-harness/blob/main/docs/adr/README.md">ADR index</a> -&gt; Routing, extraction, eval scoring, three-layer permissions</p><p><a href="https://github.com/Dhivakarnath/ap-exception-harness/blob/main/docs/post-build-report.md">Post-build report</a> -&gt; Failures compiled from the incident log</p><p><a href="https://github.com/Dhivakarnath/ap-exception-harness/blob/main/docs/incident-log.md">Incident log</a> -&gt; Append-only trail (INC-001–026)</p><p><a href="https://github.com/Dhivakarnath/ap-exception-harness/blob/main/docs/roi-framing.md">ROI framing</a> -&gt; Measured vs cited vs modeled — how not to mix them</p><p><a href="https://github.com/Dhivakarnath/ap-exception-harness/blob/main/docs/v2-integration-roadmap.md">v2 integration roadmap</a> -&gt; Connector swaps; if the supervisor must change, v1 was wrong</p><p><a href="https://github.com/Dhivakarnath/ap-exception-harness/blob/main/backend/evals/results.json">evals results</a> -&gt; The gated scorecard this post quotes</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=346dfaf18943" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>