<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by tapas mohanty on Medium]]></title>
        <description><![CDATA[Stories by tapas mohanty on Medium]]></description>
        <link>https://medium.com/@mtapas.mohanty95?source=rss-8a68848285a4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*wWuoRK4OORjojW_G.</url>
            <title>Stories by tapas mohanty on Medium</title>
            <link>https://medium.com/@mtapas.mohanty95?source=rss-8a68848285a4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 06:48:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mtapas.mohanty95/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Your Coding Agent Passed the Tests. Did It Break the Behavior?]]></title>
            <link>https://medium.com/@mtapas.mohanty95/your-coding-agent-passed-the-tests-did-it-break-the-behavior-8f60e34d89a4?source=rss-8a68848285a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/8f60e34d89a4</guid>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[coding-agents]]></category>
            <category><![CDATA[eval]]></category>
            <category><![CDATA[agentic]]></category>
            <dc:creator><![CDATA[tapas mohanty]]></dc:creator>
            <pubDate>Wed, 21 Oct 2026 12:00:46 GMT</pubDate>
            <atom:updated>2026-10-21T12:00:46.123Z</atom:updated>
            <content:encoded><![CDATA[<p>Coding agents are getting good at changing code.</p><p>Give one a task, and it can inspect a codebase, refactor a module, update a prompt, run tests, and come back with a summary of what it changed. That can save a lot of time.</p><p>But there’s a problem I keep coming back to:</p><p><strong>How do we know the change still behaves the way it should?</strong></p><p>This becomes especially important when the code being changed is itself an AI system.</p><p>Imagine a customer support agent handling a refund request. It needs to acknowledge the customer’s frustration, explain the policy, and ask for approval before issuing a refund.</p><p>A coding agent updates the support prompt to make the response more empathetic. The new answer sounds better, and the existing tests still pass.</p><p>But somewhere in the rewrite, the agent stops asking for approval(refund).</p><p>The response is friendlier. The behavior is wrong.</p><p>A basic test that checks whether the answer contains an apology might pass. A test that checks whether the agent followed the required sequence of actions would catch the problem.</p><p>That difference matters.</p><h3>The problem with testing AI-generated changes</h3><p>Traditional software testing gives us useful ways to check whether code behaves as expected. But AI systems introduce another layer of complexity: behavior can change even when the application still runs and the output looks reasonable.</p><p>A prompt change might affect which tool an agent calls, whether it follows an approval requirement, or how it handles an edge case. These regressions aren’t always obvious from the final text.</p><p>And when a coding agent is making those changes, the loop becomes more complicated:</p><ol><li>The coding agent changes the application code or prompt.</li><li>It runs the available tests.</li><li>The tests pass, so it reports that the work is complete.</li><li>A behavior that was important to the original system has quietly changed.</li></ol><p>The issue isn’t that coding agents can’t write good code. It’s that passing the tests we happened to write doesn’t prove that the behavior we care about has been preserved.</p><p>I wanted a way to make those behavioral changes easier to see.</p><h3>Show the agent what actually changed</h3><p>This is the idea behind <a href="https://github.com/tap222/assay-evals">Assay Evals</a>, an open-source project I’m building for behavioral regression testing of AI systems and agents.</p><p>Instead of treating an evaluation as a single pass-or-fail result, the goal is to make it easier to understand what changed between runs.</p><p>In the refund example, useful feedback shouldn’t stop at saying that the new run failed. It should identify the missing approval action and show how the new trajectory differs from the expected behavior.</p><p>That gives the coding agent something concrete to work with.</p><p>It can revisit the prompt, fix the instruction that caused the regression, and run the evaluation again. The objective isn’t just to get a green test result. It’s to restore the required behavior without losing the improvement it was trying to make.</p><p>For this example, that means keeping the more empathetic response while ensuring that the agent still requests approval(refund).</p><p>This is a much more useful feedback loop than asking an agent to keep modifying a prompt until the output looks right.</p><h3>One question we shouldn’t overlook: who ran the evaluation?</h3><p>There’s another detail that becomes important when coding agents start working alongside developers and CI pipelines.</p><p>When an evaluation appears in a dashboard, how do we know who triggered it?</p><p>Was it a developer testing a change locally? Was it GitHub Actions running the test suite? Or was Claude Code or Google Antigravity repeatedly changing a prompt in a background session?</p><p>Without that context, a test result tells only part of the story.</p><p>For example, suppose the same evaluation runs five times while a coding agent works on a prompt. Looking at the results without knowing who triggered them makes it harder to connect each run to the change that caused it.</p><p>Recording attribution alongside the evaluation can help answer basic but important questions:</p><ul><li>Which agent or developer ran this evaluation?</li><li>Which session and branch did it belong to?</li><li>Which commit was being tested?</li><li>Was the run part of a local development loop or a CI pipeline?</li></ul><p>That information gives the team a more useful history of how a change evolved. When a regression appears, engineers have more context for investigating it.</p><p>It also makes the evaluation dashboard useful for understanding the development process, not just the final score.</p><h3>Bringing Assay into the coding agent’s workflow</h3><p>I’ve been working on bringing this workflow directly into Assay Evals through native skills for coding agents.</p><p>The idea is straightforward: make it easier for an agent to use evaluations as part of its normal development loop, rather than expecting a developer to manually inspect every result and explain what to do next.</p><p>For supported environments, the setup looks like this:</p><p><strong>For Google Antigravity:</strong></p><pre>assay init --antigravity</pre><p><strong>For Claude Code:</strong></p><pre>assay init --claude-code</pre><p>These commands set up an Assay skill in the relevant project directory. Passing --global installs it for use across local projects.</p><p>The skill gives the coding agent instructions for inspecting evaluation results, reviewing behavioral differences, and investigating regressions before presenting its changes for review.</p><p>The safety boundary is important here: <strong>the agent can investigate and fix a regression, but it must not run </strong><strong>assay accept to approve its own changed behavior.</strong></p><p>That final approval should remain a deliberate human decision. Otherwise, an agent could change the application, see that the baseline no longer matches, and simply redefine the new behavior as correct.</p><p>That would defeat the purpose of regression testing.</p><p>I’ve also included a runnable example in the repository:</p><p><a href="https://github.com/tap222/assay-evals/tree/main/examples/antigravity">Assay Antigravity example</a></p><p>It demonstrates a Gemini-powered support agent with approval gates and prompt regression detection.</p><p>The example is useful because it brings the discussion back to a concrete engineering problem: making a change is one thing; showing that the change still respects the system’s behavioral requirements is another.</p><h3>The part I think matters most</h3><p>I don’t think coding agents need to become perfect before we can trust them with meaningful engineering work.</p><p>But we do need a better way to check what they do.</p><p>When an agent refactors a regular function, a conventional test suite may be enough to catch the important regressions. When it changes the prompt or behavior of another AI system, we also need to check things such as tool usage, required actions, approval gates, and the sequence of decisions.</p><p>And when something breaks, the feedback needs to be specific enough to help fix it.</p><p>That’s what I’m trying to make easier with Assay: compare behavior against an established baseline, identify the meaningful differences, and feed that information back into the development process.</p><p>The question I keep coming back to isn’t whether the coding agent finished the task or whether its tests passed.</p><p>It’s this:</p><p><strong>Did the change preserve the behavior that mattered?</strong></p><p>As coding agents take on more of the implementation work, that question deserves a clear answer.</p><p><strong>Project:</strong> <a href="https://github.com/tap222/assay-evals">Assay Evals on GitHub</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8f60e34d89a4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Your AI Agent Said “Success” The Database Says Otherwise]]></title>
            <link>https://medium.com/@mtapas.mohanty95/your-ai-agent-said-success-the-database-says-otherwise-c498a3a0e166?source=rss-8a68848285a4------2</link>
            <guid isPermaLink="false">https://medium.com/p/c498a3a0e166</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[software-testing]]></category>
            <category><![CDATA[ai-in-softwareengineering]]></category>
            <dc:creator><![CDATA[tapas mohanty]]></dc:creator>
            <pubDate>Mon, 19 Oct 2026 12:56:28 GMT</pubDate>
            <atom:updated>2026-10-19T12:56:28.258Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Why testing AI agents requires verifying what actually happened, not just what the agent reported.</em></p><p>Imagine asking an AI agent to update a customer’s subscription.</p><p>The agent calls a tool. The tool returns a success response. The agent tells the user:</p><p>“Your subscription has been updated successfully.”</p><p>The logs look fine. The tool call completed. The final answer is confident.</p><p>But the database still contains the old subscription.</p><p>What happened?</p><p>Perhaps the SQL UPDATE matched zero rows. Perhaps the API accepted the request but rejected the operation later. Perhaps a browser agent clicked a button, but the form submission failed because the session had expired.</p><p>From the agent’s perspective, the task is complete.</p><p>From the application’s perspective, nothing changed.</p><p>This is one of the important testing problems we need to solve as AI agents move from answering questions to taking actions.</p><h3>The problem with trusting a successful tool call</h3><p>Traditional application code usually has explicit success and failure conditions. An HTTP response code, a database result, or an exception can tell us something about what happened.</p><p>Agentic systems add another layer of uncertainty.</p><p>An agent decides which tool to call, supplies its arguments, interprets the response, and decides whether to continue or stop. Each step can appear successful while the overall task remains incomplete.</p><p>Consider this example:</p><pre>result = await db.execute(<br>    &quot;&quot;&quot;<br>    UPDATE subscriptions<br>    SET plan = &#39;premium&#39;<br>    WHERE customer_id = :customer_id<br>    &quot;&quot;&quot;,<br>    {&quot;customer_id&quot;: customer_id},<br>)</pre><pre>return {&quot;success&quot;: True}</pre><p>The update might affect one row, several rows, or zero rows. Returning success: True without checking the result makes all three outcomes look identical to the caller.</p><p>The problem is not necessarily the LLM. The tool’s success contract is too weak.</p><p>If an agent is allowed to treat a tool response as proof of completion, the application can report a successful action that never happened.</p><h3>Why checking the final answer is not enough</h3><p>Suppose an evaluation checks only this:</p><pre>assert &quot;updated successfully&quot; in response</pre><p>The test passes when the agent produces the expected sentence.</p><p>It says nothing about whether the database changed.</p><p>Even checking whether the agent called the correct tool is insufficient. A tool can be called with the correct arguments and still fail to produce the intended result.</p><p>For action-taking agents, there are at least three different things to verify:</p><ol><li><strong>Intent:</strong> Did the agent choose the right action and arguments?</li><li><strong>Execution:</strong> Did the tool actually perform the operation?</li><li><strong>Outcome:</strong> Does the real system now have the expected state?</li></ol><p>These are related, but they are not interchangeable.</p><p>A correct tool call does not guarantee a successful operation. A successful operation does not automatically guarantee the intended end state either.</p><h3>Verify the outcome independently</h3><p>A stronger approach is to check the result against the system of record.</p><p>For a database update, inspect the affected-row count and verify the relevant fields. For an external API, retrieve the resource and confirm its persisted state. For a browser agent, check the resulting page or the underlying record rather than relying on the click event.</p><p>For example:</p><pre>result = await update_subscription(customer_id, &quot;premium&quot;)</pre><pre>if result.affected_rows != 1:<br>    raise AssertionError(<br>        &quot;Expected exactly one subscription to change&quot;<br>    )</pre><pre>subscription = await get_subscription(customer_id)</pre><pre>assert subscription.plan == &quot;premium&quot;</pre><p>This is illustrative code; the actual result fields depend on the database driver and application’s data-access layer.</p><p>The key principle is independent verification. The check should obtain evidence from a source that can establish whether the requested action really took effect.</p><p>For eventually consistent systems, verification may need to wait for a final state rather than treating an intermediate pending status as completion.</p><p>And when the evidence is inconclusive, the system should say so. “Unverified” is more honest — and often safer — than “Done.”</p><h3>Test whether the verification itself works</h3><p>There is a further problem: a test can contain a success check that looks meaningful but does not detect failure.</p><p>A useful negative test deliberately simulates an action that reports success without changing anything.</p><p>Then ask: does the verification still pass?</p><p>If it does, the check is not proving the behavior we care about.</p><p>This is the same idea behind testing failure paths in ordinary software. We should not only demonstrate that a system works when everything goes right. We should also demonstrate that our tests catch cases where an operation silently does nothing.</p><h3>Where AI evaluation fits</h3><p>These problems are part of a broader shift in AI testing.</p><p>An evaluation should not stop at whether the answer sounds correct. It should also examine tool selection, tool arguments, action order, side effects, and the resulting application state.</p><p>This matters particularly for agents that handle refunds, update records, manage bookings, send communications, or trigger other consequential operations.</p><p>A useful regression test should establish what the agent was expected to do, record what it actually did, and verify whether the intended outcome occurred.</p><p>That gives us evidence we can compare across prompt changes, model upgrades, tool changes, and application releases.</p><h3>Building this into Assay</h3><p>This is one of the problems I have been exploring while building <a href="https://github.com/tap222/assay-evals">Assay Evals</a>, an open-source project for behavioral regression testing of AI applications and agents.</p><p>Assay focuses on evaluating more than the final response. It can check agent trajectories, expected tool calls, end states, and behavioral changes between runs.</p><p>The project’s roadmap also explores stronger independent verification: reading the result back from an API, database, or another system of record, and treating an inconclusive outcome separately from verified success.</p><p>You can explore the implementation and roadmap in the <a href="https://github.com/tap222/assay-evals">Assay repository</a>.</p><h3>The engineering principle</h3><p>As AI agents become more capable of changing the world around them, our definition of success must become stricter.</p><p>A successful model response is not necessarily a successful tool call. A successful tool call is not necessarily a successful business operation.</p><p><strong>The right question is not “Did the agent say it worked?” It is “What evidence proves that it worked?”</strong></p><p>That distinction is essential to making agentic systems reliable.</p><p>Because in production, confidence is not evidence, a completed tool call is not proof of an outcome, and a green evaluation is only as useful as the behavior it actually verifies.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c498a3a0e166" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>