Sitemap
Ishi Singhal

Ishi Singhal

IIT Bombay grad, now in early stage investing. Writing about the ideas and companies shaping technology.

Evaluating Agents

Bringing trust and reliability to non-deterministic systems

4 min readMar 2, 2026

--

Press enter or click to view image in full size
Source — Anthropic

What are evals?

An eval (short for evaluations) performs the same function as the name suggests: to test whether the AI system generated a satisfactory output.

How evals have evolved over time

Benchmark tests and RLHF were basically evals for LLMs like ChatGPT. This was relatively simpler as you were just grading one output. Did the model solve the math problem? Did the model spew out any toxic content? These questions are more straightforward and hence easier to test.

Get Ishi Singhal’s stories in your inbox

Join Medium for free to get updates from this writer.

Compare this with an agent, for example a travel planning and booking bot. The agent is carrying out so many steps here, from understanding your prompt to pulling the right hotel and flight data, and maybe even making the payment on your behalf. This is an extremely complex flow with multiple possible points of failure. One wrong step also compounds in the entire flow, if the agent did not understand your preferences in the first step itself, there is no way it can produce the desired outcome. Consider the following scenarios -

  • The agent might not orchestrate the flow well. Maybe it booked the flights without even confirming whether a hotel was available
  • It might lose context mid-conversation and displays options that are over your mentioned budget
  • API calls might have errors — wrong dates, destinations, etc. Similarly, tool calls are often not accurate
  • The agent might tell you in the chat that it has made the reservation but did not actually implement the payment step

These are just a few examples which, in best cases might negatively affect user experience and in the worst, generate a completely wrong outcome like exceeding the budget.

Press enter or click to view image in full size
Source — Google

So why is it so tough to generate automatic tests to judge the agent’s performance?

  • Agents are non-deterministic. Give it the same prompt twice and it will give different responses each time.
  • Further, the context, prompt, required tool calling, and the expected action are vastly different with an infinite number of possibilities.
  • They also have numerous steps until they reach the final output. Without the detailed logs of each step, it is impossible to diagnose a failure.
  • The final output is also not the sole north-star metric. For example, an agent processed the user’s refund, showing a satisfactory output, but it never retrieved the order number of the user.

The evals have to be generated such that the agent never fails. An agent that works 99% of the time will still do 100 wrong bookings for every 10,000 users, this is not at all a desirable outcome.

How do we do evals today?

Most evals today are written by humans who made the agent. They spend a lot of time crafting these in order to try and cover every possible scenario where it might fail. However, some teams have started using agents to generate some evals as well, for example — an agent might keep making test cases from each time your code failed in production.

LLM-as-a-judge is the closest approach today to automating writing evals. This approach leverages a different LLM from your agent to evaluate its output. It tests whether the agent gave the desired outcome, spewed out any toxic content, operated within its constraints, etc. However, this is again a non-deterministic test and hence cannot be relied on completely.

𝜏-Bench

Tau-bench is one of the most popular ways to judge whether the agent is successfully executing the task. It is a benchmark created by Sierra AI Research and adopted by the likes of Anthropic for agents. Most other benchmarks do semantic assessment (did the agent say the right thing, did it hallucinate, etc), but 𝜏-Bench focuses on whether the desired change was made in the database.

For example, instead of testing whether the agent told you that it made your flight booking, it checks in the database if the booking has actually been made and if all the parameters are correct.

Press enter or click to view image in full size
Source — Sierra

For agents to be deployed on a large scale and across industries, they need to be reliable, which they are not by their nature itself. This means that one of the biggest bottlenecks to be solved in the coming years would be eval generation. What the ideal future would look like is agents generating these themselves, making it faster and more comprehensive and resulting in truly trustworthy systems.

--

--

Ishi Singhal
Ishi Singhal

IIT Bombay grad, now in early stage investing. Writing about the ideas and companies shaping technology.