DEV Community

Cover image for How to Design AI Evaluations You Can Actually Trust
Jan-Felix Schmakeit for Google AI

Posted on AI-assisted

How to Design AI Evaluations You Can Actually Trust

As part of my work at Google, we are publishing a suite of Agent Skills for Google products and technologies on GitHub. These agent skills are designed to help AI agents interact with our technologies. But how do you test that these skills are useful and work as expected? My team in Developer Relations has been focused on this question, because having reliable signals on their performance is critical to help us improve them over time.

Just as you wouldn't deploy a production API without writing unit tests, you should apply the same standard to your AI agents. As Joe Spiro showed in the Designing AI Evals post series, scaling AI tools means moving beyond "vibe testing" in a terminal. Instead, you should set up a structured, automated evaluation pipeline to benchmark your integration. The evaluations (evals) are the actions you asked the agent to perform, which are graded using scorers (for example rubrics) that assert whether the agent succeeded. We'll focus on evaluations in this post and tackle tips for scoring rubrics in the next post.

However, AI evaluations cost real tokens. You need to make sure that you use these tokens as efficiently as possible. They need to provide real value that helps you build better tools. Writing good evaluations is critical. Poor evaluations provide false signals, waste your token budget, and create noise in your metrics.

Here are five rules we learned to design better evaluations you can trust. Follow them to ensure that every token you spend produces a useful metric.

Know Your Evaluation Environment

Before writing evaluations, you need to understand the setup and limitations of your chosen framework. This includes systems like Harbor, Inspect AI, or integrations in development tools like in the Agent Development Kit. Does it use an ephemeral sandbox? What tools are available? How is the output captured

  • Tailor your graders to the environment: For example, if you can deterministically access the sandbox to evaluate code, that's an option. Alternatively, ask the agent to print its response to the console. Your framework can capture this output and pass it directly to your scorers. Adjust your grader to handle these environments.
  • Be aware of dependency limitations and access to "real" resources: If an evaluation task requires access to "real" resources (for example an authenticated gcloud session with access to a Google Cloud project), create ephemeral resources or credentials that isolate and limit access so they don't impact other evaluations. Alternatively, you could provide mock tools instead of real test credentials.
  • Evaluate plans instead of tasks that are difficult to isolate: An easier approach might be to evaluate the plan to accomplish the task, rather than the actual execution.
  • Avoid interactive prompts: Multi-turn agent sessions are complex to evaluate. When getting started, design your evaluations using one-shot prompts.

Avoid the Ceiling Effect

If your evaluations show a high baseline accuracy (i.e., without your agent tool), it might not prove its value, or the evaluation prompts are too easy.

  • Write harder prompts: You cannot measure the impact of a new agent tool or skill if the baseline model already knows the answer.
  • Require multi-step reasoning: Design prompts that reflect complex, real-world use cases where your tool can actually differentiate itself from the model's pre-training.
  • Revisit the scope of your tool: If tests are repeatedly reporting a high accuracy without using your tool, it might be time to revisit it. The underlying model and agent may have improved and are able to accomplish the task without additional help. It might be time to refocus or deprecate your tool.

The Prompt-Grader Mismatch

You cannot grade an agent on something you did not explicitly ask it to do. Your evaluation prompts and graders should be complementary. This means that they should only test for things included in the prompt.

  • Avoid scope creep: If you asked a broad question, you can expect a similarly broad response. For example, if your prompt is "How do I secure Google Cloud Run?", your grader cannot penalize the agent for missing a specific, unprompted IAM role.
  • Be explicit: If you want to evaluate specific knowledge or exact implementation details, you must state those requirements clearly in the prompt.

Grade the Destination, Not the Journey

Agents possess inherent model knowledge and might skip your custom tools entirely to arrive at the correct answer. (That on its own is some useful feedback!)

  • Do not evaluate the trajectory: Avoid writing graders that check if the agent used a specific help command or followed a rigid sequence of steps.
  • Evaluate the final answer: Grade the objective output. If you absolutely must evaluate the agent's planning phase, explicitly ask it to output a detailed execution plan and evaluate the plan instead.

Curate Your Evaluation Dataset

A strong evaluation suite tests real, diverse use cases. But testing the same capability repeatedly causes overfitting and creates noisy metrics.

  • Use real-world examples: Evaluations should include real user journeys and focus on goals users want to achieve. Consider including additional context, such as sanitized sample data, to ground the evaluations.
  • Maximize your signal: Ensure every prompt in your evaluation suite tests a distinct concept or unique capability. Think of this like code coverage for traditional tests.
  • Remove overlap: Consolidate redundant prompts. A smaller, more curated data set provides clearer metrics, prevents overfitting and saves tokens.

Summary

You cannot improve AI tools if you can't measure them accurately. If you treat AI evaluations with the same focus as traditional unit tests, you improve the quality of your metrics and get more robust signals.

By applying these five rules, you eliminate false signals that waste your token budget. Instead of generating noise, your test suite gives you actionable feedback you can use to guide your engineering decisions and improve your tools.

Figuring out what to test is only the first step. A well-designed evaluation is only useful if the scorer grading answers is reliable and returns meaningful results. In my next post, we will look at how to test. You will learn how to write lean, atomic rubrics that minimize ambiguity for an LLM grader and make every token count.

Photo by William Warby on Unsplash

Top comments (9)

Collapse
 
heinrichneb profile image
Heinrich Neb

The ceiling rule has a measurable form that saved us from ourselves: a prompt earns its place in the suite only while the baseline still fails it - and one failing baseline attempt proves less than it feels like. A single baseline failure only bounds "solvable without the tool" at ≤95 %; the bound falls with independent attempts (1 − 0.05^(1/N)), so three baseline failures push it to ≤63 %, five to ≤45 %. We ran this admission rule over an existing suite and it quietly removed cases that had been measuring model capability rather than the tool. "Write harder prompts" becomes enforceable once admission is a measurement instead of an author's impression.

And a sixth rule we learned the expensive way, since your part 2 is about scorers: prove the suite can say no. Every grader gets a known-bad twin - a deliberately wrong output that must fail - plus a recorded date of the last time the suite actually rejected something. A grader that has never failed and a grader that silently stopped running print the same green, and without the twin you cannot tell trust from decoration.

Collapse
 
mickyarun profile image
arun rajkumar

Two things I'd add, both about what makes a score trustworthy over time rather than on the day you ran it.

A result is only as portable as the rig that produced it. Harness version, dataset snapshot, prompt revision, model version — if those aren't pinned to the number then you can't honestly compare this week's 82% to last month's 79%, and people absolutely will. Most eval "regressions" I've watched teams chase turned out to be someone quietly editing the test set. Version the suite the way you version the code and treat a change to it as a change that needs review, otherwise the number is a self-reported claim rather than evidence.

The unit test analogy is a good one but it breaks in a place worth naming. Unit tests are deterministic. Evals are sampled. A suite that passes 97% of the time isn't passing, it's a measurement with an interval around it, and gating on it like a boolean means you spend months chasing noise and then start ignoring the signal when it finally matters. Worth deciding up front how many runs a number needs before anyone is allowed to act on it.

Coming from payments, where nobody trusts a figure they can't reconstruct six months later, the interesting artifact was never the score. It's whether you can rerun it and get the same one.

Collapse
 
alexshev profile image
Alex Shev

Evaluation gets much more credible when the test set includes the failures that changed a production decision. I would keep a small regression suite of real bad outputs alongside aggregate scores, then make every model or prompt change earn its way through both.

Collapse
 
eduzsh profile image
Edu Peralta

The ceiling effect is the failure mode I see most when people evaluate coding agent skills. If the base model already knows the answer, the skill looks useful and the eval teaches you nothing about whether those tokens were earned. The part I would steal immediately is grading the final sandbox state instead of whether the agent walked through your preferred sequence of tool calls.

Collapse
 
hannune profile image
Tae Kim

The prompt-grader mismatch caught us badly on a project last year. We had a tool-calling agent where the rubric penalized it for making more than two tool calls, but we'd never told the agent that in the task prompt. The agent was being creative and thorough, the grader was marking it wrong, and for two weeks we thought the model had regressed. The fix was embarrassingly simple, but the two weeks of confusion wasn't.

Collapse
 
mudassirworks profile image
Mudassir Khan

the 'evaluate plans instead of tasks' point is the one most teams skip first. we burned a sprint on multistep execution evals before realizing we had no step visibility, signal was always 'final output wrong', never which call failed.

switching to plan evals gave us granular failure modes in one pass. execution layer became separate, different harness.

ceiling effect hit us too — first eval set had 87% baseline accuracy and we declared success for 3 weeks before a user surfaced the failure.

are these generally single pass, or do you run multiturn evals for complex orchestration?

Collapse
 
kartik-nvjk profile image
Kartik N V J K

Framing an eval as an action plus a scorer that asserts success is the shift that moved my hit rate, because loose "does this look right" checks never fail loudly enough to trust. The token-efficiency angle is underrated too: I cut eval-set cost a lot just by pruning cases that never change verdict across model swaps. Which of the five rules do you see teams break most often?

Collapse
 
salparvez profile image
Salman Parvez

"Grade the destination, not the journey" is the right rule when the destination exists. The case I am in is the one where it does not yet: a construction system where the field outcome is months away and the first loop has not run. What we grade instead is the claim. Every output carries MEASURED, MODELED or ASPIRATIONAL with the evidence behind it, and the eval checks whether the label is right, not whether the number is. On the ceiling effect, our baseline problem is the mirror of yours. No model has deconstruction data, so the tool cannot fail to differentiate from pre-training, which makes overfitting to our own curated examples the thing to watch rather than an easy baseline.

Collapse
 
yune120 profile image
Yunetzi

Been burned by shiny evals that hid real flaws. Trust in AI evaluations needs transparent metrics and real user feedback - yes, even the messy ones. Curious what others think.