Google ADK 2.0 Tutorial (#4b): Running Evaluations with adk eval

Hands-on guide to running, debugging, and automating multi-turn agent evaluations with Google Cloud and ADK 2.0. Covers evaluation architecture, CLI execution via adk eval, diagnosing trajectory regressions, and fine-tuning strictness in eval_config.json.

« Previous: Part 4a | ADK101 Hub | Next: Part 5a »

The adk eval CLI in the Google Agent Development Kit (ADK) 2.0 executes your .test.json and .evalset.json test suites against your live agent package, scoring every conversational turn against criteria configured in eval_config.json (such as tool_trajectory_avg_score and response_match_score). You can run evaluations directly from your terminal using uv run adk eval, visually inspect turn-by-turn diffs in adk web, or plug evaluations into continuous integration gates with pytest.

Key Takeaways

• CLI Execution: Run ADK_EVAL_MODE=true uv run adk eval <agent_package> <evalset.json> --config_file_path=eval_config.json --print_detailed_results.
• Granular Criteria (eval_config.json): Configure per-metric pass thresholds, enforce fail_on_unexpected_tool_call: true, and define per_tool_rules (require_exact_match: true) for state-mutating tools.
• Deterministic Test Keys: Use deterministic email hashing in test environments (ADK_EVAL_MODE=true) to avoid random UUID collisions across parallel test runs.
• Decoupled State Hydration: Keep gateway agents (Gatekeeper) lean with single-responsibility identity verification, delegating full domain profile hydration exclusively to specialized sub-agents (Cameriere).
• Diagnosing Failures: Compare expected vs. actual tool invocations and model responses side-by-side to debug routing loops, missed tool calls, or prompt ambiguity.

💻 Follow along: The code and evaluation suites for this article are available on the 04b-evaluating-with-adk-run branch in the companion GitHub repository.

ADK logo

Building multi-agent systems with Google ADK 2.0 requires more than just testing single-turn responses — it demands validating complex state updates, sub-agent handoffs, and tool trajectories over multi-turn conversations.

01 · How to Run Evaluations

ADK provides three evaluation surfaces suited to different points in your development workflow:

1. Interactive via Web UI (adk web): Best for visual exploration, interactive debugging, recording live sessions into .evalset.json files, and inspecting execution traces.
2. Scripted via CLI (adk eval): Ideal for local development validation, terminal-based regression runs, and verifying behavior after upgrading foundation models.
3. Automated via Pytest (AgentEvaluator): The standard for production CI/CD pipelines. Uses ADK’s AgentEvaluator programmatically alongside unit and integration tests.

In this article, we concentrate on adk eval, diving deep into how to diagnose and resolve failures when raw recorded sessions meet live evaluation runners.

02 · Turning Recorded Conversations into Evaluations (and Why Raw Exports Fail)

You might think: “We recorded our conversation in the Web UI, exported the .evalset.json, so we are ready to test!” Well… not quite.

If you take a raw session export from the Web UI and pass it directly into an automated evaluation runner without calibrating your environment and prompts, your evaluation will almost certainly FAIL:

ADK_EVAL_MODE=true uv run adk eval \
  adk_agent_app \
  adk_agent_app/evals/eval_set_1_new_user.evalset.json \
  adk_agent_app/evals/eval_set_1a_new_user.evalset.json \
  adk_agent_app/evals/eval_set_2_existing_user.evalset.json \
  --config_file_path=adk_agent_app/evals/eval_config.json \
  --print_detailed_results

Diagnostic Inspection with --print_detailed_results

When running evaluations from the terminal, adding --print_detailed_results produces an invaluable turn-by-turn diagnostic breakdown:

********************************************************************
Eval Run Summary
eval_set_1_new_user:
  Tests passed: 1
  Tests failed: 1
********************************************************************
Eval Set Id: eval_set_1_new_user
Eval Id: Zelda, English
Overall Eval Status: FAILED
---------------------------------------------------------------------
Metric: tool_trajectory_avg_score, Status: FAILED, Score: 0.6666666666666666, Threshold: 1.0
---------------------------------------------------------------------
Metric: response_match_score, Status: PASSED, Score: 0.5557680859560247, Threshold: 0.5
---------------------------------------------------------------------
Invocation Details:
+----+-------------------------+---------------------------+---------------------------+---------------------------+---------------------------+-----------------------------+------------------------+
|    | prompt                  | expected_response         | actual_response           | expected_tool_calls       | actual_tool_calls         | tool_trajectory_avg_score   | response_match_score   |
+====+=========================+===========================+===========================+===========================+===========================+=============================+========================+
|  0 | hello                   | Hello! Welcome to Sosta.  | Hello! Welcome to Sosta.  |                           |                           | Status: PASSED, Score:      | Status: PASSED, Score: |
|    |                         | Are you an already        | Are you an already        |                           |                           | 1.0                         | 1.0                    |
|    |                         | registered user?          | registered user?          |                           |                           |                             |                        |
+----+-------------------------+---------------------------+---------------------------+---------------------------+---------------------------+-----------------------------+------------------------+
|  1 | no, but I would like to | Great! I can help you     | Hello! I can help you with| args={'agent_name':       | args={'agent_name':       | Status: PASSED, Score:      | Status: PASSED, Score: |
|    | register                | with that. To get         | that. What is your full   | 'Registratore'}           | 'Registratore'}           | 1.0                         | 0.65                   |
|    |                         | started, what is your     | name?                     | name='transfer_to_agent'  | name='transfer_to_agent'  |                             |                        |
|    |                         | full name?                |                           |                           |                           |                             |                        |
+----+-------------------------+---------------------------+---------------------------+---------------------------+---------------------------+-----------------------------+------------------------+
|  2 | Zelda Ailine Luconi     | Thank you, Zelda Ailine.  | Thanks, Zelda Ailine!     |                           |                           | Status: PASSED, Score:      | Status: PASSED, Score: |
|    |                         | What is your email        | What is your email        |                           |                           | 1.0                         | 0.82                   |
|    |                         | address?                  | address?                  |                           |                           |                             |                        |
+----+-------------------------+---------------------------+---------------------------+---------------------------+---------------------------+-----------------------------+------------------------+
|  3 | zelda.luconi@gmail.com  | Got it. What is your      | Got it. What is your      |                           |                           | Status: PASSED, Score:      | Status: PASSED, Score: |
|    |                         | preferred language?       | preferred language?       |                           |                           | 1.0                         | 1.0                    |
+----+-------------------------+---------------------------+---------------------------+---------------------------+---------------------------+-----------------------------+------------------------+
|  4 | English                 | Understood. Do you have   | Great. Do you have any    |                           |                           | Status: PASSED, Score:      | Status: PASSED, Score: |
|    |                         | any culinary preferences, | culinary or dietary       |                           |                           | 1.0                         | 0.68                   |
|    |                         | such as Vegetarian or     | preferences, like Vegan   |                           |                           |                             |                        |
|    |                         | Gluten-free?              | or Gluten-free?           |                           |                           |                             |                        |
+----+-------------------------+---------------------------+---------------------------+---------------------------+---------------------------+-----------------------------+------------------------+
|  5 | I am Vegan, love pasta  | Thanks for sharing your   | Great! Lastly, what type  |                           |                           | Status: PASSED, Score:      | Status: PASSED, Score: |
|    |                         | culinary preferences!     | of vehicle do you drive   |                           |                           | 1.0                         | 0.80                   |
|    |                         | Lastly, what type of      | (e.g., Electric,          |                           |                           |                             |                        |
|    |                         | vehicle do you drive?     | Gasoline, Hybrid)?        |                           |                           |                             |                        |
+----+-------------------------+---------------------------+---------------------------+---------------------------+---------------------------+-----------------------------+------------------------+
|  6 | Electric car            | Wonderful! Your           | Hello Zelda Ailine Luconi!| args={'email':            | args={'vehicle_type':     | Status: FAILED, Score:      | Status: FAILED, Score: |
|    |                         | registration is complete. | I am Cameriere, your      | 'zelda.luconi@gmail.com', | 'Electric',               | 0.0                         | 0.25                   |
|    |                         | I'm now handing you over  | Dining Concierge. Are you | 'full_name': 'Zelda       | 'culinary_preferences':   |                             |                        |
|    |                         | to Cameriere.             | traveling with companions?| Ailine Luconi',           | ['Vegan'], 'full_name':   |                             |                        |
|    |                         |                           | And what is your          | 'preferred_language':     | 'Zelda Ailine Luconi',    |                             |                        |
|    |                         |                           | destination?              | 'English',                | 'email':                  |                             |                        |
|    |                         |                           |                           | 'vehicle_type':           | 'zelda.luconi@gmail.com'} |                             |                        |
|    |                         |                           |                           | 'Electric',               | name='save_new_user',     |                             |                        |
|    |                         |                           |                           | 'culinary_preferences':   | name='transfer_to_agent', |                             |                        |
|    |                         |                           |                           | ['Vegan']}                | name='extract_user_profile|                             |                        |
|    |                         |                           |                           | name='save_new_user'      | ' (args={'user_id':       |                             |                        |
|    |                         |                           |                           |                           | 'usr_7c0adf'})            |                             |                        |
+----+-------------------------+---------------------------+---------------------------+---------------------------+---------------------------+-----------------------------+------------------------+
|  7 | I am traveling alone    | Perfect, I have loaded    | Great! So, to summarize:  | args={'agent_name':       |                           | Status: FAILED, Score:      | Status: PASSED, Score: |
|    | and going to Rome       | your details. Here is a   | * User ID: usr_7c0adf     | 'Cameriere'}              |                           | 0.0                         | 0.53                   |
|    |                         | summary of what I have    | * Preferences: Vegan      | name='transfer_to_agent', |                           |                             |                        |
|    |                         | gathered so far...        | * Destination: Rome       | args={'user_id':          |                           |                             |                        |
|    |                         |                           | Is this summary correct?  | 'usr_7c0adf'} name='extra |                           |                             |                        |
|    |                         |                           |                           | ct_user_profile'          |                           |                             |                        |
+----+-------------------------+---------------------------+---------------------------+---------------------------+---------------------------+-----------------------------+------------------------+
|  8 | yes                     | Great! All details are    | Wonderful! I will pass    |                           |                           | Status: PASSED, Score:      | Status: PASSED, Score: |
|    |                         | confirmed. Passing to     | these details to          |                           |                           | 1.0                         | 0.78                   |
|    |                         | Suggeritore.              | Suggeritore.              |                           |                           |                             |                        |
+----+-------------------------+---------------------------+---------------------------+---------------------------+---------------------------+-----------------------------+------------------------+

Analyzing this diagnostic table reveals the key reasons why raw evaluation recordings fail out of the box:

1. Turn Misalignment during Sub-Agent Transfers (Turn 6 vs. Turn 7): Registratore saves the user, transfers control to Cameriere, and Cameriere immediately executes extract_user_profile in Turn 6. The raw recording expected the transfer and profile extraction to be spread out over two turns.
2. Missing Dynamic Keys: Dynamic values (like randomly generated user IDs) cause downstream tool parameter mismatches.
3. Overly Strict Natural Language Matching: Exact string matching fails on harmless phrasing variations.

03 · Five Common Evaluation Pitfalls & How to Fix Them

1. Relative Imports When Executing from CLI

When running agent modules via adk eval, the command executes from your workspace root. If your agent modules use absolute or top-level relative imports that rely on specific PYTHONPATH setups, imports fail immediately.

The Fix: Use explicit package-relative imports inside your submodules:

# adk_agent_app/tools/gatekeeper_tools.py
from ..config import logger  # Resolves cleanly inside adk_agent_app

2. Resolving Dynamic Parameter Mismatches with Deterministic Hashing

When tools generate random database IDs (e.g. usr_ + UUID hex), downstream tools like extract_user_profile(user_id='...') fail trajectory matching because the ID is different on every run.

Returning a single hardcoded ID like "usr_f93207" also fails when multiple evaluation cases run in parallel, because concurrent test cases (e.g., Zelda and Flora) overwrite each other's documents in Firestore!

The Fix: Implement deterministic, email-based MD5 hashing in test mode (ADK_EVAL_MODE=true):

# adk_agent_app/tools/registratore_agent_tools.py
import hashlib, os, uuid

def get_user_id(email: str) -> str:
    """Deterministic email-based ID when running under ADK eval CLI."""
    if os.getenv("ADK_EVAL_MODE") == "true":
        clean_email = email.strip().lower()
        email_hash = hashlib.md5(clean_email.encode()).hexdigest()[:6]
        return f"usr_{email_hash}"
    return f"usr_{uuid.uuid4().hex[:6]}"

With this setup:

• zelda.luconi@gmail.com deterministically generates usr_7c0adf
• f.rossi@gmail.com deterministically generates usr_ba7db8

Both IDs are stable, repeatable, and completely collision-free across concurrent test runs.

3. Session State Pollution Across Evaluation Cases

If multiple test cases run under a shared generic session key like "user_id": "user", the second test case inherits cached session variables from the first.

Get Zelda Ailine Luconi’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The Fix: Assign distinct user_id identifiers in session_input for each evaluation case:

// Flora_English test case
"session_input": {
  "app_name": "adk_agent_app",
  "user_id": "flora_test_user"
}

// Zelda, English test case
"session_input": {
  "app_name": "adk_agent_app",
  "user_id": "zelda_test_user"
}

4. Decoupling Profile Hydration: The Anti-Pattern of Duplicate State Loading

In multi-agent architectures, agents must adhere to the Single Responsibility Principle (SRP). A common pitfall is having the entry gateway (Gatekeeper) read the user's complete profile and populate session state (user:name, user:culinary_preferences, user:vehicle_type), only for the downstream specialist (Cameriere) to immediately re-read the exact same Firestore document and overwrite the exact same keys via extract_user_profile.

This creates architectural redundancy, double database reads, and brittle evaluations. Furthermore, when control is handed off from Registratore or Gatekeeper, the conversational history already contains the user's profile details. Without an explicit rule, the LLM takes a "conversational shortcut," assuming it already knows the user without executing extract_user_profile, which causes tool_trajectory_avg_score to fail.

The Fix (Approccio A — Lean Gateway & Specialized Hydrator):

1. Keep Gatekeeper.is_registered_user lean: It only checks if user_id exists in Firestore and sets state["verified_user_id"] = user_id. It leaves profile hydration entirely to downstream agents.

# adk_agent_app/tools/gatekeeper_tools.py
async def is_registered_user(user_id: str, tool_context: ToolContext) -> Dict[str, Any]:
    # 1. Verify existence in Firestore
    user_doc = db.collection("users").document(user_id).get()
    if not user_doc.exists:
        return {"status": "not_found", "exists": False, "message": "User ID does not exist."}


# 2. Store verified ID in session memory (no redundant profile hydration)
    tool_context.state["verified_user_id"] = user_id
    tool_context.state["workflow_step"] = "user_verified"
    return {
        "status": "success",
        "exists": True,
        "verified_user_id": user_id,
        "message": f"User '{user_id}' verified. Profile hydration delegated to Cameriere.",
    }

1. Enforce mandatory profile extraction in cameriere_agent.py: Instruct Cameriere to execute extract_user_profile immediately upon receiving control, greeting the user in the language they are currently speaking and checking their profile for saved language preferences:

1. AUTOMATIC PROFILE HYDRATION:
   - Whenever control is transferred to you from Gatekeeper or Registratore (on your first turn in the conversation), you MUST IMMEDIATELY execute `extract_user_profile` using the verified User ID before replying, even if the user's details already appear in the conversation history.
   - LANGUAGE & GREETING:
     * Always speak in the language the user is CURRENTLY using in the conversation.
     * In your initial reply, greet the user by their full name (from the profile returned by `extract_user_profile`).
     * If their saved preferred language differs from the language the user is currently speaking, note that their saved preferred language is different and ask whether they prefer to continue in their current language or switch to their saved language.

5. Guarding Against Placeholder Sub-Agents

When Cameriere confirms the summary, the next logical step is handing off to Suggeritore. However, before Part 5a, suggeritore_agent is still commented out as a placeholder. If Cameriere tries to call transfer_to_agent(agent_name='Suggeritore'), ADK fails because the sub-agent is not yet registered.

The Fix: Explicitly instruct Cameriere to confirm the handoff in text without invoking transfer_to_agent, and ensure the golden .evalset.json reflects zero tool calls on that final confirmation turn:

4. HANDOFF PREPARATION:
   - Summarize all gathered details clearly: User ID, Profile Preferences, Companion Restrictions, and Destination, and ask the user if the summary is correct.
   - Once the user confirms the summary (e.g., "yes"), reply briefly in text confirming that you will pass all these details to Suggeritore to prepare their personalized recommendations for their destination (do NOT call `transfer_to_agent`).

04 · Tackling Trajectory Strictness in eval_config.json

A fundamental challenge in evaluating multi-turn agents is balancing conversational flexibility with operational safety:

• In normal dialog, we want to allow natural linguistic variations (response_match_score: 0.50).
• In state-mutating actions (such as creating records via save_new_user or altering preferences via update_dietary_preferences), we cannot tolerate missing arguments or spurious tool calls. An incorrect parameter payload corrupts the database.

ADK 2.0 allows you to address this strictness dilemma directly inside eval_config.json:

{
  "criteria": {
    "tool_trajectory_avg_score": {
      "threshold": 1.0,
      "match_type": "EXACT",
      "fail_on_unexpected_tool_call": true,
      "per_tool_rules": {
        "save_new_user": {
          "require_exact_match": true,
          "description": "State-mutating database write requires exact arguments match"
        },
        "update_dietary_preferences": {
          "require_exact_match": true,
          "description": "State-mutating profile update requires exact arguments match"
        },
        "extract_user_profile": {
          "require_exact_match": true,
          "description": "Session hydration check"
        },
        "transfer_to_agent": {
          "require_exact_match": true,
          "description": "Strict agent routing control"
        }
      }
    },
    "response_match_score": 0.50
  }
}

Key Dimensions Explained:

• match_type: "EXACT": Ensures that every expected tool invocation executes in the exact sequence expected.
• fail_on_unexpected_tool_call: true: Automatically fails the evaluation if the agent hallucinates an unprompted tool call or calls a tool out of sequence.
• per_tool_rules with require_exact_match: true: Enforces strict argument validation on high-stakes, state-mutating functions (save_new_user, update_dietary_preferences), guaranteeing that data persistence contracts are upheld without compromising dialog naturalness.

05 · Binding eval_config.json in Pytest CI/CD Gates

It is a bit too early, but to execute evaluations automatically in your CI/CD test runners, bind eval_config.json via ADK's AgentEvaluator:

# tests/test_evals.py
from pathlib import Path
import pytest
from google.adk.evaluation.agent_evaluator import AgentEvaluator

@pytest.mark.asyncio
async def test_new_user_onboarding_evalset():
    """Run regression evaluations using custom scoring criteria and strict trajectory rules."""
    evals_dir = Path("adk_agent_app/evals")
    eval_file = evals_dir / "eval_set_1_new_user.evalset.json"
    config_file = evals_dir / "eval_config.json"
    result = await AgentEvaluator.evaluate(
        agent_module="adk_agent_app.agent",
        eval_dataset_file_path_or_dir=str(eval_file),
        eval_config_file_path=str(config_file),
        num_runs=1,
    )
    assert result.overall_eval_status.name == "PASSED"

06 · The Final Verification

With deterministic hashing in place, prompt instructions calibrated, and eval_config.json configured, running both evaluation suites yields 100% pass rates:

ADK_EVAL_MODE=true uv run adk eval adk_agent_app \
      adk_agent_app/evals/eval_set_1_new_user.evalset.json \
      --config_file_path=adk_agent_app/evals/eval_config.json

*********************************************************************
Eval Run Summary
eval_set_1_new_user:
  Tests passed: 2
  Tests failed: 0
*********************************************************************

ADK_EVAL_MODE=true uv run adk eval adk_agent_app \
    adk_agent_app/evals/eval_set_2_existing_user.evalset.json \
   --config_file_path=adk_agent_app/evals/eval_config.json

*********************************************************************
Eval Run Summary
eval_set_2_existing_user:
  Tests passed: 1
  Tests failed: 0
*********************************************************************

💰 Cost Note: Evaluations Aren’t Free

Unlike unit tests that execute locally for free, evaluation runs call your LLM once per turn, per test case:

• A 10-case evaluation suite with 4 turns per case requires 40 model API calls.
• Optimize your pipeline: Run full evaluation suites on pull requests and merges, rather than on every commit.
• Keep commit hooks fast: Run single-turn unit tests during local development, reserving full evaluation suites for milestone validation.

Summary & Next Steps

Testing conversational agents in Google ADK 2.0 requires balancing text flexibility with operational rigor:

1. Trace the Trajectory: Always verify the sequence and arguments of tool invocations, not just the final text response.
2. Isolate State: Use deterministic hash IDs and independent session inputs across test cases.
3. Control Strictness: Use eval_config.json with per_tool_rules to demand exact argument validation on database mutations while allowing natural phrasing in user dialogue.

In the next article, we will describe the 3rd Pillar of the Digital Assembly line, discussing MCP, A2A and RAGs.

« Previous: Part 4a | ADK101 Hub | Next: Part 5a »

Frequently Asked Questions (FAQ)

Q: How do you run an evaluation suite from the command line in Google ADK 2.0?
A: Run ADK_EVAL_MODE=true uv run adk eval <path_to_agent_folder> <path_to_evalset.json> --config_file_path=<path_to_eval_config.json> --print_detailed_results.

Q: Why does my evaluation fail even when the agent generates the right answer?
A: Generative models rarely produce exact reference strings. Lower response_match_score from 0.80 to 0.50 in eval_config.json to allow natural phrasing variations while keeping tool trajectory checks strict.

Q: Why shouldn’t Gatekeeper hydrate the entire user profile directly in is_registered_user?
A: In clean multi-agent architecture, the gateway agent (Gatekeeper) is solely responsible for authentication and routing (verified_user_id). Populating profile details in Gatekeeper when Cameriere immediately re-reads them via extract_user_profile causes redundant database queries and state pollution. Keeping Gatekeeper lean and delegating domain hydration to Cameriere guarantees clear ownership and deterministic evaluation traces.

Q: How can I enforce strict validation only on state-mutating tools?
A: Set fail_on_unexpected_tool_call: true and configure per_tool_rules in eval_config.json with "require_exact_match": true on critical tools (e.g., save_new_user, update_dietary_preferences).