When you need this. You change an assistant’s instruction and want to know whether it still reads the right documentation and gives a complete answer. We will write those expectations as evaluation cases, then compare tool use and response text separately.
A tool trajectory is the sequence of tool calls and arguments in a turn. A reference answer describes the response the case expects. ADK’s AgentEvaluator runs an agent on the cases, records the actual invocations and applies the selected metrics.
For the docs assistant, reading cli.md is a trajectory expectation. Explaining that a rebuild drops the index and recreates it from the docs folder is an answer expectation. A passing tool check cannot tell us whether the response includes that explanation.
Prepare the evaluation project
You need Python 3.12 or later and a Gemini API key. Use Download code, extract the ZIP, and open its adk-agent-eval-lesson directory. Keep the Python files and fixtures together. All commands in this lesson run from that directory.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txtOn Windows PowerShell, activate with .venv\Scripts\Activate.ps1.
The Python client reads exported environment variables. The bundled .env.example documents them; it is not loaded by the client. Set GOOGLE_API_KEY privately, then select the Gemini API in this terminal:
unset GOOGLE_GENAI_USE_VERTEXAI DOCS_MODEL
export GOOGLE_GENAI_USE_ENTERPRISE=0Sending a request to the model uses your provider account. The offline checks later in the lesson need no key.
Use a separate virtual environment for this lab. Its requirements include google-adk[eval] and the metric dependencies. Installing the evaluation extra prepares the runtime; it does not run any cases.
The rebuild case uses this documentation as its source:
---
title: CLI reference
---
# CLI reference
## ask
`docs-assistant ask QUESTION` prints an answer with citation links.
## index
`docs-assistant index --rebuild` drops the search index and builds it again from the docs folder.Define the agent under test
The agent has two local tools. list_pages returns the available Markdown filenames; read_page reads one selected file. Its instruction asks it to list the pages, read the relevant one and answer from that page.
"""The docs assistant under evaluation: it answers from the project's docs pages."""
from pathlib import Path
from google.adk import Agent
from google.genai import types
DOCS = Path(__file__).with_name("docs")
def list_pages() -> dict:
"""List the docs pages by path."""
return {"pages": sorted(page.name for page in DOCS.glob("*.md"))}
def read_page(path: str) -> dict:
"""Return the Markdown of one docs page, by the path list_pages returned."""
page = DOCS / path
if page.parent != DOCS or not page.is_file():
return {"error": f"No page {path!r}. Call list_pages for valid paths."}
return {"path": path, "content": page.read_text()}
root_agent = Agent(
name="docs_assistant",
model="gemini-3.8-flash", # or LiteLlm(model="ollama_chat/gemma4:e2b")
generate_content_config=types.GenerateContentConfig(max_output_tokens=1024, thinking_config=types.ThinkingConfig(thinking_level="low")),
instruction=("Answer questions about the docs-assistant project from its docs only. "
"Call list_pages, then read_page on the one page that answers. Reply in "
"one sentence and end it with the page path in parentheses. "
"For a rebuild question, include the command, that it drops the search index, "
"and that it builds the index again from the docs folder."),
tools=[list_pages, read_page],
)For rebuild questions, the instruction asks for the command, its drop-and-recreate behavior and the source folder. These details are part of the answer contract, not merely a preference for wording.
A tool reads outside its folder. read_page("../agent.py") returns an error instead of reading that file. The bundled failure test also covers a missing page.
Write the eval set
An eval case holds a conversation, expected response and expected intermediate tool calls. The bundled file has one case about rebuilding the index and another about the required Python version:
{
"eval_set_id": "docs_agent_smoke",
"name": "docs_agent_smoke",
"description": "Two questions whose answer is on exactly one docs page.",
"eval_cases": [
{
"eval_id": "rebuild_index",
"conversation": [
{
"invocation_id": "rebuild_index-1",
"user_content": {"role": "user", "parts": [{"text": "How do I rebuild the search index?"}]},
"final_response": {"role": "model", "parts": [{"text": "Run docs-assistant index --rebuild, which drops the search index and builds it again from the docs folder (cli.md)."}]},
"intermediate_data": {
"tool_uses": [
{"name": "list_pages", "args": {}},
{"name": "read_page", "args": {"path": "cli.md"}}
],
"intermediate_responses": []
}
}
],
"session_input": {"app_name": "docs_agent", "user_id": "dev", "state": {}}
},
{
"eval_id": "python_version",
"conversation": [
{
"invocation_id": "python_version-1",
"user_content": {"role": "user", "parts": [{"text": "Which Python version does the assistant need?"}]},
"final_response": {"role": "model", "parts": [{"text": "The assistant needs Python 3.12 or later (install.md)."}]},
"intermediate_data": {
"tool_uses": [
{"name": "list_pages", "args": {}},
{"name": "read_page", "args": {"path": "install.md"}}
],
"intermediate_responses": []
}
}
],
"session_input": {"app_name": "docs_agent", "user_id": "dev", "state": {}}
}
]
}intermediate_data.tool_uses specifies both tool names and arguments. The rebuild case expects list_pages, then read_page with {"path":"cli.md"}. Its reference answer includes the command and what the command does.
Write expectations from the task requirements. Copying whatever an agent did into the expected trajectory can turn an existing mistake into the behavior your test rewards.
These are small development cases. Add unsupported questions and other document tasks before using an evaluation to make a release decision. Keep a separate held-out set you do not use to tune the instruction.
Choose the criteria
The first configuration uses exact tool-trajectory matching and ROUGE-1 word overlap for the response. Both scoring methods run without a judge model; generating the agent answers still calls Gemini.
{
"criteria": {
"tool_trajectory_avg_score": {"threshold": 1.0, "match_type": "EXACT"},
"response_match_score": 0.8
}
}EXACT requires the same calls, arguments and order. IN_ORDER permits extra calls if the expected sequence still appears. ANY_ORDER permits extra calls and ignores ordering. By default, all three compare arguments.
ROUGE-1 compares shared words with the reference. It is easy to recompute, but it can penalize a useful paraphrase or reward an incomplete answer that repeats familiar words. Read the response before deciding what a low score means.
A correct paraphrase fails ROUGE. The bundled scorer test checks answers that include the rebuild command yet average below the response threshold. That demonstrates a word-overlap limitation. Also compare the answer’s explanation with the full contract; a correct command can still omit required effects.
An extra tool call under EXACT. If a case expects only read_page, an actual list_pages followed by read_page fails EXACT but passes IN_ORDER. Choose whether the extra call is acceptable before selecting the match type.
A different argument. A call to read_page("troubleshooting.md") does not match read_page("cli.md") under any match type. ignore_args=True would compare names only and miss that error.
Run the evaluation
The wrapper runs every case twice with AgentEvaluator.evaluate_eval_set, then saves the result. Read the call near the top of the source to see agent_module, the case set, criteria and num_runs together:
"""Evaluate the docs assistant on Gemini twice: ROUGE-1 response match, then an LLM judge."""
import asyncio
import json
import hashlib
import sys
from pathlib import Path
from google.adk.evaluation.agent_evaluator import AgentEvaluator
from google.adk.evaluation.eval_config import get_evaluation_criteria_or_default
from google.adk.evaluation.eval_set import EvalSet
from google.adk.evaluation.local_eval_set_results_manager import LocalEvalSetResultsManager
HERE = Path(__file__).parent
CONFIGS = {"rouge": "test_config.json", "judge": "judge_config.json"}
async def evaluate(name: str) -> None:
eval_set = EvalSet.model_validate_json((HERE / "docs_agent.test.json").read_text())
history = HERE / "eval-results"
before = history_versions(history)
original_failure = None
try:
await AgentEvaluator.evaluate_eval_set(
agent_module="docs_agent",
eval_set=eval_set,
eval_config=get_evaluation_criteria_or_default(str(HERE / CONFIGS[name])),
num_runs=2,
print_detailed_results=False,
app_name="docs_agent",
eval_set_results_manager=LocalEvalSetResultsManager(agents_dir=str(history)),
)
print(f"{name}: PASSED")
except BaseException as failure:
original_failure = failure
if isinstance(failure, AssertionError):
print(f"{name}: FAILED\n{failure}")
raise
finally:
# Preserve the provider/evaluation failure even if its history is absent or incomplete.
try:
after = history_versions(history)
current = [path for path, version in after.items() if before.get(path) != version]
if current:
newest = max(current, key=lambda path: path.stat().st_mtime_ns)
(HERE / f"{name}-result.json").write_text(
json.dumps(compact(json.loads(newest.read_text())), indent=1) + "\n")
else:
print("No current-run evaluation history was written; saved results were not replaced.", file=sys.stderr)
except Exception as history_error:
if original_failure is None:
raise
print(f"Current-run history could not be saved ({type(history_error).__name__}); original failure retained.", file=sys.stderr)
# Keep raw histories, including incomplete runs, for inspection.
def history_versions(folder):
"""Distinguish new/changed results from histories left by an earlier run."""
return {path: (path.stat().st_mtime_ns, hashlib.sha256(path.read_bytes()).hexdigest())
for path in folder.rglob("eval_history/*.json")}
def compact(value):
"""Drop null fields and opaque thought signatures so the saved result stays readable."""
if isinstance(value, dict):
return {k: compact(v) for k, v in value.items() if v is not None and k != "thought_signature"}
if isinstance(value, list):
return [compact(v) for v in value]
return value
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser(description="Optional fresh multi-case agent evaluation; judge mode also calls an LLM judge.")
parser.add_argument("criterion", choices=CONFIGS)
parser.add_argument("--evaluate", action="store_true")
options = parser.parse_args()
if not options.evaluate:
parser.error("evaluation is opt-in: use --evaluate in a separate copy")
asyncio.run(evaluate(options.criterion))The native evaluation CLI uses the same agent, eval set and configuration:
adk eval docs_agent docs_agent.test.json --config_file_path test_config.json --log_level errorThis runs the cases once with the pinned CLI. Read its case-status summary; the process can exit zero while reporting a failed metric. Use the Python wrapper when you want the explicit two-run configuration and a nonzero exit for failed thresholds. From a separate copy of the lab, run:
python run_eval.py rouge --evaluateThe Python command starts fresh agent requests and can replace rouge-result.json in that copy. The explicit --evaluate flag selects that activity. Inspect the printed metric failures and per-invocation records in eval-results/, including the actual answer and tool calls.
The wrapper raises on failed thresholds and keeps the failure as a nonzero exit. For each failure, ask which contract was missed: the tool sequence, tool arguments, response content or metric’s approximation of that content. Do not lower a threshold solely to make a failing run pass.
Compare meaning with an LLM judge
When word overlap is a poor fit, final_response_match_v2 uses a model to compare response meaning with the reference. The supplied judge configuration makes that choice explicit:
{
"criteria": {
"tool_trajectory_avg_score": {"threshold": 1.0, "match_type": "EXACT"},
"final_response_match_v2": {
"threshold": 0.8,
"judge_model_options": {"judge_model": "gemini-3.8-flash", "num_samples": 3}
}
}
}judge_model selects the judge and num_samples specifies repeated judgments per invocation. You can run the same evaluation activity with python run_eval.py judge --evaluate in a separate copy. It generates fresh agent answers and adds judge requests; it does not re-score the earlier answers in place.
A judge is another measurement with its own variability. Compare its decisions with human review, especially when a response contains the right command but misses a required warning or effect. The two known questions here are enough to explore the metric, not to estimate general answer quality.
Re-score the sample invocations offline
The downloadable lab also includes saved invocations for learning how the scorers behave. These checks recompute trajectory and ROUGE scores using ADK’s real evaluator classes:
python -m unittest -v test_eval_results.py test_client.pyThey use TrajectoryEvaluator and RougeEvaluator on saved actual/expected pairs. No new agent or judge response is generated. This is useful for testing a scorer change; it cannot tell you how a new instruction will answer fresh questions.
The sample rebuild answers below illustrate why the response needs human inspection alongside its score:
Run `docs-assistant index --rebuild` to drop the search index and rebuild it from the docs folder. (cli.md)
You can rebuild the search index by running `docs-assistant index --rebuild` (cli.md).Try changing the expected page argument in a separate test copy while keeping the tool name. Predict the trajectory score, then run the matching offline check. This distinguishes a tool being called from the tool being used on the right input.
Read the explanation
Trajectory and response criteria check different requirements. Reading the right page does not guarantee a complete answer.
Read trajectory and response scores
AgentEvaluator first runs inference: it creates sessions, sends the case turns to the agent and records the final response plus invocation events. The metrics then compare actual and expected invocations.
The trajectory metric extracts function calls and compares their names, arguments and order. ROUGE compares final-response text. A case’s metric result aggregates its runs, so inspect the individual runs as well as the mean; one repeat can pass while the case average fails.
Keep unit tests for local tools, contract checks for expected event shapes and evaluations for task behavior. Each catches a different kind of error. More passing development cases do not replace held-out questions or review of the actual answers.
Continue with Choose an ADK workflow pattern to compare the mechanisms from this module.
References
The evaluation APIs and metric behavior were checked against the installed google-adk 2.12.0 source. The requirements file pins ADK and the offline scorer.
- ADK evaluation: cases and evaluation runs.
- ADK evaluation criteria: trajectory and response metrics.
- ADK 2.0 overview: runtime events.
- rouge-score: the word-overlap implementation.