Test an agent against simulated users

A fixed eval case replays the same user messages whatever the agent says. That breaks as soon as the agent asks a question back: the next recorded message no longer fits. ADK’s user simulator solves this by letting a model play the user. You give it a scenario (how the conversation starts, what the user wants and how they behave), and it writes each next user turn from the agent’s last reply. We give the docs assistant two scenarios, a vague bug report from a novice and a precise question from an expert, then score both conversations. In the recorded run, the agent answered the vague report with a question, and the simulated user supplied the symptom:

When you need this. A docs assistant that asks “what error do you see?” before it answers can’t be tested with one fixed prompt, because the second user message depends on the question. Any agent that clarifies, confirms or collects details over several turns needs conversations that react to it.

Invocation 1 of 2
  prompt:                     The docs assistant isn't working for me.
  actual_response:            What specific error or symptom are you encountering when trying to use the docs
                              assistant?
  actual_tool_calls:          (none)
  hallucinations_v1:          PASSED (1.0)
  per_turn_user_simulator_quality_v1: PASSED (1.0)

Invocation 2 of 2
  prompt:                     It's saying that it can't reach Ollama.
  actual_response:            Check that `OLLAMA_API_BASE` points to a running Ollama server
                              (troubleshooting.md).
  actual_tool_calls:          list_pages()
                              read_page(
                                path='troubleshooting.md')
  hallucinations_v1:          PASSED (1.0)
  per_turn_user_simulator_quality_v1: PASSED (1.0)

Lab setup. Download the complete lab ZIP, extract it and open a terminal in adk-user-simulation/. Keep the bundled code, tests and fixtures together. Create the pinned environment before running the commands below:

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-lock.txt

1. Let the agent ask before it answers

Download /examples/adk-user-simulation/ and install requirements-lock.txt, which includes the [eval] extra. The agent is the docs assistant with one new sentence in its instruction: when the question doesn’t say what is failing, ask about the symptom first. That sentence is what makes a multi-turn test necessary.

"""The docs assistant: it answers questions about the docs-assistant project from its docs pages."""
import os
from pathlib import Path

from google.adk import Agent
from google.adk.models.lite_llm import LiteLlm
from google.genai import types

# Gemini through ADK's native client; set DOCS_MODEL=ollama_chat/gemma4:e2b to run Gemma locally.
MODEL = os.environ.get("DOCS_MODEL", "gemini-3.8-flash")
DOCS = Path(__file__).resolve().parent / "docs"


def list_pages() -> dict:
    """List the docs pages by path."""
    return {"pages": sorted(page.name for page in DOCS.glob("*.md"))}


def read_page(path: str) -> dict:
    """Return the Markdown of one docs page, by the path list_pages returned."""
    page = DOCS / path
    if page.parent != DOCS or not page.is_file():
        return {"error": f"No page {path!r}. Call list_pages for valid paths."}
    return {"path": path, "content": page.read_text(encoding="utf-8")}


root_agent = Agent(
    name="docs_assistant",
    model=LiteLlm(model=MODEL) if MODEL.startswith("ollama_chat/") else MODEL,
    instruction=("Answer questions about the docs-assistant project from its docs only. "
                 "Call list_pages, then read_page on the one page that answers. Reply in "
                 "one sentence and end it with the page path in parentheses. If the question "
                 "does not say what is failing, ask one short question about the symptom first."),
    tools=[list_pages, read_page],
    generate_content_config=types.GenerateContentConfig(
        temperature=0.2, thinking_config=types.ThinkingConfig(thinking_level="low")),
)

No eval extra. adk eval in an environment with plain google-adk exits with Eval module is not installed. Install google-adk[eval] in the environment that runs evals.

A .env in a parent folder. The CLI walked up from the agent folder to a .env that selects Agent Platform, so model calls went to Vertex AI and returned 404 NOT_FOUND. Put a .env in the agent folder, or set ADK_DISABLE_LOAD_DOTENV=1.

# A venv with google-adk but without the [eval] extra
$ adk eval docs_assistant scenarios --config_file_path eval_config.json
Error: Eval module is not installed, please install via `pip install "google-adk[eval]"`.

# A .env in a parent folder that selects Agent Platform (Vertex AI)
$ adk eval docs_assistant scenarios --config_file_path eval_config.json
2026-10-21 20:25:39,030 - INFO - google_llm.py:263 - Sending out request, model: gemini-3.8-flash, backend: GoogleLLMVariant.VERTEX_AI, stream: False
2026-10-21 20:25:39,690 - ERROR - local_eval_service.py:638 - Inference failed for eval case `6be7706e` with error 404 NOT_FOUND. {'error': {'code': 404, 'message': 'Publisher model `projects/PROJECT_ID/locations/us-central1/publi
2026-10-21 20:25:39,722 - ERROR - local_eval_service.py:638 - Inference failed for eval case `1ae10b45` with error 404 NOT_FOUND. {'error': {'code': 404, 'message': 'Publisher model `projects/PROJECT_ID/locations/us-central1/publi

2. Write the scenarios

Each scenario is a ConversationScenario. starting_prompt is sent verbatim as the first user turn. conversation_plan tells the simulator what the user wants and when to stop; it is supplied to the simulator, rather than directly to the agent. The simulator can include details from the plan in its user messages. user_persona sets how the user behaves: NOVICE waits to be asked for details and can’t correct the agent, EXPERT gives details up front and challenges wrong facts.

{
  "scenarios": [
    {
      "starting_prompt": "The docs assistant isn't working for me.",
      "conversation_plan": "When the agent asks what is failing, say it reports that it cannot reach Ollama. After it answers, ask which setting tells it where the Ollama server is. End the conversation once you have the setting name.",
      "user_persona": "NOVICE"
    },
    {
      "starting_prompt": "How do I make the assistant use a local model instead of Gemini?",
      "conversation_plan": "After the answer, ask which Python version the project needs. If the agent states a fact you have reason to doubt, ask it to check. End once you know both the setting and the Python version.",
      "user_persona": "EXPERT"
    }
  ]
}

Write plans the way a tester writes steps: what to say when the agent asks, and the condition for ending. “End once you have the setting name” gives the simulator a stop rule. Without one, it keeps talking until the turn cap.

A turn cap shorter than the plan. With max_allowed_invocations: 1, both conversations stopped after the starting prompt. The replies passed hallucinations_v1, but per_turn_user_simulator_quality_v1 scored 0.0. Set the cap above the longest plan, and keep a stop rule in every plan.

# Run one-turn-limit
$ adk eval docs_assistant scenarios --config_file_path config-one-turn.json --print_detailed_results
  Tests passed: 0
  Tests failed: 2
Metric: hallucinations_v1, Status: PASSED, Score: 1.0, Threshold: 0.8
Metric: per_turn_user_simulator_quality_v1, Status: FAILED, Score: 0.0, Threshold: 0.8

3. Add the scenarios to an eval set

adk eval_set create docs_assistant scenarios
adk eval_set add_eval_case docs_assistant scenarios \
--scenarios_file scenarios.json --session_input_file session_input.json

add_eval_case writes docs_assistant/scenarios.evalset.json with one case per scenario. Each case has a conversation_scenario and no conversation: there are no expected turns, because nobody knows the turns in advance. The command also expands the persona name into the persona’s full list of behaviors and the rubrics used to check them, so the eval set records exactly what the simulator was told. session_input.json names the app and user the sessions are created for.

4. Configure the simulator and the metrics

{
  "criteria": {
    "hallucinations_v1": {
      "threshold": 0.8,
      "evaluate_intermediate_nl_responses": true,
      "judge_model_options": {"judge_model": "gemini-3.8-flash", "num_samples": 1}
    },
    "per_turn_user_simulator_quality_v1": {
      "threshold": 0.8,
      "judge_model_options": {"judge_model": "gemini-3.8-flash", "num_samples": 1}
    }
  },
  "user_simulator_config": {
    "model": "gemini-3.8-flash",
    "max_allowed_invocations": 6
  }
}

Two metrics fit cases without reference answers:

• hallucinations_v1 asks a judge model whether each sentence of each agent reply is supported by the context the agent had, including tool results. With evaluate_intermediate_nl_responses, the clarifying question is checked too.
• per_turn_user_simulator_quality_v1 checks the simulator instead of the agent: did each user turn follow the plan and the persona, and did the conversation end when the plan said? A failing score means the test itself went off script, so read it before you trust the agent’s scores.

user_simulator_config pins the simulator to gemini-3.8-flash (the 2.12 default is gemini-2.5-flash; adk.dev lists gemini-flash-latest) and caps the conversation at six turns. Both judges use num_samples: 1 to keep the cost of the example low; more samples let you measure variation between judge scores.

Reference metrics on scenario cases. Without a config file, adk eval uses tool_trajectory_avg_score and response_match_score. Both raise expected_invocations is needed and every case is NOT_EVALUATED. Use metrics that need no reference, as in eval_config.json.

# Run default-criteria
$ adk eval docs_assistant scenarios --print_detailed_results
  Tests passed: 0
  Tests failed: 2
Metric: tool_trajectory_avg_score, Status: NOT_EVALUATED, Score: None, Threshold: 1.0
Metric: response_match_score, Status: NOT_EVALUATED, Score: None, Threshold: 0.8
ValueError: expected_invocations is needed by this metric.
ValueError: expected_invocations is required for this metric.

A metric that needs Agent Platform. multi_turn_task_success_v1 calls the Gen AI evaluation service on Agent Platform. With an AI Studio key the call returned 403 PERMISSION_DENIED and the metric stayed NOT_EVALUATED. Run it with a Google Cloud project, or score task success with your own metric.

# Run task-success-metric
$ adk eval docs_assistant scenarios --config_file_path config-task-success.json --print_detailed_results
  Tests passed: 0
  Tests failed: 2
Metric: multi_turn_task_success_v1, Status: NOT_EVALUATED, Score: None, Threshold: 0.8
google.genai.errors.ClientError: 403 PERMISSION_DENIED. {'error': {'code': 403, 'message': 'Requests to this API aiplatform.googleapis.com method google.cloud.aiplatform.v1beta1.EvaluationService.EvaluateInstances are bl ...

5. Run the evaluation

Run adk eval with eval_config.json to drive the scenarios and score the generated turns. The agent, simulated user and model-based metrics make model requests; the offline tests below replay saved replies:

ADK_DISABLE_LOAD_DOTENV=1 adk eval docs_assistant scenarios --config_file_path eval_config.json --print_detailed_results

ADK_DISABLE_LOAD_DOTENV=1 prevents the adk CLI from loading a parent .env; it does not select a backend. This recorded run used the key-based Gemini configuration already exported in the process. Both scenarios passed in this recorded run. In the novice conversation, the agent asked for the symptom without calling a tool, the simulator answered from its plan, and the agent then read troubleshooting.md. The simulator then ended the conversation without the follow-up question in its plan: the answer already named the setting it was meant to ask for. The judges’ scores per turn are in the saved result:

[
  {
    "eval_id": "6be7706e",
    "status": 1,
    "turns": [
      {
        "user": "How do I make the assistant use a local model instead of Gemini?",
        "scores": {
          "hallucinations_v1": 1,
          "per_turn_user_simulator_quality_v1": 1
        }
      },
      {
        "user": "Which Python version does the project require?",
        "scores": {
          "hallucinations_v1": 1,
          "per_turn_user_simulator_quality_v1": 1
        }
      }
    ]
  },
  {
    "eval_id": "1ae10b45",
    "status": 1,
    "turns": [
      {
        "user": "The docs assistant isn't working for me.",
        "scores": {
          "hallucinations_v1": 1,
          "per_turn_user_simulator_quality_v1": 1
        }
      },
      {
        "user": "It's saying that it can't reach Ollama.",
        "scores": {
          "hallucinations_v1": 1,
          "per_turn_user_simulator_quality_v1": 1
        }
      }
    ]
  }
]

safety_v1 and adk eval_set generate_eval_cases, which writes scenarios for you, also use the Agent Platform evaluation service, so this lesson did not run them.

How the simulator chooses the next user turn

For each scenario, the eval service creates a fresh session and sends starting_prompt. After each agent reply, the LlmBackedUserSimulator builds a prompt from the plan, the persona’s behavior instructions and the conversation so far (without tool calls, unless include_function_calls is set), and asks its model for the next user message. When the model decides the plan is done, it writes the stop signal </finished> and the conversation ends. If it reaches max_allowed_invocations first, the conversation is cut off; the starting prompt counts as the first invocation.

Only after the conversation ends do the metrics run. per_turn_user_simulator_quality_v1 scores the first turn by comparing it with starting_prompt, scores each later turn with a judge, and then checks the end: if the conversation stopped without the stop signal, it marks the last turn as failed. That is why a cap can fail a run whose replies were all correct. Both the agent and the simulator are models, so the same scenario can take a different path on the next run; repeat the set before you treat a single failure as a regression.

Test the recorded results offline

test_scenarios.py loads the eval set and the saved EvalSetResult files with ADK’s own classes, and runs the reference-based evaluators on the recorded turns to show the error they raise.

python -m unittest -v test_scenarios.py

test_a_one_turn_cap_ends_the_conversation_before_the_plan_is_done (test_scenarios.Scenarios.test_a_one_turn_cap_ends_the_conversation_before_the_plan_is_done) ... ok
test_add_eval_case_expands_the_persona_name_into_behaviors (test_scenarios.Scenarios.test_add_eval_case_expands_the_persona_name_into_behaviors) ... ok
test_both_scenarios_pass_hallucination_and_simulator_quality (test_scenarios.Scenarios.test_both_scenarios_pass_hallucination_and_simulator_quality) ... ok
test_reference_metrics_cannot_score_a_scenario_case (test_scenarios.Scenarios.test_reference_metrics_cannot_score_a_scenario_case) ... ok
test_the_task_success_metric_was_not_evaluated_without_agent_platform (test_scenarios.Scenarios.test_the_task_success_metric_was_not_evaluated_without_agent_platform) ... ok
test_the_vague_question_gets_a_clarifying_question_before_any_tool_call (test_scenarios.Scenarios.test_the_vague_question_gets_a_clarifying_question_before_any_tool_call) ... ok

----------------------------------------------------------------------
Ran 6 tests in 0.000s

OK

All 6 tests pass.

"""Check the scenario eval set and the recorded `adk eval` results offline, with ADK's own classes.

Run: python -m unittest -v test_scenarios.py
"""
import json
import unittest
import warnings
from pathlib import Path

warnings.filterwarnings("ignore")

from google.adk.evaluation.eval_case import get_all_tool_calls  # noqa: E402
from google.adk.evaluation.eval_metrics import EvalMetric  # noqa: E402
from google.adk.evaluation.eval_result import EvalSetResult  # noqa: E402
from google.adk.evaluation.eval_set import EvalSet  # noqa: E402
from google.adk.evaluation.final_response_match_v1 import RougeEvaluator  # noqa: E402
from google.adk.evaluation.trajectory_evaluator import TrajectoryEvaluator  # noqa: E402

HERE = Path(__file__).resolve().parent
EVAL_SET = EvalSet.model_validate_json((HERE / "docs_assistant/scenarios.evalset.json").read_text())
RESULTS = {p.stem: EvalSetResult.model_validate_json(p.read_text()) for p in (HERE / "results").glob("*.json")}


def case(run: str, starting_prompt: str):
    ids = {c.eval_id for c in EVAL_SET.eval_cases if c.conversation_scenario.starting_prompt == starting_prompt}
    return next(r for r in RESULTS[run].eval_case_results if r.eval_id in ids)


def scores(result) -> dict:
    return {m.metric_name: m.score for m in result.overall_eval_metric_results}


def turns(result) -> list:
    return [i.actual_invocation for i in result.eval_metric_result_per_invocation]


VAGUE = "The docs assistant isn't working for me."
LOCAL = "How do I make the assistant use a local model instead of Gemini?"


class Scenarios(unittest.TestCase):
    def test_add_eval_case_expands_the_persona_name_into_behaviors(self):
        personas = {c.conversation_scenario.user_persona.id: c.conversation_scenario.user_persona
                    for c in EVAL_SET.eval_cases}
        self.assertEqual(set(personas), {"NOVICE", "EXPERT"})
        self.assertTrue(all(p.behaviors for p in personas.values()))
        self.assertTrue(all(not c.conversation for c in EVAL_SET.eval_cases), "scenario cases have no expected turns")

    def test_the_vague_question_gets_a_clarifying_question_before_any_tool_call(self):
        first, second = turns(case("simulated-users", VAGUE))[:2]
        self.assertEqual(get_all_tool_calls(first.intermediate_data), [])
        self.assertTrue(first.final_response.parts[0].text.rstrip().endswith("?"))
        self.assertIn("Ollama", second.user_content.parts[0].text)  # written by the simulated user
        self.assertIn("troubleshooting.md", second.final_response.parts[0].text)

    def test_both_scenarios_pass_hallucination_and_simulator_quality(self):
        for prompt in (VAGUE, LOCAL):
            got = scores(case("simulated-users", prompt))
            self.assertGreaterEqual(got["hallucinations_v1"], 0.8)
            self.assertGreaterEqual(got["per_turn_user_simulator_quality_v1"], 0.8)

    def test_reference_metrics_cannot_score_a_scenario_case(self):
        actual = turns(case("simulated-users", LOCAL))
        trajectory = TrajectoryEvaluator(eval_metric=EvalMetric(
            metric_name="tool_trajectory_avg_score", criterion={"threshold": 1.0}))
        with self.assertRaisesRegex(ValueError, "expected_invocations"):
            trajectory.evaluate_invocations(actual, None)
        with self.assertRaisesRegex(ValueError, "expected_invocations"):
            RougeEvaluator(EvalMetric(metric_name="response_match_score", threshold=0.8)).evaluate_invocations(actual, None)
        self.assertIsNone(scores(case("default-criteria", LOCAL))["tool_trajectory_avg_score"])

    def test_a_one_turn_cap_ends_the_conversation_before_the_plan_is_done(self):
        for prompt in (VAGUE, LOCAL):
            result = case("one-turn-limit", prompt)
            self.assertEqual(len(turns(result)), 1)
            self.assertEqual(scores(result)["per_turn_user_simulator_quality_v1"], 0.0)
            self.assertEqual(scores(result)["hallucinations_v1"], 1.0)

    def test_the_task_success_metric_was_not_evaluated_without_agent_platform(self):
        for prompt in (VAGUE, LOCAL):
            self.assertIsNone(scores(case("task-success-metric", prompt))["multi_turn_task_success_v1"])
        self.assertIn("403 PERMISSION_DENIED", (HERE / "run-output.txt").read_text())


if __name__ == "__main__":
    unittest.main()

"""Run the scenario eval set through `adk eval` under four configurations and keep the results.

Run from this folder with GOOGLE_API_KEY set, in a venv with google-adk[eval]: python record.py
Writes run-output.txt and results/<run>.json (the EvalSetResult, compacted).
ADK_DISABLE_LOAD_DOTENV=1 stops `adk` from loading a .env found in a parent folder.
"""
import json
import os
import re
import shutil
import subprocess
import sys
from pathlib import Path

HERE = Path(__file__).resolve().parent
ADK = str(Path(sys.executable).with_name("adk"))
RUNS = {
    "simulated-users": ["--config_file_path", "eval_config.json"],
    "default-criteria": [],
    "one-turn-limit": ["--config_file_path", "config-one-turn.json"],
    "task-success-metric": ["--config_file_path", "config-task-success.json"],
}


def compact(value):
    """Drop nulls and opaque thought signatures so the saved result stays readable."""
    if isinstance(value, dict):
        return {k: compact(v) for k, v in value.items() if v is not None and k != "thought_signature"}
    if isinstance(value, list):
        return [compact(v) for v in value]
    return value


def clean(line: str) -> str:
    """Drop progress-bar prefixes and cut long API error bodies."""
    line = re.sub(r"^.*?(\d{4}-\d\d-\d\d \d\d:)", r"\1", line)
    return line if len(line) <= 220 else line[:220] + " ..."


def main() -> None:
    env = {**os.environ, "ADK_DISABLE_LOAD_DOTENV": "1", "PYTHONWARNINGS": "ignore"}
    history = HERE / "docs_assistant" / ".adk" / "eval_history"
    (HERE / "results").mkdir(exist_ok=True)
    output = []
    for name, flags in RUNS.items():
        shutil.rmtree(history, ignore_errors=True)
        command = ["adk", "eval", "docs_assistant", "scenarios", *flags, "--print_detailed_results"]
        result = subprocess.run([ADK, *command[1:]], cwd=HERE, env=env, capture_output=True, text=True, timeout=900)
        errors = [clean(line) for line in result.stderr.splitlines() if re.search(r"(Error|Exception):|ERROR -", line)]
        text = result.stdout.strip() + ("\n" + "\n".join(dict.fromkeys(errors)) if errors else "")
        output.append(f"# Run {name}\n$ {' '.join(command)}\n{text}\n")
        saved = sorted(history.glob("*.json"))
        if saved:
            (HERE / "results" / f"{name}.json").write_text(json.dumps(compact(json.loads(saved[-1].read_text())), indent=1))
    shutil.rmtree(HERE / "docs_assistant" / ".adk", ignore_errors=True)
    (HERE / "run-output.txt").write_text("\n".join(output))


if __name__ == "__main__":
    main()

You add a third scenario whose plan has four follow-up questions, and keep max_allowed_invocations at 6. The agent asks one clarifying question at the start.

• Inject tool failures with environment simulation: keep the user fixed and make the tools fail instead.
• Write a custom eval metric and optimize the instruction: score what the built-in metrics can’t, then improve the instruction against it.

References

Official sources checked October 21, 2026, against google-adk 2.12.0.

• ADK: user simulation
• ADK: evaluation criteria
• ADK: evaluate agents