# Run simulated-users $ adk eval docs_assistant scenarios --config_file_path eval_config.json --print_detailed_results Using evaluation criteria: criteria={'hallucinations_v1': BaseCriterion(threshold=0.8, include_intermediate_responses_in_final=False, evaluate_intermediate_nl_responses=True, judge_model_options={'judge_model': 'gemini-3.8-flash', 'num_samples': 1}), 'per_turn_user_simulator_quality_v1': BaseCriterion(threshold=0.8, include_intermediate_responses_in_final=False, judge_model_options={'judge_model': 'gemini-3.8-flash', 'num_samples': 1})} custom_metrics=None user_simulator_config=LlmBackedUserSimulatorConfig(type='llm_backed', model='gemini-3.8-flash', model_configuration=GenerateContentConfig( thinking_config=ThinkingConfig( include_thoughts=True, thinking_budget=10240 ) ), max_allowed_invocations=6, custom_instructions=None, include_function_calls=False) live_model_config=None ********************************************************************* Eval Run Summary scenarios: Tests passed: 2 Tests failed: 0 ******************************************************************** Eval Set Id: scenarios Eval Id: 6be7706e Overall Eval Status: PASSED --------------------------------------------------------------------- Metric: hallucinations_v1, Status: PASSED, Score: 1.0, Threshold: 0.8 --------------------------------------------------------------------- Metric: per_turn_user_simulator_quality_v1, Status: PASSED, Score: 1.0, Threshold: 0.8 --------------------------------------------------------------------- Metric: tool_call_count_v1, Status: INFORMATIONAL, Score: 3.0, Threshold: None --------------------------------------------------------------------- Metric: inference_call_count_v1, Status: INFORMATIONAL, Score: 4.0, Threshold: None --------------------------------------------------------------------- Metric: token_usage_v1, Status: INFORMATIONAL, Score: 5073.5, Threshold: None Token breakdown: total: 5073.5 input: 4319 prompt: 4319 cached: n/a tool use: n/a output: 754.5 candidates: 67.5 reasoning: 687 --------------------------------------------------------------------- Metric: invocation_duration_v1, Status: INFORMATIONAL, Score: 11.531500000000001, Threshold: None --------------------------------------------------------------------- Invocation Details: Invocation 1 of 2 prompt: How do I make the assistant use a local model instead of Gemini? actual_response: Set `OLLAMA_API_BASE` to use a local model instead of Gemini (install.md). actual_tool_calls: list_pages() read_page( path='cli.md') read_page( path='install.md') read_page( path='troubleshooting.md') hallucinations_v1: PASSED (1.0) per_turn_user_simulator_quality_v1: PASSED (1.0) tool_call_count_v1: 4 inference_call_count_v1: 5 token_usage_v1: 4205 (in 2753, out 1452) invocation_duration_v1: 18.40s Invocation 2 of 2 prompt: Which Python version does the project require? actual_response: The docs-assistant project requires Python 3.12 or later (install.md). actual_tool_calls: list_pages() read_page( path='install.md') hallucinations_v1: PASSED (1.0) per_turn_user_simulator_quality_v1: PASSED (1.0) tool_call_count_v1: 2 inference_call_count_v1: 3 token_usage_v1: 5942 (in 5885, out 57) invocation_duration_v1: 4.67s ******************************************************************** Eval Set Id: scenarios Eval Id: 1ae10b45 Overall Eval Status: PASSED --------------------------------------------------------------------- Metric: hallucinations_v1, Status: PASSED, Score: 1.0, Threshold: 0.8 --------------------------------------------------------------------- Metric: per_turn_user_simulator_quality_v1, Status: PASSED, Score: 1.0, Threshold: 0.8 --------------------------------------------------------------------- Metric: tool_call_count_v1, Status: INFORMATIONAL, Score: 1.0, Threshold: None --------------------------------------------------------------------- Metric: inference_call_count_v1, Status: INFORMATIONAL, Score: 2.0, Threshold: None --------------------------------------------------------------------- Metric: token_usage_v1, Status: INFORMATIONAL, Score: 883.5, Threshold: None Token breakdown: total: 883.5 input: 723.5 prompt: 723.5 cached: n/a tool use: n/a output: 160 candidates: 34.5 reasoning: 125.5 --------------------------------------------------------------------- Metric: invocation_duration_v1, Status: INFORMATIONAL, Score: 4.3185, Threshold: None --------------------------------------------------------------------- Invocation Details: Invocation 1 of 2 prompt: The docs assistant isn't working for me. actual_response: What specific error or symptom are you encountering when trying to use the docs assistant? actual_tool_calls: (none) hallucinations_v1: PASSED (1.0) per_turn_user_simulator_quality_v1: PASSED (1.0) tool_call_count_v1: 0 inference_call_count_v1: 1 token_usage_v1: 290 (in 177, out 113) invocation_duration_v1: 2.00s Invocation 2 of 2 prompt: It's saying that it can't reach Ollama. actual_response: Check that `OLLAMA_API_BASE` points to a running Ollama server (troubleshooting.md). actual_tool_calls: list_pages() read_page( path='troubleshooting.md') hallucinations_v1: PASSED (1.0) per_turn_user_simulator_quality_v1: PASSED (1.0) tool_call_count_v1: 2 inference_call_count_v1: 3 token_usage_v1: 1477 (in 1270, out 207) invocation_duration_v1: 6.63s # Run default-criteria $ adk eval docs_assistant scenarios --print_detailed_results Using evaluation criteria: criteria={'tool_trajectory_avg_score': 1.0, 'response_match_score': 0.8} custom_metrics=None user_simulator_config=None live_model_config=None ********************************************************************* Eval Run Summary scenarios: Tests passed: 0 Tests failed: 2 ******************************************************************** Eval Set Id: scenarios Eval Id: 6be7706e Overall Eval Status: NOT_EVALUATED --------------------------------------------------------------------- Metric: tool_trajectory_avg_score, Status: NOT_EVALUATED, Score: None, Threshold: 1.0 --------------------------------------------------------------------- Metric: response_match_score, Status: NOT_EVALUATED, Score: None, Threshold: 0.8 --------------------------------------------------------------------- Metric: tool_call_count_v1, Status: INFORMATIONAL, Score: 2.5, Threshold: None --------------------------------------------------------------------- Metric: inference_call_count_v1, Status: INFORMATIONAL, Score: 3.5, Threshold: None --------------------------------------------------------------------- Metric: token_usage_v1, Status: INFORMATIONAL, Score: 1366.0, Threshold: None Token breakdown: total: 1366 input: 1307.5 prompt: 1307.5 cached: n/a tool use: n/a output: 58.5 candidates: 58.5 reasoning: n/a --------------------------------------------------------------------- Metric: invocation_duration_v1, Status: INFORMATIONAL, Score: 5.263999999999999, Threshold: None --------------------------------------------------------------------- Invocation Details: Invocation 1 of 2 prompt: How do I make the assistant use a local model instead of Gemini? actual_response: To make the assistant use a local model instead of Gemini, set `OLLAMA_API_BASE` (install.md). actual_tool_calls: list_pages() read_page( path='cli.md') read_page( path='install.md') tool_trajectory_avg_score: NOT_EVALUATED (None) response_match_score: NOT_EVALUATED (None) tool_call_count_v1: 3 inference_call_count_v1: 4 token_usage_v1: 1195 (in 1122, out 73) invocation_duration_v1: 5.95s Invocation 2 of 2 prompt: What Python version does the project require? actual_response: The project requires Python 3.12 or later (install.md). actual_tool_calls: list_pages() read_page( path='install.md') tool_trajectory_avg_score: NOT_EVALUATED (None) response_match_score: NOT_EVALUATED (None) tool_call_count_v1: 2 inference_call_count_v1: 3 token_usage_v1: 1537 (in 1493, out 44) invocation_duration_v1: 4.58s ******************************************************************** Eval Set Id: scenarios Eval Id: 1ae10b45 Overall Eval Status: NOT_EVALUATED --------------------------------------------------------------------- Metric: tool_trajectory_avg_score, Status: NOT_EVALUATED, Score: None, Threshold: 1.0 --------------------------------------------------------------------- Metric: response_match_score, Status: NOT_EVALUATED, Score: None, Threshold: 0.8 --------------------------------------------------------------------- Metric: tool_call_count_v1, Status: INFORMATIONAL, Score: 1.0, Threshold: None --------------------------------------------------------------------- Metric: inference_call_count_v1, Status: INFORMATIONAL, Score: 2.0, Threshold: None --------------------------------------------------------------------- Metric: token_usage_v1, Status: INFORMATIONAL, Score: 1065.5, Threshold: None Token breakdown: total: 1065.5 input: 861.5 prompt: 861.5 cached: n/a tool use: n/a output: 204 candidates: 34.5 reasoning: 169.5 --------------------------------------------------------------------- Metric: invocation_duration_v1, Status: INFORMATIONAL, Score: 4.036, Threshold: None --------------------------------------------------------------------- Invocation Details: Invocation 1 of 2 prompt: The docs assistant isn't working for me. actual_response: What specific error or symptom are you encountering when trying to use the docs assistant? actual_tool_calls: (none) tool_trajectory_avg_score: NOT_EVALUATED (None) response_match_score: NOT_EVALUATED (None) tool_call_count_v1: 0 inference_call_count_v1: 1 token_usage_v1: 395 (in 177, out 218) invocation_duration_v1: 2.78s Invocation 2 of 2 prompt: It reports that it cannot reach Ollama. actual_response: Check that `OLLAMA_API_BASE` points to a running Ollama server (troubleshooting.md). actual_tool_calls: list_pages() read_page( path='troubleshooting.md') tool_trajectory_avg_score: NOT_EVALUATED (None) response_match_score: NOT_EVALUATED (None) tool_call_count_v1: 2 inference_call_count_v1: 3 token_usage_v1: 1736 (in 1546, out 190) invocation_duration_v1: 5.30s 2026-10-21 20:22:36,890 - ERROR - local_eval_service.py:436 - Metric evaluation failed for metric `tool_trajectory_avg_score` for eval case id '6be7706e' with following error `expected_invocations is needed by this metri ... ValueError: expected_invocations is needed by this metric. 2026-10-21 20:22:36,890 - ERROR - local_eval_service.py:436 - Metric evaluation failed for metric `response_match_score` for eval case id '6be7706e' with following error `expected_invocations is required for this metric. ... ValueError: expected_invocations is required for this metric. 2026-10-21 20:22:36,891 - ERROR - local_eval_service.py:436 - Metric evaluation failed for metric `tool_trajectory_avg_score` for eval case id '1ae10b45' with following error `expected_invocations is needed by this metri ... 2026-10-21 20:22:36,891 - ERROR - local_eval_service.py:436 - Metric evaluation failed for metric `response_match_score` for eval case id '1ae10b45' with following error `expected_invocations is required for this metric. ... # Run one-turn-limit $ adk eval docs_assistant scenarios --config_file_path config-one-turn.json --print_detailed_results Using evaluation criteria: criteria={'hallucinations_v1': BaseCriterion(threshold=0.8, include_intermediate_responses_in_final=False, evaluate_intermediate_nl_responses=True, judge_model_options={'judge_model': 'gemini-3.8-flash', 'num_samples': 1}), 'per_turn_user_simulator_quality_v1': BaseCriterion(threshold=0.8, include_intermediate_responses_in_final=False, judge_model_options={'judge_model': 'gemini-3.8-flash', 'num_samples': 1})} custom_metrics=None user_simulator_config=LlmBackedUserSimulatorConfig(type='llm_backed', model='gemini-3.8-flash', model_configuration=GenerateContentConfig( thinking_config=ThinkingConfig( include_thoughts=True, thinking_budget=10240 ) ), max_allowed_invocations=1, custom_instructions=None, include_function_calls=False) live_model_config=None ********************************************************************* Eval Run Summary scenarios: Tests passed: 0 Tests failed: 2 ******************************************************************** Eval Set Id: scenarios Eval Id: 6be7706e Overall Eval Status: FAILED --------------------------------------------------------------------- Metric: hallucinations_v1, Status: PASSED, Score: 1.0, Threshold: 0.8 --------------------------------------------------------------------- Metric: per_turn_user_simulator_quality_v1, Status: FAILED, Score: 0.0, Threshold: 0.8 --------------------------------------------------------------------- Metric: tool_call_count_v1, Status: INFORMATIONAL, Score: 3.0, Threshold: None --------------------------------------------------------------------- Metric: inference_call_count_v1, Status: INFORMATIONAL, Score: 4.0, Threshold: None --------------------------------------------------------------------- Metric: token_usage_v1, Status: INFORMATIONAL, Score: 1191.0, Threshold: None Token breakdown: total: 1191 input: 1122 prompt: 1122 cached: n/a tool use: n/a output: 69 candidates: 69 reasoning: n/a --------------------------------------------------------------------- Metric: invocation_duration_v1, Status: INFORMATIONAL, Score: 6.235, Threshold: None --------------------------------------------------------------------- Invocation Details: Invocation 1 of 1 prompt: How do I make the assistant use a local model instead of Gemini? actual_response: Set `OLLAMA_API_BASE` to use a local model instead of Gemini (install.md). actual_tool_calls: list_pages() read_page( path='cli.md') read_page( path='install.md') hallucinations_v1: PASSED (1.0) per_turn_user_simulator_quality_v1: FAILED (0.0) tool_call_count_v1: 3 inference_call_count_v1: 4 token_usage_v1: 1191 (in 1122, out 69) invocation_duration_v1: 6.24s ******************************************************************** Eval Set Id: scenarios Eval Id: 1ae10b45 Overall Eval Status: FAILED --------------------------------------------------------------------- Metric: hallucinations_v1, Status: PASSED, Score: 1.0, Threshold: 0.8 --------------------------------------------------------------------- Metric: per_turn_user_simulator_quality_v1, Status: FAILED, Score: 0.0, Threshold: 0.8 --------------------------------------------------------------------- Metric: tool_call_count_v1, Status: INFORMATIONAL, Score: 0.0, Threshold: None --------------------------------------------------------------------- Metric: inference_call_count_v1, Status: INFORMATIONAL, Score: 1.0, Threshold: None --------------------------------------------------------------------- Metric: token_usage_v1, Status: INFORMATIONAL, Score: 244.0, Threshold: None Token breakdown: total: 244 input: 177 prompt: 177 cached: n/a tool use: n/a output: 67 candidates: 9 reasoning: 58 --------------------------------------------------------------------- Metric: invocation_duration_v1, Status: INFORMATIONAL, Score: 2.1, Threshold: None --------------------------------------------------------------------- Invocation Details: Invocation 1 of 1 prompt: The docs assistant isn't working for me. actual_response: What specific error or symptom are you experiencing? actual_tool_calls: (none) hallucinations_v1: PASSED (1.0) per_turn_user_simulator_quality_v1: FAILED (0.0) tool_call_count_v1: 0 inference_call_count_v1: 1 token_usage_v1: 244 (in 177, out 67) invocation_duration_v1: 2.10s # Run task-success-metric $ adk eval docs_assistant scenarios --config_file_path config-task-success.json --print_detailed_results Using evaluation criteria: criteria={'multi_turn_task_success_v1': BaseCriterion(threshold=0.8, include_intermediate_responses_in_final=False)} custom_metrics=None user_simulator_config=LlmBackedUserSimulatorConfig(type='llm_backed', model='gemini-3.8-flash', model_configuration=GenerateContentConfig( thinking_config=ThinkingConfig( include_thoughts=True, thinking_budget=10240 ) ), max_allowed_invocations=6, custom_instructions=None, include_function_calls=False) live_model_config=None ********************************************************************* Eval Run Summary scenarios: Tests passed: 0 Tests failed: 2 ******************************************************************** Eval Set Id: scenarios Eval Id: 1ae10b45 Overall Eval Status: NOT_EVALUATED --------------------------------------------------------------------- Metric: multi_turn_task_success_v1, Status: NOT_EVALUATED, Score: None, Threshold: 0.8 --------------------------------------------------------------------- Metric: tool_call_count_v1, Status: INFORMATIONAL, Score: 1.0, Threshold: None --------------------------------------------------------------------- Metric: inference_call_count_v1, Status: INFORMATIONAL, Score: 2.0, Threshold: None --------------------------------------------------------------------- Metric: token_usage_v1, Status: INFORMATIONAL, Score: 844.5, Threshold: None Token breakdown: total: 844.5 input: 676 prompt: 676 cached: n/a tool use: n/a output: 168.5 candidates: 31.5 reasoning: 137 --------------------------------------------------------------------- Metric: invocation_duration_v1, Status: INFORMATIONAL, Score: 4.402, Threshold: None --------------------------------------------------------------------- Invocation Details: Invocation 1 of 2 prompt: The docs assistant isn't working for me. actual_response: What specific error message or symptom are you encountering? actual_tool_calls: (none) multi_turn_task_success_v1: NOT_EVALUATED (None) tool_call_count_v1: 0 inference_call_count_v1: 1 token_usage_v1: 250 (in 177, out 73) invocation_duration_v1: 2.31s Invocation 2 of 2 prompt: It's telling me that it can't reach Ollama. actual_response: Check that `OLLAMA_API_BASE` points to a running Ollama server (troubleshooting.md). actual_tool_calls: list_pages() read_page( path='troubleshooting.md') multi_turn_task_success_v1: NOT_EVALUATED (None) tool_call_count_v1: 2 inference_call_count_v1: 3 token_usage_v1: 1439 (in 1175, out 264) invocation_duration_v1: 6.50s ******************************************************************** Eval Set Id: scenarios Eval Id: 6be7706e Overall Eval Status: NOT_EVALUATED --------------------------------------------------------------------- Metric: multi_turn_task_success_v1, Status: NOT_EVALUATED, Score: None, Threshold: 0.8 --------------------------------------------------------------------- Metric: tool_call_count_v1, Status: INFORMATIONAL, Score: 3.0, Threshold: None --------------------------------------------------------------------- Metric: inference_call_count_v1, Status: INFORMATIONAL, Score: 4.0, Threshold: None --------------------------------------------------------------------- Metric: token_usage_v1, Status: INFORMATIONAL, Score: 5863.5, Threshold: None Token breakdown: total: 5863.5 input: 4997 prompt: 4997 cached: n/a tool use: n/a output: 866.5 candidates: 66 reasoning: 800.5 --------------------------------------------------------------------- Metric: invocation_duration_v1, Status: INFORMATIONAL, Score: 11.546500000000002, Threshold: None --------------------------------------------------------------------- Invocation Details: Invocation 1 of 2 prompt: How do I make the assistant use a local model instead of Gemini? actual_response: Set `OLLAMA_API_BASE` to use a local model instead of Gemini (install.md). actual_tool_calls: list_pages() read_page( path='cli.md') read_page( path='install.md') read_page( path='troubleshooting.md') multi_turn_task_success_v1: NOT_EVALUATED (None) tool_call_count_v1: 4 inference_call_count_v1: 5 token_usage_v1: 5193 (in 3681, out 1512) invocation_duration_v1: 16.95s Invocation 2 of 2 prompt: Which Python version does the project require? actual_response: The project requires Python 3.12 or later (install.md). actual_tool_calls: list_pages() read_page( path='install.md') multi_turn_task_success_v1: NOT_EVALUATED (None) tool_call_count_v1: 2 inference_call_count_v1: 3 token_usage_v1: 6534 (in 6313, out 221) invocation_duration_v1: 6.14s 2026-10-21 20:24:00,505 - ERROR - _evals_metric_handlers.py:1108 - Error processing metric multi_turn_task_success_v1 for case None: 403 PERMISSION_DENIED. {'error': {'code': 403, 'message': 'Requests to this API aiplatf ... google.genai.errors.ClientError: 403 PERMISSION_DENIED. {'error': {'code': 403, 'message': 'Requests to this API aiplatform.googleapis.com method google.cloud.aiplatform.v1beta1.EvaluationService.EvaluateInstances are bl ... 2026-10-21 20:24:01,318 - ERROR - _evals_metric_handlers.py:1108 - Error processing metric multi_turn_task_success_v1 for case None: 403 PERMISSION_DENIED. {'error': {'code': 403, 'message': 'Requests to this API aiplatf ...