# Run partial-metric-info $ adk eval docs_assistant train --config_file_path config-partial-metric-info.json pydantic_core._pydantic_core.ValidationError: 2 validation errors for EvalConfig # Run metric-not-on-path $ adk eval docs_assistant train --config_file_path eval_config.json Using evaluation criteria: criteria={'cited_page_was_read': 1.0} custom_metrics={'cited_page_was_read': CustomMetricConfig(code_config=CodeConfig(name='citation_metric.cited_page_was_read'), metric_info=MetricInfo(metric_name='cited_page_was_read', description='The answer cites the expected page, and the agent read that page.', metric_value_info=MetricValueInfo(interval=Interval(min_value=0.0, open_at_min=False, max_value=1.0, open_at_max=False)), requires_threshold=True), description='')} user_simulator_config=None live_model_config=None ********************************************************************* Eval Run Summary train: Tests passed: 0 Tests failed: 4 2026-10-21 21:04:00,126 - ERROR - local_eval_service.py:436 - Metric evaluation failed for metric `cited_page_was_read` for eval case id 'wrong-citation' with following error `Could not import custom metric function from ... ModuleNotFoundError: No module named 'citation_metric' ImportError: Could not import custom metric function from citation_metric.cited_page_was_read 2026-10-21 21:04:00,135 - ERROR - local_eval_service.py:436 - Metric evaluation failed for metric `cited_page_was_read` for eval case id 'python-version' with following error `Could not import custom metric function from ... 2026-10-21 21:04:00,138 - ERROR - local_eval_service.py:436 - Metric evaluation failed for metric `cited_page_was_read` for eval case id 'rebuild-index' with following error `Could not import custom metric function from ... 2026-10-21 21:04:00,141 - ERROR - local_eval_service.py:436 - Metric evaluation failed for metric `cited_page_was_read` for eval case id 'ask-output' with following error `Could not import custom metric function from cit ... # Run baseline-train $ PYTHONPATH=. adk eval docs_assistant train --config_file_path eval_config.json --print_detailed_results Using evaluation criteria: criteria={'cited_page_was_read': 1.0} custom_metrics={'cited_page_was_read': CustomMetricConfig(code_config=CodeConfig(name='citation_metric.cited_page_was_read'), metric_info=MetricInfo(metric_name='cited_page_was_read', description='The answer cites the expected page, and the agent read that page.', metric_value_info=MetricValueInfo(interval=Interval(min_value=0.0, open_at_min=False, max_value=1.0, open_at_max=False)), requires_threshold=True), description='')} user_simulator_config=None live_model_config=None ********************************************************************* Eval Run Summary train: Tests passed: 0 Tests failed: 4 ******************************************************************** Eval Set Id: train Eval Id: wrong-citation Overall Eval Status: FAILED --------------------------------------------------------------------- Metric: cited_page_was_read, Status: FAILED, Score: 0.0, Threshold: 1.0 --------------------------------------------------------------------- Metric: tool_call_count_v1, Status: INFORMATIONAL, Score: 2.0, Threshold: None --------------------------------------------------------------------- Metric: inference_call_count_v1, Status: INFORMATIONAL, Score: 3.0, Threshold: None --------------------------------------------------------------------- Metric: token_usage_v1, Status: INFORMATIONAL, Score: 644.0, Threshold: None Token breakdown: total: 644 input: 587 prompt: 587 cached: n/a tool use: n/a output: 57 candidates: 57 reasoning: n/a --------------------------------------------------------------------- Metric: invocation_duration_v1, Status: INFORMATIONAL, Score: 4.692, Threshold: None --------------------------------------------------------------------- Invocation Details: Invocation 1 of 1 prompt: The assistant cites the wrong page. What should I do? expected_response: Rebuild the search index with `docs-assistant index --rebuild` (troubleshooting.md). actual_response: If the assistant cites the wrong page, rebuild the search index by running: ```bash docs-assistant index --rebuild ``` expected_tool_calls: (none) actual_tool_calls: list_pages() read_page( path='troubleshooting.md') cited_page_was_read: FAILED (0.0) tool_call_count_v1: 2 inference_call_count_v1: 3 token_usage_v1: 644 (in 587, out 57) invocation_duration_v1: 4.69s ******************************************************************** Eval Set Id: train Eval Id: ask-output Overall Eval Status: FAILED --------------------------------------------------------------------- Metric: cited_page_was_read, Status: FAILED, Score: 0.0, Threshold: 1.0 --------------------------------------------------------------------- Metric: tool_call_count_v1, Status: INFORMATIONAL, Score: 2.0, Threshold: None --------------------------------------------------------------------- Metric: inference_call_count_v1, Status: INFORMATIONAL, Score: 3.0, Threshold: None --------------------------------------------------------------------- Metric: token_usage_v1, Status: INFORMATIONAL, Score: 611.0, Threshold: None Token breakdown: total: 611 input: 564 prompt: 564 cached: n/a tool use: n/a output: 47 candidates: 47 reasoning: n/a --------------------------------------------------------------------- Metric: invocation_duration_v1, Status: INFORMATIONAL, Score: 4.123, Threshold: None --------------------------------------------------------------------- Invocation Details: Invocation 1 of 1 prompt: What does the ask command print? expected_response: `docs-assistant ask QUESTION` prints an answer with citation links (cli.md). actual_response: The `ask` command (`docs-assistant ask QUESTION`) prints an answer with citation links. expected_tool_calls: (none) actual_tool_calls: list_pages() read_page( path='cli.md') cited_page_was_read: FAILED (0.0) tool_call_count_v1: 2 inference_call_count_v1: 3 token_usage_v1: 611 (in 564, out 47) invocation_duration_v1: 4.12s ******************************************************************** Eval Set Id: train Eval Id: rebuild-index Overall Eval Status: FAILED --------------------------------------------------------------------- Metric: cited_page_was_read, Status: FAILED, Score: 0.0, Threshold: 1.0 --------------------------------------------------------------------- Metric: tool_call_count_v1, Status: INFORMATIONAL, Score: 2.0, Threshold: None --------------------------------------------------------------------- Metric: inference_call_count_v1, Status: INFORMATIONAL, Score: 3.0, Threshold: None --------------------------------------------------------------------- Metric: token_usage_v1, Status: INFORMATIONAL, Score: 632.0, Threshold: None Token breakdown: total: 632 input: 567 prompt: 567 cached: n/a tool use: n/a output: 65 candidates: 65 reasoning: n/a --------------------------------------------------------------------- Metric: invocation_duration_v1, Status: INFORMATIONAL, Score: 5.399, Threshold: None --------------------------------------------------------------------- Invocation Details: Invocation 1 of 1 prompt: How do I rebuild the search index? expected_response: Run `docs-assistant index --rebuild` (cli.md). actual_response: To rebuild the search index, run: ```bash docs-assistant index --rebuild ``` This will drop the existing search index and rebuild it from the docs folder. expected_tool_calls: (none) actual_tool_calls: list_pages() read_page( path='cli.md') cited_page_was_read: FAILED (0.0) tool_call_count_v1: 2 inference_call_count_v1: 3 token_usage_v1: 632 (in 567, out 65) invocation_duration_v1: 5.40s ******************************************************************** Eval Set Id: train Eval Id: python-version Overall Eval Status: FAILED --------------------------------------------------------------------- Metric: cited_page_was_read, Status: FAILED, Score: 0.0, Threshold: 1.0 --------------------------------------------------------------------- Metric: tool_call_count_v1, Status: INFORMATIONAL, Score: 2.0, Threshold: None --------------------------------------------------------------------- Metric: inference_call_count_v1, Status: INFORMATIONAL, Score: 3.0, Threshold: None --------------------------------------------------------------------- Metric: token_usage_v1, Status: INFORMATIONAL, Score: 597.0, Threshold: None Token breakdown: total: 597 input: 557 prompt: 557 cached: n/a tool use: n/a output: 40 candidates: 40 reasoning: n/a --------------------------------------------------------------------- Metric: invocation_duration_v1, Status: INFORMATIONAL, Score: 4.592, Threshold: None --------------------------------------------------------------------- Invocation Details: Invocation 1 of 1 prompt: Which Python version do I need? expected_response: Python 3.12 or later (install.md). actual_response: You need **Python 3.12 or later**. expected_tool_calls: (none) actual_tool_calls: list_pages() read_page( path='install.md') cited_page_was_read: FAILED (0.0) tool_call_count_v1: 2 inference_call_count_v1: 3 token_usage_v1: 597 (in 557, out 40) invocation_duration_v1: 4.59s # Run baseline-validation $ PYTHONPATH=. adk eval docs_assistant validation --config_file_path eval_config.json --print_detailed_results Using evaluation criteria: criteria={'cited_page_was_read': 1.0} custom_metrics={'cited_page_was_read': CustomMetricConfig(code_config=CodeConfig(name='citation_metric.cited_page_was_read'), metric_info=MetricInfo(metric_name='cited_page_was_read', description='The answer cites the expected page, and the agent read that page.', metric_value_info=MetricValueInfo(interval=Interval(min_value=0.0, open_at_min=False, max_value=1.0, open_at_max=False)), requires_threshold=True), description='')} user_simulator_config=None live_model_config=None ********************************************************************* Eval Run Summary validation: Tests passed: 0 Tests failed: 3 ******************************************************************** Eval Set Id: validation Eval Id: local-model Overall Eval Status: FAILED --------------------------------------------------------------------- Metric: cited_page_was_read, Status: FAILED, Score: 0.0, Threshold: 1.0 --------------------------------------------------------------------- Metric: tool_call_count_v1, Status: INFORMATIONAL, Score: 3.0, Threshold: None --------------------------------------------------------------------- Metric: inference_call_count_v1, Status: INFORMATIONAL, Score: 4.0, Threshold: None --------------------------------------------------------------------- Metric: token_usage_v1, Status: INFORMATIONAL, Score: 1022.0, Threshold: None Token breakdown: total: 1022 input: 934 prompt: 934 cached: n/a tool use: n/a output: 88 candidates: 88 reasoning: n/a --------------------------------------------------------------------- Metric: invocation_duration_v1, Status: INFORMATIONAL, Score: 6.014, Threshold: None --------------------------------------------------------------------- Invocation Details: Invocation 1 of 1 prompt: How do I point the assistant at a local model? expected_response: Set `OLLAMA_API_BASE` (install.md). actual_response: To point the assistant at a local model, set the `OLLAMA_API_BASE` environment variable to your local model endpoint (such as your local Ollama server address) instead of using Gemini. expected_tool_calls: (none) actual_tool_calls: list_pages() read_page( path='cli.md') read_page( path='install.md') cited_page_was_read: FAILED (0.0) tool_call_count_v1: 3 inference_call_count_v1: 4 token_usage_v1: 1022 (in 934, out 88) invocation_duration_v1: 6.01s ******************************************************************** Eval Set Id: validation Eval Id: ollama-unreachable Overall Eval Status: FAILED --------------------------------------------------------------------- Metric: cited_page_was_read, Status: FAILED, Score: 0.0, Threshold: 1.0 --------------------------------------------------------------------- Metric: tool_call_count_v1, Status: INFORMATIONAL, Score: 2.0, Threshold: None --------------------------------------------------------------------- Metric: inference_call_count_v1, Status: INFORMATIONAL, Score: 3.0, Threshold: None --------------------------------------------------------------------- Metric: token_usage_v1, Status: INFORMATIONAL, Score: 644.0, Threshold: None Token breakdown: total: 644 input: 593 prompt: 593 cached: n/a tool use: n/a output: 51 candidates: 51 reasoning: n/a --------------------------------------------------------------------- Metric: invocation_duration_v1, Status: INFORMATIONAL, Score: 5.725, Threshold: None --------------------------------------------------------------------- Invocation Details: Invocation 1 of 1 prompt: The assistant can't reach Ollama. What do I check? expected_response: Check that `OLLAMA_API_BASE` points to a running Ollama server (troubleshooting.md). actual_response: Check that the `OLLAMA_API_BASE` environment variable points to a running Ollama server. expected_tool_calls: (none) actual_tool_calls: list_pages() read_page( path='troubleshooting.md') cited_page_was_read: FAILED (0.0) tool_call_count_v1: 2 inference_call_count_v1: 3 token_usage_v1: 644 (in 593, out 51) invocation_duration_v1: 5.72s ******************************************************************** Eval Set Id: validation Eval Id: drop-index Overall Eval Status: FAILED --------------------------------------------------------------------- Metric: cited_page_was_read, Status: FAILED, Score: 0.0, Threshold: 1.0 --------------------------------------------------------------------- Metric: tool_call_count_v1, Status: INFORMATIONAL, Score: 3.0, Threshold: None --------------------------------------------------------------------- Metric: inference_call_count_v1, Status: INFORMATIONAL, Score: 4.0, Threshold: None --------------------------------------------------------------------- Metric: token_usage_v1, Status: INFORMATIONAL, Score: 1012.0, Threshold: None Token breakdown: total: 1012 input: 933 prompt: 933 cached: n/a tool use: n/a output: 79 candidates: 79 reasoning: n/a --------------------------------------------------------------------- Metric: invocation_duration_v1, Status: INFORMATIONAL, Score: 6.795, Threshold: None --------------------------------------------------------------------- Invocation Details: Invocation 1 of 1 prompt: What command drops the search index? expected_response: `docs-assistant index --rebuild` drops the search index and builds it again (cli.md). actual_response: The command that drops the search index (and rebuilds it from the docs folder) is: ```bash docs-assistant index --rebuild ``` expected_tool_calls: (none) actual_tool_calls: list_pages() read_page( path='cli.md') read_page( path='troubleshooting.md') cited_page_was_read: FAILED (0.0) tool_call_count_v1: 3 inference_call_count_v1: 4 token_usage_v1: 1012 (in 933, out 79) invocation_duration_v1: 6.79s # Run optimized-validation $ PYTHONPATH=. DOCS_INSTRUCTION_FILE=optimized-instruction.txt adk eval docs_assistant validation --config_file_path eval_config.json --print_detailed_results Using evaluation criteria: criteria={'cited_page_was_read': 1.0} custom_metrics={'cited_page_was_read': CustomMetricConfig(code_config=CodeConfig(name='citation_metric.cited_page_was_read'), metric_info=MetricInfo(metric_name='cited_page_was_read', description='The answer cites the expected page, and the agent read that page.', metric_value_info=MetricValueInfo(interval=Interval(min_value=0.0, open_at_min=False, max_value=1.0, open_at_max=False)), requires_threshold=True), description='')} user_simulator_config=None live_model_config=None ********************************************************************* Eval Run Summary validation: Tests passed: 3 Tests failed: 0 ******************************************************************** Eval Set Id: validation Eval Id: drop-index Overall Eval Status: PASSED --------------------------------------------------------------------- Metric: cited_page_was_read, Status: PASSED, Score: 1.0, Threshold: 1.0 --------------------------------------------------------------------- Metric: tool_call_count_v1, Status: INFORMATIONAL, Score: 2.0, Threshold: None --------------------------------------------------------------------- Metric: inference_call_count_v1, Status: INFORMATIONAL, Score: 3.0, Threshold: None --------------------------------------------------------------------- Metric: token_usage_v1, Status: INFORMATIONAL, Score: 1587.0, Threshold: None Token breakdown: total: 1587 input: 1533 prompt: 1533 cached: n/a tool use: n/a output: 54 candidates: 54 reasoning: n/a --------------------------------------------------------------------- Metric: invocation_duration_v1, Status: INFORMATIONAL, Score: 6.679, Threshold: None --------------------------------------------------------------------- Invocation Details: Invocation 1 of 1 prompt: What command drops the search index? expected_response: `docs-assistant index --rebuild` drops the search index and builds it again (cli.md). actual_response: The command that drops the search index and rebuilds it is `docs-assistant index --rebuild` (cli.md). expected_tool_calls: (none) actual_tool_calls: list_pages() read_page( path='cli.md') cited_page_was_read: PASSED (1.0) tool_call_count_v1: 2 inference_call_count_v1: 3 token_usage_v1: 1587 (in 1533, out 54) invocation_duration_v1: 6.68s ******************************************************************** Eval Set Id: validation Eval Id: ollama-unreachable Overall Eval Status: PASSED --------------------------------------------------------------------- Metric: cited_page_was_read, Status: PASSED, Score: 1.0, Threshold: 1.0 --------------------------------------------------------------------- Metric: tool_call_count_v1, Status: INFORMATIONAL, Score: 2.0, Threshold: None --------------------------------------------------------------------- Metric: inference_call_count_v1, Status: INFORMATIONAL, Score: 3.0, Threshold: None --------------------------------------------------------------------- Metric: token_usage_v1, Status: INFORMATIONAL, Score: 1967.0, Threshold: None Token breakdown: total: 1967 input: 1766 prompt: 1766 cached: n/a tool use: n/a output: 201 candidates: 53 reasoning: 148 --------------------------------------------------------------------- Metric: invocation_duration_v1, Status: INFORMATIONAL, Score: 5.617, Threshold: None --------------------------------------------------------------------- Invocation Details: Invocation 1 of 1 prompt: The assistant can't reach Ollama. What do I check? expected_response: Check that `OLLAMA_API_BASE` points to a running Ollama server (troubleshooting.md). actual_response: Check that `OLLAMA_API_BASE` points to a running Ollama server (troubleshooting.md). expected_tool_calls: (none) actual_tool_calls: list_pages() read_page( path='troubleshooting.md') cited_page_was_read: PASSED (1.0) tool_call_count_v1: 2 inference_call_count_v1: 3 token_usage_v1: 1967 (in 1766, out 201) invocation_duration_v1: 5.62s ******************************************************************** Eval Set Id: validation Eval Id: local-model Overall Eval Status: PASSED --------------------------------------------------------------------- Metric: cited_page_was_read, Status: PASSED, Score: 1.0, Threshold: 1.0 --------------------------------------------------------------------- Metric: tool_call_count_v1, Status: INFORMATIONAL, Score: 2.0, Threshold: None --------------------------------------------------------------------- Metric: inference_call_count_v1, Status: INFORMATIONAL, Score: 3.0, Threshold: None --------------------------------------------------------------------- Metric: token_usage_v1, Status: INFORMATIONAL, Score: 1809.0, Threshold: None Token breakdown: total: 1809 input: 1631 prompt: 1631 cached: n/a tool use: n/a output: 178 candidates: 64 reasoning: 114 --------------------------------------------------------------------- Metric: invocation_duration_v1, Status: INFORMATIONAL, Score: 5.869, Threshold: None --------------------------------------------------------------------- Invocation Details: Invocation 1 of 1 prompt: How do I point the assistant at a local model? expected_response: Set `OLLAMA_API_BASE` (install.md). actual_response: To point the assistant at a local model instead of Gemini, set the `OLLAMA_API_BASE` environment variable to your running local model server (install.md). expected_tool_calls: (none) actual_tool_calls: list_pages() read_page( path='install.md') cited_page_was_read: PASSED (1.0) tool_call_count_v1: 2 inference_call_count_v1: 3 token_usage_v1: 1809 (in 1631, out 178) invocation_duration_v1: 5.87s # Run optimized-after-docs-change (install.md now says Python 3.13) > Which Python version do I need? call list_pages {} call read_page {'path': 'install.md'} answer You need Python 3.13 or later (install.md).