Generate traces and grade evaluations with agents-cli eval

Define evaluation cases, generate conversation traces and grade the resulting responses. You will separate CLI completion from the quality gate, using local checks for case shape, command accuracy and coverage alongside the instructions for model-based evaluation.

Problem

When you need this. Manually chatting with docuctl_docs after editing its prompt or tool docstrings does not tell you whether a change broke multi-turn follow-ups or reintroduced removed docuctl 1.x commands. Running agents-cli eval run runs inference and grading in one step, but decoupling agents-cli eval generate from agents-cli eval grade lets you capture agent traces once and iterate on deterministic checks and LLM-as-judge rubrics without re-running the agent. In this lesson, you will build a three-case dataset (docuctl-eval-dataset.json) covering both single-turn and continued-conversation (N+1) shapes, generate populated traces, score them with eval_config.yaml, and write a CI gate that catches dropped cases and threshold regressions.

Concepts

agents-cli eval splits evaluation into trace collection (eval generate) and trace scoring (eval grade) around the Gemini platform evaluation schema (eval_cases).

Shape A (prompt)

A single-turn evaluation case that defines a top-level prompt ({"role": "user", "parts": [...]}) and an optional reference.response, with no prior agent_data.turns.

Shape B (agent_data.turns, the N+1 pattern)

A continued-conversation evaluation case that omits top-level prompt and provides prior turns in agent_data.turns whose final entry is a user message (author: "user"). eval generate seeds the prior events into a fresh ADK session and dispatches the trailing user message to record the N+1 response.

Action-only event stripping

Before sending traces to the Vertex AI Evaluation Service, eval grade strips events that contain only actions (state_delta) and no content, preventing 400 Required field is not set errors.

Exit code vs evaluation gate

agents-cli eval generate and eval grade can finish successfully while the resulting coverage or quality is insufficient. A CI gate must inspect coverage and summary_metrics in results_*.json.

Architecture

Trace generation supplies cases to the agent and writes the observed conversation. Grading reads those traces and produces metrics. The final gate reads both coverage and scores; a completed CLI process is only one part of that decision. The downloaded Python helper tests the dataset and gate rules locally.

Working code

Download the complete lesson lab, extract it, and open its agents-cli-eval-grade directory. It includes the source, tests and fixtures. With Python 3.12+ and uv installed, prepare the local environment:

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-lock.txt
LAB_DIR="$PWD"

Replace the project path below with the directory created in Lesson 1. Use the model credentials configured there for model activities; use your cloud credentials for cloud activities.

export PROJECT_DIR="/absolute/path/to/docuctl-docs"
cd "$PROJECT_DIR"
mkdir -p tests/eval/datasets
cp "$LAB_DIR/docuctl-eval-dataset.json" tests/eval/datasets/docuctl-eval.json
cp "$LAB_DIR/eval_config.yaml" tests/eval/eval_config.yaml

Keep this extracted lab open for the offline checks. For the CLI activity, copy docuctl-eval-dataset.json into the generated project’s tests/eval/datasets/docuctl-eval.json and copy eval_config.yaml to tests/eval/eval_config.yaml. Run the following agents-cli commands from that project with Lesson 1’s model authentication configured.

Author single-turn and continued eval cases

agents-cli eval generate accepts evaluation datasets in the Gemini platform evaluation format (eval_cases). Each case in eval_cases must use one of two mutually exclusive shapes:

• Shape A (single-turn prompt): sets top-level prompt ({"role": "user", "parts": [{"text": "..."}]}) and an optional reference.response ground truth.
• Shape B (continued conversation, the N+1 pattern): omits prompt and instead provides prior conversation history in agent_data.turns, where the final event in turns is a user message (author: "user"). During eval generate, the CLI seeds the prior events into a fresh ADK session and sends that trailing user message to /run_sse to record the N+1 agent response.

Inspect docuctl-eval-dataset.json, which combines two Shape A cases (rollback_prod_r41 and create_ci_token) with one Shape B continued-conversation case (continued_staging_rollback):

{
  "eval_cases": [
    {
      "eval_case_id": "rollback_prod_r41",
      "prompt": {
        "role": "user",
        "parts": [{"text": "I use docuctl 2.x. How do I roll production back to r41?"}]
      },
      "reference": {
        "response": {
          "role": "model",
          "parts": [{"text": "Run `docuctl rollback --env production --to r41`. Note that `docuctl release revert` was removed in 2.0 and database migrations are not rolled back automatically."}]
        }
      }
    },
    {
      "eval_case_id": "create_ci_token",
      "prompt": {
        "role": "user",
        "parts": [{"text": "How do I create a deploy token for staging CI and check when tokens expire?"}]
      },
      "reference": {
        "response": {
          "role": "model",
          "parts": [{"text": "Run `docuctl tokens create --name ci-docs-api --scope deploy:staging` and store it as `DOCUCTL_TOKEN`. Tokens expire after 90 days; check expiry with `docuctl tokens list`."}]
        }
      }
    },
    {
      "eval_case_id": "continued_staging_rollback",
      "agent_data": {
        "turns": [
          {
            "turn_index": 0,
            "events": [
              {
                "author": "user",
                "content": {
                  "role": "user",
                  "parts": [{"text": "I use docuctl 2.x. How do I roll production back to r41?"}]
                }
              },
              {
                "author": "docuctl_docs",
                "content": {
                  "role": "model",
                  "parts": [{"text": "Run `docuctl rollback --env production --to r41`."}]
                }
              },
              {
                "author": "user",
                "content": {
                  "role": "user",
                  "parts": [{"text": "And on staging?"}]
                }
              }
            ]
          }
        ]
      },
      "reference": {
        "response": {
          "role": "model",
          "parts": [{"text": "Run `docuctl rollback --env staging --to r41`."}]
        }
      }
    }
  ]
}

Generate populated agent traces

To run docuctl_docs over the dataset and write the populated traces (responses plus full agent_data.turns tool-call trajectories) to disk, run agents-cli eval generate:

# Generate populated traces against the local ADK server
agents-cli eval generate \
  --dataset tests/eval/datasets/docuctl-eval.json \
  --output artifacts/traces/docuctl-traces.json \
  --concurrency 2

When --url is omitted, eval generate boots the project’s local FastAPI server on port 18080, queries /apps/app/app-info to discover root_agent_name (docuctl_docs), rewrites any generic "agent" event authors in Shape B cases to "docuctl_docs", dispatches each case to /run_sse, serializes the returned conversation events and writes artifacts/traces/docuctl-traces.json.

Inspect each case’s populated responses and agent_data.turns. Confirm that the intended user question was sent, the expected tool was called, and a final response exists. A saved JSON file without those events is not a completed agent run.

Grade traces with custom and judge metrics

Once artifacts/traces/docuctl-traces.json is on disk, agents-cli eval grade scores those traces without calling the agent again. Configure the metrics in tests/eval/eval_config.yaml:

metrics_to_run:
  - command_flag_accuracy
  - agent_turn_count

custom_metrics:
  - name: command_flag_accuracy
    custom_function: |
      def evaluate(instance):
          resp = str(instance.get("response") or "")
          has_v2_cmd = ("docuctl rollback" in resp) or ("docuctl tokens" in resp)
          used_removed = "release revert" in resp and "removed" not in resp.lower()
          score = 1.0 if (has_v2_cmd and not used_removed) else 0.0
          return {"score": score, "explanation": f"has_v2_cmd={has_v2_cmd}, used_removed={used_removed}"}
  - name: agent_turn_count
    custom_function: |
      def evaluate(instance):
          turns = (instance.get("agent_data") or {}).get("turns", [])
          return {"score": float(len(turns))}

eval_config.yaml selects the metrics to run in metrics_to_run:

The supplied configuration uses two local custom metrics: command_flag_accuracy checks for selected command strings, and agent_turn_count counts recorded turns. Neither establishes that an answer is factually correct or useful. Review the answer against its source page as well. These metric functions do not call a judge model; generating the agent traces does call Gemini. Adding managed or model-based metrics introduces separate model or cloud requests.

Before grading, eval grade removes any actions-only events (events with a state_delta but no content), because the managed API rejects content-less events with 400 Required field is not set.

Here is eval_pipeline.py, which implements the Shape A / Shape B history splitter (split_case_history), the deterministic custom metrics, and the CI threshold gate (check_eval_gate):

"""Trace generation validation, custom metrics, and CI threshold gate for agents-cli eval."""

from __future__ import annotations

from typing import Any
import math


def split_case_history(case: dict[str, Any]) -> tuple[list[dict[str, Any]], dict[str, Any]]:
    """Validate an eval case (Shape A prompt or Shape B agent_data.turns N+1) and split history.

    Mirrors `google.agents.cli.eval.cmd_generate.split_case_history`.
    """
    prompt = case.get("prompt")
    agent_data = case.get("agent_data") or {}
    turns = agent_data.get("turns") or []

    if prompt and turns:
        raise ValueError(
            "Case has both top-level 'prompt' and agent_data.turns; ambiguous."
        )
    if prompt:
        return [], prompt

    prior_events: list[dict[str, Any]] = []
    for turn in turns:
        prior_events.extend(turn.get("events") or [])
    if not prior_events or prior_events[-1].get("author") != "user":
        raise ValueError(
            "Case has no user message to send (missing prompt and no trailing "
            "user event in agent_data.turns)."
        )
    last_event = dict(prior_events.pop())
    content = last_event.get("content")
    if not content:
        raise ValueError("Trailing user event has no content to send to /run_sse.")
    return prior_events, content


def evaluate_command_flag_accuracy(instance: dict[str, Any]) -> dict[str, Any]:
    """Deterministic custom metric checking docuctl 2.x commands and rejecting removed 1.x commands."""
    resp = str(instance.get("response") or "")
    has_v2_cmd = ("docuctl rollback" in resp) or ("docuctl tokens" in resp)
    used_removed = "release revert" in resp and "removed" not in resp.lower()
    score = 1.0 if (has_v2_cmd and not used_removed) else 0.0
    return {
        "score": score,
        "explanation": f"has_v2_cmd={has_v2_cmd}, used_removed={used_removed}",
    }


def evaluate_agent_turn_count(instance: dict[str, Any]) -> dict[str, float]:
    """Count conversation turns in agent_data.turns."""
    turns = (instance.get("agent_data") or {}).get("turns", [])
    return {"score": float(len(turns))}


def check_eval_gate(
    summary_metrics: list[dict[str, Any]],
    thresholds: dict[str, float],
    *,
    expected_cases: int,
) -> dict[str, float]:
    """Enforce zero dropped/errored cases and minimum mean_score thresholds on eval grade output."""
    by_name = {m["metric_name"]: m for m in summary_metrics}
    checked: dict[str, float] = {}
    for metric_name, min_score in thresholds.items():
        if metric_name not in by_name:
            raise ValueError(f"Missing required metric {metric_name!r} in summary_metrics.")
        entry = by_name[metric_name]
        valid = int(entry.get("num_cases_valid") or 0)
        errored = int(entry.get("num_cases_error") or 0)
        if errored > 0 or valid != expected_cases:
            raise ValueError(
                f"Metric {metric_name!r} graded {valid}/{expected_cases} cases "
                f"(num_cases_error={errored}); refusing unexpected coverage or grading errors."
            )
        mean_score = float(entry["mean_score"])
        if not math.isfinite(mean_score) or mean_score < min_score:
            raise ValueError(
                f"Metric {metric_name!r} mean_score {mean_score:.4f} is non-finite or below "
                f"threshold {min_score:.4f}."
            )
        checked[metric_name] = round(mean_score, 4)
    return checked

check_eval_gate requires the exact expected case count and finite scores. Its thresholds are minimum scores for higher-is-better metrics. A turn count is a diagnostic, not a quality score; check completion before interpreting fewer turns as an efficiency improvement.

Run agents-cli eval grade against the saved trace file:

agents-cli eval grade \
  --traces artifacts/traces/docuctl-traces.json \
  --config tests/eval/eval_config.yaml \
  --output artifacts/eval_results \
  --project "$GOOGLE_CLOUD_PROJECT"

The supplied earlier summary illustrates the result format. It is not a new evaluation of the docuctl revision:

{
  "cli_version": "1.11.0",
  "generate": {
    "command": "agents-cli eval generate --dataset tests/eval/datasets/shipctl-eval.json --output artifacts/traces/shipctl-traces.json",
    "server_url": "http://127.0.0.1:18080",
    "discovered_root_agent_name": "shipctl_docs",
    "cases_populated": 3
  },
  "grade": {
    "command": "agents-cli eval grade --traces artifacts/traces/shipctl-traces.json --config tests/eval/eval_config.yaml --output artifacts/eval_results --project vertex-ai-experminent",
    "exit_code": 0,
    "qps": 15.0,
    "summary_metrics": [
      {
        "metric_name": "command_flag_accuracy",
        "num_cases_total": 3,
        "num_cases_valid": 3,
        "num_cases_error": 0,
        "mean_score": 1.0,
        "stdev_score": 0.0
      },
      {
        "metric_name": "agent_turn_count",
        "num_cases_total": 3,
        "num_cases_valid": 3,
        "num_cases_error": 0,
        "mean_score": 1.3333,
        "stdev_score": 0.5774
      }
    ],
    "case_results": [
      {
        "eval_case_id": "rollback_prod_r41",
        "command_flag_accuracy": 1.0,
        "agent_turn_count": 1.0,
        "tool_called": "read_page(page='rollback')"
      },
      {
        "eval_case_id": "create_ci_token",
        "command_flag_accuracy": 1.0,
        "agent_turn_count": 1.0,
        "tool_called": "read_page(page='tokens')"
      },
      {
        "eval_case_id": "continued_staging_rollback",
        "command_flag_accuracy": 1.0,
        "agent_turn_count": 2.0,
        "tool_called": null
      }
    ]
  },
  "failures": {
    "ambiguous_prompt_and_agent_data": {
      "eval_case_id": "broken_ambiguous_case",
      "error": "Case has both top-level 'prompt' and agent_data.turns; ambiguous."
    },
    "dropped_case_silent_exit_zero": {
      "cli_exit_code": 0,
      "warning": "Inference summary: 2/3 succeeded, 1 failed. Warning: Artifact contains only the 2 successful case(s); 1 failed case(s) were dropped.",
      "summary_metrics_with_drop": [
        {
          "metric_name": "command_flag_accuracy",
          "num_cases_total": 2,
          "num_cases_valid": 2,
          "num_cases_error": 0,
          "mean_score": 1.0,
          "stdev_score": 0.0
        }
      ]
    }
  }
}

Across all 3 cases, command_flag_accuracy achieves mean_score: 1.0000 (stdev_score: 0.0000), and agent_turn_count averages 1.3333 (1 turn for each of the two Shape A cases and 2 turns for the Shape B continued conversation).

Failure cases

Run the offline verification suite to test Shape A and Shape B splitting, the custom metric functions, and the two failure boundaries:

cd "$LAB_DIR"
"$LAB_DIR/.venv/bin/python" -m unittest -v test_eval_grade.py

Guard against two failure modes in automated evaluation pipelines:

Ambiguous case with both prompt and agent_data.turns. If an evaluation case defines both a top-level prompt and non-empty agent_data.turns, split_case_history cannot tell whether to start a fresh single-turn session or continue an existing multi-turn conversation, and raises ValueError:

Case has both top-level 'prompt' and agent_data.turns; ambiguous.

Use prompt for single-turn cases (Shape A) and agent_data.turns ending with a user event for continued conversations (Shape B), never both in the same case.

Dropped or low-scoring cases still exit with code 0. Both agents-cli eval generate and agents-cli eval grade exit with code 0 as long as the CLI process itself finishes—even if a case timed out during eval generate (Warning: Artifact contains only the 2 successful case(s); 1 failed case(s) were dropped) or scored 0.0 during eval grade. Relying on the shell exit code alone in CI will pass a degraded agent. Always parse summary_metrics from the saved results_*.json file with a check such as check_eval_gate() to verify that num_cases_valid == expected_cases, num_cases_error == 0, and each required higher-is-better metric has a finite mean_score that meets its minimum threshold.

Deployment

Use the evaluation gate as a separate release check. Retain the case set, populated traces, grading configuration and coverage report with the candidate revision. A deployment should not proceed merely because an evaluation command returned a successful exit status.

Scaling

Plan evaluation volume across cases, conversation turns, repeated trials and judge requests. Deterministic checks run locally; agent execution and model-based grading may make paid requests. Start with a small reviewed dataset and account for incomplete execution before increasing the batch.

Key takeaways

• Separating agents-cli eval generate from agents-cli eval grade lets you collect traces once and iterate on eval_config.yaml metrics without repeating agent inference.
• Shape A cases test single-turn prompts (prompt); Shape B cases (agent_data.turns ending with author: "user") seed prior history into an ADK session to test turn N+1.
• Because eval generate and eval grade return exit code 0 even when cases drop or score poorly, CI pipelines must validate num_cases_valid, num_cases_error, and mean_score in results_*.json.

Best practices

• Combine fast, local deterministic custom_function checks (such as command_flag_accuracy and agent_turn_count) with model-based rubrics (multi_turn_task_success and multi_turn_tool_use_quality).
• Keep Shape A and Shape B strictly mutually exclusive in every dataset file and validate the dataset in unit tests before running eval generate.
• Record judge settings and inspect repeated judgements; temperature zero does not guarantee identical grading.

References

Commands reviewed against Google Agents CLI 1.11.0 and the linked official documentation on October 23, 2026.

• Google Agents CLI evaluation guide: eval generate, eval grade, dataset schemas, and eval_config.yaml.
• Vertex AI GenAI Evaluation Service documentation: managed single-turn and multi-turn evaluation metrics.