Synthesize user simulations and compare evaluation runs

Use simulated conversations to discover candidate failure cases, then compare a targeted change with the baseline. The local lab checks metric differences and regression decisions; the CLI activities show where new agent and simulator requests enter the workflow.

Problem

When you need this. Hand-written evaluation cases test the questions you remembered to write down, but real developers ask compound questions—such as asking how to create a staging token and then asking how to revoke an active token, a topic tokens.md does not cover. Without multi-turn user simulation and baseline-to-candidate comparison, an agent can loop through unrelated documentation pages or regress on existing commands when you edit its prompt. In this lesson, you will synthesize multi-turn conversations with agents-cli eval dataset synthesize, diagnose a tool-loop loss cluster with agents-cli eval analyze, test a proposed instruction change, and inspect the differences with agents-cli eval compare.

Concepts

The agents-cli Quality Flywheel closes the loop between discovering multi-turn failures and verifying that a targeted fix improves the failing cluster without breaking existing capabilities.

User simulation (eval dataset synthesize)

Runs an LLM-driven user persona against your live ADK agent for up to --max-turns turns per scenario and saves already-populated traces (responses and agent_data.turns) ready for eval grade.

Loss clustering (eval analyze)

Groups low-scoring multi-turn cases (multi_turn_task_success or multi_turn_tool_use_quality) into candidate failure clusters so you can map failures to tool docstrings, system instructions, or agent architecture.

Recursive score diff (eval compare)

An in-process dictionary comparator that walks baseline and candidate results_*.json files and reports [baseline, candidate, "+delta"] for each changed numeric leaf.

Non-regression gate

A check over the eval compare diff ensuring that improving a target metric (such as multi_turn_tool_use_quality) does not degrade a held-out metric (such as command_flag_accuracy).

Architecture

Synthesis produces conversations to review, analysis groups low-scoring cases, and comparison pairs baseline and candidate metrics. The feedback path is a proposed agent change followed by another evaluation. Keep the original cases and metric meanings fixed when interpreting the comparison.

Working code

Download the complete lesson lab, extract it, and open its agents-cli-eval-compare directory. It includes the source, tests and fixtures. With Python 3.12+ and uv installed, prepare the local environment:

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-lock.txt
LAB_DIR="$PWD"

Replace the project path below with the directory created in Lesson 1. Use the model credentials configured there for model activities; use your cloud credentials for cloud activities.

export PROJECT_DIR="/absolute/path/to/docuctl-docs"
cd "$PROJECT_DIR"

For live synthesis, continue in the authenticated generated project from Lesson 1. Simulation runs both the user simulator and your agent and may incur model charges. The extracted lab runs local comparison tests without simulation. Its scenario and score files are supplied examples, not a new experiment on docuctl.

Synthesize multi-turn user simulations

Instead of writing every multi-turn conversation by hand, agents-cli eval dataset synthesize inspects the ADK agent declared in agents-cli-manifest.yaml (agent_directory: app) and runs a model-based user simulator against your agent for up to --max-turns turns per scenario:

# Generate 3 multi-turn simulated conversations with up to 4 turns each
agents-cli eval dataset synthesize \
  -n 3 \
  --max-turns 4 \
  --instruction "Developers asking about docuctl 1.x vs 2.x rollback, token expiry, and unbuilt features" \
  --output artifacts/traces/synthesized-docuctl.json

Because eval dataset synthesize runs your agent live during the simulation, its output file (artifacts/traces/synthesized-docuctl.json) already contains populated traces (responses and agent_data.turns). You pass that file directly to agents-cli eval grade --traces artifacts/traces/synthesized-docuctl.json—do not pass synthesized traces back through eval generate.

Inspect the three supplied scenario examples and their tool trajectories in synthesized-scenarios.json:

{
  "synthesize_command": "agents-cli eval dataset synthesize -n 3 --max-turns 4 --instruction \"Developers asking about shipctl 1.x vs 2.x rollback, token expiry, and unbuilt features\" --output artifacts/traces/synthesized-shipctl.json",
  "eval_cases": [
    {
      "eval_case_id": "synth_v1_revert_migration",
      "scenario": "Developer tries to use removed shipctl 1.x `release revert` command and asks about database migrations.",
      "turns_generated": 2,
      "tools_called": ["read_page(page='rollback')"]
    },
    {
      "eval_case_id": "synth_token_revocation_boundary",
      "scenario": "Developer asks how to create a staging CI token and then asks how to revoke an active token (which tokens.md does not cover).",
      "turns_generated": 2,
      "baseline_tools_called": [
        "read_page(page='tokens')",
        "read_page(page='config')",
        "read_page(page='deploy')",
        "read_page(page='rollback')"
      ],
      "candidate_tools_called": [
        "read_page(page='tokens')"
      ]
    },
    {
      "eval_case_id": "synth_prod_approval_flow",
      "scenario": "Developer deploys r42 to staging and then asks what approval command is required for production.",
      "turns_generated": 2,
      "tools_called": ["read_page(page='deploy')"]
    }
  ]
}

The token-revocation scenario illustrates an important boundary: tokens.md documents creation and expiry, but not revocation. A useful answer should identify that gap instead of inventing a command. Use this scenario to design a check; the supplied sequence is a teaching fixture, rather than a saved model response.

Cluster failure modes and fix the agent

After running and grading your own baseline, inspect the failing cases before changing the instruction. For managed multi-turn metrics, the CLI can group failures with eval analyze; use the result path produced by your grading run and your own cloud project:

agents-cli eval analyze \
  --eval-result $BASELINE_RESULT \
  --metric multi_turn_tool_use_quality \
  --top-k 2 \
  --project "$GOOGLE_CLOUD_PROJECT"

Following the agents-cli Quality Flywheel, map each failure symptom to the narrowest fix layer:

Compare baseline and candidate runs

Keep the baseline result, change one instruction, then generate and grade a candidate on the same cases with the same model and metrics. Keep development cases separate from held-out checks. Save the two metric dictionaries as baseline.json and candidate.json, then compare the baseline and candidate metric summaries with agents-cli eval compare:

agents-cli eval compare \
  artifacts/eval_results/baseline.json \
  artifacts/eval_results/candidate.json

agents-cli eval compare runs completely in-process (no network calls) and computes a recursive dictionary diff with dotted key paths and signed deltas (+0.3333, +1.0, -0.3333). Here is compare_flywheel.py, which implements diff_eval_results, validate_analyze_metric, and verify_no_regressions:

"""Quality Flywheel diffing, loss-cluster validation, and regression gates for agents-cli eval."""

from __future__ import annotations

from typing import Any

SUPPORTED_ANALYZE_METRICS = frozenset(
    {"multi_turn_task_success", "multi_turn_tool_use_quality"}
)


def diff_eval_results(
    base: dict[str, Any], cand: dict[str, Any], prefix: str = ""
) -> dict[str, Any]:
    """Compute a recursive diff between two eval result dicts (mirrors cmd_compare._diff)."""
    differences: dict[str, Any] = {}
    all_keys = set(base.keys()) | set(cand.keys())
    for key in sorted(all_keys):
        full_key = f"{prefix}{key}" if not prefix else f"{prefix}.{key}"
        base_val = base.get(key)
        cand_val = cand.get(key)

        if base_val == cand_val:
            continue

        if isinstance(base_val, dict) and isinstance(cand_val, dict):
            nested = diff_eval_results(base_val, cand_val, prefix=full_key)
            differences.update(nested["differences"])
            continue

        entry: dict[str, Any] = {"baseline": base_val, "candidate": cand_val}
        if isinstance(base_val, (int, float)) and isinstance(cand_val, (int, float)):
            delta = round(cand_val - base_val, 4)
            entry["delta"] = f"+{delta}" if delta >= 0 else str(delta)
        differences[full_key] = entry

    if prefix:
        return {"differences": differences}

    return {
        "baseline_keys": sorted(base.keys()),
        "candidate_keys": sorted(cand.keys()),
        "differences": differences,
        "changed_keys": sorted(differences.keys()),
        "unchanged_keys": sorted(k for k in all_keys if k not in differences),
    }


def validate_analyze_metric(metric_name: str) -> str:
    """Validate that the metric passed to `agents-cli eval analyze` is supported."""
    if metric_name not in SUPPORTED_ANALYZE_METRICS:
        supported = ", ".join(sorted(SUPPORTED_ANALYZE_METRICS))
        raise ValueError(
            f"Unsupported metric {metric_name!r} for 'agents-cli eval analyze'. "
            f"Currently supported: {supported}."
        )
    return metric_name


def verify_no_regressions(
    comparison: dict[str, Any],
    higher_is_better_metrics: tuple[str, ...] = (
        "command_flag_accuracy",
        "multi_turn_task_success",
        "multi_turn_tool_use_quality",
    ),
    max_allowed_drop: float = 0.0,
) -> dict[str, float]:
    """Verify that no higher-is-better metric regressed beyond max_allowed_drop."""
    diffs = comparison.get("differences") or {}
    deltas: dict[str, float] = {}
    regressions: list[str] = []

    for metric in higher_is_better_metrics:
        key = f"{metric}.mean_score"
        if key not in diffs:
            deltas[metric] = 0.0
            continue
        entry = diffs[key]
        delta = round(float(entry["candidate"]) - float(entry["baseline"]), 4)
        deltas[metric] = delta
        if delta < -abs(max_allowed_drop):
            regressions.append(
                f"{metric}.mean_score regressed from {entry['baseline']} to "
                f"{entry['candidate']} ({delta:+.4f})"
            )

    if regressions:
        raise ValueError("Regression detected: " + "; ".join(regressions))
    return deltas

Inspect the recorded baseline, candidate, and agents-cli eval compare output in recorded-eval-compare.json:

{
  "cli_version": "1.11.0",
  "baseline": {
    "command_flag_accuracy": {
      "mean_score": 0.6667,
      "stdev_score": 0.5774,
      "num_cases_total": 3,
      "num_cases_valid": 3,
      "num_cases_error": 0
    },
    "multi_turn_task_success": {
      "mean_score": 3.6667,
      "stdev_score": 1.1547,
      "num_cases_total": 3,
      "num_cases_valid": 3,
      "num_cases_error": 0
    },
    "multi_turn_tool_use_quality": {
      "mean_score": 0.6667,
      "stdev_score": 0.5774,
      "num_cases_total": 3,
      "num_cases_valid": 3,
      "num_cases_error": 0
    },
    "agent_turn_count": {
      "mean_score": 2.3333,
      "stdev_score": 0.5774,
      "num_cases_total": 3,
      "num_cases_valid": 3,
      "num_cases_error": 0
    }
  },
  "candidate": {
    "command_flag_accuracy": {
      "mean_score": 1.0,
      "stdev_score": 0.0,
      "num_cases_total": 3,
      "num_cases_valid": 3,
      "num_cases_error": 0
    },
    "multi_turn_task_success": {
      "mean_score": 4.6667,
      "stdev_score": 0.5774,
      "num_cases_total": 3,
      "num_cases_valid": 3,
      "num_cases_error": 0
    },
    "multi_turn_tool_use_quality": {
      "mean_score": 1.0,
      "stdev_score": 0.0,
      "num_cases_total": 3,
      "num_cases_valid": 3,
      "num_cases_error": 0
    },
    "agent_turn_count": {
      "mean_score": 2.0,
      "stdev_score": 0.0,
      "num_cases_total": 3,
      "num_cases_valid": 3,
      "num_cases_error": 0
    }
  },
  "compare_output": {
    "command": "agents-cli eval compare artifacts/eval_results/baseline.json artifacts/eval_results/candidate.json",
    "exit_code": 0,
    "differences": {
      "agent_turn_count.mean_score": {
        "baseline": 2.3333,
        "candidate": 2.0,
        "delta": "-0.3333"
      },
      "agent_turn_count.stdev_score": {
        "baseline": 0.5774,
        "candidate": 0.0,
        "delta": "-0.5774"
      },
      "command_flag_accuracy.mean_score": {
        "baseline": 0.6667,
        "candidate": 1.0,
        "delta": "+0.3333"
      },
      "command_flag_accuracy.stdev_score": {
        "baseline": 0.5774,
        "candidate": 0.0,
        "delta": "-0.5774"
      },
      "multi_turn_task_success.mean_score": {
        "baseline": 3.6667,
        "candidate": 4.6667,
        "delta": "+1.0"
      },
      "multi_turn_task_success.stdev_score": {
        "baseline": 1.1547,
        "candidate": 0.5774,
        "delta": "-0.5773"
      },
      "multi_turn_tool_use_quality.mean_score": {
        "baseline": 0.6667,
        "candidate": 1.0,
        "delta": "+0.3333"
      },
      "multi_turn_tool_use_quality.stdev_score": {
        "baseline": 0.5774,
        "candidate": 0.0,
        "delta": "-0.5774"
      }
    }
  },
  "loss_cluster": {
    "command": "agents-cli eval analyze --eval-result artifacts/eval_results/baseline_full.json --metric multi_turn_tool_use_quality --top-k 2",
    "top_cluster": "Redundant read_page calls across unrelated pages when a sub-question is not covered in the target page",
    "fix_applied": "Updated INSTRUCTION to stop after reading the relevant topic page and state when a sub-question is undocumented instead of scanning all pages."
  },
  "failures": {
    "unsupported_analyze_metric": {
      "metric": "command_flag_accuracy",
      "error": "Unsupported metric 'command_flag_accuracy' for 'agents-cli eval analyze'. Currently supported: multi_turn_task_success, multi_turn_tool_use_quality."
    },
    "regressed_candidate": {
      "differences": {
        "multi_turn_tool_use_quality.mean_score": {
          "baseline": 0.6667,
          "candidate": 1.0,
          "delta": "+0.3333"
        },
        "command_flag_accuracy.mean_score": {
          "baseline": 1.0,
          "candidate": 0.6667,
          "delta": "-0.3333"
        }
      },
      "error": "Regression detected: command_flag_accuracy.mean_score regressed from 1.0 to 0.6667 (-0.3333)"
    }
  }
}

The supplied baseline and candidate dictionaries demonstrate signed deltas. Their numbers do not establish that an instruction change improved a real Gemini agent. A valid improvement claim needs the corresponding baseline and candidate traces, configurations, case membership and grader results. eval compare only compares the dictionaries it is given.

Failure cases

Run the offline verification suite to test the recursive diff logic, the eval analyze metric validator, and the regression gate:

cd "$LAB_DIR"
"$LAB_DIR/.venv/bin/python" -m unittest -v test_eval_compare.py

Two failure boundaries commonly occur during the Quality Flywheel:

Unsupported metric passed to eval analyze. The experimental agents-cli eval analyze command only supports clustering against two managed multi-turn metrics: multi_turn_task_success and multi_turn_tool_use_quality. Passing a custom metric name such as --metric command_flag_accuracy fails validation:

Unsupported metric 'command_flag_accuracy' for 'agents-cli eval analyze'. Currently supported: multi_turn_task_success, multi_turn_tool_use_quality.

Inspect per-case explanation strings directly in results_*.json when debugging custom metrics, and reserve eval analyze for multi_turn_task_success and multi_turn_tool_use_quality.

Candidate improves target metric but regresses held-out metric. agents-cli eval compare prints the JSON diff and exits with 0 even when a candidate prompt improves multi_turn_tool_use_quality (+0.3333) while causing command_flag_accuracy to drop from 1.0 to 0.6667 (-0.3333). Always pass the diff output through a gate such as verify_no_regressions() so any negative delta on a higher-is-better metric blocks the change before deployment.

Deployment

Use reviewed simulation cases as regression inputs for the next candidate. Keep baseline and candidate result files with their source revisions and metric configuration. The comparison informs the release decision; it does not deploy the candidate or prove that simulated users represent production traffic.

Scaling

Each additional scenario or turn can increase both agent and simulator requests. Set scenario and turn limits, keep failed or incomplete runs, and review representative failure clusters before expanding the evaluation. A single simulated conversation does not estimate the failure rate of real users.

Key takeaways

• agents-cli eval dataset synthesize produces already-populated multi-turn traces (responses and agent_data.turns) that go directly into agents-cli eval grade.
• agents-cli eval analyze clusters low-scoring multi-turn trajectories (multi_turn_task_success and multi_turn_tool_use_quality) so you can fix tool docstrings or system instructions at the right layer.
• agents-cli eval compare computes an in-process recursive diff with signed deltas, which must be paired with a non-regression check (verify_no_regressions()) to block regressions in CI.

Best practices

• Review synthesized conversations before adding them to your permanent regression suite so simulator hallucinations do not become false ground truth.
• Use the trace to locate the failing boundary before changing tool descriptions, instructions or model settings. Re-evaluate the targeted change.
• Track both quality metrics (higher is better) and efficiency diagnostics such as agent_turn_count (fewer turns are useful only when the required task still completes) when comparing baseline and candidate runs.

References

Commands reviewed against Google Agents CLI 1.11.0 and the linked official documentation on October 23, 2026.

• Google Agents CLI evaluation guide: eval dataset synthesize, eval compare, eval analyze, and eval optimize.
• Vertex AI user simulation and multi-turn evaluation: multi-turn task success and tool-use quality rubrics.