Calibrate task graders before comparing policies

Test a proposed agent grader against authored counterexamples before using its score to select a policy. Apply the prerequisite trial metrics to a fixed calculation fixture and state which success event matters for the task.

Problem

When you need this. An apparent improvement may come from grader mistakes or variation between trials.

An answer contains a docuctl command and a citation. A cheap grader calls that a success. But the command is invented, or the citation belongs to the wrong documentation version. Tuning the agent against that grader could reward exactly the behavior the application should refuse.

You will compare this format heuristic with the existing task contract on authored synthetic answers. You will then calculate two different repeated-trial questions: whether at least one attempt succeeds, and whether every attempt succeeds. No model judge or language model is called in this lab.

Concepts

Calibration here compares a proposed grader’s decisions with explicit reference labels. A false positive accepts a response the task rejects; a false negative rejects a response the reference accepts. The cheap format grader checks only command-shaped text and a nonempty citation.

The repeated-trial benchmark owns the probability definitions and estimators. This workshop applies that vocabulary after checking the grader. An at-least-one-success event can be relevant when a person chooses among attempts; success on every attempt is a different requirement for repeated actions.

Architecture

Authored response contrasts feed both the format grader and the existing task-contract grader. Their decisions are compared with the same labels to expose disagreements. A separate Boolean fixture illustrates trial-event calculations; it is not a record of repeated model execution.

Working code

Construct counterexamples to the proposed grader

The format heuristic accepts an answer when its citation is nonempty and its text contains docuctl . That is a useful shape check, but it never asks which command the task requires or whether the cited page matches the requested version.

judge-cases.json contains correct answers and targeted failures: a wrong-version source, an invented command, a missing citation and an unsupported refusal. Each row has an explicit reference_label. These labels were authored for this controlled exercise; they are not ratings from a human study.

[
  {
    "id": "correct",
    "case": {
      "id": "rollback-direct",
      "family": "rollback",
      "question": "rollback staging",
      "version": "v2",
      "expected_doc": "rollback-v2",
      "expected_command": "docuctl rollback --env staging --release LAST_GOOD"
    },
    "result": {
      "stop": "final",
      "answer": "docuctl rollback --env staging --release LAST_GOOD",
      "citation": "rollback-v2"
    },
    "reference_label": true
  },
  {
    "id": "wrong-version",
    "case": {
      "id": "rollback-direct",
      "family": "rollback",
      "question": "rollback staging",
      "version": "v2",
      "expected_doc": "rollback-v2",
      "expected_command": "docuctl rollback --env staging --release LAST_GOOD"
    },
    "result": {
      "stop": "final",
      "answer": "docuctl release create --target staging",
      "citation": "deploy-v1"
    },
    "reference_label": false
  },
  {
    "id": "invented-command",
    "case": {
      "id": "rollback-direct",
      "family": "rollback",
      "question": "rollback staging",
      "version": "v2",
      "expected_doc": "rollback-v2",
      "expected_command": "docuctl rollback --env staging --release LAST_GOOD"
    },
    "result": {
      "stop": "final",
      "answer": "docuctl undo --env staging",
      "citation": "rollback-v2"
    },
    "reference_label": false
  },
  {
    "id": "missing-source",
    "case": {
      "id": "rollback-direct",
      "family": "rollback",
      "question": "rollback staging",
      "version": "v2",
      "expected_doc": "rollback-v2",
      "expected_command": "docuctl rollback --env staging --release LAST_GOOD"
    },
    "result": {
      "stop": "final",
      "answer": "docuctl rollback --env staging --release LAST_GOOD",
      "citation": ""
    },
    "reference_label": false
  },
  {
    "id": "unsupported-refusal",
    "case": {
      "id": "rollback-direct",
      "family": "rollback",
      "question": "rollback staging",
      "version": "v2",
      "expected_doc": "rollback-v2",
      "expected_command": "docuctl rollback --env staging --release LAST_GOOD"
    },
    "result": {
      "stop": "final",
      "answer": "I cannot answer.",
      "citation": ""
    },
    "reference_label": false
  },
  {
    "id": "second-correct",
    "case": {
      "id": "deploy-direct",
      "family": "deploy",
      "question": "deploy staging",
      "version": "v2",
      "expected_doc": "deploy-v2",
      "expected_command": "docuctl deploy --env staging"
    },
    "result": {
      "stop": "final",
      "answer": "docuctl deploy --env staging",
      "citation": "deploy-v2"
    },
    "reference_label": true
  }
]

The task-contract grader reuses outcome() from the first module. It requires a final result with the expected command and source. That exact-match rule fits this narrow command lookup task. It would be too restrictive for a task where several paraphrases or alternative commands are valid; those tasks need a different contract and reviewed reference answers.

Compare the two graders against the same labels

Download the complete lab and enter agent-engineering/:

python3.12 -m venv .venv
source .venv/bin/activate
python judge_trials.py
python -m unittest -v test_judge.py

"""Compare a cheap grader with reference labels; distinguish any from all trials."""
from math import comb
from core import load, outcome


def format_grader(result):
    return bool(result.get('citation')) and 'docuctl ' in result.get('answer', '')


def confusion(rows, grade):
    counts = {key: 0 for key in ('TP', 'FP', 'FN', 'TN')}
    for row in rows:
        predicted, expected = bool(grade(row)), row['reference_label']
        counts[('T' if predicted == expected else 'F') + ('P' if predicted else 'N')] += 1
    return counts


def trial_metrics(passed, k):
    if not passed or any(type(x) is not bool for x in passed):
        raise ValueError('trials must be a nonempty list of Boolean outcomes')
    n, c = len(passed), sum(passed)
    if type(k) is not int or not 1 <= k <= n:
        raise ValueError('k must lie between 1 and the number of trials')
    # HumanEval finite-sample estimator, not1-(1-observed_rate)**k.
    any_k = 1 - (comb(n - c, k) / comb(n, k) if n - c >= k else 0)
    return {'passed': c, 'trials': n, 'pass_at_k': any_k,
            'all_k_iid_plugin': (c / n) ** k,
            'observed_any': any(passed), 'observed_all': all(passed)}


def main():
    rows = load('judge-cases.json')
    print('Reference set:', len(rows), 'authored synthetic answers; no model judge calls')
    for name, grade in [('format', lambda row: format_grader(row['result'])),
                        ('task contract', lambda row: outcome(row['case'], row['result']))]:
        c = confusion(rows, grade)
        print(name + ':', ' '.join(f'{key}={value}' for key, value in c.items()))
    score = trial_metrics([True, True, False, True], 2)
    print('Calculation fixture: 3 of 4 pass; any=True; all=False')
    print(f"pass@2={score['pass_at_k']:.4f}; all 2 IID plug-in={score['all_k_iid_plugin']:.4f}")
    print('The all-2 estimate assumes independent, unchanged trials; four fixture values are not a reliability study.')


if __name__ == '__main__':
    main()

confusion() compares each predicted result with its reference label. A false positive is an answer the grader accepts even though the reference rejects it. A false negative is an answer the grader rejects despite a positive reference.

Reference set: 6 authored synthetic answers; no model judge calls
format: TP=2 FP=2 FN=0 TN=2
task contract: TP=2 FP=0 FN=0 TN=4
Calculation fixture: 3 of 4 pass; any=True; all=False
pass@2=1.0000; all 2 IID plug-in=0.5625
The all-2 estimate assumes independent, unchanged trials; four fixture values are not a reliability study.

The format grader accepts the wrong-version and invented-command answers. They look like commands with sources, but they do not satisfy the task. The task-contract grader agrees with all labels in this small reference fixture.

That agreement is a calibration check, not a general quality guarantee. The labels intentionally exercise the contract’s known boundaries. More diverse cases, label review and disagreement inspection are needed before using a grader to guide a wider agent experiment.

test_a_well_formatted_wrong_answer_is_a_false_positive isolates the invented command. If someone changes the reporting code so that this answer no longer counts as a false positive, the test fails. A high aggregate score must not hide the failure that motivated the check.

Ask which repeated-trial event matters to the application

Suppose a fixed task has trial outcomes True, True, False, True. The observed batch contains a success, but it also contains a failure. “At least one succeeded” and “all succeeded” answer different operational questions.

For a coding assistant whose user can choose among attempts, an at-least-one-success measure may be useful. For an action performed repeatedly without a person selecting a result, failures on later attempts still matter. A retry policy also consumes work; a higher any-success score does not remove that cost.

trial_metrics() applies the two estimators introduced in the repeated-trial benchmark. Read the returned fields as different questions: pass_at_k estimates an at-least-one-success event from the finite observed batch; all_k_iid_plugin substitutes the observed rate into an all-success calculation under an independent, unchanged-probability assumption. The fixed four-outcome example lets you check the distinction before attaching either number to an agent comparison.

The displayed Booleans are a calculation fixture. They were not generated by repeated model trials, so neither number estimates this assistant’s actual reliability.

Preserve failures when reporting an improvement

test_any_success_and_all_success_are_different checks both observed events and both calculated values. The function also rejects an empty trial list, non-Boolean outcomes and a k outside the available trial count. Missing or incomplete runs should be reported explicitly rather than silently converted to successful attempts.

Before comparing two policies, freeze the grader version, task set, per-task trial count and request budget. Inspect disagreements on actual traces. If the grader changes after seeing a candidate’s outputs, rerun both policies under that same revised rule before attributing the score difference to the candidate.

Design exercise: propose a counterexample that the exact command-and-citation contract should reject but the format grader accepts. Then choose a repeated-trial metric for a read-only suggestion tool and for an automatically repeated action. State the event, assumptions and extra execution cost for each.

The harness module applies the same separation to code edits: the completion message is recorded, while a fresh process checks what the candidate actually does.

Failure cases

The invented-command and wrong-version responses look like supported commands to the format heuristic. The independent task contract rejects them. Preserving those false positives matters more than a high aggregate agreement count; an invalid or empty trial input must also refuse rather than manufacture a score.

Deployment

Version the grader with the task set before using it to compare policies. If a counterexample changes the grading rule, evaluate both baseline and candidate under the revised rule. A model-based judge would require its own reviewed calibration examples and measured disagreements; this lab applies deterministic graders only.

Scaling

A wider task needs more diverse counterexamples and reviewed labels, not merely more runs of the same examples. Inspect false-positive and false-negative categories before expanding automated evaluation. Repeated attempts also consume work; a larger any-success value alone does not justify the added requests.

Key takeaways

• A format heuristic can reward a task failure.
• Calibration depends on the reference labels and their coverage.
• Repeated-trial metrics answer different events and retain their stated assumptions.

Best practices

• Keep a counterexample for every known false-positive boundary.
• Freeze grader and task identities for a policy comparison.
• Report incomplete trials rather than converting them into success.

References

• Chen et al., Evaluating Large Language Models Trained on Code: functional-correctness evaluation and the finite-sample pass-at-k estimator.
• Anthropic, Demystifying evals for AI agents: grader design, task outcomes and repeated-trial evaluation.

References checked October 24, 2026. The lab uses authored reference labels and Boolean calculation inputs, with no model-based grading.