Verify edits independently of completion claims

Benchmark controlled repair submissions by executing their behavior independently. Record completion claims as observations, then compare them with fresh checker results without treating the wording as confirmation that a task succeeded.

Problem

When you need this. A completion message needs to be checked against the edited files and their behavior.

Three repair submissions say the work is done. One leaves the bug unchanged, one fixes the boundary and one reverses the comparison. Their completion messages look the same; their behavior is different.

You will build a small benchmark around the restricted repair harness. It records each candidate’s claim and independently executes its expression in a fresh workspace. The candidates are controlled local fixtures, so the result compares verification behavior rather than a model’s coding success rate.

Concepts

A completion claim is a statement made by a candidate. An outcome check observes whether the required behavior holds. The two can disagree in either direction: a broken candidate can claim success, and a valid repair can omit that claim.

A trial identity keeps a submitted candidate, its claim and its execution result aligned. This fixture compares three controlled expressions through the existing restricted repair harness. It does not evaluate fresh model-written patches or the richer repository-tool loop from the preceding lesson.

Architecture

The benchmark supplies each candidate and its claim to a new harness attempt. The independent checker runs in the fresh workspace and returns status separately from the claim. A report retains both fields, allowing the unchanged defect, valid fix and reversed comparison to be compared on behavior.

Working code

Keep the submitted artifact distinct from the report about it

Each candidate has an ID, a comparison expression and reported_done. Only the expression is passed into the execution harness. The claim is retained in the report, where a reviewer can compare it with the measured result.

[
  {
    "id": "unchanged",
    "expression": "attempt <= max_attempts",
    "reported_done": true
  },
  {
    "id": "boundary-fix",
    "expression": "attempt < max_attempts",
    "reported_done": true
  },
  {
    "id": "wrong-direction",
    "expression": "attempt > max_attempts",
    "reported_done": true
  }
]

This matters even when a policy reports structured JSON rather than natural language. A Boolean field called done is still a claim from the submitter. It cannot replace checking the resulting file, command behavior or state change required by the task.

The task specification and its checker already exist in the harness lab. Reuse them. If the benchmark copied and subtly changed the expected boundary, comparing its results with the earlier repair exercise would become ambiguous.

Grade each candidate in a fresh execution

Download the complete lab and enter agent-engineering/:

python3.12 -m venv .venv
source .venv/bin/activate
python benchmark.py
python -m unittest -v test_benchmark.py

"""Grade actual candidate behavior instead of trusting a completion message."""
from terminal_harness import check_candidate
from core import load


def benchmark(candidates):
    if not candidates:
        raise ValueError('benchmark needs at least one candidate')
    ids = [c['id'] for c in candidates]
    if len(ids) != len(set(ids)):
        raise ValueError('candidate IDs must be unique')
    return [{'id': c['id'], 'reported_done': c['reported_done'],
             'grade': check_candidate(c['expression'])} for c in candidates]


def main():
    candidates = load('benchmark-candidates.json')
    for row in benchmark(candidates):
        grade = row['grade']
        print(f"{row['id']}: says_done={row['reported_done']}; actual={grade['outcome']}; {grade['passed']}/{grade['total']}")
    print('Three controlled candidate patches; these are environment checks, not model success rates.')


if __name__ == '__main__':
    main()

benchmark() first requires a nonempty candidate list and unique IDs. It then calls check_candidate() for every expression. The harness owns grammar validation, temporary files, subprocess execution, timeout handling and the independent behavioral checks.

Unique IDs prevent a later aggregation step from confusing two submissions. They do not make a candidate correct. The same ID must continue to identify the same expression and result when the report is exported or compared with another run.

Each invocation of the harness creates a new candidate directory. A successful candidate cannot leave a passing file that an unchanged candidate later reuses. This benchmark therefore checks the artifact actually submitted for each row, rather than the last file left in a shared workspace.

Inspect disagreement between the claim and the result

unchanged: says_done=True; actual=failed; 4/6
boundary-fix: says_done=True; actual=passed; 6/6
wrong-direction: says_done=True; actual=failed; 2/6
Three controlled candidate patches; these are environment checks, not model success rates.

The unchanged expression says it is done while still failing the equality boundary. The wrong-direction expression also says it is done and fails other required inputs. The intended strict comparison passes all supplied checks.

The successful-check count helps locate partial behavior, but the task passes only when all required checks pass. An expression that works on most of these inputs still violates the contract. Choosing a lower threshold after seeing the failing candidate would change the task rather than repair the bug.

test_done_message_does_not_override_failing_environment verifies the first disagreement directly. test_valid_behavior_passes_even_without_a_done_claim sets the successful candidate’s claim to false and expects it to pass anyway. Those two tests establish that grading depends on the executed behavior in both directions.

This result says nothing about how often a language model finds the fix. A model evaluation would need actual recorded trials, a fixed request budget, the submitted artifacts and the same independent checks. Treating these hand-authored candidates as model samples would invent model results.

Retain rejected and incomplete rows

The reused harness has distinct rejected_patch, failed, timeout, checker_error and passed outcomes. The benchmark retains the complete returned grade. A malformed expression is not quietly dropped from the candidate count, and a timed-out run is not described as a correct implementation that merely took longer.

The demonstration’s rows are accepted comparisons, so rejection and timeout behavior are exercised in the harness tests rather than fabricated in the displayed benchmark output. Keep this distinction when interpreting the report.

A checker failure deserves investigation before comparing candidates. If its environment or dependencies are broken, the report has an execution problem rather than useful results about the repair. Preserve the failed row and fix the environment; do not selectively rerun only the candidates expected to win.

Specify the next environment outcome before adding tools

The same principle extends beyond code repair. A file-edit task needs the changed file inspected and tested. A database task needs the resulting rows checked. A deployment task needs a separately observed service state. A completion statement is useful context, but each task needs its own concrete outcome.

This benchmark implements only the restricted local repair environment. Its program has no repository-editing agent, deployment tool or general shell sandbox. Extending its scope means adding the corresponding execution boundary and independent outcome checks before running new actions.

Design exercise: add an unchanged candidate whose claim is false and a valid candidate whose claim is false. Predict the behavioral results before running them. Then specify a new task where the correct outcome is a filesystem change: name the file, its expected content, a failure case and the fresh-state setup the checker would require.

Failure cases

The unchanged candidate and the valid fix can carry the same completion message. Only their executed checks distinguish them. Conversely, omitting a done claim must not fail a candidate whose behavior passes. Duplicate or empty trial inputs refuse before a misleading summary is produced.

Deployment

Use this claim/outcome separation when collecting results from a coding tool. Keep candidate identity, checker revision, logs and execution status with each report. A production evaluator additionally needs isolation and task-representative hidden checks; this controlled comparison supplies neither a general security sandbox nor a model leaderboard.

Scaling

More candidates increase workspace and checker executions. Bound those resources independently of how many submissions claim success. When extending to model-generated edits, report failed and incomplete attempts, keep the same checker for compared candidates and account for any task exposure during development.

Key takeaways

• Completion wording and task success are independent observations.
• The same harness and checker must evaluate all compared candidates.
• No executed trials means no basis for a success rate.

Best practices

• Retain claims and measured outcomes in separate fields.
• Refuse duplicate trial identities.
• Keep incomplete executions in the report.

References

• Anthropic, Demystifying evals for AI agents: distinguish transcripts and final environment outcomes, and evaluate coding tasks through their implemented behavior.
• Chen et al., Evaluating Large Language Models Trained on Code: functional correctness is measured through execution rather than surface similarity alone.

References checked October 24, 2026. The displayed benchmark uses controlled local candidate expressions and actual harness execution.