Curate traces before selecting an improvement

Prepare task cases and their actual decision traces before selecting a retrieval improvement. Clean known sensitive patterns, deduplicate versioned questions and keep task families within one development or evaluation split.

Problem

When you need this. Duplicate or invalid trajectories would distort the cases used to select an improvement.

The assistant misses “restore the staging release.” You add an alias and the score improves. But the test set contains the same question under a new ID, and another example differs only in capitalization. The apparent improvement may reflect repeated cases rather than a better policy.

You will prepare the workshop’s synthetic documentation tasks before selecting a retrieval change. The program cleans known sensitive patterns, deduplicates versioned questions, assigns whole task families to splits and runs the scripted policy on those cleaned cases. It can save each case together with the decisions and observations it actually produced.

Concepts

A task family groups related examples that could reveal the same answer or retrieval strategy. An encoding question and its paraphrase should not land on opposite sides of the development/evaluation split. Splitting whole families reduces one form of contamination between development and evaluation. Renaming a duplicate does not create a new independent task.

A trace records the decisions and observations produced by executing a case. It must correspond to the cleaned question supplied to the policy, not a differently redacted display copy. Pattern-based redaction handles the declared examples only; it is not a general detector for every kind of sensitive data.

Architecture

The curation program cleans each question, computes its deduplication identity and assigns its declared family to a split. It then executes the existing scripted policy on the retained cleaned cases and writes case/trace pairs. Selection in the next lesson consumes these separated tasks rather than raw repeated questions.

Working code

Assign related questions to the same split

A task family represents related questions about the same operation, such as rollback. Direct wording and a paraphrase of that operation stay together. Splitting individual rows randomly could put one wording in development and its near-copy in test.

The fixture contains an explicit family assignment:

{
  "train": [
    "deploy",
    "config",
    "old-deploy"
  ],
  "development": [
    "rollback",
    "tokens",
    "migrate"
  ],
  "test": [
    "releases",
    "logs",
    "health"
  ]
}

These are authored synthetic cases, not production conversations. The assignment is fixed before the retrieval candidate is selected. Training data is available for future policy work; the next lesson uses development data for selection and test data for the frozen candidate’s result.

A family label is a human decision in this lab. The program verifies that a named family appears in only one split. It cannot discover that two differently named families are really paraphrases. Review the task descriptions and expected sources before trusting the assignment.

Clean and deduplicate before executing the cases

Download the complete lab and enter agent-engineering/:

python3.12 -m venv .venv
source .venv/bin/activate
python curate_traces.py --save-dir curated
python -m unittest -v test_curation.py

redact() replaces email-shaped text and the value in a TOKEN= pattern in each question. The cleaned question, rather than its original sensitive value, is passed to the policy. This ordering matters: removing a value only from the final report would leave it in the policy’s earlier search decision.

partition() fingerprints the cleaned, case-folded, whitespace-normalized question together with its documentation version. The case ID is excluded. Renaming a copied example therefore cannot make it a new case. Version remains part of the key because the same question against different documentation versions can legitimately require different commands.

An exact duplicate with the same family and expected result is retained once. A duplicate with a conflicting family, source or command raises an error. Silently keeping whichever label happened to arrive first would hide a dataset disagreement.

"""Deduplicate task families, redact two known fields, and split before tuning."""
import argparse
import hashlib
import json
from pathlib import Path
import re
from core import load, run, scripted, task_from


def redact(text):
    text = re.sub(r'[\w.+-]+@[\w.-]+\.[A-Za-z]{2,}', '[email]', text)
    return re.sub(r'\bTOKEN=[A-Za-z0-9_-]+', 'TOKEN=[removed]', text)


def partition(cases, family_splits):
    family_owner = {}
    for split, families in family_splits.items():
        for family in families:
            if family in family_owner:
                raise ValueError('a task family appears in more than one split')
            family_owner[family] = split
    output = {name: [] for name in family_splits}
    seen = {}
    for case in cases:
        clean = {**case, 'question': redact(case['question'])}
        # Case IDs are not a deduplication key: changing an ID must not leak a copy.
        fingerprint = hashlib.sha256(json.dumps([
            ' '.join(clean['question'].casefold().split()), clean['version']]).encode()).hexdigest()
        if fingerprint in seen:
            old = seen[fingerprint]
            if (old['family'], old['expected_doc'], old['expected_command']) != (
                    clean['family'], clean['expected_doc'], clean['expected_command']):
                raise ValueError('duplicate question has conflicting family or labels')
            continue
        if clean['family'] not in family_owner:
            raise ValueError('task family has no assigned split')
        seen[fingerprint] = clean
        output[family_owner[clean['family']]].append(clean)
    if any(not rows for rows in output.values()):
        raise ValueError('every split must contain at least one case')
    return output


def curate(cases, family_splits):
    """Run only the deduplicated, question-redacted cases in each fixed split."""
    splits = partition(cases, family_splits)
    return {name: [{'case': case, 'result': run(task_from(case), scripted)}
                   for case in rows] for name, rows in splits.items()}


def save(records, directory):
    directory = Path(directory)
    directory.mkdir(parents=True, exist_ok=True)
    for split, rows in records.items():
        if split not in {'train', 'development', 'test'}:
            raise ValueError('unknown split name')
        (directory / (split + '.json')).write_text(json.dumps(rows, indent=2) + '\n')


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--save-dir', help='Write one local trace file per split')
    args = parser.parse_args()
    cases = load('cases.json')
    records = curate(cases, load('splits.json'))
    if args.save_dir:
        save(records, args.save_dir)
    print('Input:', len(cases), 'synthetic questions;', sum(map(len, records.values())), 'fresh scripted traces')
    for name, rows in records.items():
        print(name + ':', len(rows), 'questions;', len({r['case']['family'] for r in rows}), 'families')
    print('Redaction:', redact('owner@example.invalid TOKEN=demo_secret'))
    print('Split by task family before selecting a policy; keep the test set out of selection.')


if __name__ == '__main__':
    main()

These rules do not detect arbitrary secrets, personal information or semantic duplicates. The documents are bundled fictional material, and only the question passes through the narrow redactor. Real traces can also contain sensitive tool results and provider messages; they need a reviewed handling policy before collection or reuse.

Save the input beside the behavior it produced

curate() first partitions the cleaned cases. It then invokes the existing runtime once per retained case using the scripted policy. Each saved record contains case and result; the result includes the stop state, trace, tool count and turn count.

The command writes train.json, development.json and test.json under the requested local directory. Open a record and compare case.question with the first search decision’s query. They should match the cleaned input. Do not attach an old trace from a differently worded question to a new case merely because its ID looks familiar.

Input: 18 synthetic questions; 18 fresh scripted traces
train: 6 questions; 3 families
development: 6 questions; 3 families
test: 6 questions; 3 families
Redaction: [email] TOKEN=[removed]
Split by task family before selecting a policy; keep the test set out of selection.

The summary reports retained questions and families separately. A split with many paraphrases of one operation offers less task variety than the same number of independent families. Neither count establishes how representative these fixtures are of future requests.

The saved records preserve failed runs as well as final answers. Keeping only successful traces would remove the failed trajectories needed to diagnose the current policy. This program executes scripted decisions; it does not collect or synthesize model reasoning.

Reject ambiguous data instead of dropping it

test_a_family_cannot_cross_the_split assigns a development family to test as well and expects refusal. test_unknown_family_cannot_silently_drop checks that an unassigned case raises an error instead of disappearing from the counts.

test_relabelled_duplicates_are_rejected gives an identical question a contradictory expected source. test_renaming_an_exact_copy_does_not_add_a_case changes only its ID and verifies that the case total stays fixed. Together they separate contradictory data from harmless duplication.

The saved-trace regression inserts an email and a token-shaped value into a question, runs curation, reads the saved JSON and checks the actual search query. It would fail if execution occurred before question redaction, even if the summary printed a cleaned example afterwards.

Freeze the data boundary before tuning

In a real experiment, keep a dataset version, family assignment and policy version with each trace batch. Review disagreements before looking at candidate results. If a failure inspires a new case, decide whether it is development material or belongs in a later evaluation set; repeatedly tuning against it makes it part of development.

Design exercise: add a duplicate question with a new ID and then change its expected command. Predict which version is deduplicated and which is refused. Next add a paraphrase with different wording: explain why this fingerprint cannot detect it and how family review should prevent it from crossing the split.

Continue with retrieval-policy selection, which reuses this partition rather than defining a second dataset protocol.

Failure cases

An exact duplicate under a new ID must not add a second apparent success. A more subtle error is running the unredacted question and saving only a redacted copy; the saved trace would then describe a different input. The tests check both boundaries.

Deployment

Use a reviewed curation policy before saving traces from real users. Keep the raw-data access boundary separate from the cleaned development corpus, and review patterns and retained fields for the actual data involved. The synthetic cleaner demonstrates known patterns and does not authorize collecting production conversations.

Scaling

Deduplication and family assignment need consistent identities across ingestion partitions. Store the curation revision with the retained cases and inspect which examples were removed. For large or semantically similar datasets, exact normalization is only one layer; additional reviewed similarity and privacy controls may be needed.

Key takeaways

• New IDs do not make duplicate tasks independent observations.
• Split related task families before tuning.
• A retained trace must describe the exact cleaned input that was executed.

Best practices

• Record reasons for excluded or merged cases.
• Reject unknown family assignments instead of dropping them.
• Test redaction before policy execution, not only in the saved display.

References

• Anthropic, Demystifying evals for AI agents: tasks, trials, transcripts and final environment outcomes are distinct evaluation objects.

Reference checked October 24, 2026. Split checks, cleaning behavior and saved trace contents are defined by the bundled implementation; the fixtures are intentionally small and synthetic.