Tune retrieval policy on separated task families

Select retrieval settings using development task families, then evaluate the frozen choice on different families. Inspect both source hits and returned passages so the tuning decision remains separate from held-out evaluation.

Problem

When you need this. You need to select a retrieval policy without tuning it on the held-out task family.

A search-and-read policy cannot cite a source it never retrieves. Before changing the policy prompt or adding another worker, inspect whether the expected page appears in its search results. If retrieval is the bottleneck, changing the controller may leave the failure untouched.

You will compare a small, declared set of retrieval configurations for the fictional documentation assistant. Selection uses development tasks only. The selected configuration is then evaluated on different task families, with both source hits and the number of returned passages reported.

Concepts

A candidate configuration changes a declared retrieval setting, such as result count or query aliases. Selection chooses among those candidates using development data. Held-out evaluation examines the frozen choice on tasks not used for that decision.

A tie rule is part of selection policy. This lab prefers less returned context when development quality ties; that preference is not a measured model-cost saving. Source retrieval is also separate from answer correctness: finding the expected page is necessary for this task, but a later policy can still use it incorrectly.

Architecture

The experiment loads separated task families and enumerates the declared retrieval candidates. Development scores and a fixed tie rule select one configuration. The selected configuration then runs on different families, with expected-source hits and returned-passage counts reported separately.

Working code

Optimize a named failure rather than a general score

The curation lesson provides fixed train, development and test family assignments. Keep those assignments. The optimization target here is source recall: for each question, does the returned list contain its expected document ID?

This deliberately stops before answer generation. A correct page anywhere in the candidates is a retrieval success, even if the current scripted controller would read another candidate first. Source recall is therefore a diagnostic measure, not the assistant’s end-to-end answer score.

The candidates change two things: whether the search expands documentation aliases, and how many ranked passages it returns. The alias glossary maps alternate wording to search terms from the bundled documentation vocabulary. It is not populated with expected document IDs from the test labels.

Download the complete lab and enter agent-engineering/:

python3.12 -m venv .venv
source .venv/bin/activate
python optimize_retrieval.py
python -m unittest -v test_optimization.py

This runs local lexical search. It makes no embedding, reranker or language-model request. The retrieval implementation is reused from the agent-design lab; its ranking mechanics are not replaced by a new framework.

Declare the candidate set and tie rule before scoring

Candidate is immutable and records a name, alias-expansion flag and top_k. CANDIDATES fixes the alternatives before select() receives the development cases.

"""Select a real retrieval configuration on development cases, then evaluate once."""
from dataclasses import dataclass
from core import Documents, load
from curate_traces import partition


@dataclass(frozen=True)
class Candidate:
    name: str
    expand: bool
    top_k: int


CANDIDATES = (Candidate('literal-k1', False, 1), Candidate('glossary-k1', True, 1),
              Candidate('glossary-k2', True, 2))


def evaluate(candidate, cases):
    if not cases:
        raise ValueError('cannot score an empty case set')
    docs, passed, returned = Documents(), 0, 0
    for case in cases:
        hits = docs.search(case['question'], case['version'],
                           expand=candidate.expand, top_k=candidate.top_k)
        passed += case['expected_doc'] in [hit['id'] for hit in hits]
        returned += len(hits)
    return {'hits': passed, 'total': len(cases), 'returned': returned}


def select(development, *, candidates=CANDIDATES):
    if not candidates:
        raise ValueError('at least one candidate is required')
    scores = [(candidate, evaluate(candidate, development)) for candidate in candidates]
    # Maximize hits, then prefer less context; the name makes remaining ties stable.
    chosen, _ = min(scores, key=lambda row: (-row[1]['hits'], row[1]['returned'], row[0].name))
    return chosen, scores


def main():
    splits = partition(load('cases.json'), load('splits.json'))
    chosen, scores = select(splits['development'])
    for candidate, score in scores:
        print(f"development {candidate.name}: {score['hits']}/{score['total']} source hits; {score['returned']} passages")
    # Test labels are first used after selection has returned a frozen candidate.
    score = evaluate(chosen, splits['test'])
    print('selected:', chosen.name)
    print(f"held-out {chosen.name}: {score['hits']}/{score['total']} source hits; {score['returned']} passages")
    print('Metric: correct source in returned passages, not answer quality or model reliability.')


if __name__ == '__main__':
    main()

evaluate() returns successful source hits, case count and total passages returned. select() first maximizes hits, then prefers fewer passages. The candidate name breaks any remaining tie deterministically. It does not select by alphabetical order before evaluating task performance.

The secondary rule expresses a real constraint: extra passages create more text to read. This lab counts passages rather than tokens, so the measure is only a coarse proxy for context work. A larger passage can cost more than several short ones; token and latency measurements belong in the later system experiment.

There is no training step in this program. The training split remains unused. Calling the operation “selection” is precise: it chooses among existing retrieval configurations rather than updating model weights.

Read the development result before the test result

development literal-k1: 5/6 source hits; 6 passages
development glossary-k1: 5/6 source hits; 6 passages
development glossary-k2: 6/6 source hits; 12 passages
selected: glossary-k2
held-out glossary-k2: 6/6 source hits; 12 passages
Metric: correct source in returned passages, not answer quality or model reliability.

On these development cases, returning another candidate recovers an expected source that the smaller lists miss. Alias expansion alone does not improve the aggregate hit count. The selected policy therefore includes both expansion and the larger candidate list, according to the declared objective and tie rule.

That result does not prove that both changes were necessary. The candidate set does not include a literal-search configuration with the larger list. If you want to isolate the contribution of alias expansion, declare that additional comparison and use development data. Do not use the held-out result to decide retrospectively which ablation to call successful.

chosen is returned before evaluate(chosen, splits['test']) runs. The test result uses the same object and search implementation. No selection callback sees the test cases. That code boundary makes the intended protocol visible, although a local program cannot prevent a person from repeatedly inspecting and tuning against the test file.

Check selection behavior with deliberately misleading inputs

test_ties_prefer_less_context presents two configurations that find the same expected pages. It reverses their input order and verifies that the smaller returned context wins. This guards against a selector that simply takes the first candidate.

test_test_labels_do_not_change_the_frozen_selection changes every test expectation, selects again from unchanged development cases and confirms that the chosen configuration stays the same. The altered test score then falls, demonstrating that the test labels affect reporting without leaking into selection.

An empty development set raises an error rather than producing a meaningless winner. An unsupported candidate count is rejected by the search configuration check. Neither is treated as a low score that could accidentally beat another invalid candidate.

Carry the selected policy into an end-to-end experiment

The next useful experiment is to connect the selected candidate list to a reading policy and evaluate the final command and citation. A source that appears second in the list still needs to be read or selected correctly. That experiment should preserve the test families and report the additional tool work.

The current result is a small synthetic demonstration. It provides neither a confidence interval for live traffic nor a demonstration that a model will choose the right passage. Keep the limited metric next to the result so a retrieval win is not mistaken for a complete agent improvement.

Design exercise: add the missing larger-list literal-search candidate and predict what it could tell you. Write the selection rule before running it. If its source hits tie with the selected candidate, explain which measured quantity chooses the winner and which important costs remain unmeasured.

Continue with grader calibration before trusting a broader success score.

Failure cases

Changing held-out labels must not change the already selected configuration. The empty-development case must refuse selection, while a development tie follows the declared context preference. These checks prevent evaluation data or arbitrary candidate order from silently choosing the winner.

Deployment

Retain the selected retrieval settings, development-family membership and tie rule with the application configuration. Evaluate the fixed candidate before attaching it to a broader policy. If source content or task requirements change, decide which identities and evaluations must be renewed instead of reusing an old selection score.

Scaling

The comparison workload grows with candidates and development cases. Narrow the declared search space using diagnosed retrieval failures before increasing it. More tuning attempts also increase the opportunity to overfit development cases; preserve family separation and use the held-out result as evaluation results, not another tuning input.

Key takeaways

• Retrieval selection uses development data, not held-out labels.
• A tie rule is an explicit policy choice.
• Expected-source hits do not measure a model’s final answer quality.

Best practices

• Freeze the candidate list and tie rule before scoring.
• Keep the selected configuration identifiable.
• Inspect returned context alongside retrieval success.

References

• Anthropic, Demystifying evals for AI agents: inspect the behavior being graded and separate component test results from task outcomes.

Reference checked October 24, 2026. Candidate scores are actual results from the bundled local search, using the fixed synthetic family splits.