Problem
When you need this. Recent turns, matching turns and summary notes keep different information. Compare them on the same questions and token allowance before choosing a rule for your application.
The earlier builds in this series each choose context with one rule. The context budget manager keeps the newest complete turns. Context compaction adds cue-word notes from removed turns. A retriever-style rule would keep the turns that share words with the question. Which one should an application use?
One example cannot answer that. Each rule wins on a conversation that suits it and fails on one that does not. Comparing them needs a fixed evaluation set: the same cases, the same budget, the same counter and the same grader for every strategy, with the per-case results kept, not only the totals.
We will build that comparison. strategies.py holds four selection strategies. evaluate.py loads 12 cases, runs every strategy on every case and grades which facts reach the request. The report is recorded, and a test checks that a fresh run still matches it.
Success check: compare the four strategies under the same input budget and explain a missed or stale fact from each strategy’s per-case results.
You need: Python 3.11 or later and the pinned Gemma tokenizer files from the context budget manager build; prepare.py downloads them once, with no account and no model weights. The tests grade recorded model answers and call no model. Recording new answers needs Ollama with gemma4:e2b.
Concepts
What the grader measures
Each case lists required facts: exact sentences or phrases from the conversation that a correct answer depends on. Some cases also list stale facts: earlier statements that a later turn replaced. The grader looks for each fact in the selected messages by exact text match.
Fact recall is the number of required facts found divided by the number required. A complete case is one where every required fact was found. A case with a stale fact is reported separately, because a replaced decision in the request can lead to a confident wrong answer.
This is a check on the request, not on an answer. It tells you what a model would be able to see. It does not call a model, and a strategy with full recall can still produce a wrong answer. Grading answers needs a model, repeated trials and a separate grader.
Development and held-out splits
The development split is the set of cases you look at while you write the strategies. The held-out split is kept aside and run only after the strategies are fixed. A strategy that does well on development cases but worse on held-out cases was probably shaped to the development cases.
In this build, the held-out split is not blind. The author reports writing the 6 development cases first, using them while writing strategies.py, and writing the 6 held-out cases afterwards, knowing the strategies and their cue list. One commit adds both splits, so the history cannot confirm that order. The cue limit matches words in two held-out cases and in neither the development cases nor the compaction build’s fixture. One held-out fact, in held-pronoun, depends on it; a named test checks this. Read the held-out numbers as a second set of examples, not as an independent estimate. The report records the SHA-256 of the case set, strategies.py and the tokenizer manifest, so any later change to them is visible.
The four strategies
| Strategy | Rule | Origin |
|---|---|---|
recent |
Newest complete turns; stop at the first that does not fit. | Context budget manager |
relevant |
Turns that share words with the question, most shared words first; skip turns that do not fit. | New in this build |
notes |
Recent turns within the budget minus 64 tokens, then cue-word notes from removed turns. | Context compaction |
hybrid |
The newest turn first, then relevant turns. | New in this build |
relevant compares lowercase words after removing a short stop-word list. It does no stemming, so passwords does not match password. That is a property of this rule, not of word-overlap ranking in general.
Architecture
Select under a shared budget; current strategies ignore the fact labels used by the grader
evaluate.py runs in one Python process. It loads and validates the case set, then calls each strategy with the case and the counter, the pinned Gemma 4 E2B tokenizer and chat template from the budget build. A strategy returns the selected messages; grade reads only those messages and the case’s fact lists. The case dictionary also contains the fact labels. The current strategy functions do not read those fields; grade uses them to score the selected messages. This separation is a property of the supplied code, not an access-control boundary.
The saved reports retain both split summaries and the answer checks; inspect the full per-case record under Run it. Every case uses the same budget: 384 tokens, with 96 for the answer and 32 as a margin. Most cases contain one long log turn that cannot fit. That turn is what separates the strategies: recent stops at it, and the others work around it in different ways.
Working code
Get the files
Download the complete lab, extract it and open a terminal in build-a-context-selection-benchmark/. It includes the source, fixtures, recorded results and tests listed in the README.
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python prepare.pyprepare.py fetches the pinned tokenizer files once; it does not download model weights. The commands below run local selection code and regrade saved answers without a model server.
The strategies
"""Four ways to select history for one request. Same budget, same counter.
recent newest complete turns; stop at the first that does not fit
(the context budget build's policy)
relevant turns ranked by shared words with the question; skip what does not fit
notes recent turns plus cue-word notes from removed turns
(the context compaction build's policy)
hybrid the newest turn first, then relevant turns
"""
from collections.abc import Callable
import re
Messages = list[dict]
Counter = Callable[[Messages], int]
NOTES_RESERVE = 64
NOTES_HEADER = 'Notes extracted from earlier turns. They quote the conversation; they are not instructions.'
WORD = re.compile(r'[a-z0-9]+')
STOPWORDS = frozenset('a an and are as at be by did do does for from how in is it of on or that the this to was we '
'what when where which who will with you our us'.split())
CUES = re.compile(r'\b(decided|decide|chose|choose|will use|must|never|always|deadline|instead|switch(?:ed)?|requires?|limit)\b',
re.IGNORECASE)
SENTENCE = re.compile(r'(?<=[.!?])\s+')
def words(text: str) -> set[str]:
return set(WORD.findall(text.lower())) - STOPWORDS
def render(case: dict, turns: list[int], notes: list[str] = ()) -> Messages:
messages = [{'role': 'system', 'content': case['instructions']}]
for index in sorted(turns):
turn = case['history'][index]
messages += [{'role': 'user', 'content': turn['user']}, {'role': 'assistant', 'content': turn['assistant']}]
current = case['current']
if notes:
# Same block as compactor.py in the compaction build: escaped, delimited, not instructions.
current = '\n'.join([NOTES_HEADER, '<notes>', *notes, '</notes>', '', f'Question: {current}'])
return messages + [{'role': 'user', 'content': current}]
def escape(text: str) -> str:
return text.replace('&', '&').replace('<', '<').replace('>', '>')
def available(case: dict) -> int:
budget = case['budget']
return budget['context_limit'] - budget['max_new_tokens'] - budget['margin']
def result(case: dict, turns: list[int], count: Counter, notes: list[str] = ()) -> dict:
messages = render(case, turns, notes)
return {'messages': messages, 'turns': sorted(turns), 'notes': len(notes), 'input_tokens': count(messages)}
def recent(case: dict, count: Counter) -> dict:
kept: list[int] = []
for index in range(len(case['history']) - 1, -1, -1):
if count(render(case, [index, *kept])) > available(case):
break
kept.insert(0, index)
return result(case, kept, count)
def _fill(case: dict, count: Counter, order: list[int], kept: list[int]) -> list[int]:
for index in order:
if index not in kept and count(render(case, [*kept, index])) <= available(case):
kept = [*kept, index]
return kept
def relevant(case: dict, count: Counter) -> dict:
query = words(case['current'])
def overlap(index):
turn = case['history'][index]
return len(query & words(turn['user'] + ' ' + turn['assistant']))
order = sorted((i for i in range(len(case['history'])) if overlap(i)), key=lambda i: (-overlap(i), -i))
return result(case, _fill(case, count, order, []), count)
def hybrid(case: dict, count: Counter) -> dict:
newest = len(case['history']) - 1
kept = _fill(case, count, [newest], []) if newest >= 0 else []
query = words(case['current'])
order = sorted((i for i in range(newest) if query & words(' '.join(case['history'][i].values()))),
key=lambda i: (-len(query & words(' '.join(case['history'][i].values()))), -i))
return result(case, _fill(case, count, order, kept), count)
def notes(case: dict, count: Counter) -> dict:
kept: list[int] = []
for index in range(len(case['history']) - 1, -1, -1):
if count(render(case, [index, *kept])) > available(case) - NOTES_RESERVE:
break
kept.insert(0, index)
candidates = []
for index in range(len(case['history']) - len(kept)):
for role in ('user', 'assistant'):
for sentence in SENTENCE.split(case['history'][index][role].strip()):
line = f'- [turn {index + 1}, {role}] {escape(sentence)}'
if CUES.search(sentence) and line not in candidates:
candidates.append(line)
chosen: list[str] = []
for line in reversed(candidates):
if count(render(case, kept, [line, *chosen])) > available(case):
break
chosen.insert(0, line)
return result(case, kept, count, chosen)
STRATEGIES = {'recent': recent, 'relevant': relevant, 'notes': notes, 'hybrid': hybrid}Every strategy renders turns in conversation order, whatever order it chose them in. notes renders its notes exactly as the compaction build does: escaped, inside one <notes> block, after a line that says they are not instructions. recent and the recent-turn part of notes reimplement the loops from the budget and compaction builds in standard-library Python. Counting still uses the prepared tokenizer files from the budget build. _fill adds turns in a given order and skips any turn that does not fit, which is what lets relevant and hybrid work around a long turn.
The case set
Each case has an ID, a split, a kind, a conversation, a question, a budget, required facts and stale facts. This is the development case with a corrected decision:
{
"id": "dev-correction",
"split": "dev",
"kind": "corrected decision",
"instructions": "Answer only from the conversation. If a fact is missing, say so.",
"current": "Which database will we use?",
"history": [
{
"user": "We will use MySQL for the first version.",
"assistant": "MySQL noted."
},
{
"user": "Switch to PostgreSQL instead of MySQL.",
"assistant": "Switched to PostgreSQL."
},
{
"user": "Here is the migration log: 09:00 migrate retry 1 of 5 after a timeout. 09:01 ...",
"assistant": "The migration log shows only retries."
}
],
"budget": {
"context_limit": 384,
"max_new_tokens": 96,
"margin": 32
},
"required": [
"Switch to PostgreSQL instead of MySQL."
],
"stale": [
"We will use MySQL"
]
}load_cases rejects a case whose required or stale fact does not occur in its history. Without that check, a typo in a fact would make a case impossible to pass, and the report would blame the strategies.
The runner and grader
"""Run every strategy on every case and grade the selected context.
The graders check whether named facts reach the request. They do not call a
model and they do not grade an answer. The run is deterministic: one trial.
"""
import hashlib
import json
from pathlib import Path
import platform
from tokenizer import count_request, counter
from strategies import STRATEGIES, escape
HERE = Path(__file__).parent
CASE_FIELDS = {'id', 'split', 'kind', 'instructions', 'current', 'history', 'budget', 'required', 'stale'}
class CaseError(ValueError):
pass
def load_cases(path: Path) -> list[dict]:
cases = json.loads(path.read_text())
seen = set()
for case in cases:
if set(case) != CASE_FIELDS:
raise CaseError(f'{case.get("id")}: fields must be {sorted(CASE_FIELDS)}')
if case['id'] in seen:
raise CaseError(f'duplicate case id {case["id"]}')
seen.add(case['id'])
if case['split'] not in ('dev', 'held_out'):
raise CaseError(f'{case["id"]}: split must be dev or held_out')
if not case['required']:
raise CaseError(f'{case["id"]}: a case needs at least one required fact')
text = ' '.join(t['user'] + ' ' + t['assistant'] for t in case['history'])
for fact in case['required'] + case['stale']:
if fact not in text:
raise CaseError(f'{case["id"]}: fact {fact!r} does not occur in the history')
return cases
def grade(case: dict, selected: dict) -> dict:
text = '\n'.join(m['content'] for m in selected['messages'])
# Notes are escaped, so look for both spellings of each fact.
present = lambda fact: fact in text or escape(fact) in text
found = [fact for fact in case['required'] if present(fact)]
stale = [fact for fact in case['stale'] if present(fact)]
budget = case['budget']
return {
'case': case['id'],
'split': case['split'],
'kind': case['kind'],
'turns': selected['turns'],
'notes': selected['notes'],
'input_tokens': selected['input_tokens'],
'within_budget': selected['input_tokens'] + budget['max_new_tokens'] + budget['margin'] <= budget['context_limit'],
'required_found': len(found),
'required_total': len(case['required']),
'missing': [fact for fact in case['required'] if fact not in found],
'stale_included': stale,
}
def summarize(rows: list[dict]) -> dict:
found = sum(r['required_found'] for r in rows)
total = sum(r['required_total'] for r in rows)
return {
'cases': len(rows),
'facts_found': found,
'facts_total': total,
'fact_recall': round(found / total, 3),
'complete_cases': sum(r['required_found'] == r['required_total'] for r in rows),
'cases_with_stale_facts': sum(bool(r['stale_included']) for r in rows),
'over_budget': sum(not r['within_budget'] for r in rows),
'mean_input_tokens': round(sum(r['input_tokens'] for r in rows) / len(rows), 1),
}
def run(path: Path = HERE / 'cases.json') -> dict:
cases = load_cases(path)
digest = lambda file: hashlib.sha256(file.read_bytes()).hexdigest()
report = {
'case_set': path.name,
'case_set_sha256': digest(path),
'strategies_sha256': digest(HERE / 'strategies.py'),
'tokenizer_manifest_sha256': digest(HERE / 'tokenizer-manifest.json'),
'environment': {'python': platform.python_version(), 'implementation': platform.python_implementation()},
'counter': counter().profile(),
'trials': 1,
'grader': 'exact substring match of required and stale facts, raw or escaped, in the selected messages',
'summary': {},
'cases': {},
}
for name, strategy in STRATEGIES.items():
rows = [grade(case, strategy(case, count_request)) for case in cases]
report['cases'][name] = rows
report['summary'][name] = {split: summarize([r for r in rows if r['split'] == split])
for split in ('dev', 'held_out')}
return report
if __name__ == '__main__':
print(json.dumps(run(), indent=2, ensure_ascii=False))The report records the case-set file name, the SHA-256 of the case set, strategies.py and the tokenizer manifest, the Python version, the counter, the number of trials and the grader rule. The strategies and the counter are deterministic, so one trial gives the same result every time, and no timing is recorded. The test that compares a fresh run with the recording ignores the environment block, because the results must not depend on it.
Run it
From the directory with the files, run:
python3 evaluate.pyThe recorded summary, by strategy and split:
| Strategy | Split | Facts found | Complete cases | Cases with stale facts | Mean input tokens |
|---|---|---|---|---|---|
recent | dev | 2 of 7 | 1 of 6 | 0 | 49.8 |
recent | held-out | 2 of 7 | 1 of 6 | 0 | 54.2 |
relevant | dev | 5 of 7 | 4 of 6 | 1 | 60.7 |
relevant | held-out | 4 of 7 | 4 of 6 | 1 | 58.7 |
notes | dev | 6 of 7 | 5 of 6 | 1 | 84.7 |
notes | held-out | 6 of 7 | 5 of 6 | 1 | 96 |
hybrid | dev | 5 of 7 | 4 of 6 | 1 | 66.7 |
hybrid | held-out | 4 of 7 | 4 of 6 | 1 | 67 |
Inspect the complete recorded report
{
"case_set": "cases.json",
"case_set_sha256": "30d77344974e5d711695d6c556255f4b221a1992608be81a300306cb483d4d44",
"strategies_sha256": "b703fc62d351ab4ba7f51886d67385418650f2162147be52aaddd9f6048010b2",
"tokenizer_manifest_sha256": "13fdb279d284653bed8b0af8b344eac7c2d0cc5e7a76c0cbc23ae250f5bd3181",
"environment": {
"python": "3.11.11",
"implementation": "CPython"
},
"counter": {
"model": "google/gemma-4-E2B-it",
"revision": "3e22461f65e89153144f8adb70e3b8c2cc9845a7",
"template_sha256": "0a2c8073c878ab1da004bee933a998606537bbb62016310352c7285c3f01c5b5",
"enable_thinking": false,
"count_scope": "chat template with generation prefix; tokenizer files only, no inference"
},
"trials": 1,
"grader": "exact substring match of required and stale facts, raw or escaped, in the selected messages",
"summary": {
"recent": {
"dev": {
"cases": 6,
"facts_found": 2,
"facts_total": 7,
"fact_recall": 0.286,
"complete_cases": 1,
"cases_with_stale_facts": 0,
"over_budget": 0,
"mean_input_tokens": 49.8
},
"held_out": {
"cases": 6,
"facts_found": 2,
"facts_total": 7,
"fact_recall": 0.286,
"complete_cases": 1,
"cases_with_stale_facts": 0,
"over_budget": 0,
"mean_input_tokens": 54.2
}
},
"relevant": {
"dev": {
"cases": 6,
"facts_found": 5,
"facts_total": 7,
"fact_recall": 0.714,
"complete_cases": 4,
"cases_with_stale_facts": 1,
"over_budget": 0,
"mean_input_tokens": 60.7
},
"held_out": {
"cases": 6,
"facts_found": 4,
"facts_total": 7,
"fact_recall": 0.571,
"complete_cases": 4,
"cases_with_stale_facts": 1,
"over_budget": 0,
"mean_input_tokens": 58.7
}
},
"notes": {
"dev": {
"cases": 6,
"facts_found": 6,
"facts_total": 7,
"fact_recall": 0.857,
"complete_cases": 5,
"cases_with_stale_facts": 1,
"over_budget": 0,
"mean_input_tokens": 84.7
},
"held_out": {
"cases": 6,
"facts_found": 6,
"facts_total": 7,
"fact_recall": 0.857,
"complete_cases": 5,
"cases_with_stale_facts": 1,
"over_budget": 0,
"mean_input_tokens": 96.0
}
},
"hybrid": {
"dev": {
"cases": 6,
"facts_found": 5,
"facts_total": 7,
"fact_recall": 0.714,
"complete_cases": 4,
"cases_with_stale_facts": 1,
"over_budget": 0,
"mean_input_tokens": 66.7
},
"held_out": {
"cases": 6,
"facts_found": 4,
"facts_total": 7,
"fact_recall": 0.571,
"complete_cases": 4,
"cases_with_stale_facts": 1,
"over_budget": 0,
"mean_input_tokens": 67.0
}
}
},
"cases": {
"recent": [
{
"case": "dev-latest",
"split": "dev",
"kind": "fact in the newest turn",
"turns": [
1
],
"notes": 0,
"input_tokens": 58,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": []
},
{
"case": "dev-old-decision",
"split": "dev",
"kind": "decision behind a long turn",
"turns": [
2
],
"notes": 0,
"input_tokens": 54,
"within_budget": true,
"required_found": 0,
"required_total": 1,
"missing": [
"We chose PostgreSQL for the application."
],
"stale_included": []
},
{
"case": "dev-correction",
"split": "dev",
"kind": "corrected decision",
"turns": [],
"notes": 0,
"input_tokens": 35,
"within_budget": true,
"required_found": 0,
"required_total": 1,
"missing": [
"Switch to PostgreSQL instead of MySQL."
],
"stale_included": []
},
{
"case": "dev-pronoun",
"split": "dev",
"kind": "fact stated with a pronoun",
"turns": [],
"notes": 0,
"input_tokens": 35,
"within_budget": true,
"required_found": 0,
"required_total": 1,
"missing": [
"exports run hourly"
],
"stale_included": []
},
{
"case": "dev-spread",
"split": "dev",
"kind": "facts spread across turns",
"turns": [
2,
3
],
"notes": 0,
"input_tokens": 81,
"within_budget": true,
"required_found": 1,
"required_total": 2,
"missing": [
"400 days of history"
],
"stale_included": []
},
{
"case": "dev-no-overlap",
"split": "dev",
"kind": "fact with no shared words",
"turns": [],
"notes": 0,
"input_tokens": 36,
"within_budget": true,
"required_found": 0,
"required_total": 1,
"missing": [
"blue-green rollout"
],
"stale_included": []
},
{
"case": "held-latest",
"split": "held_out",
"kind": "fact in the newest turn",
"turns": [
1
],
"notes": 0,
"input_tokens": 62,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": []
},
{
"case": "held-old-decision",
"split": "held_out",
"kind": "decision behind a long turn",
"turns": [
2
],
"notes": 0,
"input_tokens": 51,
"within_budget": true,
"required_found": 0,
"required_total": 1,
"missing": [
"decided to store uploads in object storage"
],
"stale_included": []
},
{
"case": "held-correction",
"split": "held_out",
"kind": "corrected decision",
"turns": [],
"notes": 0,
"input_tokens": 38,
"within_budget": true,
"required_found": 0,
"required_total": 1,
"missing": [
"use Redis streams for the job queue"
],
"stale_included": []
},
{
"case": "held-pronoun",
"split": "held_out",
"kind": "fact stated with a pronoun",
"turns": [],
"notes": 0,
"input_tokens": 36,
"within_budget": true,
"required_found": 0,
"required_total": 1,
"missing": [
"invoice retry limit to three attempts"
],
"stale_included": []
},
{
"case": "held-spread",
"split": "held_out",
"kind": "facts spread across turns",
"turns": [
2,
3
],
"notes": 0,
"input_tokens": 84,
"within_budget": true,
"required_found": 1,
"required_total": 2,
"missing": [
"at least 14 characters"
],
"stale_included": []
},
{
"case": "held-buried",
"split": "held_out",
"kind": "fact inside a long turn",
"turns": [
1
],
"notes": 0,
"input_tokens": 54,
"within_budget": true,
"required_found": 0,
"required_total": 1,
"missing": [
"gateway must allow 200 connections"
],
"stale_included": []
}
],
"relevant": [
{
"case": "dev-latest",
"split": "dev",
"kind": "fact in the newest turn",
"turns": [
1
],
"notes": 0,
"input_tokens": 58,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": []
},
{
"case": "dev-old-decision",
"split": "dev",
"kind": "decision behind a long turn",
"turns": [
0
],
"notes": 0,
"input_tokens": 61,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": []
},
{
"case": "dev-correction",
"split": "dev",
"kind": "corrected decision",
"turns": [
0
],
"notes": 0,
"input_tokens": 57,
"within_budget": true,
"required_found": 0,
"required_total": 1,
"missing": [
"Switch to PostgreSQL instead of MySQL."
],
"stale_included": [
"We will use MySQL"
]
},
{
"case": "dev-pronoun",
"split": "dev",
"kind": "fact stated with a pronoun",
"turns": [
1
],
"notes": 0,
"input_tokens": 57,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": []
},
{
"case": "dev-spread",
"split": "dev",
"kind": "facts spread across turns",
"turns": [
0,
2
],
"notes": 0,
"input_tokens": 95,
"within_budget": true,
"required_found": 2,
"required_total": 2,
"missing": [],
"stale_included": []
},
{
"case": "dev-no-overlap",
"split": "dev",
"kind": "fact with no shared words",
"turns": [],
"notes": 0,
"input_tokens": 36,
"within_budget": true,
"required_found": 0,
"required_total": 1,
"missing": [
"blue-green rollout"
],
"stale_included": []
},
{
"case": "held-latest",
"split": "held_out",
"kind": "fact in the newest turn",
"turns": [
1
],
"notes": 0,
"input_tokens": 62,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": []
},
{
"case": "held-old-decision",
"split": "held_out",
"kind": "decision behind a long turn",
"turns": [
0
],
"notes": 0,
"input_tokens": 61,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": []
},
{
"case": "held-correction",
"split": "held_out",
"kind": "corrected decision",
"turns": [
0,
1
],
"notes": 0,
"input_tokens": 94,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": [
"decided to use RabbitMQ"
]
},
{
"case": "held-pronoun",
"split": "held_out",
"kind": "fact stated with a pronoun",
"turns": [
1
],
"notes": 0,
"input_tokens": 61,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": []
},
{
"case": "held-spread",
"split": "held_out",
"kind": "facts spread across turns",
"turns": [],
"notes": 0,
"input_tokens": 37,
"within_budget": true,
"required_found": 0,
"required_total": 2,
"missing": [
"at least 14 characters",
"expire after 30 minutes"
],
"stale_included": []
},
{
"case": "held-buried",
"split": "held_out",
"kind": "fact inside a long turn",
"turns": [],
"notes": 0,
"input_tokens": 37,
"within_budget": true,
"required_found": 0,
"required_total": 1,
"missing": [
"gateway must allow 200 connections"
],
"stale_included": []
}
],
"notes": [
{
"case": "dev-latest",
"split": "dev",
"kind": "fact in the newest turn",
"turns": [
1
],
"notes": 0,
"input_tokens": 58,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": []
},
{
"case": "dev-old-decision",
"split": "dev",
"kind": "decision behind a long turn",
"turns": [
2
],
"notes": 1,
"input_tokens": 97,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": []
},
{
"case": "dev-correction",
"split": "dev",
"kind": "corrected decision",
"turns": [],
"notes": 3,
"input_tokens": 110,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": [
"We will use MySQL"
]
},
{
"case": "dev-pronoun",
"split": "dev",
"kind": "fact stated with a pronoun",
"turns": [],
"notes": 1,
"input_tokens": 78,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": []
},
{
"case": "dev-spread",
"split": "dev",
"kind": "facts spread across turns",
"turns": [
2,
3
],
"notes": 1,
"input_tokens": 129,
"within_budget": true,
"required_found": 2,
"required_total": 2,
"missing": [],
"stale_included": []
},
{
"case": "dev-no-overlap",
"split": "dev",
"kind": "fact with no shared words",
"turns": [],
"notes": 0,
"input_tokens": 36,
"within_budget": true,
"required_found": 0,
"required_total": 1,
"missing": [
"blue-green rollout"
],
"stale_included": []
},
{
"case": "held-latest",
"split": "held_out",
"kind": "fact in the newest turn",
"turns": [
1
],
"notes": 0,
"input_tokens": 62,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": []
},
{
"case": "held-old-decision",
"split": "held_out",
"kind": "decision behind a long turn",
"turns": [
2
],
"notes": 1,
"input_tokens": 96,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": []
},
{
"case": "held-correction",
"split": "held_out",
"kind": "corrected decision",
"turns": [],
"notes": 1,
"input_tokens": 85,
"within_budget": true,
"required_found": 0,
"required_total": 1,
"missing": [
"use Redis streams for the job queue"
],
"stale_included": [
"decided to use RabbitMQ"
]
},
{
"case": "held-pronoun",
"split": "held_out",
"kind": "fact stated with a pronoun",
"turns": [],
"notes": 1,
"input_tokens": 82,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": []
},
{
"case": "held-spread",
"split": "held_out",
"kind": "facts spread across turns",
"turns": [
2,
3
],
"notes": 1,
"input_tokens": 131,
"within_budget": true,
"required_found": 2,
"required_total": 2,
"missing": [],
"stale_included": []
},
{
"case": "held-buried",
"split": "held_out",
"kind": "fact inside a long turn",
"turns": [
1
],
"notes": 2,
"input_tokens": 120,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": []
}
],
"hybrid": [
{
"case": "dev-latest",
"split": "dev",
"kind": "fact in the newest turn",
"turns": [
1
],
"notes": 0,
"input_tokens": 58,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": []
},
{
"case": "dev-old-decision",
"split": "dev",
"kind": "decision behind a long turn",
"turns": [
0,
2
],
"notes": 0,
"input_tokens": 80,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": []
},
{
"case": "dev-correction",
"split": "dev",
"kind": "corrected decision",
"turns": [
0
],
"notes": 0,
"input_tokens": 57,
"within_budget": true,
"required_found": 0,
"required_total": 1,
"missing": [
"Switch to PostgreSQL instead of MySQL."
],
"stale_included": [
"We will use MySQL"
]
},
{
"case": "dev-pronoun",
"split": "dev",
"kind": "fact stated with a pronoun",
"turns": [
1
],
"notes": 0,
"input_tokens": 57,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": []
},
{
"case": "dev-spread",
"split": "dev",
"kind": "facts spread across turns",
"turns": [
0,
2,
3
],
"notes": 0,
"input_tokens": 112,
"within_budget": true,
"required_found": 2,
"required_total": 2,
"missing": [],
"stale_included": []
},
{
"case": "dev-no-overlap",
"split": "dev",
"kind": "fact with no shared words",
"turns": [],
"notes": 0,
"input_tokens": 36,
"within_budget": true,
"required_found": 0,
"required_total": 1,
"missing": [
"blue-green rollout"
],
"stale_included": []
},
{
"case": "held-latest",
"split": "held_out",
"kind": "fact in the newest turn",
"turns": [
1
],
"notes": 0,
"input_tokens": 62,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": []
},
{
"case": "held-old-decision",
"split": "held_out",
"kind": "decision behind a long turn",
"turns": [
0,
2
],
"notes": 0,
"input_tokens": 77,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": []
},
{
"case": "held-correction",
"split": "held_out",
"kind": "corrected decision",
"turns": [
0,
1
],
"notes": 0,
"input_tokens": 94,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": [
"decided to use RabbitMQ"
]
},
{
"case": "held-pronoun",
"split": "held_out",
"kind": "fact stated with a pronoun",
"turns": [
1
],
"notes": 0,
"input_tokens": 61,
"within_budget": true,
"required_found": 1,
"required_total": 1,
"missing": [],
"stale_included": []
},
{
"case": "held-spread",
"split": "held_out",
"kind": "facts spread across turns",
"turns": [
3
],
"notes": 0,
"input_tokens": 54,
"within_budget": true,
"required_found": 0,
"required_total": 2,
"missing": [
"at least 14 characters",
"expire after 30 minutes"
],
"stale_included": []
},
{
"case": "held-buried",
"split": "held_out",
"kind": "fact inside a long turn",
"turns": [
1
],
"notes": 0,
"input_tokens": 54,
"within_budget": true,
"required_found": 0,
"required_total": 1,
"missing": [
"gateway must allow 200 connections"
],
"stale_included": []
}
]
}
}Read the failures, not only the totals
recent finds 2 of 7 development facts. In most cases a long log turn sits between the question and the fact, and recent stops there. It never includes a stale fact, because it never reaches the older turns that hold them.
notes finds the most facts on both splits, 6 of 7 and 6 of 7, but it includes a stale fact on each split. One of its held-out facts, in held-pronoun, depends on the cue limit, which no development case tests. In held-correction, the correction “Actually, use Redis streams…” has no cue word, so the note keeps only the replaced RabbitMQ decision: 0 of 1 required facts and one stale fact.
relevant drops from 5 development facts to 4 held-out facts. In held-spread, the question says password and session, and the turns say Passwords and Sessions. Without stemming, no turn shares a word with the question. In dev-no-overlap, the question asks how to deploy and the fact says rollout; all four strategies miss it.
hybrid matches relevant on facts found, and it uses more tokens because it always tries the newest turn first.
Compare the recorded model answers
Fact recall shows what each strategy puts in the request. The saved answers show what the model answered from it. The recording command, answer_check.py record, sends each strategy’s request for the 6 held-out cases to gemma4:e2b, the Gemma 4 E2B model under the Apache 2.0 license, through a local Ollama server pinned by digest, 5 seeds each. Only held-out cases are sent, because the development cases shaped the strategies.
answer-keys.json was written from each case’s required and stale facts before any answer was recorded. An answer passes if it contains every required fact and presents no replaced decision as current. It fails otherwise, and an answer that says the fact is missing is counted separately, because saying so is better than guessing. It is incomplete if there is no answer to grade. These are the outcome labels from the evaluation foundations.
Inspect answer_check.py and answer-keys.json
"""Send each strategy's held-out requests to the pinned Gemma 4 E2B model and grade the answers.
python answer_check.py record # needs Ollama with gemma4:e2b and the pinned tokenizer
python answer_check.py # offline: grades the recorded answers again
Only the six held-out cases are sent: the development cases shaped the
strategies, so answers on them would say less. Each case and strategy is
sent with five seeds. answer-keys.json, written from the cases' required
and stale facts before any answer was recorded, defines the grade:
incomplete no answer to grade
fail a required fact is missing, or a replaced decision is presented as current
pass every required fact, and no replaced decision presented as current
An answer that says a fact is missing is still a fail, but it is counted
separately: saying "not in the conversation" is better than guessing.
120 answers from a small model illustrate how the strategies behave; they
are not a measure of answer quality.
"""
import json
from pathlib import Path
import re
import sys
HERE = Path(__file__).parent
KEYS = json.loads((HERE / 'answer-keys.json').read_text())['keys']
SENTENCE = re.compile(r'(?<=[.!?])\s+|\n+')
REPLACED = re.compile(r'replac|instead|not |no longer|switch|changed|actually|previous|originally|earlier', re.I)
SAYS_MISSING = re.compile(r"(do not|don't|does not|doesn't) (have|say|mention|state|specify|contain)|not (mention|state|specif|provide|includ|contain)|no information|not enough information|missing|not in the conversation|cannot (find|determine)|provide (more |the )?(context|conversation)",
re.I)
def stale_as_current(case_id: str, text: str) -> bool:
for pattern in KEYS[case_id]['stale']:
for sentence in SENTENCE.split(text):
if re.search(pattern, sentence, re.I) and not REPLACED.search(sentence) \
and not any(re.search(m, sentence, re.I) for m in KEYS[case_id]['must']):
return True
return False
def grade(case_id: str, answer: dict) -> tuple[str, str]:
text = answer['answer']
if not text.strip():
return 'incomplete', 'no answer'
if stale_as_current(case_id, text):
return 'fail', 'presents a replaced decision as current'
missing = [m for m in KEYS[case_id]['must'] if not re.search(m, text, re.I)]
if missing:
return 'fail', 'says the fact is missing' if SAYS_MISSING.search(text) else 'required fact absent'
return 'pass', 'every required fact'
def summarize(recorded: dict) -> dict:
out = {}
for strategy, cases in recorded['answers'].items():
verdicts = [grade(case_id, a) for case_id, answers in cases.items() for a in answers]
out[strategy] = {**{v: [g[0] for g in verdicts].count(v) for v in ('pass', 'fail', 'incomplete')},
'said_missing': sum(g[1] == 'says the fact is missing' for g in verdicts),
'stale_as_current': sum(g[1] == 'presents a replaced decision as current' for g in verdicts),
'cases_passed_every_trial': sum(all(grade(c, a)[0] == 'pass' for a in answers)
for c, answers in cases.items())}
return out
def record() -> dict:
from answer_model import MODEL, AnswerModel
from evaluate import load_cases
from strategies import STRATEGIES
from tokenizer import count_request
cases = [c for c in load_cases(HERE / 'cases.json') if c['split'] == 'held_out']
model = AnswerModel()
answers = {}
for name, strategy in STRATEGIES.items():
answers[name] = {}
for case in cases:
messages = strategy(case, count_request)['messages']
answers[name][case['id']] = [model.answer(messages, [], num_ctx=case['budget']['context_limit'],
num_predict=case['budget']['max_new_tokens'], seed=seed)
for seed in MODEL['seeds']]
return {'model': MODEL, 'answers': answers}
if __name__ == '__main__':
if sys.argv[1:] == ['record']:
(HERE / 'answer-result.json').write_text(json.dumps(record(), indent=1, ensure_ascii=False) + '\n')
recorded = json.loads((HERE / 'answer-result.json').read_text())
graded = {s: {c: [grade(c, a)[0] for a in answers] for c, answers in cases.items()}
for s, cases in recorded['answers'].items()}
print(json.dumps({'summary': summarize(recorded), 'graded': graded}, indent=2)){
"note": "Written from each held-out case's required and stale facts before any answer was recorded. A pattern is a case-insensitive regular expression; every 'must' pattern has to match.",
"keys": {
"held-latest": {"must": ["mateo"], "stale": []},
"held-old-decision": {"must": ["object storage"], "stale": []},
"held-correction": {"must": ["redis"], "stale": ["rabbitmq"]},
"held-pronoun": {"must": ["\\bthree\\b|\\b3\\b"], "stale": []},
"held-spread": {"must": ["\\b14\\b", "\\b30\\b"], "stale": []},
"held-buried": {"must": ["\\b200\\b"], "stale": []}
}
}Run the following command to load the saved answers and apply the fixed string and regular-expression checks. It makes no model request.
python3 answer_check.py| Strategy | Pass | Fail | Of which said missing | Replaced decision as current | Cases passed in every trial |
|---|---|---|---|---|---|
recent | 5 of 30 | 25 | 24 | 0 | 1 of 6 |
relevant | 20 of 30 | 10 | 8 | 0 | 4 of 6 |
notes | 25 of 30 | 5 | 0 | 5 | 5 of 6 |
hybrid | 20 of 30 | 10 | 9 | 0 | 4 of 6 |
Read every recorded answer
{
"model": {
"checked_on": "2026-10-20",
"publisher_model": "google/gemma-4-E2B-it",
"license": "Apache-2.0",
"ollama_tag": "gemma4:e2b",
"ollama_digest": "b37049369adfe3d2b653af0ab301a062ca5cbe96ace6aa0d3f2559ee9b563fc2",
"ollama_version": "0.32.14",
"quantization": "Q4_K_M",
"parameters": "4.6B total (E2B: about 2B effective)",
"options": {
"temperature": 1.0,
"top_k": 64,
"top_p": 0.95
},
"think": false,
"trials": 5,
"seeds": [
1,
2,
3,
4,
5
]
},
"answers": {
"recent": {
"held-latest": [
{
"seed": 1,
"answer": "Mateo is on call this week.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 62,
"answer_tokens": 9
},
{
"seed": 2,
"answer": "The on-call engineer this week is Mateo.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 62,
"answer_tokens": 11
},
{
"seed": 3,
"answer": "The on-call engineer this week is Mateo.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 62,
"answer_tokens": 11
},
{
"seed": 4,
"answer": "Mateo is on call this week.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 62,
"answer_tokens": 9
},
{
"seed": 5,
"answer": "The on-call engineer this week is Mateo.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 62,
"answer_tokens": 11
}
],
"held-old-decision": [
{
"seed": 1,
"answer": "I do not have information about where uploads are stored.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 51,
"answer_tokens": 12
},
{
"seed": 2,
"answer": "I do not have information about where uploads are stored.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 51,
"answer_tokens": 12
},
{
"seed": 3,
"answer": "I do not have information about where uploads are stored.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 51,
"answer_tokens": 12
},
{
"seed": 4,
"answer": "I do not have information about where uploads are stored.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 51,
"answer_tokens": 12
},
{
"seed": 5,
"answer": "I do not have information about where uploads are stored.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 51,
"answer_tokens": 12
}
],
"held-correction": [
{
"seed": 1,
"answer": "Please provide more context so I can answer your question.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 38,
"answer_tokens": 12
},
{
"seed": 2,
"answer": "I do not have that information.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 38,
"answer_tokens": 8
},
{
"seed": 3,
"answer": "I do not have that information.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 38,
"answer_tokens": 8
},
{
"seed": 4,
"answer": "Please provide more context so I can answer your question.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 38,
"answer_tokens": 12
},
{
"seed": 5,
"answer": "I do not have that information.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 38,
"answer_tokens": 8
}
],
"held-pronoun": [
{
"seed": 1,
"answer": "I do not have that information.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 36,
"answer_tokens": 8
},
{
"seed": 2,
"answer": "I do not have that information.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 36,
"answer_tokens": 8
},
{
"seed": 3,
"answer": "I do not have that information.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 36,
"answer_tokens": 8
},
{
"seed": 4,
"answer": "I do not have that information.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 36,
"answer_tokens": 8
},
{
"seed": 5,
"answer": "I do not have that information.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 36,
"answer_tokens": 8
}
],
"held-spread": [
{
"seed": 1,
"answer": "I do not have information about the password and session rules.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 84,
"answer_tokens": 13
},
{
"seed": 2,
"answer": "I do not have information about the password and session rules.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 84,
"answer_tokens": 13
},
{
"seed": 3,
"answer": "I do not have information about the password and session rules.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 84,
"answer_tokens": 13
},
{
"seed": 4,
"answer": "I do not have information about the password and session rules.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 84,
"answer_tokens": 13
},
{
"seed": 5,
"answer": "I do not have information about the password and session rules.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 84,
"answer_tokens": 13
}
],
"held-buried": [
{
"seed": 1,
"answer": "What connection limit must the gateway allow?",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 54,
"answer_tokens": 9
},
{
"seed": 2,
"answer": "I do not have that information.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 54,
"answer_tokens": 8
},
{
"seed": 3,
"answer": "I do not have that information.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 54,
"answer_tokens": 8
},
{
"seed": 4,
"answer": "Please provide more context about what you are referring to so I can answer your question.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 54,
"answer_tokens": 18
},
{
"seed": 5,
"answer": "I do not have that information.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 54,
"answer_tokens": 8
}
]
},
"relevant": {
"held-latest": [
{
"seed": 1,
"answer": "Mateo is on call this week.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 62,
"answer_tokens": 9
},
{
"seed": 2,
"answer": "The on-call engineer this week is Mateo.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 62,
"answer_tokens": 11
},
{
"seed": 3,
"answer": "The on-call engineer this week is Mateo.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 62,
"answer_tokens": 11
},
{
"seed": 4,
"answer": "Mateo is on call this week.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 62,
"answer_tokens": 9
},
{
"seed": 5,
"answer": "The on-call engineer this week is Mateo.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 62,
"answer_tokens": 11
}
],
"held-old-decision": [
{
"seed": 1,
"answer": "You decided to store uploads in object storage.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 61,
"answer_tokens": 10
},
{
"seed": 2,
"answer": "We decided to store uploads in object storage.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 61,
"answer_tokens": 10
},
{
"seed": 3,
"answer": "We decided to store uploads in object storage.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 61,
"answer_tokens": 10
},
{
"seed": 4,
"answer": "We decided to store uploads in object storage.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 61,
"answer_tokens": 10
},
{
"seed": 5,
"answer": "We decided to store uploads in object storage.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 61,
"answer_tokens": 10
}
],
"held-correction": [
{
"seed": 1,
"answer": "We use Redis streams for the job queue.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 94,
"answer_tokens": 10
},
{
"seed": 2,
"answer": "We use Redis streams for the job queue.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 94,
"answer_tokens": 10
},
{
"seed": 3,
"answer": "We use Redis streams for the job queue.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 94,
"answer_tokens": 10
},
{
"seed": 4,
"answer": "We use Redis streams for the job queue.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 94,
"answer_tokens": 10
},
{
"seed": 5,
"answer": "We use Redis streams for the job queue.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 94,
"answer_tokens": 10
}
],
"held-pronoun": [
{
"seed": 1,
"answer": "Invoices retry three times.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 61,
"answer_tokens": 7
},
{
"seed": 2,
"answer": "Invoices retry three times.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 61,
"answer_tokens": 7
},
{
"seed": 3,
"answer": "Invoices retry three times.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 61,
"answer_tokens": 7
},
{
"seed": 4,
"answer": "Invoices retry three times.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 61,
"answer_tokens": 7
},
{
"seed": 5,
"answer": "Invoices retry three times.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 61,
"answer_tokens": 7
}
],
"held-spread": [
{
"seed": 1,
"answer": "I do not have any information about password and session rules.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 37,
"answer_tokens": 13
},
{
"seed": 2,
"answer": "Please provide more context so I can answer your question.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 37,
"answer_tokens": 12
},
{
"seed": 3,
"answer": "Please provide the context for what you are referring to.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 37,
"answer_tokens": 12
},
{
"seed": 4,
"answer": "I do not have any information about password and session rules.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 37,
"answer_tokens": 13
},
{
"seed": 5,
"answer": "Please provide the conversation you are referring to so I can answer your question.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 37,
"answer_tokens": 16
}
],
"held-buried": [
{
"seed": 1,
"answer": "What connection limit must the gateway allow?",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 37,
"answer_tokens": 9
},
{
"seed": 2,
"answer": "Please provide the context for your question.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 37,
"answer_tokens": 9
},
{
"seed": 3,
"answer": "Please provide more context.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 37,
"answer_tokens": 6
},
{
"seed": 4,
"answer": "What connection limit must the gateway allow?",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 37,
"answer_tokens": 9
},
{
"seed": 5,
"answer": "Please provide more context.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 37,
"answer_tokens": 6
}
]
},
"notes": {
"held-latest": [
{
"seed": 1,
"answer": "Mateo is on call this week.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 62,
"answer_tokens": 9
},
{
"seed": 2,
"answer": "The on-call engineer this week is Mateo.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 62,
"answer_tokens": 11
},
{
"seed": 3,
"answer": "The on-call engineer this week is Mateo.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 62,
"answer_tokens": 11
},
{
"seed": 4,
"answer": "Mateo is on call this week.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 62,
"answer_tokens": 9
},
{
"seed": 5,
"answer": "The on-call engineer this week is Mateo.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 62,
"answer_tokens": 11
}
],
"held-old-decision": [
{
"seed": 1,
"answer": "We decided to store uploads in object storage.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 96,
"answer_tokens": 10
},
{
"seed": 2,
"answer": "We decided to store uploads in object storage.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 96,
"answer_tokens": 10
},
{
"seed": 3,
"answer": "We decided to store uploads in object storage.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 96,
"answer_tokens": 10
},
{
"seed": 4,
"answer": "We decided to store uploads in object storage.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 96,
"answer_tokens": 10
},
{
"seed": 5,
"answer": "We decided to store uploads in object storage.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 96,
"answer_tokens": 10
}
],
"held-correction": [
{
"seed": 1,
"answer": "We decided to use RabbitMQ for the job queue.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 85,
"answer_tokens": 12
},
{
"seed": 2,
"answer": "We decided to use RabbitMQ for the job queue.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 85,
"answer_tokens": 12
},
{
"seed": 3,
"answer": "We decided to use RabbitMQ for the job queue.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 85,
"answer_tokens": 12
},
{
"seed": 4,
"answer": "We decided to use RabbitMQ for the job queue.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 85,
"answer_tokens": 12
},
{
"seed": 5,
"answer": "We decided to use RabbitMQ for the job queue.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 85,
"answer_tokens": 12
}
],
"held-pronoun": [
{
"seed": 1,
"answer": "She set the invoice retry limit to three attempts.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 82,
"answer_tokens": 11
},
{
"seed": 2,
"answer": "She set the invoice retry limit to three attempts.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 82,
"answer_tokens": 11
},
{
"seed": 3,
"answer": "She set the invoice retry limit to three attempts.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 82,
"answer_tokens": 11
},
{
"seed": 4,
"answer": "She set the invoice retry limit to three attempts.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 82,
"answer_tokens": 11
},
{
"seed": 5,
"answer": "She set the invoice retry limit to three attempts.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 82,
"answer_tokens": 11
}
],
"held-spread": [
{
"seed": 1,
"answer": "Passwords must have at least 14 characters, and sessions must expire after 30 minutes of inactivity.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 131,
"answer_tokens": 23
},
{
"seed": 2,
"answer": "Passwords must have at least 14 characters, and sessions must expire after 30 minutes of inactivity.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 131,
"answer_tokens": 23
},
{
"seed": 3,
"answer": "Passwords must have at least 14 characters, and sessions must expire after 30 minutes of inactivity.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 131,
"answer_tokens": 23
},
{
"seed": 4,
"answer": "Passwords must have at least 14 characters, and sessions must expire after 30 minutes of inactivity.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 131,
"answer_tokens": 23
},
{
"seed": 5,
"answer": "Passwords must have at least 14 characters, and sessions must expire after 30 minutes of inactivity.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 131,
"answer_tokens": 23
}
],
"held-buried": [
{
"seed": 1,
"answer": "The gateway must allow 200 connections.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 120,
"answer_tokens": 11
},
{
"seed": 2,
"answer": "The gateway must allow 200 connections.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 120,
"answer_tokens": 11
},
{
"seed": 3,
"answer": "The gateway must allow 200 connections.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 120,
"answer_tokens": 11
},
{
"seed": 4,
"answer": "The gateway must allow 200 connections.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 120,
"answer_tokens": 11
},
{
"seed": 5,
"answer": "The gateway must allow 200 connections.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 120,
"answer_tokens": 11
}
]
},
"hybrid": {
"held-latest": [
{
"seed": 1,
"answer": "Mateo is on call this week.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 62,
"answer_tokens": 9
},
{
"seed": 2,
"answer": "The on-call engineer this week is Mateo.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 62,
"answer_tokens": 11
},
{
"seed": 3,
"answer": "The on-call engineer this week is Mateo.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 62,
"answer_tokens": 11
},
{
"seed": 4,
"answer": "Mateo is on call this week.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 62,
"answer_tokens": 9
},
{
"seed": 5,
"answer": "The on-call engineer this week is Mateo.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 62,
"answer_tokens": 11
}
],
"held-old-decision": [
{
"seed": 1,
"answer": "We decided to store uploads in object storage.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 77,
"answer_tokens": 10
},
{
"seed": 2,
"answer": "We decided to store uploads in object storage.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 77,
"answer_tokens": 10
},
{
"seed": 3,
"answer": "We decided to store uploads in object storage.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 77,
"answer_tokens": 10
},
{
"seed": 4,
"answer": "We decided to store uploads in object storage.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 77,
"answer_tokens": 10
},
{
"seed": 5,
"answer": "We decided to store uploads in object storage.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 77,
"answer_tokens": 10
}
],
"held-correction": [
{
"seed": 1,
"answer": "We use Redis streams for the job queue.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 94,
"answer_tokens": 10
},
{
"seed": 2,
"answer": "We use Redis streams for the job queue.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 94,
"answer_tokens": 10
},
{
"seed": 3,
"answer": "We use Redis streams for the job queue.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 94,
"answer_tokens": 10
},
{
"seed": 4,
"answer": "We use Redis streams for the job queue.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 94,
"answer_tokens": 10
},
{
"seed": 5,
"answer": "We use Redis streams for the job queue.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 94,
"answer_tokens": 10
}
],
"held-pronoun": [
{
"seed": 1,
"answer": "Invoices retry three times.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 61,
"answer_tokens": 7
},
{
"seed": 2,
"answer": "Invoices retry three times.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 61,
"answer_tokens": 7
},
{
"seed": 3,
"answer": "Invoices retry three times.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 61,
"answer_tokens": 7
},
{
"seed": 4,
"answer": "Invoices retry three times.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 61,
"answer_tokens": 7
},
{
"seed": 5,
"answer": "Invoices retry three times.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 61,
"answer_tokens": 7
}
],
"held-spread": [
{
"seed": 1,
"answer": "Please provide more context so I can answer your question.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 54,
"answer_tokens": 12
},
{
"seed": 2,
"answer": "I do not have any information about password and session rules.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 54,
"answer_tokens": 13
},
{
"seed": 3,
"answer": "I do not have any information about password and session rules.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 54,
"answer_tokens": 13
},
{
"seed": 4,
"answer": "Please provide more context so I can answer your question.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 54,
"answer_tokens": 12
},
{
"seed": 5,
"answer": "I do not have any information about password and session rules.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 54,
"answer_tokens": 13
}
],
"held-buried": [
{
"seed": 1,
"answer": "What connection limit must the gateway allow?",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 54,
"answer_tokens": 9
},
{
"seed": 2,
"answer": "I do not have that information.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 54,
"answer_tokens": 8
},
{
"seed": 3,
"answer": "I do not have that information.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 54,
"answer_tokens": 8
},
{
"seed": 4,
"answer": "Please provide more context about what you are referring to so I can answer your question.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 54,
"answer_tokens": 18
},
{
"seed": 5,
"answer": "I do not have that information.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 54,
"answer_tokens": 8
}
]
}
}
}Inspect every grade
{
"summary": {
"recent": {
"pass": 5,
"fail": 25,
"incomplete": 0,
"said_missing": 24,
"stale_as_current": 0,
"cases_passed_every_trial": 1
},
"relevant": {
"pass": 20,
"fail": 10,
"incomplete": 0,
"said_missing": 8,
"stale_as_current": 0,
"cases_passed_every_trial": 4
},
"notes": {
"pass": 25,
"fail": 5,
"incomplete": 0,
"said_missing": 0,
"stale_as_current": 5,
"cases_passed_every_trial": 5
},
"hybrid": {
"pass": 20,
"fail": 10,
"incomplete": 0,
"said_missing": 9,
"stale_as_current": 0,
"cases_passed_every_trial": 4
}
},
"graded": {
"recent": {
"held-latest": [
"pass",
"pass",
"pass",
"pass",
"pass"
],
"held-old-decision": [
"fail",
"fail",
"fail",
"fail",
"fail"
],
"held-correction": [
"fail",
"fail",
"fail",
"fail",
"fail"
],
"held-pronoun": [
"fail",
"fail",
"fail",
"fail",
"fail"
],
"held-spread": [
"fail",
"fail",
"fail",
"fail",
"fail"
],
"held-buried": [
"fail",
"fail",
"fail",
"fail",
"fail"
]
},
"relevant": {
"held-latest": [
"pass",
"pass",
"pass",
"pass",
"pass"
],
"held-old-decision": [
"pass",
"pass",
"pass",
"pass",
"pass"
],
"held-correction": [
"pass",
"pass",
"pass",
"pass",
"pass"
],
"held-pronoun": [
"pass",
"pass",
"pass",
"pass",
"pass"
],
"held-spread": [
"fail",
"fail",
"fail",
"fail",
"fail"
],
"held-buried": [
"fail",
"fail",
"fail",
"fail",
"fail"
]
},
"notes": {
"held-latest": [
"pass",
"pass",
"pass",
"pass",
"pass"
],
"held-old-decision": [
"pass",
"pass",
"pass",
"pass",
"pass"
],
"held-correction": [
"fail",
"fail",
"fail",
"fail",
"fail"
],
"held-pronoun": [
"pass",
"pass",
"pass",
"pass",
"pass"
],
"held-spread": [
"pass",
"pass",
"pass",
"pass",
"pass"
],
"held-buried": [
"pass",
"pass",
"pass",
"pass",
"pass"
]
},
"hybrid": {
"held-latest": [
"pass",
"pass",
"pass",
"pass",
"pass"
],
"held-old-decision": [
"pass",
"pass",
"pass",
"pass",
"pass"
],
"held-correction": [
"pass",
"pass",
"pass",
"pass",
"pass"
],
"held-pronoun": [
"pass",
"pass",
"pass",
"pass",
"pass"
],
"held-spread": [
"fail",
"fail",
"fail",
"fail",
"fail"
],
"held-buried": [
"fail",
"fail",
"fail",
"fail",
"fail"
]
}
}
}The answers follow the fact-recall rows closely. When a required fact reached the request, the model used it; when it did not, the model mostly said so instead of guessing: 24 of recent’s 25 failed answers say the information is missing. The exception is the stale note. notes passes 25 of 30 answers, the most, and fails exactly the 5 held-correction answers, which all give the replaced RabbitMQ decision that its note kept. Where both the correction and the replaced decision reached the request, in relevant and hybrid, the model gave the correction in 5 and 5 of 5 answers: a stale fact did harm only when it arrived without its correction.
These are 120 answers from one small model on 6 held-out cases that were not written blind: an illustration, not a ranking. They do show the two failure modes a fact-recall gate predicts, a missing fact and a stale one, appearing in answers.
Check your understanding
Read the explanation
A held-out split measures a strategy on cases that did not shape it. Once you tune on it, it becomes a development split, and you need fresh held-out cases.
Failure cases
Run the tests:
python3 -m unittest -v test_evaluate.py test_answer_check.pyThe suite has 19 tests. The first runs evaluate.py and compares its output, apart from the environment block, with the recording. Others check that each split has 6 cases, that notes uses the compaction build’s escaped <notes> block, that an escaped fact still counts as found, and that the held-out notes result depends on the cue limit. The rest cover these cases:
| Case | Expected behavior |
|---|---|
| The newest turns hide an older fact behind a long turn | recent keeps only the turns after the long turn and misses the fact. |
| The question and the fact share no words | relevant finds no required fact in dev-no-overlap and held-spread. |
| A correction has no cue word | notes misses the correction and includes the replaced decision. |
| A replaced fact reaches the request | The grader lists it in stale_included, separately from missing facts. |
| A selection exceeds the budget | Every strategy stays within budget on every case. |
| A case names a fact the history never states | Reject the case set. Also reject duplicate IDs, unknown splits, cases without required facts and extra fields. |
| The case set or the code changes after recording | A recorded SHA-256 hash no longer matches its file. |
| A strategy changes the cases | No strategy changes its input. |
| Answer keys that do not match the held-out cases | Keys cover exactly the held-out cases. |
| An answer that presents a replaced decision as current | Grade it fail. |
| An answer with only some of the required facts | Grade it fail; every required fact is needed. |
| An answer that says the fact is missing | Grade it fail and count it separately; an empty answer is incomplete. |
| An answer model that is not the pinned build | Refuse before sending any request. |
Inspect the tests
import copy
import hashlib
import json
from pathlib import Path
import re
import subprocess
import sys
import tempfile
import unittest
from tokenizer import count_request
from evaluate import CaseError, grade, load_cases, run
import strategies
from strategies import STRATEGIES, notes, recent, relevant
HERE = Path(__file__).parent
def cases():
return {case['id']: case for case in load_cases(HERE / 'cases.json')}
class EvaluationTest(unittest.TestCase):
def write_cases(self, data):
handle = tempfile.NamedTemporaryFile('w', suffix='.json', delete=False)
json.dump(data, handle)
handle.close()
self.addCleanup(Path(handle.name).unlink)
return Path(handle.name)
def test_fixture_matches_recorded_result(self):
output = subprocess.run([sys.executable, 'evaluate.py'], cwd=HERE, check=True,
capture_output=True, text=True).stdout
fresh = json.loads(output)
recorded = json.loads((HERE / 'evaluation-result.json').read_text())
# The environment says where the recording was made; the results must not depend on it.
self.assertEqual(set(recorded.pop('environment')), {'python', 'implementation'})
fresh.pop('environment')
self.assertEqual(fresh, recorded)
def test_case_set_and_code_hashes_are_recorded(self):
recorded = json.loads((HERE / 'evaluation-result.json').read_text())
for key, name in (('case_set_sha256', 'cases.json'), ('strategies_sha256', 'strategies.py'),
('tokenizer_manifest_sha256', 'tokenizer-manifest.json')):
self.assertEqual(recorded[key], hashlib.sha256((HERE / name).read_bytes()).hexdigest(), name)
def test_splits_are_reported_separately(self):
report = run()
for summary in report['summary'].values():
self.assertEqual(summary['dev']['cases'], 6)
self.assertEqual(summary['held_out']['cases'], 6)
def test_every_selection_fits_the_budget(self):
for name, rows in run()['cases'].items():
for row in rows:
with self.subTest(strategy=name, case=row['case']):
self.assertTrue(row['within_budget'])
def test_recent_stops_at_a_long_turn(self):
result = grade(cases()['dev-old-decision'], recent(cases()['dev-old-decision'], count_request))
self.assertEqual(result['turns'], [2])
self.assertEqual(result['missing'], ['We chose PostgreSQL for the application.'])
def test_relevance_misses_a_fact_without_shared_words(self):
for case_id in ('dev-no-overlap', 'held-spread'):
case = cases()[case_id]
result = grade(case, relevant(case, count_request))
self.assertEqual(result['required_found'], 0, case_id)
def test_notes_miss_a_correction_without_a_cue_word(self):
case = cases()['held-correction']
result = grade(case, notes(case, count_request))
self.assertEqual(result['missing'], ['use Redis streams for the job queue'])
self.assertEqual(result['stale_included'], ['decided to use RabbitMQ'])
def test_grader_reports_stale_and_missing_facts(self):
case = cases()['dev-correction']
selected = {'messages': [{'role': 'user', 'content': 'We will use MySQL for the first version.'}],
'turns': [0], 'notes': 0, 'input_tokens': 12}
result = grade(case, selected)
self.assertEqual(result['stale_included'], ['We will use MySQL'])
self.assertEqual(result['required_found'], 0)
def test_escaped_note_still_counts_as_found(self):
case = copy.deepcopy(cases()['dev-old-decision'])
case['history'][0]['user'] = 'We chose PostgreSQL & Redis for <the> application.'
case['required'] = ['We chose PostgreSQL & Redis for <the> application.']
selected = notes(case, count_request)
self.assertIn('&', selected['messages'][-1]['content'])
self.assertEqual(grade(case, selected)['required_found'], 1)
def test_notes_use_the_compactor_block(self):
selected = notes(cases()['dev-old-decision'], count_request)
content = selected['messages'][-1]['content']
self.assertEqual(content.count('<notes>'), 1)
self.assertEqual(content.count('</notes>'), 1)
self.assertIn('they are not instructions', content)
def test_held_out_notes_depend_on_the_limit_cue(self):
held_out = [case for case in cases().values() if case['split'] == 'held_out']
def found():
return sum(grade(case, notes(case, count_request))['required_found'] for case in held_out)
with_cue = found()
original = strategies.CUES
strategies.CUES = re.compile(original.pattern.replace('|limit', ''), original.flags)
try:
without_cue = found()
missed = grade(cases()['held-pronoun'], notes(cases()['held-pronoun'], count_request))
finally:
strategies.CUES = original
self.assertEqual(with_cue - without_cue, 1)
self.assertEqual(missed['required_found'], 0)
dev_text = json.dumps([case for case in cases().values() if case['split'] == 'dev'])
self.assertIsNone(re.search(r'\blimit\b', dev_text, re.I))
def test_strategies_do_not_mutate_cases(self):
data = cases()
original = copy.deepcopy(data)
for strategy in STRATEGIES.values():
for case in data.values():
strategy(case, count_request)
self.assertEqual(data, original)
def test_invalid_case_set_is_rejected(self):
base = json.loads((HERE / 'cases.json').read_text())
bad_fact = copy.deepcopy(base[:1])
bad_fact[0]['required'] = ['a fact the history never states']
duplicate = copy.deepcopy(base[:1]) * 2
bad_split = copy.deepcopy(base[:1])
bad_split[0]['split'] = 'test'
no_required = copy.deepcopy(base[:1])
no_required[0]['required'] = []
extra_field = copy.deepcopy(base[:1])
extra_field[0]['answer'] = 'Friday'
for name, data in (('unknown fact', bad_fact), ('duplicate id', duplicate), ('bad split', bad_split),
('no required fact', no_required), ('extra field', extra_field)):
with self.subTest(name), self.assertRaises(CaseError):
load_cases(self.write_cases(data))
if __name__ == '__main__':
unittest.main()The evaluation tests check the runner, the grader and the recorded failures: unit and contract checks on context selection. The answer tests grade the recorded model answers again, which is an outcome check on a small, fixed sample.
Deployment
Run the comparison before a change ships
The deployable unit here is the comparison, not a service. Run evaluate.py in your test pipeline whenever a selection strategy, its settings, the counter or the case set changes. Keep evaluation-result.json under version control, and review a change to it like a code change: which cases changed, which facts were gained or lost, which stale facts appeared.
The counter is pinned to Gemma 4 E2B. For another model, use its counter and rerun the full set: a different counter changes what fits.
Grow the case set carefully
Add cases from real conversations, with personal data removed, and label their required and stale facts. Assign each new case to a split when you add it, and do not move cases between splits after you have seen results. When you change the held-out cases, the recorded hash changes and the test fails; update the recording in the same change so the review shows both.
Answers were recorded once from gemma4:e2b through Ollama 0.32.14. No server deployment was made. The tests pass on Python 3.11 and 3.15.
Scaling
Cost of the comparison
The run makes one strategy call per case per strategy, and each strategy makes several counter calls. With a real tokenizer, counting dominates, as the budget build measured. This build records no timing: it depends on the machine and would not transfer.
From fact recall to answer quality
Fact recall is cheap and deterministic, which makes it a good first gate. Answer grading costs one model call per case, strategy and trial: 120 calls here. Run the cheap gate on every change and the answer check on a fixed sample before a release, and keep the fact-recall report beside the answers: when an answer is wrong, the fact-recall row shows whether the fact was missing from the request or ignored by the model.
Key takeaways
- Compare strategies on one fixed case set, with the same budget, counter and grader for each.
- Keep per-case results. The totals hide why a strategy failed; the cases explain it.
- Report stale facts separately from missing facts: a strategy can find more facts and still send a replaced decision.
- Name the limits of the grader. Fact recall measures what reaches the request; the recorded answers show that a stale fact in the request can become the answer.
Best practices
- Record the hashes of the case set and the strategy code with the results. Have someone who has not seen the strategies write the held-out cases, or say that the split is not blind.
- Reject cases whose facts do not occur in the conversation, so a labeling error cannot look like a strategy failure.
- Write a named test for each failure pattern you find, so a later change that fixes or breaks it shows up in review.
- Do not rank strategies from a handful of synthetic cases; build a case set from your own traffic.
References
- Build a context budget manager: the
recentpolicy. - Build context compaction with information-loss tests: the
notespolicy and the exact-match probes. - Gemma 4 E2B model card and Ollama chat API: the model and server used to record the answers. Checked October 20, 2026.
- Python
hashlib: the SHA-256 hashes recorded for the case set and code. Checked October 20, 2026. - Anthropic: Demystifying evals for AI agents: code-based, model-based and human graders, multiple trials because model outputs vary, and reading transcripts rather than only scores. It does not cover held-out splits. Checked October 20, 2026.
The strategies, case set, recorded report and tests belong to this build. No external source validates these results.