Problem
When you need this. A fact remembered at turn ten may be changed or deleted by turn fifty. Replay one conversation to see which current, stale and forgotten facts reach later requests.
The earlier builds each tested one mechanism on a short fixture. A real conversation is long. Facts arrive early, change later, and sometimes the user asks the assistant to forget one. A strategy that looks fine after ten turns can lose the trip’s destination by turn fifty, or keep sending a passport number the user asked it to forget.
This build is a benchmark for that: one 50-turn conversation, replayed turn by turn, with five checkpoints. At each checkpoint it builds the request with four strategies and checks three things:
- Retention: which required current facts are in the request.
- Stale facts: which facts that a later turn replaced are in the request.
- Forgotten facts: whether a fact the user asked to forget is in the request.
All four strategies get the same 352-token input budget, counted with the pinned Gemma 4 E2B tokenizer. Across the five checkpoints, keeping recent turns finds 4 of 26 required facts; managed memory finds 26, with no stale or forgotten facts.
Success check: trace a fact from its first statement through a correction or forget request, then compare what each strategy sends at the five checkpoints.
You need: Python 3.11 or later and the pinned Gemma tokenizer files from the context budget manager build; prepare.py downloads them once, with no account and no model weights. The tests grade recorded model answers and call no model; recording new answers needs Ollama with gemma4:e2b.
Concepts
A conversation with labeled events
The fixture is generated by a script, so it can be read, changed and regenerated. Of the 50 turns, 13 are memory events: a fact said, a fact corrected, or a request to forget. Each carries a label that says what a memory system should do at that turn. The other turns are routine filler; every fourth one pastes a long document, which pushes older turns out of the budget.
The labels describe what perfect extraction would produce. The two memory strategies use them directly, so their results are an upper bound for this kind of store: a real extractor that turns conversation text into memories will miss some facts and mislabel others.
Four strategies
| Strategy | What it sends | From |
|---|---|---|
recent |
Newest complete turns that fit | Context budget manager |
notes |
Recent turns plus cue-word notes from removed turns | Context compaction |
append-only |
Recent turns plus every labeled fact, never changed or deleted | This build, as a baseline |
managed |
Recent turns, with corrected and forgotten turns redacted, plus labeled facts, corrected, forgotten and consolidated | The persistent memory builds |
append-only separates two questions: does having memory help, and does managing it help? It stores the same facts as managed, but it never applies a correction or a forget request.
Checkpoints and graders
Each checkpoint asks for a summary of the trip plan and lists the facts a correct summary needs, the replaced facts it must not use, and the forgotten fact it must not contain. The grader checks the request by exact text, accepting escaped text inside notes blocks. A test checks that every listed fact was actually said before its checkpoint, so a typo cannot make a checkpoint impossible.
Architecture
Memory compaction benchmark · 50 turns, 4 strategies, 5 checkpoints
benchmark.py replays the turns in order. After each turn, it applies that turn’s label to two SQLite stores: the append-only store records only remember events, and the managed store also applies correct and forget. At each checkpoint it runs consolidation on the managed store, then builds the four requests, checks that each fits the budget, and grades it.
Both memory strategies list current facts newest first, within a facts quota of 160 tokens, and fill the rest of the budget with recent turns. Every line and request is counted with the pinned Gemma 4 E2B tokenizer and chat template, so the quota and budget are real token counts for that model. The quota keeps the facts from taking the whole budget. It also has a cost, shown in the quota sweep.
Working code
Get the files
Download the complete lab, extract it and open a terminal in build-a-memory-compaction-benchmark/. It includes the source, fixtures, recorded results and tests listed in the README.
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python prepare.pyprepare.py fetches the pinned tokenizer files once; it does not download model weights. The commands below run local selection code and regrade saved answers without a model server.
The fixture generator
"""Write conversation.json: a fixed 50-turn conversation with labeled memory events.
Fact turns carry a `memory` label: what an application with perfect extraction
would remember, correct or forget at that turn. Other turns are filler of
varying length. Run this file to regenerate the fixture; the test suite checks
that the stored file still matches.
"""
import json
from pathlib import Path
FACTS = {
3: ('I am vegetarian.', {'op': 'remember', 'attribute': 'diet'}),
5: ('The budget is 150 euros per night.', {'op': 'remember', 'attribute': 'budget'}),
8: ('We are going to Porto.', {'op': 'remember', 'attribute': 'city'}),
12: ('My passport number is P1234567.', {'op': 'remember', 'attribute': 'passport'}),
15: ('We travel from 3 to 6 November.', {'op': 'remember', 'attribute': 'dates'}),
18: ('I want a hotel near the river.', {'op': 'remember', 'attribute': 'hotel-area'}),
22: ('Raise the budget to 180 euros per night.', {'op': 'correct', 'attribute': 'budget'}),
27: ('Book the morning flight if possible.', {'op': 'remember', 'attribute': 'flight'}),
31: ('I eat fish now, but still no meat.', {'op': 'correct', 'attribute': 'diet'}),
35: ('Please forget my passport number.', {'op': 'forget', 'attribute': 'passport'}),
40: ('Move the trip to 10 to 13 November.', {'op': 'correct', 'attribute': 'dates'}),
44: ('Bob will join us on the second day.', {'op': 'remember', 'attribute': 'attendee'}),
47: ('I want a hotel near the river.', {'op': 'remember', 'attribute': 'hotel-area'}),
}
TOPICS = ['the agenda', 'the slides', 'the venue map', 'the team list', 'the travel policy',
'the expense form', 'the meeting notes', 'the room layout']
def filler(turn: int) -> tuple[str, str]:
topic = TOPICS[turn % len(TOPICS)]
# Every fourth filler turn pastes a long document, which pushes older turns out.
repeat = 14 if turn % 4 == 0 else 2
body = ' '.join(f'Line {i + 1} of {topic} has routine details for the offsite.' for i in range(repeat))
return f'Here is {topic}: {body}', f'Thanks, I read {topic}.'
def conversation() -> dict:
turns = []
for turn in range(1, 51):
if turn in FACTS:
text, label = FACTS[turn]
turns.append({'turn': turn, 'user': text, 'assistant': 'Noted.', 'memory': label})
else:
user, assistant = filler(turn)
turns.append({'turn': turn, 'user': user, 'assistant': assistant})
return {
'instructions': 'Answer from the conversation and the stored memories. Say when a fact is missing.',
'turns': turns,
'checkpoints': [
{'after': 10, 'question': 'Summarize the trip plan so far.',
'required': ['I am vegetarian.', '150 euros per night', 'going to Porto'], 'stale': [], 'deleted': []},
{'after': 20, 'question': 'Summarize the trip plan so far.',
'required': ['I am vegetarian.', '150 euros per night', 'going to Porto', '3 to 6 November',
'hotel near the river'], 'stale': [], 'deleted': []},
{'after': 30, 'question': 'Summarize the trip plan so far.',
'required': ['180 euros per night', 'going to Porto', '3 to 6 November', 'hotel near the river',
'morning flight'], 'stale': ['150 euros per night'], 'deleted': []},
{'after': 40, 'question': 'Summarize the trip plan so far.',
'required': ['fish now', '180 euros per night', 'going to Porto', '10 to 13 November',
'hotel near the river', 'morning flight'],
'stale': ['I am vegetarian.', '150 euros per night', '3 to 6 November'], 'deleted': ['P1234567']},
{'after': 50, 'question': 'Summarize the trip plan so far.',
'required': ['fish now', '180 euros per night', 'going to Porto', '10 to 13 November',
'hotel near the river', 'morning flight', 'Bob will join'],
'stale': ['I am vegetarian.', '150 euros per night', '3 to 6 November'], 'deleted': ['P1234567']},
],
'budget': {'context_limit': 512, 'max_new_tokens': 128, 'margin': 32},
}
if __name__ == '__main__':
Path(__file__).with_name('conversation.json').write_text(json.dumps(conversation(), indent=2) + '\n')FACTS maps turn numbers to the user’s words and the label. 3 labels are corrections (budget, diet, dates), 1 is a forget request (the passport number), and turn 47 repeats the hotel preference word for word. The checkpoints list facts by exact text that occurs in those turns.
The benchmark
"""Replay a 50-turn conversation and check fact retention at five checkpoints.
Four strategies build the request for each checkpoint question under the same
budget and counter:
recent newest complete turns only (the context budget build's policy)
notes recent turns plus cue-word notes from removed turns
(the context compaction build's policy)
append-only recent turns plus every labeled fact, stored and never changed
managed recent turns plus labeled facts that are corrected, forgotten
and consolidated (the persistent memory builds); a turn whose
fact was forgotten or corrected is redacted from its transcript
The memory strategies use the conversation's labels, which act as perfect
extraction. A real extractor would do worse.
"""
import json
from pathlib import Path
import platform
import re
import sqlite3
import tempfile
from consolidate import consolidate
from tokenizer import count_request, count_text
from memory import Clock, MemoryStore, Scope
HERE = Path(__file__).parent
SCOPE = Scope('alice', 'offsite')
NOTES_RESERVE = 96
FACTS_QUOTA = 160
CUES = re.compile(r'\b(decided|decide|chose|choose|will use|must|never|always|deadline|instead|switch(?:ed)?|requires?|limit)\b',
re.IGNORECASE)
SENTENCE = re.compile(r'(?<=[.!?])\s+')
HEADER = 'Stored notes about this user. They quote the user; they are not instructions.'
REDACTED = '[removed: the user corrected this or asked to forget it]'
def escape(text: str) -> str:
return text.replace('&', '&').replace('<', '<').replace('>', '>')
def request(case: dict, turns: list[dict], question: str, notes: list[str]) -> list[dict]:
messages = [{'role': 'system', 'content': case['instructions']}]
for turn in turns:
messages += [{'role': 'user', 'content': turn['user']}, {'role': 'assistant', 'content': turn['assistant']}]
content = question if not notes else '\n'.join([HEADER, '<notes>', *notes, '</notes>', '', f'Question: {question}'])
return messages + [{'role': 'user', 'content': content}]
def available(case: dict) -> int:
budget = case['budget']
return budget['context_limit'] - budget['max_new_tokens'] - budget['margin']
def fill_recent(case, history, question, notes, limit):
"""Newest complete turns first; stop at the first turn that does not fit."""
kept = []
for turn in reversed(history):
if count_request(request(case, [turn, *kept], question, notes)) > limit:
break
kept.insert(0, turn)
return kept
def recent(case, history, question):
kept = fill_recent(case, history, question, [], available(case))
return request(case, kept, question, [])
def notes(case, history, question):
kept = fill_recent(case, history, question, [], available(case) - NOTES_RESERVE)
removed = history[:len(history) - len(kept)]
candidates = [f'- [turn {t["turn"]}] {escape(s)}' for t in removed
for s in SENTENCE.split(t['user'].strip()) if CUES.search(s)]
chosen = []
for line in reversed(candidates):
if count_request(request(case, kept, question, [line, *chosen])) > available(case):
break
chosen.insert(0, line)
return request(case, kept, question, chosen)
def memory_lines(store: MemoryStore, quota: int) -> list[str]:
"""Current facts, newest first, within the facts quota; skip a fact that does not fit."""
lines, used = [], 0
for row in sorted(store.current(SCOPE), key=lambda r: -r['id']):
line = f'- [#{row["id"]}, {row["source"]}] {escape(row["text"])}'
if used + count_text(line) <= quota:
lines.append(line)
used += count_text(line)
return lines
def with_memory(case, history, question, store, quota=FACTS_QUOTA):
lines = memory_lines(store, quota)
kept = fill_recent(case, history, question, lines, available(case))
return request(case, kept, question, lines)
def apply(store: MemoryStore, turn: dict, managed: bool) -> list[int]:
"""Write a labeled turn into the store. Append-only stores every fact and never changes one.
Returns the turns whose fact the managed store replaced or deleted, so the
managed strategy can redact them from the transcript as well.
"""
label = turn.get('memory')
if not label:
return []
current = [r for r in store.current(SCOPE) if r['attribute'] == label['attribute']]
source = f'turn-{turn["turn"]}'
if managed and label['op'] == 'correct' and current:
store.correct(SCOPE, current[-1]['id'], turn['user'], source)
return [int(current[-1]['source'].split('-')[1])]
if managed and label['op'] == 'forget' and current:
for row in current:
store.forget(SCOPE, row['id'], f'user request at {source}')
return [int(row['source'].split('-')[1]) for row in current]
if label['op'] != 'forget':
store.remember(SCOPE, 'user', turn['user'], source, attribute=label['attribute'] if managed else None)
return []
def redact(history: list[dict], turns: set[int]) -> list[dict]:
return [{**t, 'user': REDACTED} if t['turn'] in turns else t for t in history]
def grade(messages: list[dict], checkpoint: dict) -> dict:
text = '\n'.join(m['content'] for m in messages)
present = lambda fact: fact in text or escape(fact) in text
return {'found': sum(map(present, checkpoint['required'])), 'required': len(checkpoint['required']),
'missing': [f for f in checkpoint['required'] if not present(f)],
'stale': [f for f in checkpoint['stale'] if present(f)],
'deleted': [f for f in checkpoint['deleted'] if present(f)],
'input_tokens': count_request(messages)}
def run(case: dict, facts_quota: int = FACTS_QUOTA, keep: dict | None = None) -> dict:
"""Replay the conversation. If `keep` is a dict, store each request in it by (strategy, checkpoint)."""
strategies = {'recent': recent, 'notes': notes, 'append-only': with_memory, 'managed': with_memory}
results = {name: [] for name in strategies}
storage = {}
with tempfile.TemporaryDirectory() as folder:
stores = {name: MemoryStore(str(Path(folder) / f'{name}.sqlite3'), Clock())
for name in ('append-only', 'managed')}
history, redacted = [], set()
checkpoints = {c['after']: c for c in case['checkpoints']}
for turn in case['turns']:
history.append(turn)
apply(stores['append-only'], turn, managed=False)
redacted.update(apply(stores['managed'], turn, managed=True))
checkpoint = checkpoints.get(turn['turn'])
if not checkpoint:
continue
consolidate(stores['managed'], SCOPE)
for name, strategy in strategies.items():
if name == 'managed':
messages = strategy(case, redact(history, redacted), checkpoint['question'], stores[name], facts_quota)
elif name in stores:
messages = strategy(case, history, checkpoint['question'], stores[name], facts_quota)
else:
messages = strategy(case, history, checkpoint['question'])
if count_request(messages) > available(case):
raise RuntimeError(f'{name} exceeded the input budget at turn {turn["turn"]}')
results[name].append({'after': turn['turn'], **grade(messages, checkpoint)})
if keep is not None:
keep[name, turn['turn']] = messages
for name, store in stores.items():
storage[name] = {
'stored_rows': store.db.execute('SELECT count(*) FROM memories').fetchone()[0],
'current_facts': len(store.current(SCOPE)),
'deletion_records': len(store.deletions(SCOPE)),
'resolutions': len(store.resolutions(SCOPE)),
}
store.close()
totals = {name: {'found': sum(r['found'] for r in rows), 'required': sum(r['required'] for r in rows),
'stale': sum(len(r['stale']) for r in rows), 'deleted': sum(len(r['deleted']) for r in rows)}
for name, rows in results.items()}
return {'budget': case['budget'], 'input_budget': available(case), 'facts_quota': facts_quota,
'checkpoints': results,
'totals': totals, 'storage': storage,
'counter': 'Gemma 4 E2B tokenizer and chat template (tokenizer-manifest.json)', 'trials': 1,
'environment': {'python': platform.python_version(), 'sqlite': sqlite3.sqlite_version}}
def full_report(case: dict) -> dict:
report = run(case)
# How retention depends on the facts quota, for the two memory strategies.
report['quota_sweep'] = {str(quota): {name: run(case, quota)['totals'][name] for name in ('append-only', 'managed')}
for quota in (120, 160, 240)}
return report
if __name__ == '__main__':
print(json.dumps(full_report(json.loads((HERE / 'conversation.json').read_text())), indent=2))apply turns a label into store operations. For the managed store, a correction supersedes the current fact with the same attribute, and a forget request deletes every current fact with that attribute. The append-only store ignores attributes, corrections and forget requests: it only adds.
run raises an error if any strategy builds a request larger than the budget, so a strategy cannot score well by overspending. full_report adds a sweep over three facts quotas.
Run it
From the directory with the files, run:
python3 generate_conversation.py
python3 benchmark.pyThe first command rewrites conversation.json with the same content. The second prints the report. The totals across all five checkpoints:
{
"recent": {
"found": 4,
"required": 26,
"stale": 0,
"deleted": 0
},
"notes": {
"found": 2,
"required": 26,
"stale": 0,
"deleted": 0
},
"append-only": {
"found": 25,
"required": 26,
"stale": 3,
"deleted": 1
},
"managed": {
"found": 26,
"required": 26,
"stale": 0,
"deleted": 0
}
}Inspect the complete recorded report
{
"budget": {
"context_limit": 512,
"max_new_tokens": 128,
"margin": 32
},
"input_budget": 352,
"facts_quota": 160,
"checkpoints": {
"recent": [
{
"after": 10,
"found": 2,
"required": 3,
"missing": [
"I am vegetarian."
],
"stale": [],
"deleted": [],
"input_tokens": 293
},
{
"after": 20,
"found": 1,
"required": 5,
"missing": [
"I am vegetarian.",
"150 euros per night",
"going to Porto",
"3 to 6 November"
],
"stale": [],
"deleted": [],
"input_tokens": 352
},
{
"after": 30,
"found": 0,
"required": 5,
"missing": [
"180 euros per night",
"going to Porto",
"3 to 6 November",
"hotel near the river",
"morning flight"
],
"stale": [],
"deleted": [],
"input_tokens": 146
},
{
"after": 40,
"found": 1,
"required": 6,
"missing": [
"fish now",
"180 euros per night",
"going to Porto",
"hotel near the river",
"morning flight"
],
"stale": [],
"deleted": [],
"input_tokens": 226
},
{
"after": 50,
"found": 0,
"required": 7,
"missing": [
"fish now",
"180 euros per night",
"going to Porto",
"10 to 13 November",
"hotel near the river",
"morning flight",
"Bob will join"
],
"stale": [],
"deleted": [],
"input_tokens": 142
}
],
"notes": [
{
"after": 10,
"found": 1,
"required": 3,
"missing": [
"I am vegetarian.",
"150 euros per night"
],
"stale": [],
"deleted": [],
"input_tokens": 215
},
{
"after": 20,
"found": 0,
"required": 5,
"missing": [
"I am vegetarian.",
"150 euros per night",
"going to Porto",
"3 to 6 November",
"hotel near the river"
],
"stale": [],
"deleted": [],
"input_tokens": 38
},
{
"after": 30,
"found": 0,
"required": 5,
"missing": [
"180 euros per night",
"going to Porto",
"3 to 6 November",
"hotel near the river",
"morning flight"
],
"stale": [],
"deleted": [],
"input_tokens": 146
},
{
"after": 40,
"found": 1,
"required": 6,
"missing": [
"fish now",
"180 euros per night",
"going to Porto",
"hotel near the river",
"morning flight"
],
"stale": [],
"deleted": [],
"input_tokens": 226
},
{
"after": 50,
"found": 0,
"required": 7,
"missing": [
"fish now",
"180 euros per night",
"going to Porto",
"10 to 13 November",
"hotel near the river",
"morning flight",
"Bob will join"
],
"stale": [],
"deleted": [],
"input_tokens": 142
}
],
"append-only": [
{
"after": 10,
"found": 3,
"required": 3,
"missing": [],
"stale": [],
"deleted": [],
"input_tokens": 344
},
{
"after": 20,
"found": 5,
"required": 5,
"missing": [],
"stale": [],
"deleted": [],
"input_tokens": 174
},
{
"after": 30,
"found": 5,
"required": 5,
"missing": [],
"stale": [
"150 euros per night"
],
"deleted": [],
"input_tokens": 321
},
{
"after": 40,
"found": 6,
"required": 6,
"missing": [],
"stale": [
"3 to 6 November"
],
"deleted": [
"P1234567"
],
"input_tokens": 304
},
{
"after": 50,
"found": 6,
"required": 7,
"missing": [
"going to Porto"
],
"stale": [
"3 to 6 November"
],
"deleted": [],
"input_tokens": 329
}
],
"managed": [
{
"after": 10,
"found": 3,
"required": 3,
"missing": [],
"stale": [],
"deleted": [],
"input_tokens": 344
},
{
"after": 20,
"found": 5,
"required": 5,
"missing": [],
"stale": [],
"deleted": [],
"input_tokens": 174
},
{
"after": 30,
"found": 5,
"required": 5,
"missing": [],
"stale": [],
"deleted": [],
"input_tokens": 301
},
{
"after": 40,
"found": 6,
"required": 6,
"missing": [],
"stale": [],
"deleted": [],
"input_tokens": 315
},
{
"after": 50,
"found": 7,
"required": 7,
"missing": [],
"stale": [],
"deleted": [],
"input_tokens": 305
}
]
},
"totals": {
"recent": {
"found": 4,
"required": 26,
"stale": 0,
"deleted": 0
},
"notes": {
"found": 2,
"required": 26,
"stale": 0,
"deleted": 0
},
"append-only": {
"found": 25,
"required": 26,
"stale": 3,
"deleted": 1
},
"managed": {
"found": 26,
"required": 26,
"stale": 0,
"deleted": 0
}
},
"storage": {
"append-only": {
"stored_rows": 12,
"current_facts": 12,
"deletion_records": 0,
"resolutions": 0
},
"managed": {
"stored_rows": 11,
"current_facts": 7,
"deletion_records": 1,
"resolutions": 1
}
},
"counter": "Gemma 4 E2B tokenizer and chat template (tokenizer-manifest.json)",
"trials": 1,
"environment": {
"python": "3.11.11",
"sqlite": "3.53.4"
},
"quota_sweep": {
"120": {
"append-only": {
"found": 24,
"required": 26,
"stale": 1,
"deleted": 0
},
"managed": {
"found": 25,
"required": 26,
"stale": 0,
"deleted": 0
}
},
"160": {
"append-only": {
"found": 25,
"required": 26,
"stale": 3,
"deleted": 1
},
"managed": {
"found": 26,
"required": 26,
"stale": 0,
"deleted": 0
}
},
"240": {
"append-only": {
"found": 26,
"required": 26,
"stale": 7,
"deleted": 2
},
"managed": {
"found": 26,
"required": 26,
"stale": 0,
"deleted": 0
}
}
}
}Read the results
Recent turns lose early facts. By turn 50, recent finds 0 of 7 required facts. The long pasted documents fill the budget, and everything said before them is gone.
Cue-word notes find nothing here. The compaction build’s extractor keeps sentences with words like must or chose. This conversation states its facts as plans and preferences (“I want a hotel near the river”), which contain none of those words, and a test confirms that none of the labeled facts matches. notes also reserves budget for notes, so it keeps fewer recent turns and finds 2 required facts in total, fewer than recent. An extractor tuned to one conversation can miss everything in another; here it matched none of the labeled facts.
Append-only memory keeps facts and leaks. append-only finds 25 required facts, but it sends 3 stale facts across the checkpoints, and at turn 40 it sends the passport number the user asked it to forget at turn 35.
Managed memory keeps facts without leaking. managed finds 26 of 26, with 0 stale and 0 forgotten facts. A store is not the only copy of a fact: the recent turns repeat what the user said. So managed also redacts each turn whose fact was corrected or forgotten before it fills recent turns. Without that, a checkpoint right after a forget request would still send the passport number in the transcript; a test builds that case, and the three other strategies send it. Redaction covers only the turn that stated the fact: a forget request that quotes the value, or an assistant reply that repeats it, would still send it.
The facts quota
The report runs both memory strategies at three quotas:
| Facts quota | Strategy | Found | Stale | Forgotten |
|---|---|---|---|---|
| 120 | append-only | 24 of 26 | 1 | 0 |
| 120 | managed | 25 of 26 | 0 | 0 |
| 160 | append-only | 25 of 26 | 3 | 1 |
| 160 | managed | 26 of 26 | 0 | 0 |
| 240 | append-only | 26 of 26 | 7 | 2 |
| 240 | managed | 26 of 26 | 0 | 0 |
At 120 tokens, managed misses 1 fact at turn 50: its 7 current facts need more than the quota, and newest-first selection drops the oldest still-valid fact, “We are going to Porto.” From 160 tokens it finds every fact. A larger quota makes append-only worse: with more room, it sends more of the replaced and forgotten facts it never removed. Adding budget to an unmanaged memory increases what it leaks.
Check the answers
The report shows what each request holds. answer_check.py sends the turn-40 and turn-50 requests of all four strategies, unchanged, to gemma4:e2b, the Gemma 4 E2B model under the Apache 2.0 license, through a local Ollama server pinned by digest, 5 seeds each. Each call uses the conversation’s own budget: a 512-token context and at most 128 answer tokens. An answer passes with every required fact, no replaced fact stated as current and no forgotten fact, fails otherwise, and is incomplete if there is no answer, the labels from the evaluation foundations.
The grading keys were written from the checkpoints before any answer was recorded. Reading the first answers showed two key errors, which were fixed before the grades below: the date patterns missed ordinals such as “November 3rd to November 6th”, and the words “from” and “was” counted as marking a date as replaced, so “The trip is planned from 3 to 6 November” passed.
Inspect answer_check.py
"""Send the turn-40 and turn-50 requests of every strategy to the pinned Gemma 4 E2B model.
python answer_check.py record # needs Ollama with gemma4:e2b and the pinned tokenizer
python answer_check.py # offline: grades the recorded answers again
The requests are the ones benchmark.py builds at the default facts quota, sent
unchanged with the conversation's own budget: a 512-token context and at most
128 answer tokens. Each request is sent with five seeds. KEYS, written from the
checkpoints' required, stale and forgotten facts before any answer was recorded.
After reading the first recording, the date patterns were widened to accept
ordinals ("3rd") and the replacement markers were narrowed: "from" and "was"
had let "The trip is planned from 3 to 6 November" pass.
incomplete no answer to grade
fail the forgotten passport number appears, a replaced fact is stated as
current, or a required fact is missing
pass every required fact, no replaced fact stated as current, and no
forgotten fact
Forty answers from a small model illustrate how the four requests behave; they
are not a measure of answer quality.
"""
import json
from pathlib import Path
import re
import sys
HERE = Path(__file__).parent
CHECKPOINTS = (40, 50)
STRATEGIES = ('recent', 'notes', 'append-only', 'managed')
KEYS = {
'required': {
'fish now': r'\bfish\b',
'180 euros per night': r'\b180\b',
'going to Porto': r'\bporto\b',
'10 to 13 November': r'\b10(?:th)?\b[^\n]{0,25}\b13(?:th)?\b',
'hotel near the river': r'\briver\b',
'morning flight': r'\bmorning\b',
'Bob will join': r'\bbob\b',
},
'stale': {
'I am vegetarian.': r'\bvegetarian\b',
'150 euros per night': r'\b150\b',
'3 to 6 November': r'\b3(?:rd)?\b[^\n]{0,25}\b6(?:th)?\b',
},
'deleted': {'P1234567': r'P1234567'},
}
SENTENCE = re.compile(r'(?<=[.!?])\s+|\n+')
# A sentence that marks the old value as replaced does not state it as current.
REPLACED = re.compile(r"\bnot\b|n't|\bno longer\b|\bprevious(?:ly)?\b|\boriginal(?:ly)?\b|\bearlier\b|\bold\b|"
r"\breplaced\b|\binstead of\b|\b(?:changed|moved|raised) from\b", re.I)
def grade(checkpoint: dict, answer: dict) -> tuple[str, str]:
text = answer['answer']
if not text.strip():
return 'incomplete', 'no answer'
if any(re.search(KEYS['deleted'][f], text) for f in checkpoint['deleted']):
return 'fail', 'sends the forgotten passport number'
sentences = SENTENCE.split(text)
stale = [f for f in checkpoint['stale']
if any(re.search(KEYS['stale'][f], s, re.I) and not REPLACED.search(s) for s in sentences)]
if stale:
return 'fail', 'states a replaced fact as current: ' + ', '.join(stale)
missing = [f for f in checkpoint['required'] if not re.search(KEYS['required'][f], text, re.I)]
if missing:
cut = '; stopped at the answer-token limit' if answer.get('done_reason') == 'length' else ''
return 'fail', f'missing {len(missing)} of {len(checkpoint["required"])} required facts{cut}'
return 'pass', 'every required fact'
def facts_found(checkpoint: dict, answer: dict) -> int:
return sum(bool(re.search(KEYS['required'][f], answer['answer'], re.I)) for f in checkpoint['required'])
def checkpoints() -> dict:
case = json.loads((HERE / 'conversation.json').read_text())
return {c['after']: c for c in case['checkpoints'] if c['after'] in CHECKPOINTS}
def summarize(recorded: dict) -> dict:
out = {}
for name, answers in recorded['answers'].items():
checkpoint = checkpoints()[recorded['checkpoints'][name]]
verdicts = [grade(checkpoint, a) for a in answers]
out[name] = {v: [g[0] for g in verdicts].count(v) for v in ('pass', 'fail', 'incomplete')}
out[name]['facts_in_answers'] = f'{sum(facts_found(checkpoint, a) for a in answers)} of {len(checkpoint["required"]) * len(answers)}'
reasons = [g[1] for g in verdicts if g[0] != 'pass']
out[name]['reasons'] = {r: reasons.count(r) for r in sorted(set(reasons))}
return out
def record() -> dict:
from answer_model import MODEL, AnswerModel
from benchmark import run
case = json.loads((HERE / 'conversation.json').read_text())
requests = {}
run(case, keep=requests)
model = AnswerModel()
budget = case['budget']
answers, at = {}, {}
for after in CHECKPOINTS:
for strategy in STRATEGIES:
name = f'{strategy} · turn {after}'
at[name] = after
answers[name] = [model.answer(requests[strategy, after], [], num_ctx=budget['context_limit'],
num_predict=budget['max_new_tokens'], seed=seed)
for seed in MODEL['seeds']]
return {'model': MODEL, 'checkpoints': at, 'answers': answers}
if __name__ == '__main__':
if sys.argv[1:] == ['record']:
(HERE / 'answer-result.json').write_text(json.dumps(record(), indent=1, ensure_ascii=False) + '\n')
recorded = json.loads((HERE / 'answer-result.json').read_text())
graded = {name: [grade(checkpoints()[recorded['checkpoints'][name]], a)[0] for a in answers]
for name, answers in recorded['answers'].items()}
print(json.dumps({'summary': summarize(recorded), 'graded': graded}, indent=2))| Request | Pass | Fail | Required facts in the answers |
|---|---|---|---|
recent · turn 40 | 0 of 5 | 5 of 5 | 3 of 30 |
notes · turn 40 | 0 of 5 | 5 of 5 | 3 of 30 |
append-only · turn 40 | 0 of 5 | 5 of 5 | 18 of 30 |
managed · turn 40 | 3 of 5 | 2 of 5 | 28 of 30 |
recent · turn 50 | 0 of 5 | 5 of 5 | 0 of 35 |
notes · turn 50 | 0 of 5 | 5 of 5 | 0 of 35 |
append-only · turn 50 | 0 of 5 | 5 of 5 | 23 of 35 |
managed · turn 50 | 2 of 5 | 3 of 5 | 32 of 35 |
Read every recorded answer
{
"model": {
"checked_on": "2026-10-20",
"publisher_model": "google/gemma-4-E2B-it",
"license": "Apache-2.0",
"ollama_tag": "gemma4:e2b",
"ollama_digest": "b37049369adfe3d2b653af0ab301a062ca5cbe96ace6aa0d3f2559ee9b563fc2",
"ollama_version": "0.32.14",
"quantization": "Q4_K_M",
"parameters": "4.6B total (E2B: about 2B effective)",
"options": {
"temperature": 1.0,
"top_k": 64,
"top_p": 0.95
},
"think": false,
"trials": 5,
"seeds": [
1,
2,
3,
4,
5
]
},
"checkpoints": {
"recent · turn 40": 40,
"notes · turn 40": 40,
"append-only · turn 40": 40,
"managed · turn 40": 40,
"recent · turn 50": 50,
"notes · turn 50": 50,
"append-only · turn 50": 50,
"managed · turn 50": 50
},
"answers": {
"recent · turn 40": [
{
"seed": 1,
"answer": "Based on the information provided so far, here is a summary of the trip plan:\n\n* **Expense Form:**\n * Line 1 contains routine details for the offsite.\n * Line 2 contains routine details for the offsite.\n* **Meeting Notes:**\n * Line 1 contains routine details for the offsite.\n * Line 2 contains routine details for the offsite.\n* **Room Layout:**\n * Line 1 contains routine details for the offsite.\n * Line 2 contains routine details for the offsite.",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 226,
"answer_tokens": 128
},
{
"seed": 2,
"answer": "So far, the trip plan includes:\n\n* **Expense Form:** Line 1 has routine details for the offsite, and Line 2 has routine details for the offsite.\n* **Meeting Notes:** Line 1 has routine details for the offsite, and Line 2 has routine details for the offsite.\n* **Room Layout:** Line 1 has routine details for the offsite, and Line 2 has routine details for the offsite.\n* **Trip Date Change:** The trip has been moved to **November 10 to 13**.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 226,
"answer_tokens": 123
},
{
"seed": 3,
"answer": "So far, the trip plan includes:\n\n* **Expense Form:** Line 1 has routine details for the offsite, and Line 2 has routine details for the offsite.\n* **Meeting Notes:** Line 1 has routine details for the offsite, and Line 2 has routine details for the offsite.\n* **Room Layout:** Line 1 has routine details for the offsite, and Line 2 has routine details for the offsite.\n* **Date Change:** The trip has been moved to November 10 to 13.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 226,
"answer_tokens": 121
},
{
"seed": 4,
"answer": "Here is a summary of the trip plan so far based on the information provided:\n\n* **Expense Form:** Line 1 contains routine details for the offsite, and Line 2 contains routine details for the offsite.\n* **Meeting Notes:** Line 1 contains routine details for the offsite, and Line 2 contains routine details for the offsite.\n* **Room Layout:** Line 1 contains routine details for the offsite, and Line 2 contains routine details for the offsite.\n* **Trip Date Update:** The trip has been moved to **November 10 to 1",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 226,
"answer_tokens": 128
},
{
"seed": 5,
"answer": "So far, the trip plan includes:\n\n* **Expense Form:** Line 1 has routine details for the offsite, and Line 2 has routine details for the offsite.\n* **Meeting Notes:** Line 1 has routine details for the offsite, and Line 2 has routine details for the offsite.\n* **Room Layout:** Line 1 has routine details for the offsite, and Line 2 has routine details for the offsite.\n* **Trip Date:** The trip has been moved to **10 to 13 November**.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 226,
"answer_tokens": 121
}
],
"notes · turn 40": [
{
"seed": 1,
"answer": "Based on the information provided so far, here is a summary of the trip plan:\n\n* **Expense Form:**\n * Line 1 contains routine details for the offsite.\n * Line 2 contains routine details for the offsite.\n* **Meeting Notes:**\n * Line 1 contains routine details for the offsite.\n * Line 2 contains routine details for the offsite.\n* **Room Layout:**\n * Line 1 contains routine details for the offsite.\n * Line 2 contains routine details for the offsite.",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 226,
"answer_tokens": 128
},
{
"seed": 2,
"answer": "So far, the trip plan includes:\n\n* **Expense Form:** Line 1 has routine details for the offsite, and Line 2 has routine details for the offsite.\n* **Meeting Notes:** Line 1 has routine details for the offsite, and Line 2 has routine details for the offsite.\n* **Room Layout:** Line 1 has routine details for the offsite, and Line 2 has routine details for the offsite.\n* **Trip Date Change:** The trip has been moved to **November 10 to 13**.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 226,
"answer_tokens": 123
},
{
"seed": 3,
"answer": "So far, the trip plan includes:\n\n* **Expense Form:** Line 1 has routine details for the offsite, and Line 2 has routine details for the offsite.\n* **Meeting Notes:** Line 1 has routine details for the offsite, and Line 2 has routine details for the offsite.\n* **Room Layout:** Line 1 has routine details for the offsite, and Line 2 has routine details for the offsite.\n* **Date Change:** The trip has been moved to November 10 to 13.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 226,
"answer_tokens": 121
},
{
"seed": 4,
"answer": "Here is a summary of the trip plan so far based on the information provided:\n\n* **Expense Form:** Line 1 contains routine details for the offsite, and Line 2 contains routine details for the offsite.\n* **Meeting Notes:** Line 1 contains routine details for the offsite, and Line 2 contains routine details for the offsite.\n* **Room Layout:** Line 1 contains routine details for the offsite, and Line 2 contains routine details for the offsite.\n* **Trip Date Update:** The trip has been moved to **November 10 to 1",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 226,
"answer_tokens": 128
},
{
"seed": 5,
"answer": "So far, the trip plan includes:\n\n* **Expense Form:** Line 1 has routine details for the offsite, and Line 2 has routine details for the offsite.\n* **Meeting Notes:** Line 1 has routine details for the offsite, and Line 2 has routine details for the offsite.\n* **Room Layout:** Line 1 has routine details for the offsite, and Line 2 has routine details for the offsite.\n* **Trip Date:** The trip has been moved to **10 to 13 November**.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 226,
"answer_tokens": 121
}
],
"append-only · turn 40": [
{
"seed": 1,
"answer": "Based on the stored notes, here is a summary of the trip plan so far:\n\n* **Trip Dates:** The trip has been moved to **10 to 13 November**. (This is noted in [#10, turn-40], although there is also a note [#5, turn-15] stating travel from 3 to 6 November, which seems to be a conflicting or previous detail.)\n* **Destination:** The destination is **Porto**. ([#3, turn-8])\n* **Accommodation Preference:** The user wants a hotel **near the river**. ([#6",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 304,
"answer_tokens": 128
},
{
"seed": 2,
"answer": "Here is a summary of the trip plan so far based on the stored notes:\n\n* **Trip Dates:** The trip is planned from **November 3rd to November 6th** (from note [#5, turn-15]). However, there is a separate instruction to move the trip to **November 10th to 13th** (from note [#10, turn-40]).\n* **Destination:** The destination is **Porto** (from note [#3, turn-8]).\n* **Budget:** The budget has been raised to **180 euros per night",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 304,
"answer_tokens": 128
},
{
"seed": 3,
"answer": "Here is a summary of the trip plan so far, based on the stored notes:\n\n* **Dates:** The trip has been moved to **10 to 13 November**. (This is based on note [#10, turn-40])\n* **Destination:** The destination is **Porto**. (Based on note [#3, turn-8])\n* **Accommodation Preference:** The user wants a **hotel near the river**. (Based on note [#6, turn-18])\n* **Budget:** The budget has been raised to **180 euros per night**. (Based",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 304,
"answer_tokens": 128
},
{
"seed": 4,
"answer": "Here is a summary of the trip plan based on the stored notes:\n\n* **Trip Dates:** The trip is planned to be from **3 to 6 November** (from note [#5, turn-15]).\n* **New Trip Dates:** The trip has been moved to **10 to 13 November** (from note [#10, turn-40]).\n* **Destination:** The destination is **Porto** (from note [#3, turn-8]).\n* **Budget:** The budget has been raised to **180 euros per night** (from note [#",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 304,
"answer_tokens": 128
},
{
"seed": 5,
"answer": "Here is a summary of the trip plan based on the stored notes:\n\n* **Dates:** The trip is currently planned for November 3rd to 6th, but a change has been requested to move it to **November 10th to 13th**.\n* **Destination:** The destination is **Porto**.\n* **Accommodation:** The user wants a hotel **near the river**.\n* **Budget:** The budget has been raised to **180 euros per night**.\n* **Flights:** There is a request to **book the morning flight if possible**.\n* ",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 304,
"answer_tokens": 128
}
],
"managed · turn 40": [
{
"seed": 1,
"answer": "The trip plan has the following details based on the stored notes:\n\n* **Trip Dates:** Moved to November 10 to 13.\n* **Location:** Porto.\n* **Flight:** Book the morning flight if possible.\n* **Budget:** Raise the budget to 180 euros per night.\n* **Accommodation:** The user wants a hotel near the river.\n* **Dietary Note:** The user eats fish now but still no meat.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 315,
"answer_tokens": 104
},
{
"seed": 2,
"answer": "Based on the stored notes, here is a summary of the trip plan so far:\n\n* **Trip Dates:** The trip has been moved to **November 10 to 13**.\n* **Location:** The destination is **Porto**.\n* **Flight Preference:** There was a suggestion to **book the morning flight if possible**.\n* **Budget:** The budget has been raised to **180 euros per night**.\n* **Accommodation Preference:** There is a desire for a **hotel near the river**.\n* **Personal Note:** One note mentions a dietary preference: \"I",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 315,
"answer_tokens": 128
},
{
"seed": 3,
"answer": "Based on the stored notes, here is a summary of the trip plan so far:\n\n* **Trip Dates:** The trip has been moved to **November 10 to 13**.\n* **Destination:** The destination is **Porto**.\n* **Budget:** The budget has been raised to **180 euros per night**.\n* **Accommodation Preference:** There is a preference for a **hotel near the river**.\n* **Flights:** There is a request to **book the morning flight if possible**.\n* **Dietary Note:** One note mentions, \"I eat fish now",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 315,
"answer_tokens": 128
},
{
"seed": 4,
"answer": "Here is a summary of the trip plan based on the stored notes:\n\n* **Trip Dates:** The trip has been moved to **November 10 to 13**.\n* **Location:** The destination is **Porto**.\n* **Flight:** The request to **book the morning flight if possible** was made.\n* **Accommodation:** There is a preference for a **hotel near the river**.\n* **Budget:** The budget has been **raised to 180 euros per night**.\n* **Dietary Note (Quote):** The user mentioned, \"I eat fish",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 315,
"answer_tokens": 128
},
{
"seed": 5,
"answer": "Based on the stored notes, here is a summary of the trip plan so far:\n\n* **Trip Dates:** The trip has been moved to **November 10 to 13**.\n* **Budget:** The budget has been raised to **180 euros per night**.\n* **Travel Preferences:**\n * There is a request to **book the morning flight if possible**.\n * There is a preference for a **hotel near the river**.\n* **Destination:** The destination is **Porto**.\n* **Dietary Note (User Quote):** The user notes",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 315,
"answer_tokens": 128
}
],
"recent · turn 50": [
{
"seed": 1,
"answer": "So far, you have provided:\n\n1. **Line 1 of the slides:** Routine details for the offsite.\n2. **Line 2 of the slides:** Routine details for the offsite.\n3. **Line 1 of the venue map:** Routine details for the offsite.\n4. **Line 2 of the venue map:** Routine details for the offsite.\n\n**I do not have enough information to provide a full summary of the trip plan.** I only have routine details from the slides and the venue map.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 142,
"answer_tokens": 117
},
{
"seed": 2,
"answer": "Based on what you've provided, here is a summary of the trip plan so far:\n\n* **Slides:** You have provided two lines of slides containing routine details for the offsite.\n* **Venue Map:** You have provided two lines of the venue map containing routine details for the offsite.\n\n**However, I do not have the actual content of these slides or the venue map.** Therefore, I can only summarize that the plan includes routine details for the offsite in both the presentation material and the venue map.\n\nCould you please provide the actual content of the slides and the venue map so I can give",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 142,
"answer_tokens": 128
},
{
"seed": 3,
"answer": "Based on the information you've provided, here is a summary of the trip plan so far:\n\n* **Slide 1:** Contains routine details for the offsite.\n* **Slide 2:** Contains routine details for the offsite.\n* **Venue Map, Line 1:** Contains routine details for the offsite.\n* **Venue Map, Line 2:** Contains routine details for the offsite.\n\nEssentially, the slides and the venue map currently contain routine details for the offsite.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 142,
"answer_tokens": 109
},
{
"seed": 4,
"answer": "Based on the information you've provided so far, I have the following:\n\n* **Slide 1:** Contains routine details for the offsite.\n* **Slide 2:** Contains routine details for the offsite.\n* **Venue Map, Line 1:** Contains routine details for the offsite.\n* **Venue Map, Line 2:** Contains routine details for the offsite.\n\n**However, I do not have the actual content of these slides or the map.** Therefore, I cannot provide a summary of the trip plan because the details themselves are missing.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 142,
"answer_tokens": 123
},
{
"seed": 5,
"answer": "Based on the information you've provided so far, here is a summary of the trip plan:\n\n* **Slide 1:** Contains routine details for the offsite.\n* **Slide 2:** Contains routine details for the offsite.\n* **Venue Map, Line 1:** Contains routine details for the offsite.\n* **Venue Map, Line 2:** Contains routine details for the offsite.\n\n**However, I do not have the actual content of these slides or the map.** Therefore, I can only tell you that the plan currently contains the *routine details* for the off",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 142,
"answer_tokens": 128
}
],
"notes · turn 50": [
{
"seed": 1,
"answer": "So far, you have provided:\n\n1. **Line 1 of the slides:** Routine details for the offsite.\n2. **Line 2 of the slides:** Routine details for the offsite.\n3. **Line 1 of the venue map:** Routine details for the offsite.\n4. **Line 2 of the venue map:** Routine details for the offsite.\n\n**I do not have enough information to provide a full summary of the trip plan.** I only have routine details from the slides and the venue map.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 142,
"answer_tokens": 117
},
{
"seed": 2,
"answer": "Based on what you've provided, here is a summary of the trip plan so far:\n\n* **Slides:** You have provided two lines of slides containing routine details for the offsite.\n* **Venue Map:** You have provided two lines of the venue map containing routine details for the offsite.\n\n**However, I do not have the actual content of these slides or the venue map.** Therefore, I can only summarize that the plan includes routine details for the offsite in both the presentation material and the venue map.\n\nCould you please provide the actual content of the slides and the venue map so I can give",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 142,
"answer_tokens": 128
},
{
"seed": 3,
"answer": "Based on the information you've provided, here is a summary of the trip plan so far:\n\n* **Slide 1:** Contains routine details for the offsite.\n* **Slide 2:** Contains routine details for the offsite.\n* **Venue Map, Line 1:** Contains routine details for the offsite.\n* **Venue Map, Line 2:** Contains routine details for the offsite.\n\nEssentially, the slides and the venue map currently contain routine details for the offsite.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 142,
"answer_tokens": 109
},
{
"seed": 4,
"answer": "Based on the information you've provided so far, I have the following:\n\n* **Slide 1:** Contains routine details for the offsite.\n* **Slide 2:** Contains routine details for the offsite.\n* **Venue Map, Line 1:** Contains routine details for the offsite.\n* **Venue Map, Line 2:** Contains routine details for the offsite.\n\n**However, I do not have the actual content of these slides or the map.** Therefore, I cannot provide a summary of the trip plan because the details themselves are missing.",
"tool_calls": [],
"done_reason": "stop",
"prompt_tokens": 142,
"answer_tokens": 123
},
{
"seed": 5,
"answer": "Based on the information you've provided so far, here is a summary of the trip plan:\n\n* **Slide 1:** Contains routine details for the offsite.\n* **Slide 2:** Contains routine details for the offsite.\n* **Venue Map, Line 1:** Contains routine details for the offsite.\n* **Venue Map, Line 2:** Contains routine details for the offsite.\n\n**However, I do not have the actual content of these slides or the map.** Therefore, I can only tell you that the plan currently contains the *routine details* for the off",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 142,
"answer_tokens": 128
}
],
"append-only · turn 50": [
{
"seed": 1,
"answer": "Here is a summary of the trip plan based on the stored notes:\n\n* **Hotel Preference:** There is a desire to stay in a hotel near the river (mentioned twice).\n* **Attendees:** Bob will join on the second day.\n* **Dates:** There was a suggestion to move the trip to November 10 to 13, and another note mentions traveling from November 3 to 6.\n* **Flights:** There is a request to book the morning flight if possible.\n* **Budget:** The budget has been raised to 180 euros per night.\n*",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 329,
"answer_tokens": 128
},
{
"seed": 2,
"answer": "Based on the stored notes, here is a summary of the trip plan so far:\n\n* **Accommodation Preference:** There is a desire for a hotel near the river (mentioned in notes [#12] and [#6]).\n* **Travel Dates:** The travel dates have been mentioned as November 3rd to 6th (note [#5]), and there was a suggestion to move the trip to November 10th to 13th (note [#10]).\n* **Attendees:** Bob is scheduled to join the group on the second day (note [#11]).\n* **Flights:**",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 329,
"answer_tokens": 128
},
{
"seed": 3,
"answer": "Based on the stored notes, here is a summary of the trip plan so far:\n\n* **Accommodation Preference:** The user has expressed a desire for a hotel near the river twice.\n* **Attendees:** Bob will be joining on the second day.\n* **Dates:** There were requests to move the trip to November 10 to 13, and another indication of travel dates from November 3 to 6.\n* **Flights:** There was a request to book the morning flight if possible.\n* **Budget:** The budget was raised to 180 euros per night.\n",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 329,
"answer_tokens": 128
},
{
"seed": 4,
"answer": "Here is a summary of the trip plan based on the stored notes:\n\n* **Accommodation:** There is a request for a hotel near the river.\n* **Attendees:** Bob will join on the second day.\n* **Dates:** The trip is planned to be from November 3rd to November 6th.\n* **Budget:** The budget has been raised to 180 euros per night.\n* **Flights:** There is a request to book the morning flight if possible.\n* **Dietary Note:** One person currently eats fish but still avoids meat.\n* **",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 329,
"answer_tokens": 128
},
{
"seed": 5,
"answer": "Based on the stored notes, here is a summary of the trip plan so far:\n\n* **Accommodation Preference:** The user wants a hotel near the river (mentioned twice).\n* **Attendees:** Bob will join on the second day.\n* **Dates:** The trip was moved to November 10 to 13, and travel dates were set from November 3 to 6 (there is a slight conflict here, but both date ranges were noted).\n* **Flights:** It was requested to book the morning flight if possible.\n* **Dietary:** The user currently eats fish but is",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 329,
"answer_tokens": 128
}
],
"managed · turn 50": [
{
"seed": 1,
"answer": "Here is a summary of the trip plan based on the stored notes:\n\n* **Date Change:** The trip has been moved to **November 10 to 13**.\n* **Location:** The destination is **Porto**.\n* **Attendees:** **Bob** will join the group on the second day.\n* **Accommodation Preference:** There is a preference for a **hotel near the river**.\n* **Budget:** The budget has been raised to **180 euros per night**.\n* **Travel:** It is recommended to **book the morning flight if possible**.\n* ",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 305,
"answer_tokens": 128
},
{
"seed": 2,
"answer": "Based on the stored notes, here is a summary of the trip plan so far:\n\n* **Attendees:** Bob will join on the second day.\n* **Dates:** The trip has been moved to November 10 to 13.\n* **Location:** The destination is Porto.\n* **Budget:** The budget has been raised to 180 euros per night.\n* **Accommodation Preference:** There is a preference for a hotel near the river.\n* **Travel:** It is suggested to book the morning flight if possible.\n* **Dietary Note:** One person currently eats fish but still avoids meat",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 305,
"answer_tokens": 128
},
{
"seed": 3,
"answer": "Based on the stored notes, here is a summary of the trip plan so far:\n\n* **Attendees:** Bob will join on the second day.\n* **Dates:** The trip has been moved to November 10 to 13.\n* **Location:** The destination is Porto.\n* **Dietary Note:** One person eats fish but still avoids meat.\n* **Accommodation Preference:** There is a request for a hotel near the river.\n* **Travel:** It is suggested to book the morning flight if possible.\n* **Budget:** The budget has been raised to ",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 305,
"answer_tokens": 128
},
{
"seed": 4,
"answer": "Here is a summary of the trip plan based on the stored notes:\n\n* **Attendees:** Bob will join on the second day.\n* **Dates:** The trip has been moved to November 10 to 13.\n* **Location:** The destination is Porto.\n* **Dietary Notes:** One person currently eats fish but has no meat.\n* **Accommodation:** There is a preference for a hotel near the river.\n* **Flights:** It is recommended to book the morning flight if possible.\n* **Budget:** The budget has been raised to 180",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 305,
"answer_tokens": 128
},
{
"seed": 5,
"answer": "Based on the stored notes, here is a summary of the trip plan so far:\n\n* **Attendees:** Bob will join on the second day.\n* **Dates:** The trip has been moved to November 10 to 13.\n* **Location:** The destination is Porto.\n* **Dietary Note:** The user currently eats fish but still does not eat meat.\n* **Accommodation Preference:** The user wants a hotel near the river.\n* **Flight Preference:** It was suggested to book the morning flight if possible.\n* **Budget:** The budget has been raised",
"tool_calls": [],
"done_reason": "length",
"prompt_tokens": 305,
"answer_tokens": 128
}
]
}
}Inspect the grades and reasons
{
"summary": {
"recent \u00b7 turn 40": {
"pass": 0,
"fail": 5,
"incomplete": 0,
"facts_in_answers": "3 of 30",
"reasons": {
"missing 5 of 6 required facts": 3,
"missing 6 of 6 required facts; stopped at the answer-token limit": 2
}
},
"notes \u00b7 turn 40": {
"pass": 0,
"fail": 5,
"incomplete": 0,
"facts_in_answers": "3 of 30",
"reasons": {
"missing 5 of 6 required facts": 3,
"missing 6 of 6 required facts; stopped at the answer-token limit": 2
}
},
"append-only \u00b7 turn 40": {
"pass": 0,
"fail": 5,
"incomplete": 0,
"facts_in_answers": "18 of 30",
"reasons": {
"missing 2 of 6 required facts; stopped at the answer-token limit": 1,
"missing 3 of 6 required facts; stopped at the answer-token limit": 1,
"states a replaced fact as current: 3 to 6 November": 3
}
},
"managed \u00b7 turn 40": {
"pass": 3,
"fail": 2,
"incomplete": 0,
"facts_in_answers": "28 of 30",
"reasons": {
"missing 1 of 6 required facts; stopped at the answer-token limit": 2
}
},
"recent \u00b7 turn 50": {
"pass": 0,
"fail": 5,
"incomplete": 0,
"facts_in_answers": "0 of 35",
"reasons": {
"missing 7 of 7 required facts": 3,
"missing 7 of 7 required facts; stopped at the answer-token limit": 2
}
},
"notes \u00b7 turn 50": {
"pass": 0,
"fail": 5,
"incomplete": 0,
"facts_in_answers": "0 of 35",
"reasons": {
"missing 7 of 7 required facts": 3,
"missing 7 of 7 required facts; stopped at the answer-token limit": 2
}
},
"append-only \u00b7 turn 50": {
"pass": 0,
"fail": 5,
"incomplete": 0,
"facts_in_answers": "23 of 35",
"reasons": {
"states a replaced fact as current: 3 to 6 November": 5
}
},
"managed \u00b7 turn 50": {
"pass": 2,
"fail": 3,
"incomplete": 0,
"facts_in_answers": "32 of 35",
"reasons": {
"missing 1 of 7 required facts; stopped at the answer-token limit": 3
}
}
},
"graded": {
"recent \u00b7 turn 40": [
"fail",
"fail",
"fail",
"fail",
"fail"
],
"notes \u00b7 turn 40": [
"fail",
"fail",
"fail",
"fail",
"fail"
],
"append-only \u00b7 turn 40": [
"fail",
"fail",
"fail",
"fail",
"fail"
],
"managed \u00b7 turn 40": [
"pass",
"fail",
"pass",
"pass",
"fail"
],
"recent \u00b7 turn 50": [
"fail",
"fail",
"fail",
"fail",
"fail"
],
"notes \u00b7 turn 50": [
"fail",
"fail",
"fail",
"fail",
"fail"
],
"append-only \u00b7 turn 50": [
"fail",
"fail",
"fail",
"fail",
"fail"
],
"managed \u00b7 turn 50": [
"fail",
"pass",
"fail",
"pass",
"fail"
]
}
}Recent turns and notes. Both requests hold the same recent turns, because no cue-word note was found, and every answer summarizes the pasted documents. At turn 40 a few answers also give the new dates, the only fact still in the recent turns.
Append-only memory. Every answer failed. 8 of 10 gave the replaced dates, 3 to 6 November, as the plan or next to the new dates without saying which hold. The store holds both statements with nothing to say which is current, and the model cannot tell either. No answer repeated the passport number, although the turn-40 request contained it. That does not fix the leak: the model still received a number the user asked it to forget.
Managed memory. 5 of 10 answers passed, and no answer stated a replaced fact. Every failure stopped at the 128-token answer limit before its last fact; across all strategies, 27 of 40 answers stopped there. The request held every fact, but the answer reserve was too small for a summary of every fact in this model’s list style. The reserve is part of the budget: raising it lowers the input budget, so the benchmark must be run again to see what that costs. 40 answers from one small model illustrate these failures; they are not a measure of answer quality.
Check your understanding
Before you look at the sweep, predict: if the facts quota grew from 160 to 240 tokens, would append-only stop leaking the forgotten passport number? It would not. The passport fact stays in the append-only store forever, and a bigger quota only means it is sent at more checkpoints. In the sweep, the forgotten-fact count for append-only rises from 1 at quota 160 to 2 at quota 240. At quota 120 it is 0, but not because the leak is fixed: newer facts crowd the passport line out of the quota. Leaks are fixed by deleting, not by budget.
Read the explanation
Deleting a memory removes one copy. Every other place the request draws from, including the recent turns, needs its own rule for corrected and forgotten facts.
Failure cases
Run the tests:
python3 -m unittest -v test_benchmark.py test_answer_check.pyThe suite has 19 tests. The first compares a fresh run with the recording, and another grades the recorded answers again. The benchmark tests run the benchmark in the test process and check its behavior directly; the answer tests grade fixed answers and call no model:
| Case | Expected behavior |
|---|---|
| The fixture drifts from its generator | The stored conversation must equal the generator’s output. |
| A checkpoint names a fact that was never said | Every required, stale and forgotten fact occurs before its checkpoint. |
| A strategy overspends the budget | Every request fits the input budget. |
| A fact is replaced or forgotten | Managed memory never sends it, at any quota. |
| A checkpoint right after a forget request | Managed memory redacts the turn that stated the number; the strategies that send the raw transcript still send it. A turn that quotes the number again is not redacted. |
| Memory never applies forget requests | Append-only sends the forgotten fact at turn 40, and more often with a larger quota. |
| Early facts in a long conversation | Recent turns keep no required fact by turn 50. |
| Facts phrased without cue words | The cue-word extractor matches none of them and does no better than recent turns. |
| More current facts than the facts quota | The oldest current fact is dropped at quota 120; quota 160 keeps all of them. |
| A forget request in the managed store | One deletion record, and fewer current facts than the append-only store. |
| An answer that states the replaced dates as current | Grade it fail; an answer that marks them as changed passes. |
| An answer that repeats the forgotten passport number | Grade it fail. |
| An answer cut off at the answer-token limit | Grade it fail and say it stopped at the limit. |
| An answer model that is not the pinned build | Refuse before sending any request. |
To check that the tests catch real defects, we changed the benchmark one rule at a time. Ignoring forget requests in the managed store fails four tests. Sending the transcript without redaction fails the test for a checkpoint right after a forget request. Ignoring the quota fails three. Letting recent-turn selection skip a turn that does not fit, instead of stopping, fails three. Ignoring corrections is caught only by the recording comparison: consolidation’s newer-statement rule then retires the old fact anyway, so no stale fact reaches the request.
Inspect the tests
import json
from pathlib import Path
import subprocess
import sys
import unittest
import benchmark
from generate_conversation import conversation
HERE = Path(__file__).parent
def recorded():
return json.loads((HERE / 'benchmark-result.json').read_text())
_LIVE = {}
def live():
"""Run the benchmark in this process, so each test checks the code, not only the recording."""
if not _LIVE:
_LIVE.update(benchmark.full_report(case()))
return _LIVE
def case():
return json.loads((HERE / 'conversation.json').read_text())
class BenchmarkTest(unittest.TestCase):
def test_fixture_matches_recorded_result(self):
output = subprocess.run([sys.executable, 'benchmark.py'], cwd=HERE, check=True,
capture_output=True, text=True).stdout
fresh, record = json.loads(output), recorded()
# The Python and SQLite versions describe the run, not the result.
self.assertEqual(set(record.pop('environment')), {'python', 'sqlite'})
fresh.pop('environment')
self.assertEqual(fresh, record)
def test_conversation_matches_its_generator(self):
self.assertEqual(case(), conversation())
self.assertEqual(len(case()['turns']), 50)
def test_every_checked_fact_was_said_before_its_checkpoint(self):
data = case()
for checkpoint in data['checkpoints']:
said = ' '.join(t['user'] for t in data['turns'] if t['turn'] <= checkpoint['after'])
for fact in checkpoint['required'] + checkpoint['stale'] + checkpoint['deleted']:
with self.subTest(after=checkpoint['after'], fact=fact):
self.assertIn(fact, said)
def test_every_request_fits_the_budget(self):
report = live()
for name, rows in report['checkpoints'].items():
for row in rows:
with self.subTest(strategy=name, after=row['after']):
self.assertLessEqual(row['input_tokens'], report['input_budget'])
def test_managed_memory_never_sends_stale_or_forgotten_facts(self):
report = live()
for row in report['checkpoints']['managed']:
self.assertEqual((row['stale'], row['deleted']), ([], []), row['after'])
for totals in report['quota_sweep'].values():
self.assertEqual((totals['managed']['stale'], totals['managed']['deleted']), (0, 0))
def test_managed_request_omits_a_fact_forgotten_one_turn_earlier(self):
data = case()
turns = [t for t in data['turns'] if t['turn'] <= 12]
turns.append({'turn': 13, 'user': 'Please forget my passport number.', 'assistant': 'Noted.',
'memory': {'op': 'forget', 'attribute': 'passport'}})
checkpoint = {'after': 13, 'question': 'Summarize the trip plan so far.', 'required': [], 'stale': [],
'deleted': ['P1234567']}
report = benchmark.run({**data, 'turns': turns, 'checkpoints': [checkpoint]})
leaks = {name: rows[0]['deleted'] for name, rows in report['checkpoints'].items()}
self.assertEqual(leaks['managed'], [])
# The other strategies send the raw transcript, so the number is still in their recent turns.
self.assertEqual([name for name, found in leaks.items() if found], ['recent', 'notes', 'append-only'])
def test_append_only_memory_leaks_the_forgotten_fact(self):
report = live()
at_40 = next(r for r in report['checkpoints']['append-only'] if r['after'] == 40)
self.assertEqual(at_40['deleted'], ['P1234567'])
sweep = report['quota_sweep']
self.assertGreater(sweep['240']['append-only']['deleted'], sweep['160']['append-only']['deleted'])
self.assertGreater(sweep['240']['append-only']['stale'], sweep['160']['append-only']['stale'])
def test_recent_turns_lose_early_facts(self):
at_50 = next(r for r in live()['checkpoints']['recent'] if r['after'] == 50)
self.assertEqual(at_50['found'], 0)
self.assertIn('going to Porto', at_50['missing'])
def test_cue_extraction_finds_none_of_these_facts(self):
facts = [t['user'] for t in case()['turns'] if 'memory' in t]
self.assertEqual([f for f in facts if benchmark.CUES.search(f)], [])
self.assertLessEqual(live()['totals']['notes']['found'], live()['totals']['recent']['found'])
def test_a_small_facts_quota_drops_the_oldest_current_fact(self):
report = live()
self.assertEqual(report['facts_quota'], 160)
self.assertEqual(report['totals']['managed']['found'], report['totals']['managed']['required'])
small = benchmark.run(case(), 120)
at_50 = next(r for r in small['checkpoints']['managed'] if r['after'] == 50)
self.assertEqual(at_50['missing'], ['going to Porto'])
self.assertEqual(small['totals']['managed'], report['quota_sweep']['120']['managed'])
def test_forgotten_fact_is_gone_from_the_managed_store(self):
storage = live()['storage']
self.assertEqual(storage['managed']['deletion_records'], 1)
self.assertEqual(storage['append-only']['deletion_records'], 0)
self.assertLess(storage['managed']['current_facts'], storage['append-only']['current_facts'])
if __name__ == '__main__':
unittest.main()"""Offline tests for the benchmark answer check. They grade recorded answers; no model is called."""
import json
from pathlib import Path
import subprocess
import sys
import unittest
from answer_check import KEYS, checkpoints, grade
from answer_model import MODEL, AnswerModel, AnswerModelError
HERE = Path(__file__).parent
AT_40 = checkpoints()[40]
CURRENT = ('Porto, 10 to 13 November, 180 euros per night, a hotel near the river, '
'the morning flight, and she eats fish now.')
def answer(text='', done_reason='stop'):
return {'seed': 1, 'answer': text, 'tool_calls': [], 'done_reason': done_reason}
class AnswerCheckTest(unittest.TestCase):
def test_recorded_answers_grade_as_recorded(self):
output = subprocess.run([sys.executable, 'answer_check.py'], cwd=HERE, check=True,
capture_output=True, text=True).stdout
self.assertEqual(json.loads(output), json.loads((HERE / 'answer-grades.json').read_text()))
def test_keys_cover_every_checkpoint_fact(self):
for checkpoint in checkpoints().values():
for kind in ('required', 'stale', 'deleted'):
self.assertLessEqual(set(checkpoint[kind]), set(KEYS[kind]))
def test_current_facts_pass(self):
self.assertEqual(grade(AT_40, answer(CURRENT)), ('pass', 'every required fact'))
def test_replaced_dates_stated_as_current_fail(self):
stale = CURRENT + ' The trip is planned from November 3rd to November 6th.'
self.assertEqual(grade(AT_40, answer(stale)), ('fail', 'states a replaced fact as current: 3 to 6 November'))
marked = CURRENT + ' The dates were changed from 3 to 6 November.'
self.assertEqual(grade(AT_40, answer(marked))[0], 'pass')
def test_forgotten_passport_fails(self):
self.assertEqual(grade(AT_40, answer(CURRENT + ' Passport: P1234567.')),
('fail', 'sends the forgotten passport number'))
def test_cut_off_answer_names_the_limit(self):
cut = 'Porto, 10 to 13 November, 180 euros per night, a hotel near the river, the morning'
self.assertEqual(grade(AT_40, answer(cut, 'length')),
('fail', 'missing 1 of 6 required facts; stopped at the answer-token limit'))
def test_empty_answer_is_incomplete(self):
self.assertEqual(grade(AT_40, answer(' ')), ('incomplete', 'no answer'))
def test_unpinned_model_is_refused(self):
def transport(path, payload):
return {'models': [{'name': MODEL['ollama_tag'], 'digest': 'other'}]}
with self.assertRaisesRegex(AnswerModelError, 'pinned digest'):
AnswerModel(transport)
if __name__ == '__main__':
unittest.main()Deployment
Run it as a regression gate
The deployable unit is the benchmark, not a service. Run it whenever a memory rule, a quota, the counter or the extractor changes, and review changes to benchmark-result.json like code: which checkpoints gained or lost facts, and whether any stale or forgotten fact appeared. A new forgotten-fact leak should block the change.
Extend the fixture with conversations from your own application, with personal data removed, and add an extractor strategy that writes memories from text instead of labels. The gap between that strategy and managed is the cost of imperfect extraction, and it is the number this benchmark cannot show yet.
The benchmark is deterministic, so one run is enough; the answers are not, so they were recorded once from gemma4:e2b through Ollama 0.32.14 and are graded again offline. No timing is recorded. No server deployment was made. The tests pass on Python 3.11, the version of the tokenizer environment.
Scaling
Cost of the benchmark
The replay makes one store write per memory event and, at each checkpoint, a consolidation pass and four request builds. Each build counts the whole request once per turn it tries to keep, with the Gemma tokenizer and chat template, so the number of counter calls grows with the number of kept turns. Longer conversations and more checkpoints grow the cost linearly in both. The answer check adds one model call per request and seed.
What the numbers do not show
One synthetic user, one conversation and five checkpoints show how each strategy fails. They do not estimate how often each fails for real users. Report results per conversation and per checkpoint, not only totals, and keep at least one conversation you did not look at while tuning quotas.
Key takeaways
- Test memory over a long conversation with facts that change and a fact the user asks to forget.
- Retention alone is not enough: count stale facts and forgotten-fact leaks too.
- Appending memories improves retention but leaks replaced and forgotten facts, and a larger budget makes the leak worse.
- A facts quota is a trade-off: too small drops the oldest valid facts; measure it.
- A fact in the request is not a fact in the answer: unresolved old facts confuse the model, and a small answer reserve cuts the answer short.
Best practices
- Generate the fixture from a script and test that the stored file matches it.
- Check that every graded fact was said before its checkpoint.
- Fail the run if any strategy exceeds the budget.
- Treat a forgotten-fact leak as a blocking failure.
References
- Build memory consolidation with conflict tests: the version 4 store and consolidation used by the managed strategy.
- Build context compaction with information-loss tests: the cue-word extractor used by the notes strategy.
- Build a context-selection benchmark: the case-set and grader conventions this benchmark follows.
- Gemma 4 E2B model card and Ollama chat API: the model and server used to record the answers. Checked October 20, 2026.
The conversation, strategies, recorded report and tests belong to this build. No external source validates these results.