A corpus update can change the meaning of an evaluation target even when its logical source ID stays the same. Reusing the old label may score a passage whose exact text is no longer present. Refuse that stale binding before comparing retrieval results.
When you need this. Use this workflow when changed or deleted documents require you to decide which retrieval cases remain comparable across corpus snapshots.
This lab runs the existing lexical retriever on fixed development questions. It does not generate answers, load held-out questions, tune a retriever or call a model. The corpus change is explicitly synthetic, not an official Python documentation update.
Freeze the requests and make the synthetic update explicit
Reuse the baseline snapshot, development cases and lexical ranking rules from the earlier retrieval comparison. This evaluation needs SQLite FTS5, without the vector SDK.
The updated fixture retains the original write summary verbatim as its prefix, then appends clearly labelled non-factual length padding in the same paragraph. That changes source bytes, revision, paragraph content and chunk identity. The JSON summary is deleted. Other summaries retain their original text and provenance.
Run the same original questions, requested API version, candidate budget and cutoffs on both snapshots. Retrieval receives only case ID, query and version; expected target labels enter after ranking. The native SQLite FTS5 retriever quotes each extracted term, applies the version filter before top-k and preserves deterministic tie ordering.
Refuse stale targets before interpreting a score
A target binds source logical ID, API version, raw source hash and chunk-text hash. The old legacy target ID does not equal the current versioned chunk ID. The binder requires one exact current match rather than assuming that a stable logical ID preserves text.
Stale target. Binding the original labels against the updated snapshot is refused. Both changed write targets and the deleted JSON target remain unresolved at this stage. No retrieval score or hash match on another field repairs that failure.
The actual initial binding attempt reports refused with code target_binding_not_bijective. The comparison retains this attempt alongside the subsequent explicit annotations.
Reannotate retained sources through a trusted local input
The local annotation file is separate from retrieval. It names both snapshots, the exact old target, the exact new target and a reason. Here the answer-bearing write summary remains as an unchanged prefix, so the fixture operator explicitly rebinds those questions to the changed whole paragraph.
This is a trusted local labeling decision, not an automatic semantic check or a model judgment. The checker validates its consistency; it does not establish author identity, permission or factual correctness. The output keeps old and new target fields and hashes so the change is reviewable.
Changed annotation. A changed old hash, new hash or claimed current chunk ID is refused. The new target must uniquely match the updated snapshot and retain the original logical source and version. Deleted target. The JSON case cannot be repaired by silently pointing its label at a different source. It remains not comparable with an explicit deletion reason and null metrics.
Compare ranks with their case and target denominators
For a supported bound case, recall at a cutoff divides retrieved target chunks by all labeled target chunks. Reciprocal rank is the inverse of the first target rank within the cutoff, or zero for a miss. The record keeps ranks, target counts and miss flags for each case.
Changed saved ranking. The validator re-runs the exact local lexical requests against the recorded snapshot and configuration, then compares order, native scores and result completeness. A reordered, rescored or shortened ranking is refused even if its self-hash has been recomputed. This deterministic recheck runs no model.
Unsupported case. A question without a target has null recall, reciprocal rank and miss values. Its returned candidate count remains available; retrieval alone does not establish answer abstention. Changed comparison request. Different query or configuration bindings cannot be treated as the same experiment.
| Development case | Comparison | Before target ranks | After target ranks |
|---|---|---|---|
| utf8-write | paired-supported | [2] | [2] |
| replace-file | paired-supported | [1] | [1] |
| json-result-type | not-comparable | [1] | null |
| backup-schedule | unsupported-excluded | [] | [] |
The paired means include only supported cases with exact bindings in both snapshots. Deleted or unresolved cases are not assigned a convenient zero or silently removed from the report. Unsupported questions remain separate. Read the paired denominator together with the excluded cases before interpreting any mean.
| Cutoff | Before mean recall | After mean recall | Paired supported cases |
|---|---|---|---|
| 1 | 0.5 | 0.5 | 2 |
| 2 | 1 | 1 | 2 |
The actual paired rankings and means are unchanged. The deleted target is still a loss of comparable case coverage. This tiny, already-used development set and synthetic padding do not establish a quality gain, a general benchmark result or an answer-generation outcome.
Run the evaluation and inspect a refused annotation
Download the complete updated-corpus evaluation lab. It needs Python with standard-library SQLite FTS5. The evaluation runs locally without a model.
"""Fixed-development lexical evaluation across an explicitly synthetic corpus update."""
import copy
import hashlib
import json
import math
from pathlib import Path
from evaluation_bindings import bind_targets,metrics
from lexical import QuestionLexicalRetriever,question_terms
from version_documents import canonical_hash,validate_snapshot
CONFIG={'retriever':'sqlite-fts5-bm25','tokenizer':'module6-ascii-quoted-or-v1','candidate_budget':3,'cutoffs':[1,2]}
CASE_IDS={'utf8-write','replace-file','json-result-type','backup-schedule'}
PROVENANCE_SHA256='96d9cb5db4165b6e4d61408d20b3f1a6c935af51d83f1c54646e3f1ca407c7fe'
def checked_cases(cases):
if not isinstance(cases,list) or len(cases)!=4 or any(not isinstance(c,dict) or not isinstance(c.get('id'),str) for c in cases):raise ValueError('fixed_development_cases_required')
if {c['id'] for c in cases}!=CASE_IDS:raise ValueError('fixed_development_cases_required')
for case in cases:
if case.get('split')!='development' or case.get('class') not in ('answerable','unsupported'):raise ValueError('development_only')
if not isinstance(case.get('input'),str) or not isinstance(case.get('tags'),dict):raise ValueError('invalid_case')
if not isinstance(case.get('expected'),dict):raise ValueError('invalid_target_class')
targets=case['expected'].get('targets')
if not isinstance(targets,list) or (case['class']=='answerable')!=bool(targets):raise ValueError('invalid_target_class')
return cases
def requests_from_cases(cases):
return [{'case_id':c['id'],'query':c['input'],'version':c['tags']['python_version']} for c in checked_cases(cases)]
def load_inputs(root=None):
root=Path(root) if root is not None else Path(__file__).parent/'fixtures'
raw=(root/'provenance.json').read_bytes()
if hashlib.sha256(raw).hexdigest()!=PROVENANCE_SHA256:raise ValueError('provenance_hash_mismatch')
provenance=json.loads(raw)
for relative,expected in provenance['fixture_sha256'].items():
path=Path(relative)
if path.is_absolute() or '..' in path.parts or not path.parts or path.parts[0]!='fixtures':raise ValueError('invalid_fixture_path')
if hashlib.sha256((root/Path(*path.parts[1:])).read_bytes()).hexdigest()!=expected:raise ValueError('fixture_hash_mismatch')
before=json.loads((root/'baseline-snapshot.json').read_text());after=json.loads((root/'updated-snapshot.json').read_text())
validate_snapshot(before);validate_snapshot(after)
return before,after,checked_cases(json.loads((root/'development-cases.json').read_text())),json.loads((root/'reannotations.json').read_text())
def _config(config):
try:equal=canonical_hash(config)==canonical_hash(CONFIG)
except (TypeError,ValueError,OverflowError) as error:raise ValueError('unsupported_config') from error
if not equal:raise ValueError('unsupported_config')
return copy.deepcopy(config)
def _requests(requests):
if not isinstance(requests,list) or len(requests)!=4:raise ValueError('fixed_requests_required')
if any(not isinstance(r,dict) or not isinstance(r.get('case_id'),str) for r in requests):raise ValueError('case_join_mismatch')
if {r['case_id'] for r in requests}!=CASE_IDS:raise ValueError('case_join_mismatch')
for row in requests:
if not isinstance(row,dict) or set(row)!={'case_id','query','version'}:raise ValueError('labels_not_retrieval_inputs')
question_terms(row['query'])
if not isinstance(row['version'],str):raise ValueError('invalid_version')
return copy.deepcopy(requests)
def run_rankings(snapshot,requests,config=None):
validate_snapshot(snapshot);requests=_requests(requests);config=_config(CONFIG if config is None else config)
if sum(len(d['chunks']) for d in snapshot['documents'])>100:raise ValueError('corpus_bound')
retriever=QuestionLexicalRetriever(snapshot);rows=[]
try:
for request in requests:
ranked=retriever.search(request['query'],k=config['candidate_budget'],policy={'version':{'$eq':request['version']}})
rows.append({'case_id':request['case_id'],'query_terms':question_terms(request['query']),'ranked':ranked})
finally:retriever.close()
body={'snapshot_id':snapshot['snapshot_id'],'requests':requests,'request_sha256':canonical_hash(requests),'config':config,'config_sha256':canonical_hash(config),'rows':rows}
return {**body,'run_sha256':canonical_hash(body)}
def validate_run(snapshot,run):
validate_snapshot(snapshot)
try:
_requests(run['requests']);_config(run['config'])
if run['snapshot_id']!=snapshot['snapshot_id'] or run['request_sha256']!=canonical_hash(run['requests']) or run['config_sha256']!=canonical_hash(run['config']):raise ValueError()
if run['run_sha256']!=canonical_hash({k:v for k,v in run.items() if k!='run_sha256'}):raise ValueError()
expected={r['case_id']:r for r in run['requests']};rows=run['rows']
if not isinstance(rows,list) or len(rows)!=len(expected) or {r['case_id'] for r in rows}!=set(expected):raise ValueError()
chunks={c['chunk_id']:{'chunk_id':c['chunk_id'],'source_id':d['source_id'],'version':d['version'],'url':d['url'],'text':c['text']} for d in snapshot['documents'] for c in d['chunks']}
for row in rows:
request=expected[row['case_id']]
if row['query_terms']!=question_terms(request['query']) or len(row['ranked'])>run['config']['candidate_budget']:raise ValueError()
seen=set()
for item in row['ranked']:
if item['chunk_id'] in seen or item['chunk_id'] not in chunks or item['version']!=request['version']:raise ValueError()
seen.add(item['chunk_id'])
if any(item.get(k)!=v for k,v in chunks[item['chunk_id']].items()) or type(item['native_score']) not in (int,float) or not math.isfinite(item['native_score']):raise ValueError()
# This deterministic local retriever can rederive scores/order/completeness.
# A self-hash alone would accept a consistently rehashed forged ranking.
if canonical_hash(rows)!=canonical_hash(run_rankings(snapshot,run['requests'],run['config'])['rows']):raise ValueError()
except (KeyError,TypeError,ValueError,OverflowError) as exc:raise ValueError('invalid_run_binding') from exc
return run
def reannotate(before,after,cases,annotations):
"""Trusted local operator decisions, checked against both exact snapshots.
This checks consistency, not author identity, factual validity or permission.
A deleted source cannot be made supported by silently selecting another source.
"""
checked_cases(cases);bind_targets(before,cases);validate_snapshot(after)
revised=copy.deepcopy(cases);by_id={c['id']:c for c in revised};receipts=[];seen=set()
if not isinstance(annotations,list):raise ValueError('invalid_reannotations')
for annotation in annotations:
try:
if set(annotation)!={'case_id','old_snapshot_id','new_snapshot_id','old_target','new_target','reason'}:raise ValueError()
cid=annotation['case_id']
if cid in seen or cid not in by_id:raise ValueError()
seen.add(cid);case=by_id[cid]
if annotation['old_snapshot_id']!=before['snapshot_id'] or annotation['new_snapshot_id']!=after['snapshot_id']:raise ValueError()
old,new=annotation['old_target'],annotation['new_target']
if len(case['expected']['targets'])!=1 or canonical_hash(old)!=canonical_hash(case['expected']['targets'][0]):raise ValueError()
if new['source_id']!=old['source_id'] or new['source_version']!=old['source_version']:raise ValueError()
if not isinstance(annotation['reason'],str) or not annotation['reason'].strip() or len(annotation['reason'])>1024:raise ValueError()
candidate=copy.deepcopy(case);candidate['expected']['targets']=[copy.deepcopy(new)]
bound,_=bind_targets(after,[candidate])
if bound[cid]['target_chunk_ids']!=[new['chunk_id']]:raise ValueError()
case['expected']['targets']=[copy.deepcopy(new)]
receipt={**copy.deepcopy(annotation),'new_current_chunk_ids':bound[cid]['target_chunk_ids'],'old_target_sha256':canonical_hash(old),'new_target_sha256':canonical_hash(new)}
receipts.append({**receipt,'receipt_sha256':canonical_hash(receipt)})
except (KeyError,TypeError,ValueError) as exc:raise ValueError('invalid_reannotation') from exc
return revised,receipts
def _score(snapshot,run,cases):
by_id={r['case_id']:r['ranked'] for r in run['rows']};result=[]
for case in cases:
try:bindings,_=bind_targets(snapshot,[case])
except ValueError:
source_ids={d['source_id'] for d in snapshot['documents']}
missing=any(t['source_id'] not in source_ids for t in case['expected']['targets'])
result.append({'case_id':case['id'],'status':'not-comparable','reason':'deleted_target_source' if missing else 'stale_target_binding','metrics':None});continue
targets=bindings[case['id']]['target_chunk_ids']
result.append({'case_id':case['id'],'status':'supported' if targets else 'unsupported-excluded','target_chunk_ids':targets,'returned_count':len(by_id[case['id']]),'metrics':{str(k):{**metrics(by_id[case['id']],targets,k),'target_count':len(targets),'miss':not any(r['chunk_id'] in targets for r in by_id[case['id']][:k]) if targets else None} for k in CONFIG['cutoffs']}})
return result
def compare_runs(before,after,before_run,after_run,cases,annotations):
validate_run(before,before_run);validate_run(after,after_run);checked_cases(cases)
if canonical_hash(before_run['requests'])!=canonical_hash(after_run['requests']) or canonical_hash(before_run['config'])!=canonical_hash(after_run['config']):raise ValueError('comparison_context_mismatch')
if canonical_hash(before_run['requests'])!=canonical_hash(requests_from_cases(cases)):raise ValueError('case_request_mismatch')
bind_targets(before,cases)
stale=_score(after,after_run,cases);revised,receipts=reannotate(before,after,cases,annotations)
previous=_score(before,before_run,cases);current=_score(after,after_run,revised);current_by={r['case_id']:r for r in current};paired=[]
for old in previous:
new=current_by[old['case_id']]
row={'case_id':old['case_id'],'before':old,'after':new,'comparison':'paired-supported' if old['status']==new['status']=='supported' else 'not-comparable' if new['status']=='not-comparable' else 'unsupported-excluded'}
if row['comparison']=='paired-supported':row['recall_delta']={str(k):new['metrics'][str(k)]['recall_at_k']-old['metrics'][str(k)]['recall_at_k'] for k in CONFIG['cutoffs']}
paired.append(row)
paired_supported=[r for r in paired if r['comparison']=='paired-supported'];aggregate={}
for k in CONFIG['cutoffs']:
aggregate[str(k)]={'paired_supported_denominator':len(paired_supported),'before_mean_recall':math.fsum(r['before']['metrics'][str(k)]['recall_at_k'] for r in paired_supported)/len(paired_supported) if paired_supported else None,'after_mean_recall':math.fsum(r['after']['metrics'][str(k)]['recall_at_k'] for r in paired_supported)/len(paired_supported) if paired_supported else None}
return {'same_request_sha256':before_run['request_sha256'],'same_config_sha256':before_run['config_sha256'],'stale_binding_attempt':stale,'reannotation_receipts':receipts,'per_case':paired,'paired_aggregate':aggregate,'not_comparable_cases':[r['case_id'] for r in paired if r['comparison']=='not-comparable'],'unsupported_cases':[r['case_id'] for r in paired if r['comparison']=='unsupported-excluded']}python run.py --output runs/my-run/result.json
python -m unittest discover -vFixed development corpus-update evaluation (SQLite FTS5; no model)
Stale labels: refused (target_binding_not_bijective)
utf8-write: target ranks [2] -> [2]; recall@1 0.0 -> 0.0; target denominator=1
replace-file: target ranks [1] -> [1]; recall@1 1.0 -> 1.0; target denominator=1
json-result-type: not-comparable; metrics null
backup-schedule: unsupported-excluded; metrics null
Paired recall@1: 0.5 -> 0.5; supported case denominator=2
Paired recall@2: 1.0 -> 1.0; supported case denominator=2
Development-only observations; no quality gain or held-out estimate claimed.First inspect the stale binding attempt. Then compare the exact old and new annotation hashes for a retained write target. Predict what happens if you change the annotation’s current chunk ID or redirect the deleted JSON target to the write source. Run the corresponding named regression tests and explain why a valid retrieval run cannot authorize that relabeling.
Inspect the per-case rankings and metric denominators in the saved result before comparing the means.
Inspect the complete saved result
{
"schema_version": 1,
"mode": "actual-offline-sqlite-fixed-development-evaluation",
"model_called": false,
"synthetic_update": {
"description": "Retain original path-write summary; append explicit synthetic non-factual length padding; delete json-text. Not an official documentation update.",
"before_snapshot_id": "0615d22a3315e97c0a88c6c566c546877304f5a28a97e83f68dd36881d4bbd0e",
"after_snapshot_id": "a73b29dc573ae0d6aceccf24fd827a2c3b05dade0739d8f95ea0170b1654bdc9"
},
"config": {
"retriever": "sqlite-fts5-bm25",
"tokenizer": "module6-ascii-quoted-or-v1",
"candidate_budget": 3,
"cutoffs": [
1,
2
]
},
"stale_binding_refusal": {
"status": "refused",
"code": "target_binding_not_bijective"
},
"baseline_run": {
"snapshot_id": "0615d22a3315e97c0a88c6c566c546877304f5a28a97e83f68dd36881d4bbd0e",
"requests": [
{
"case_id": "utf8-write",
"query": "How do I explicitly select UTF-8 when saving text with pathlib?",
"version": "3.12"
},
{
"case_id": "replace-file",
"query": "Will write_text replace the contents of an existing file?",
"version": "3.12"
},
{
"case_id": "json-result-type",
"query": "Does json.dumps produce text or a bytes object?",
"version": "3.12"
},
{
"case_id": "backup-schedule",
"query": "How frequently does our cloud provider back up these files?",
"version": "3.12"
}
],
"request_sha256": "ef92a4d049718e959037f38b80c6624ea5c45aa6612cf8d69b5218276fc1f388",
"config": {
"retriever": "sqlite-fts5-bm25",
"tokenizer": "module6-ascii-quoted-or-v1",
"candidate_budget": 3,
"cutoffs": [
1,
2
]
},
"config_sha256": "b546787ceb372881fb58a2dd847b58dac1661c31d4ca4e9d829eb0f847eaa92c",
"rows": [
{
"case_id": "utf8-write",
"query_terms": [
"how",
"do",
"i",
"explicitly",
"select",
"utf",
"8",
"when",
"saving",
"text",
"with",
"pathlib"
],
"ranked": [
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"native_score": -0.8901430296001853
},
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"native_score": -4.904418516387322e-06
},
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"native_score": -9.765013054830289e-07
}
]
},
{
"case_id": "replace-file",
"query_terms": [
"will",
"write",
"text",
"replace",
"the",
"contents",
"of",
"an",
"existing",
"file"
],
"ranked": [
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"native_score": -2.3976548397386743
},
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"native_score": -4.084683995209214e-06
},
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"native_score": -3.3056246724488786e-06
}
]
},
{
"case_id": "json-result-type",
"query_terms": [
"does",
"json",
"dumps",
"produce",
"text",
"or",
"a",
"bytes",
"object"
],
"ranked": [
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"native_score": -3.628237730877387
},
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"native_score": -3.0341221974564053e-06
},
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"native_score": -2.263641817358195e-06
}
]
},
{
"case_id": "backup-schedule",
"query_terms": [
"how",
"frequently",
"does",
"our",
"cloud",
"provider",
"back",
"up",
"these",
"files"
],
"ranked": [
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"native_score": -0.8273874668010814
}
]
}
],
"run_sha256": "3ff3b453089f014ef37512b5b1f379f507077c9ec50b023c1e7e931e680d395e"
},
"updated_run": {
"snapshot_id": "a73b29dc573ae0d6aceccf24fd827a2c3b05dade0739d8f95ea0170b1654bdc9",
"requests": [
{
"case_id": "utf8-write",
"query": "How do I explicitly select UTF-8 when saving text with pathlib?",
"version": "3.12"
},
{
"case_id": "replace-file",
"query": "Will write_text replace the contents of an existing file?",
"version": "3.12"
},
{
"case_id": "json-result-type",
"query": "Does json.dumps produce text or a bytes object?",
"version": "3.12"
},
{
"case_id": "backup-schedule",
"query": "How frequently does our cloud provider back up these files?",
"version": "3.12"
}
],
"request_sha256": "ef92a4d049718e959037f38b80c6624ea5c45aa6612cf8d69b5218276fc1f388",
"config": {
"retriever": "sqlite-fts5-bm25",
"tokenizer": "module6-ascii-quoted-or-v1",
"candidate_budget": 3,
"cutoffs": [
1,
2
]
},
"config_sha256": "b546787ceb372881fb58a2dd847b58dac1661c31d4ca4e9d829eb0f847eaa92c",
"rows": [
{
"case_id": "utf8-write",
"query_terms": [
"how",
"do",
"i",
"explicitly",
"select",
"utf",
"8",
"when",
"saving",
"text",
"with",
"pathlib"
],
"ranked": [
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"native_score": -0.7120246369373029
},
{
"chunk_id": "f0b79b313377631927e732d10c888aa25a11a480986465651bf16193a23d1062",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.\nSYNTHETIC EVALUATION LENGTH PADDING (not Python API documentation): quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz end synthetic padding.",
"native_score": -0.32726262306934556
}
]
},
{
"case_id": "replace-file",
"query_terms": [
"will",
"write",
"text",
"replace",
"the",
"contents",
"of",
"an",
"existing",
"file"
],
"ranked": [
{
"chunk_id": "f0b79b313377631927e732d10c888aa25a11a480986465651bf16193a23d1062",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.\nSYNTHETIC EVALUATION LENGTH PADDING (not Python API documentation): quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz end synthetic padding.",
"native_score": -1.3090415613310908
},
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"native_score": -4.944802852803522e-06
}
]
},
{
"case_id": "json-result-type",
"query_terms": [
"does",
"json",
"dumps",
"produce",
"text",
"or",
"a",
"bytes",
"object"
],
"ranked": [
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"native_score": -3.550942630726448e-06
},
{
"chunk_id": "f0b79b313377631927e732d10c888aa25a11a480986465651bf16193a23d1062",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.\nSYNTHETIC EVALUATION LENGTH PADDING (not Python API documentation): quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz quartz end synthetic padding.",
"native_score": -1.6329758196466427e-06
}
]
},
{
"case_id": "backup-schedule",
"query_terms": [
"how",
"frequently",
"does",
"our",
"cloud",
"provider",
"back",
"up",
"these",
"files"
],
"ranked": []
}
],
"run_sha256": "3c6d8d5989ee1f1267389d2fbce98bbdc2b755eb83318248ba63397fb05e440f"
},
"comparison": {
"same_request_sha256": "ef92a4d049718e959037f38b80c6624ea5c45aa6612cf8d69b5218276fc1f388",
"same_config_sha256": "b546787ceb372881fb58a2dd847b58dac1661c31d4ca4e9d829eb0f847eaa92c",
"stale_binding_attempt": [
{
"case_id": "utf8-write",
"status": "not-comparable",
"reason": "stale_target_binding",
"metrics": null
},
{
"case_id": "replace-file",
"status": "not-comparable",
"reason": "stale_target_binding",
"metrics": null
},
{
"case_id": "json-result-type",
"status": "not-comparable",
"reason": "deleted_target_source",
"metrics": null
},
{
"case_id": "backup-schedule",
"status": "unsupported-excluded",
"target_chunk_ids": [],
"returned_count": 0,
"metrics": {
"1": {
"target_ranks": [],
"recall_at_k": null,
"reciprocal_rank_at_k": null,
"status": "unsupported-excluded",
"target_count": 0,
"miss": null
},
"2": {
"target_ranks": [],
"recall_at_k": null,
"reciprocal_rank_at_k": null,
"status": "unsupported-excluded",
"target_count": 0,
"miss": null
}
}
}
],
"reannotation_receipts": [
{
"case_id": "utf8-write",
"old_snapshot_id": "0615d22a3315e97c0a88c6c566c546877304f5a28a97e83f68dd36881d4bbd0e",
"new_snapshot_id": "a73b29dc573ae0d6aceccf24fd827a2c3b05dade0739d8f95ea0170b1654bdc9",
"old_target": {
"source_id": "path-write",
"chunk_id": "path-write:b71a2d18a53d86bf",
"source_version": "3.12",
"source_text_sha256": "a48b4b9f763cd6309ebe32af19c5a814adcf810c2259683da9f328ccee46948e",
"chunk_text_sha256": "a48b4b9f763cd6309ebe32af19c5a814adcf810c2259683da9f328ccee46948e"
},
"new_target": {
"source_id": "path-write",
"chunk_id": "f0b79b313377631927e732d10c888aa25a11a480986465651bf16193a23d1062",
"source_version": "3.12",
"source_text_sha256": "d4f6834b416b81968ef89b23b3e06337fa97254fcb83250f7418a5a9f10ff36e",
"chunk_text_sha256": "d4f6834b416b81968ef89b23b3e06337fa97254fcb83250f7418a5a9f10ff36e"
},
"reason": "Original answer-bearing summary is retained verbatim as prefix; explicit synthetic non-factual padding changes byte/chunk identity. Local operator explicitly rebinds this target, not a model decision.",
"new_current_chunk_ids": [
"f0b79b313377631927e732d10c888aa25a11a480986465651bf16193a23d1062"
],
"old_target_sha256": "0bcdf743b34827b0aed93b558f1e39399d8aacb20c07d2ab64b09b44887c6ef3",
"new_target_sha256": "f00a632a08e25e7b3d8d5e0ab4568d23bfd41f8e2e533f82d7677b15ab2c44e3",
"receipt_sha256": "323ea6ca8d5b41de8078ec7285621ad48e7afe664490ff610dace9cb7bd18c13"
},
{
"case_id": "replace-file",
"old_snapshot_id": "0615d22a3315e97c0a88c6c566c546877304f5a28a97e83f68dd36881d4bbd0e",
"new_snapshot_id": "a73b29dc573ae0d6aceccf24fd827a2c3b05dade0739d8f95ea0170b1654bdc9",
"old_target": {
"source_id": "path-write",
"chunk_id": "path-write:b71a2d18a53d86bf",
"source_version": "3.12",
"source_text_sha256": "a48b4b9f763cd6309ebe32af19c5a814adcf810c2259683da9f328ccee46948e",
"chunk_text_sha256": "a48b4b9f763cd6309ebe32af19c5a814adcf810c2259683da9f328ccee46948e"
},
"new_target": {
"source_id": "path-write",
"chunk_id": "f0b79b313377631927e732d10c888aa25a11a480986465651bf16193a23d1062",
"source_version": "3.12",
"source_text_sha256": "d4f6834b416b81968ef89b23b3e06337fa97254fcb83250f7418a5a9f10ff36e",
"chunk_text_sha256": "d4f6834b416b81968ef89b23b3e06337fa97254fcb83250f7418a5a9f10ff36e"
},
"reason": "Original answer-bearing summary is retained verbatim as prefix; explicit synthetic non-factual padding changes byte/chunk identity. Local operator explicitly rebinds this target, not a model decision.",
"new_current_chunk_ids": [
"f0b79b313377631927e732d10c888aa25a11a480986465651bf16193a23d1062"
],
"old_target_sha256": "0bcdf743b34827b0aed93b558f1e39399d8aacb20c07d2ab64b09b44887c6ef3",
"new_target_sha256": "f00a632a08e25e7b3d8d5e0ab4568d23bfd41f8e2e533f82d7677b15ab2c44e3",
"receipt_sha256": "d7d58cdd541b020eed42d776d853bf8aa4ac1b02c4eb43189e08f8204ff93723"
}
],
"per_case": [
{
"case_id": "utf8-write",
"before": {
"case_id": "utf8-write",
"status": "supported",
"target_chunk_ids": [
"c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7"
],
"returned_count": 3,
"metrics": {
"1": {
"target_ranks": [
2
],
"recall_at_k": 0.0,
"reciprocal_rank_at_k": 0.0,
"status": "supported",
"target_count": 1,
"miss": true
},
"2": {
"target_ranks": [
2
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 0.5,
"status": "supported",
"target_count": 1,
"miss": false
}
}
},
"after": {
"case_id": "utf8-write",
"status": "supported",
"target_chunk_ids": [
"f0b79b313377631927e732d10c888aa25a11a480986465651bf16193a23d1062"
],
"returned_count": 2,
"metrics": {
"1": {
"target_ranks": [
2
],
"recall_at_k": 0.0,
"reciprocal_rank_at_k": 0.0,
"status": "supported",
"target_count": 1,
"miss": true
},
"2": {
"target_ranks": [
2
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 0.5,
"status": "supported",
"target_count": 1,
"miss": false
}
}
},
"comparison": "paired-supported",
"recall_delta": {
"1": 0.0,
"2": 0.0
}
},
{
"case_id": "replace-file",
"before": {
"case_id": "replace-file",
"status": "supported",
"target_chunk_ids": [
"c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7"
],
"returned_count": 3,
"metrics": {
"1": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported",
"target_count": 1,
"miss": false
},
"2": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported",
"target_count": 1,
"miss": false
}
}
},
"after": {
"case_id": "replace-file",
"status": "supported",
"target_chunk_ids": [
"f0b79b313377631927e732d10c888aa25a11a480986465651bf16193a23d1062"
],
"returned_count": 2,
"metrics": {
"1": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported",
"target_count": 1,
"miss": false
},
"2": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported",
"target_count": 1,
"miss": false
}
}
},
"comparison": "paired-supported",
"recall_delta": {
"1": 0.0,
"2": 0.0
}
},
{
"case_id": "json-result-type",
"before": {
"case_id": "json-result-type",
"status": "supported",
"target_chunk_ids": [
"e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c"
],
"returned_count": 3,
"metrics": {
"1": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported",
"target_count": 1,
"miss": false
},
"2": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported",
"target_count": 1,
"miss": false
}
}
},
"after": {
"case_id": "json-result-type",
"status": "not-comparable",
"reason": "deleted_target_source",
"metrics": null
},
"comparison": "not-comparable"
},
{
"case_id": "backup-schedule",
"before": {
"case_id": "backup-schedule",
"status": "unsupported-excluded",
"target_chunk_ids": [],
"returned_count": 1,
"metrics": {
"1": {
"target_ranks": [],
"recall_at_k": null,
"reciprocal_rank_at_k": null,
"status": "unsupported-excluded",
"target_count": 0,
"miss": null
},
"2": {
"target_ranks": [],
"recall_at_k": null,
"reciprocal_rank_at_k": null,
"status": "unsupported-excluded",
"target_count": 0,
"miss": null
}
}
},
"after": {
"case_id": "backup-schedule",
"status": "unsupported-excluded",
"target_chunk_ids": [],
"returned_count": 0,
"metrics": {
"1": {
"target_ranks": [],
"recall_at_k": null,
"reciprocal_rank_at_k": null,
"status": "unsupported-excluded",
"target_count": 0,
"miss": null
},
"2": {
"target_ranks": [],
"recall_at_k": null,
"reciprocal_rank_at_k": null,
"status": "unsupported-excluded",
"target_count": 0,
"miss": null
}
}
},
"comparison": "unsupported-excluded"
}
],
"paired_aggregate": {
"1": {
"paired_supported_denominator": 2,
"before_mean_recall": 0.5,
"after_mean_recall": 0.5
},
"2": {
"paired_supported_denominator": 2,
"before_mean_recall": 1.0,
"after_mean_recall": 1.0
}
},
"not_comparable_cases": [
"json-result-type"
],
"unsupported_cases": [
"backup-schedule"
]
},
"network_connection_attempts": []
}Key takeaways
A stable logical source ID does not preserve an exact evaluation target. Refuse stale labels, make reannotation explicit, and keep deletion and unsupported cases visible beside paired denominators.
Best practices
Separate retrieval inputs from labels. Freeze requests and configuration, retain old/new target identities, and report lost comparability without inventing a quality improvement.
Read the explanation
Exact target binding, explicit local reannotation and visible comparison denominators are separate contracts.
What’s next
Continue with Evaluate Retrieval-Agent Tasks.
References
The lab reuses Compare Retrieval Configurations and Version Documents and Chunks. The synthetic update, local reannotation policy and paired-case reporting are application choices demonstrated by the downloadable code and recorded execution.