Run lexical, vector, fused and coverage-reranked retrieval over the same preserved development questions. The comparison keeps per-question results, misses and unsupported candidate behavior. It does not tune a configuration or call a model.
When you need this. Use it when you need to compare retrieval changes against fixed source targets while keeping the candidate pipeline separate from the labels used to check its output.
Prerequisites. Use the preceding hybrid and reranking labs. Review the evaluation dataset for target labels and development splits. The pinned dependencies require Python 3.12 or later.
A tie is a useful result: the unchanged UTF-8 miss tells us these scoring changes have not improved that task. The comparison keeps the same development questions and reports equal supported-query metrics. Keep the failure as the baseline rather than choosing a different question to make a configuration look better.
The baseline’s held-out result belongs to its original configuration. These development comparisons do not recompute a held-out score for the new retrieval paths. Keep that distinction when selecting a configuration; reusing a score would conceal the changed index and ranking behavior.
Freeze inputs before comparing configurations
The program fixes the configuration names, candidate budget, rank constant and evaluation cutoffs before running retrieval. Each configuration uses the same question, requested version, snapshot and saved vector recording. The lexical branch keeps its documented literal-OR question representation; a vector question remains the original complete string.
Before retrieval, the program projects each case to its ID, query and requested version. It does not pass expected sources, answer labels or target chunk IDs into either retrieval or coverage scoring. The preserved adapter exposes only development questions. Historic held-out vectors remain in the archived file but are not queried.
Bind legacy labels to the exact current chunk
The original baseline labels use earlier chunk IDs. A string comparison against current versioned IDs would silently produce false misses. After retrieval finishes, the binding step matches each target’s source ID, source version, source-text hash and exact chunk-text hash to one current chunk.
Invalid target mapping. A changed hash, missing target or ambiguous match raises ValueError. Duplicate target bindings are refused. The record retains both the legacy and current IDs, so the mapping can be inspected instead of guessed from a familiar source name.
This mapping changes the label’s identifier, not the expected content. Editing a target after looking at results would change the evaluation task; the comparison does not do that.
Read each metric before its average
Recall at a cutoff is the fraction of expected target chunks found within that cutoff. Reciprocal rank at a cutoff is the reciprocal of the first target’s rank when it falls inside the cutoff, or zero when no target does. The program also keeps target positions and missed case IDs for each configuration.
Unsupported denominator. A question with no target chunks has undefined supported-query recall and reciprocal rank here. Its metric values remain null, and it stays outside those averages. The report separately retains how many candidates each configuration returned. An empty list would not prove that an unsupported-question policy worked.
The recorded development comparison has the same supported-query metrics for all configurations. The UTF-8 writing case remains the first-rank miss. Coverage reranking does not fix it, and rank fusion does not turn the unsupported backup candidates into answers.
The accessible table below is bound directly to the recorded aggregates. Read the per-question ranks in the raw result before interpreting these means. The dataset is a small development fixture, not a held-out quality benchmark or a model-answer evaluation.
| Configuration | Cutoff | Supported cases | Mean recall | Mean reciprocal rank |
|---|---|---|---|---|
| lexical | 1 | 3 | 0.6666666666666666 | 0.6666666666666666 |
| lexical | 2 | 3 | 1 | 0.8333333333333334 |
| vector | 1 | 3 | 0.6666666666666666 | 0.6666666666666666 |
| vector | 2 | 3 | 1 | 0.8333333333333334 |
| rrf | 1 | 3 | 0.6666666666666666 | 0.6666666666666666 |
| rrf | 2 | 3 | 1 | 0.8333333333333334 |
| rrf+coverage | 1 | 3 | 0.6666666666666666 | 0.6666666666666666 |
| rrf+coverage | 2 | 3 | 1 | 0.8333333333333334 |
Execute the frozen comparison
Download the complete lab. It contains the copied retrieval and reranking implementations, preserved fixtures, exact dependency pins, independent failure tests and actual output.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.lock"""Frozen tiny-development comparison; label joins occur after retrieval."""
import hashlib
import math
from hybrid import HybridRetriever
from rerank import rerank
from version_documents import validate_snapshot
CANDIDATE_BUDGET=3
RANK_CONSTANT=60
CUTOFFS=(1,2)
CONFIGURATIONS=('lexical','vector','rrf','rrf+coverage')
def retrieve_runs(snapshot,embeddings,questions):
"""questions contains only id, query and requested version; no target labels."""
retriever=HybridRetriever(snapshot,embeddings);runs=[]
try:
for question in questions:
policy={'version':{'$eq':question['version']}}
hybrid=retriever.search(question['query'],k=CANDIDATE_BUDGET,candidate_budget=CANDIDATE_BUDGET,rank_constant=RANK_CONSTANT,policy=policy)
runs.append({'case_id':question['case_id'],'query':question['query'],'version':question['version'],'lexical_query_terms':hybrid['lexical_query_terms'],'rankings':{'lexical':hybrid['lexical'],'vector':hybrid['vector'],'rrf':hybrid['results'],'rrf+coverage':rerank(question['query'],hybrid['results'],snapshot,k=CANDIDATE_BUDGET)}})
finally:retriever.close()
return runs
def bind_targets(snapshot,cases):
"""Map legacy labels by source ID, API version and exact chunk-text hash."""
validate_snapshot(snapshot)
bindings={};receipts=[]
for case in cases:
if case['id'] in bindings:raise ValueError('duplicate_case')
if case['split']!='development':raise ValueError('development_only')
targets=[]
for target in case['expected']['targets']:
matches=[chunk for source in snapshot['documents'] if source['source_id']==target['source_id'] and source['version']==target['source_version'] and source['sha256']==target['source_text_sha256'] for chunk in source['chunks'] if hashlib.sha256(chunk['text'].encode('utf-8')).hexdigest()==target['chunk_text_sha256']]
if len(matches)!=1:raise ValueError('target_binding_not_bijective')
cid=matches[0]['chunk_id']
if cid in targets:raise ValueError('duplicate_target')
targets.append(cid)
receipts.append({'case_id':case['id'],'legacy_chunk_id':target['chunk_id'],'current_chunk_id':cid,'binding':['source_id','source_version','source_text_sha256','chunk_text_sha256']})
if case['class']=='answerable' and not targets:raise ValueError('answerable_requires_targets')
if case['class']!='answerable' and targets:raise ValueError('unsupported_must_have_no_targets')
bindings[case['id']]={'case_class':case['class'],'target_chunk_ids':targets}
return bindings,receipts
def metrics(rows,targets,k):
ids=[r['chunk_id'] for r in rows]
if len(ids)!=len(set(ids)):raise ValueError('duplicate_ranked_chunk')
ranks=[i for i,cid in enumerate(ids,1) if cid in targets]
if not targets:return {'target_ranks':[],'recall_at_k':None,'reciprocal_rank_at_k':None,'status':'unsupported-excluded'}
within=[r for r in ranks if r<=k]
return {'target_ranks':ranks,'recall_at_k':len(within)/len(targets),'reciprocal_rank_at_k':1/min(within) if within else 0.0,'status':'supported'}
def evaluate(runs,bindings):
if len({r['case_id'] for r in runs})!=len(runs) or set(bindings)!={r['case_id'] for r in runs}:raise ValueError('case_join_mismatch')
per_query=[]
for run in runs:
label=bindings[run['case_id']]
per_query.append({'case_id':run['case_id'],'case_class':label['case_class'],'target_chunk_ids':label['target_chunk_ids'],'configurations':{name:{str(k):metrics(run['rankings'][name],label['target_chunk_ids'],k) for k in CUTOFFS} for name in CONFIGURATIONS}})
supported=[r for r in per_query if r['case_class']=='answerable']
aggregate={name:{str(k):{'supported_case_count':len(supported),'mean_recall_at_k':math.fsum(r['configurations'][name][str(k)]['recall_at_k'] for r in supported)/len(supported) if supported else None,'mean_reciprocal_rank_at_k':math.fsum(r['configurations'][name][str(k)]['reciprocal_rank_at_k'] for r in supported)/len(supported) if supported else None} for k in CUTOFFS} for name in CONFIGURATIONS}
unsupported=[{'case_id':run['case_id'],'returned_candidate_counts':{name:len(rows) for name,rows in run['rankings'].items()}} for run in runs if bindings[run['case_id']]['case_class']!='answerable']
misses={name:{str(k):[r['case_id'] for r in supported if r['configurations'][name][str(k)]['recall_at_k']==0] for k in CUTOFFS} for name in CONFIGURATIONS}
return {'supported_misses':misses,'per_query':per_query,'aggregate':aggregate,'unsupported_candidate_behavior':unsupported,'supported_case_count':len(supported),'unsupported_case_count':len(unsupported)}python run.py --output scratch-result.json
python -m unittest discover -v{
"supported_case_count": 3,
"unsupported_case_count": 1,
"aggregate": {
"lexical": {
"1": {
"recall": 0.6666666666666666,
"reciprocal_rank": 0.6666666666666666
},
"2": {
"recall": 1,
"reciprocal_rank": 0.8333333333333334
}
},
"vector": {
"1": {
"recall": 0.6666666666666666,
"reciprocal_rank": 0.6666666666666666
},
"2": {
"recall": 1,
"reciprocal_rank": 0.8333333333333334
}
},
"rrf": {
"1": {
"recall": 0.6666666666666666,
"reciprocal_rank": 0.6666666666666666
},
"2": {
"recall": 1,
"reciprocal_rank": 0.8333333333333334
}
},
"rrf+coverage": {
"1": {
"recall": 0.6666666666666666,
"reciprocal_rank": 0.6666666666666666
},
"2": {
"recall": 1,
"reciprocal_rank": 0.8333333333333334
}
}
},
"supported_misses": {
"lexical": {
"1": [
"utf8-write"
],
"2": []
},
"vector": {
"1": [
"utf8-write"
],
"2": []
},
"rrf": {
"1": [
"utf8-write"
],
"2": []
},
"rrf+coverage": {
"1": [
"utf8-write"
],
"2": []
}
}
}The CLI runs each configuration in memory and then joins labels for checking. It generates no answer and makes no measured cost, latency or capacity comparison. A larger evaluation needs additional representative cases and a separate untouched validation set; do not tune these development results and present them as independent validation.
Inspect the complete saved result
{
"execution_mode": "offline-frozen-development-comparison",
"model_called": false,
"snapshot_id": "0615d22a3315e97c0a88c6c566c546877304f5a28a97e83f68dd36881d4bbd0e",
"frozen_configuration": {
"names": [
"lexical",
"vector",
"rrf",
"rrf+coverage"
],
"candidate_budget_per_branch": 3,
"rank_constant": 60,
"cutoffs": [
1,
2
],
"scope": "four original development cases; no held-out queries; no configuration tuning"
},
"label_binding_receipts": [
{
"case_id": "utf8-write",
"legacy_chunk_id": "path-write:b71a2d18a53d86bf",
"current_chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"binding": [
"source_id",
"source_version",
"source_text_sha256",
"chunk_text_sha256"
]
},
{
"case_id": "replace-file",
"legacy_chunk_id": "path-write:b71a2d18a53d86bf",
"current_chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"binding": [
"source_id",
"source_version",
"source_text_sha256",
"chunk_text_sha256"
]
},
{
"case_id": "json-result-type",
"legacy_chunk_id": "json-text:8a17e251a41ccb3b",
"current_chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"binding": [
"source_id",
"source_version",
"source_text_sha256",
"chunk_text_sha256"
]
}
],
"retrieval_runs": [
{
"case_id": "utf8-write",
"query": "How do I explicitly select UTF-8 when saving text with pathlib?",
"version": "3.12",
"lexical_query_terms": [
"how",
"do",
"i",
"explicitly",
"select",
"utf",
"8",
"when",
"saving",
"text",
"with",
"pathlib"
],
"rankings": {
"lexical": [
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"native_score": -0.8901430296001853
},
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"native_score": -4.904418516387322e-06
},
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"native_score": -9.765013054830289e-07
}
],
"vector": [
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"score": 0.637539995393079
},
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"score": 0.5762678845903342
},
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"score": 0.4948350231504922
}
],
"rrf": [
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"lexical_score": -0.8901430296001853,
"vector_score": 0.637539995393079,
"lexical_rank": 1,
"vector_rank": 1,
"lexical_contribution": 0.01639344262295082,
"vector_contribution": 0.01639344262295082,
"fused_score": 0.03278688524590164
},
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"lexical_score": -4.904418516387322e-06,
"vector_score": 0.5762678845903342,
"lexical_rank": 2,
"vector_rank": 2,
"lexical_contribution": 0.016129032258064516,
"vector_contribution": 0.016129032258064516,
"fused_score": 0.03225806451612903
},
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"lexical_score": -9.765013054830289e-07,
"vector_score": 0.4948350231504922,
"lexical_rank": 3,
"vector_rank": 3,
"lexical_contribution": 0.015873015873015872,
"vector_contribution": 0.015873015873015872,
"fused_score": 0.031746031746031744
}
],
"rrf+coverage": [
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"lexical_score": -0.8901430296001853,
"vector_score": 0.637539995393079,
"lexical_rank": 1,
"vector_rank": 1,
"lexical_contribution": 0.01639344262295082,
"vector_contribution": 0.01639344262295082,
"fused_score": 0.03278688524590164,
"incoming_rank": 1,
"rerank_features": {
"query_tokens": [
"how",
"do",
"i",
"explicitly",
"select",
"utf",
"8",
"when",
"saving",
"text",
"with",
"pathlib"
],
"matched_tokens": [
"8",
"text",
"utf",
"when"
],
"matched_count": 4,
"query_token_count": 12
},
"coverage_score": 0.3333333333333333,
"rerank_rank": 1
},
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"lexical_score": -4.904418516387322e-06,
"vector_score": 0.5762678845903342,
"lexical_rank": 2,
"vector_rank": 2,
"lexical_contribution": 0.016129032258064516,
"vector_contribution": 0.016129032258064516,
"fused_score": 0.03225806451612903,
"incoming_rank": 2,
"rerank_features": {
"query_tokens": [
"how",
"do",
"i",
"explicitly",
"select",
"utf",
"8",
"when",
"saving",
"text",
"with",
"pathlib"
],
"matched_tokens": [
"8",
"select",
"text",
"utf"
],
"matched_count": 4,
"query_token_count": 12
},
"coverage_score": 0.3333333333333333,
"rerank_rank": 2
},
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"lexical_score": -9.765013054830289e-07,
"vector_score": 0.4948350231504922,
"lexical_rank": 3,
"vector_rank": 3,
"lexical_contribution": 0.015873015873015872,
"vector_contribution": 0.015873015873015872,
"fused_score": 0.031746031746031744,
"incoming_rank": 3,
"rerank_features": {
"query_tokens": [
"how",
"do",
"i",
"explicitly",
"select",
"utf",
"8",
"when",
"saving",
"text",
"with",
"pathlib"
],
"matched_tokens": [
"select"
],
"matched_count": 1,
"query_token_count": 12
},
"coverage_score": 0.08333333333333333,
"rerank_rank": 3
}
]
}
},
{
"case_id": "replace-file",
"query": "Will write_text replace the contents of an existing file?",
"version": "3.12",
"lexical_query_terms": [
"will",
"write",
"text",
"replace",
"the",
"contents",
"of",
"an",
"existing",
"file"
],
"rankings": {
"lexical": [
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"native_score": -2.3976548397386743
},
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"native_score": -4.084683995209214e-06
},
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"native_score": -3.3056246724488786e-06
}
],
"vector": [
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"score": 0.6357498076122466
},
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"score": 0.5547892797395033
},
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"score": 0.36746132784169144
}
],
"rrf": [
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"lexical_score": -2.3976548397386743,
"vector_score": 0.6357498076122466,
"lexical_rank": 1,
"vector_rank": 1,
"lexical_contribution": 0.01639344262295082,
"vector_contribution": 0.01639344262295082,
"fused_score": 0.03278688524590164
},
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"lexical_score": -4.084683995209214e-06,
"vector_score": 0.5547892797395033,
"lexical_rank": 2,
"vector_rank": 2,
"lexical_contribution": 0.016129032258064516,
"vector_contribution": 0.016129032258064516,
"fused_score": 0.03225806451612903
},
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"lexical_score": -3.3056246724488786e-06,
"vector_score": 0.36746132784169144,
"lexical_rank": 3,
"vector_rank": 3,
"lexical_contribution": 0.015873015873015872,
"vector_contribution": 0.015873015873015872,
"fused_score": 0.031746031746031744
}
],
"rrf+coverage": [
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"lexical_score": -2.3976548397386743,
"vector_score": 0.6357498076122466,
"lexical_rank": 1,
"vector_rank": 1,
"lexical_contribution": 0.01639344262295082,
"vector_contribution": 0.01639344262295082,
"fused_score": 0.03278688524590164,
"incoming_rank": 1,
"rerank_features": {
"query_tokens": [
"will",
"write",
"text",
"replace",
"the",
"contents",
"of",
"an",
"existing",
"file"
],
"matched_tokens": [
"an",
"existing",
"file",
"of",
"text",
"the",
"write"
],
"matched_count": 7,
"query_token_count": 10
},
"coverage_score": 0.7,
"rerank_rank": 1
},
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"lexical_score": -4.084683995209214e-06,
"vector_score": 0.5547892797395033,
"lexical_rank": 2,
"vector_rank": 2,
"lexical_contribution": 0.016129032258064516,
"vector_contribution": 0.016129032258064516,
"fused_score": 0.03225806451612903,
"incoming_rank": 2,
"rerank_features": {
"query_tokens": [
"will",
"write",
"text",
"replace",
"the",
"contents",
"of",
"an",
"existing",
"file"
],
"matched_tokens": [
"file",
"text",
"the"
],
"matched_count": 3,
"query_token_count": 10
},
"coverage_score": 0.3,
"rerank_rank": 2
},
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"lexical_score": -3.3056246724488786e-06,
"vector_score": 0.36746132784169144,
"lexical_rank": 3,
"vector_rank": 3,
"lexical_contribution": 0.015873015873015872,
"vector_contribution": 0.015873015873015872,
"fused_score": 0.031746031746031744,
"incoming_rank": 3,
"rerank_features": {
"query_tokens": [
"will",
"write",
"text",
"replace",
"the",
"contents",
"of",
"an",
"existing",
"file"
],
"matched_tokens": [
"file",
"the",
"write"
],
"matched_count": 3,
"query_token_count": 10
},
"coverage_score": 0.3,
"rerank_rank": 3
}
]
}
},
{
"case_id": "json-result-type",
"query": "Does json.dumps produce text or a bytes object?",
"version": "3.12",
"lexical_query_terms": [
"does",
"json",
"dumps",
"produce",
"text",
"or",
"a",
"bytes",
"object"
],
"rankings": {
"lexical": [
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"native_score": -3.628237730877387
},
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"native_score": -3.0341221974564053e-06
},
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"native_score": -2.263641817358195e-06
}
],
"vector": [
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"score": 0.652333570145617
},
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"score": 0.42563890525675496
},
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"score": 0.4145266500154773
}
],
"rrf": [
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"lexical_score": -3.628237730877387,
"vector_score": 0.652333570145617,
"lexical_rank": 1,
"vector_rank": 1,
"lexical_contribution": 0.01639344262295082,
"vector_contribution": 0.01639344262295082,
"fused_score": 0.03278688524590164
},
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"lexical_score": -3.0341221974564053e-06,
"vector_score": 0.42563890525675496,
"lexical_rank": 2,
"vector_rank": 2,
"lexical_contribution": 0.016129032258064516,
"vector_contribution": 0.016129032258064516,
"fused_score": 0.03225806451612903
},
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"lexical_score": -2.263641817358195e-06,
"vector_score": 0.4145266500154773,
"lexical_rank": 3,
"vector_rank": 3,
"lexical_contribution": 0.015873015873015872,
"vector_contribution": 0.015873015873015872,
"fused_score": 0.031746031746031744
}
],
"rrf+coverage": [
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"lexical_score": -3.628237730877387,
"vector_score": 0.652333570145617,
"lexical_rank": 1,
"vector_rank": 1,
"lexical_contribution": 0.01639344262295082,
"vector_contribution": 0.01639344262295082,
"fused_score": 0.03278688524590164,
"incoming_rank": 1,
"rerank_features": {
"query_tokens": [
"does",
"json",
"dumps",
"produce",
"text",
"or",
"a",
"bytes",
"object"
],
"matched_tokens": [
"a",
"does",
"dumps",
"json",
"or"
],
"matched_count": 5,
"query_token_count": 9
},
"coverage_score": 0.5555555555555556,
"rerank_rank": 1
},
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"lexical_score": -3.0341221974564053e-06,
"vector_score": 0.42563890525675496,
"lexical_rank": 2,
"vector_rank": 2,
"lexical_contribution": 0.016129032258064516,
"vector_contribution": 0.016129032258064516,
"fused_score": 0.03225806451612903,
"incoming_rank": 2,
"rerank_features": {
"query_tokens": [
"does",
"json",
"dumps",
"produce",
"text",
"or",
"a",
"bytes",
"object"
],
"matched_tokens": [
"a",
"text"
],
"matched_count": 2,
"query_token_count": 9
},
"coverage_score": 0.2222222222222222,
"rerank_rank": 2
},
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"lexical_score": -2.263641817358195e-06,
"vector_score": 0.4145266500154773,
"lexical_rank": 3,
"vector_rank": 3,
"lexical_contribution": 0.015873015873015872,
"vector_contribution": 0.015873015873015872,
"fused_score": 0.031746031746031744,
"incoming_rank": 3,
"rerank_features": {
"query_tokens": [
"does",
"json",
"dumps",
"produce",
"text",
"or",
"a",
"bytes",
"object"
],
"matched_tokens": [
"a",
"text"
],
"matched_count": 2,
"query_token_count": 9
},
"coverage_score": 0.2222222222222222,
"rerank_rank": 3
}
]
}
},
{
"case_id": "backup-schedule",
"query": "How frequently does our cloud provider back up these files?",
"version": "3.12",
"lexical_query_terms": [
"how",
"frequently",
"does",
"our",
"cloud",
"provider",
"back",
"up",
"these",
"files"
],
"rankings": {
"lexical": [
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"native_score": -0.8273874668010814
}
],
"vector": [
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"score": 0.3202785226491292
},
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"score": 0.26563513588901855
},
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"score": 0.2488070454064458
}
],
"rrf": [
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"lexical_score": -0.8273874668010814,
"vector_score": 0.2488070454064458,
"lexical_rank": 1,
"vector_rank": 3,
"lexical_contribution": 0.01639344262295082,
"vector_contribution": 0.015873015873015872,
"fused_score": 0.032266458495966696
},
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"lexical_score": null,
"vector_score": 0.3202785226491292,
"lexical_rank": null,
"vector_rank": 1,
"lexical_contribution": 0.0,
"vector_contribution": 0.01639344262295082,
"fused_score": 0.01639344262295082
},
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"lexical_score": null,
"vector_score": 0.26563513588901855,
"lexical_rank": null,
"vector_rank": 2,
"lexical_contribution": 0.0,
"vector_contribution": 0.016129032258064516,
"fused_score": 0.016129032258064516
}
],
"rrf+coverage": [
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"lexical_score": -0.8273874668010814,
"vector_score": 0.2488070454064458,
"lexical_rank": 1,
"vector_rank": 3,
"lexical_contribution": 0.01639344262295082,
"vector_contribution": 0.015873015873015872,
"fused_score": 0.032266458495966696,
"incoming_rank": 1,
"rerank_features": {
"query_tokens": [
"how",
"frequently",
"does",
"our",
"cloud",
"provider",
"back",
"up",
"these",
"files"
],
"matched_tokens": [
"does"
],
"matched_count": 1,
"query_token_count": 10
},
"coverage_score": 0.1,
"rerank_rank": 1
},
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"lexical_score": null,
"vector_score": 0.3202785226491292,
"lexical_rank": null,
"vector_rank": 1,
"lexical_contribution": 0.0,
"vector_contribution": 0.01639344262295082,
"fused_score": 0.01639344262295082,
"incoming_rank": 2,
"rerank_features": {
"query_tokens": [
"how",
"frequently",
"does",
"our",
"cloud",
"provider",
"back",
"up",
"these",
"files"
],
"matched_tokens": [],
"matched_count": 0,
"query_token_count": 10
},
"coverage_score": 0.0,
"rerank_rank": 2
},
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"lexical_score": null,
"vector_score": 0.26563513588901855,
"lexical_rank": null,
"vector_rank": 2,
"lexical_contribution": 0.0,
"vector_contribution": 0.016129032258064516,
"fused_score": 0.016129032258064516,
"incoming_rank": 3,
"rerank_features": {
"query_tokens": [
"how",
"frequently",
"does",
"our",
"cloud",
"provider",
"back",
"up",
"these",
"files"
],
"matched_tokens": [],
"matched_count": 0,
"query_token_count": 10
},
"coverage_score": 0.0,
"rerank_rank": 3
}
]
}
}
],
"supported_misses": {
"lexical": {
"1": [
"utf8-write"
],
"2": []
},
"vector": {
"1": [
"utf8-write"
],
"2": []
},
"rrf": {
"1": [
"utf8-write"
],
"2": []
},
"rrf+coverage": {
"1": [
"utf8-write"
],
"2": []
}
},
"per_query": [
{
"case_id": "utf8-write",
"case_class": "answerable",
"target_chunk_ids": [
"c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7"
],
"configurations": {
"lexical": {
"1": {
"target_ranks": [
2
],
"recall_at_k": 0.0,
"reciprocal_rank_at_k": 0.0,
"status": "supported"
},
"2": {
"target_ranks": [
2
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 0.5,
"status": "supported"
}
},
"vector": {
"1": {
"target_ranks": [
2
],
"recall_at_k": 0.0,
"reciprocal_rank_at_k": 0.0,
"status": "supported"
},
"2": {
"target_ranks": [
2
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 0.5,
"status": "supported"
}
},
"rrf": {
"1": {
"target_ranks": [
2
],
"recall_at_k": 0.0,
"reciprocal_rank_at_k": 0.0,
"status": "supported"
},
"2": {
"target_ranks": [
2
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 0.5,
"status": "supported"
}
},
"rrf+coverage": {
"1": {
"target_ranks": [
2
],
"recall_at_k": 0.0,
"reciprocal_rank_at_k": 0.0,
"status": "supported"
},
"2": {
"target_ranks": [
2
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 0.5,
"status": "supported"
}
}
}
},
{
"case_id": "replace-file",
"case_class": "answerable",
"target_chunk_ids": [
"c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7"
],
"configurations": {
"lexical": {
"1": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported"
},
"2": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported"
}
},
"vector": {
"1": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported"
},
"2": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported"
}
},
"rrf": {
"1": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported"
},
"2": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported"
}
},
"rrf+coverage": {
"1": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported"
},
"2": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported"
}
}
}
},
{
"case_id": "json-result-type",
"case_class": "answerable",
"target_chunk_ids": [
"e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c"
],
"configurations": {
"lexical": {
"1": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported"
},
"2": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported"
}
},
"vector": {
"1": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported"
},
"2": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported"
}
},
"rrf": {
"1": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported"
},
"2": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported"
}
},
"rrf+coverage": {
"1": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported"
},
"2": {
"target_ranks": [
1
],
"recall_at_k": 1.0,
"reciprocal_rank_at_k": 1.0,
"status": "supported"
}
}
}
},
{
"case_id": "backup-schedule",
"case_class": "unsupported",
"target_chunk_ids": [],
"configurations": {
"lexical": {
"1": {
"target_ranks": [],
"recall_at_k": null,
"reciprocal_rank_at_k": null,
"status": "unsupported-excluded"
},
"2": {
"target_ranks": [],
"recall_at_k": null,
"reciprocal_rank_at_k": null,
"status": "unsupported-excluded"
}
},
"vector": {
"1": {
"target_ranks": [],
"recall_at_k": null,
"reciprocal_rank_at_k": null,
"status": "unsupported-excluded"
},
"2": {
"target_ranks": [],
"recall_at_k": null,
"reciprocal_rank_at_k": null,
"status": "unsupported-excluded"
}
},
"rrf": {
"1": {
"target_ranks": [],
"recall_at_k": null,
"reciprocal_rank_at_k": null,
"status": "unsupported-excluded"
},
"2": {
"target_ranks": [],
"recall_at_k": null,
"reciprocal_rank_at_k": null,
"status": "unsupported-excluded"
}
},
"rrf+coverage": {
"1": {
"target_ranks": [],
"recall_at_k": null,
"reciprocal_rank_at_k": null,
"status": "unsupported-excluded"
},
"2": {
"target_ranks": [],
"recall_at_k": null,
"reciprocal_rank_at_k": null,
"status": "unsupported-excluded"
}
}
}
}
],
"aggregate": {
"lexical": {
"1": {
"supported_case_count": 3,
"mean_recall_at_k": 0.6666666666666666,
"mean_reciprocal_rank_at_k": 0.6666666666666666
},
"2": {
"supported_case_count": 3,
"mean_recall_at_k": 1.0,
"mean_reciprocal_rank_at_k": 0.8333333333333334
}
},
"vector": {
"1": {
"supported_case_count": 3,
"mean_recall_at_k": 0.6666666666666666,
"mean_reciprocal_rank_at_k": 0.6666666666666666
},
"2": {
"supported_case_count": 3,
"mean_recall_at_k": 1.0,
"mean_reciprocal_rank_at_k": 0.8333333333333334
}
},
"rrf": {
"1": {
"supported_case_count": 3,
"mean_recall_at_k": 0.6666666666666666,
"mean_reciprocal_rank_at_k": 0.6666666666666666
},
"2": {
"supported_case_count": 3,
"mean_recall_at_k": 1.0,
"mean_reciprocal_rank_at_k": 0.8333333333333334
}
},
"rrf+coverage": {
"1": {
"supported_case_count": 3,
"mean_recall_at_k": 0.6666666666666666,
"mean_reciprocal_rank_at_k": 0.6666666666666666
},
"2": {
"supported_case_count": 3,
"mean_recall_at_k": 1.0,
"mean_reciprocal_rank_at_k": 0.8333333333333334
}
}
},
"unsupported_candidate_behavior": [
{
"case_id": "backup-schedule",
"returned_candidate_counts": {
"lexical": 1,
"vector": 3,
"rrf": 3,
"rrf+coverage": 3
}
}
],
"supported_case_count": 3,
"unsupported_case_count": 1,
"limit": "Descriptive results on three supported development questions, not a general search-quality estimate or model comparison. Unsupported questions have no retrieval target and are excluded from metric means."
}The completed command retains recall and reciprocal-rank values, including the unsupported question outside the metric means. This remains a small development comparison, not a search-quality estimate.
Compare the recorded behavior
Follow the UTF-8 writing case from its legacy target mapping to the current target ID and returned ranks. Explain the difference between finding that target within a cutoff and placing it first. Then inspect the backup case: identify its candidate counts and the absent metric values without converting either into an answer-quality score.
Read the explanation
Labels join after retrieval, and undefined metrics remain distinct from measured failures.
What’s next
Continue with Choose Chunk Boundaries.
References
The original RRF paper, section 1, was checked on 23 October 2026 for the fusion formula. The dataset lesson defines this application’s versioned targets and split. Metric definitions, label binding and the frozen configuration policy are implemented explicitly in this lab; this recording contains development-only retrieval checks.