A RAG answer can name a real Python documentation page and still use the wrong version. Path.read_text accepts a newline parameter in the Python 3.13 documentation, but the Python 3.12 signature does not. A useful question set must say which version the developer asked about and which passages support its expected handling.
We will build that dataset for the local RAG pipeline, using the same original Python API summaries plus one version-specific summary. The local program resolves the labels, checks source identities and tests split boundaries. It does not run retrieval or generate answers. Recording the pipeline’s baseline on this dataset is a separate activity.
When you need this. Use an eval dataset before changing retrieval or prompts, so you can name the questions the application should answer, the information it should decline to invent and the source version each answer must use.
State what a question can answer from this corpus
An eval dataset is a set of inputs and their expected outcomes. In this lesson the expected outcome is an answerability label plus relevant passage targets, not a model-written answer. A relevant passage supplies information needed for this particular question and documentation version.
The dataset has three question classes. Answerable questions cover facts stated in the supplied summaries, such as selecting UTF-8 when writing text. Unsupported questions ask for information this corpus does not contain, such as a cloud backup schedule. Version-specific questions require the requested documentation version, even when another version describes the same method.
The application statuses remain answered and not_in_docs. Version-specific is a question class, not a third runtime status. For example, “Can read_text take newline in Python 3.12?” has a supported negative answer. It should not be confused with an unsupported question whose answer is absent from the corpus.
[
{
"id": "read-newline-312",
"split": "held_out",
"class": "version_specific",
"input": "Can Path.read_text take a newline argument in Python 3.12?",
"expected": {
"status": "answered",
"targets": [
{
"source_id": "path-read",
"chunk_id": "path-read:1f1129b8d1011a82",
"source_version": "3.12",
"source_text_sha256": "dc097ca63486ace45279f412ea266bf754fcd2f0b1153f5d763ce905dafbc2c0",
"chunk_text_sha256": "dc097ca63486ace45279f412ea266bf754fcd2f0b1153f5d763ce905dafbc2c0"
}
]
},
"fail_if": [],
"why": "The 3.12 signature has no newline; actual 3.12 check can verify rejection.",
"tags": {
"python_version": "3.12",
"intent_family": "read-newline"
}
},
{
"id": "read-newline-313",
"split": "held_out",
"class": "version_specific",
"input": "In Python 3.13, is newline an accepted Path.read_text parameter?",
"expected": {
"status": "answered",
"targets": [
{
"source_id": "path-read-3-13",
"chunk_id": "path-read-3-13:3f86296ba6f04cc2",
"source_version": "3.13",
"source_text_sha256": "0dd5becb0c04458ad5f8bbf3fdef4f71034c584ef863bf748793bef7309b9f4f",
"chunk_text_sha256": "0dd5becb0c04458ad5f8bbf3fdef4f71034c584ef863bf748793bef7309b9f4f"
}
]
},
"fail_if": [],
"why": "Official 3.13 documentation states the added parameter; documented-only here.",
"tags": {
"python_version": "3.13",
"intent_family": "read-newline"
}
}
]“Unsupported” is relative to the bounded corpus. The missing-parent-directory question might be answerable from other Python documentation, but these summaries do not state that behavior. Its label asks the application to avoid supplying outside knowledge, rather than asserting that nobody knows the answer.
These are authored teaching questions. The backup-policy development case builds on the earlier missing-information example. The other cases were written for this dataset.
Bind each label to an exact passage and version
A source is one versioned documentation summary. A chunk is the passage produced by the existing splitter. The lab reuses RecursiveCharacterTextSplitter and the preceding pipeline’s unchanged chunk-ID function, which combines source identity with a hash of text and URL. It does not introduce another chunking strategy here.
Each relevant target carries a source ID, chunk ID, source version, source-text hash and chunk-text hash. A hash is a fingerprint used to compare the recorded bytes. The source manifest binds the complete corpus snapshot, while the dataset’s existing canonical hash identifies its question and label content.
Wrong documentation version. The controlled invalid case changes a development target to claim version 3.13 while the question and source are version 3.12. The program refuses it with TargetError. Point the label to the intended version’s passage; a source merely existing somewhere in the corpus is insufficient.
Stale target hash. A second invalid label retains its IDs but changes the expected source hash. It also gets TargetError. Review changed text and the label together. Replacing only the hash to silence the check does not show that the new passage remains relevant.
The added Python 3.13 summary follows the documented newline addition. The supplied behavior probe runs on Python 3.12; the 3.13 behavior comes from the documentation.
Check answerability without pretending to judge relevance
The RAG adapter validates answerability and passage targets. The shared Evals loader handles case IDs, splits and dataset hashing.
"""Validate RAG-specific source targets; reuse the existing Evals contracts."""
from pathlib import Path
import hashlib
import sys
from langchain_core.documents import Document
from langchain_text_splitters import RecursiveCharacterTextSplitter
from pipeline_ids import chunk_id
ROOT=Path(__file__).parent
sys.path.insert(0,str(ROOT/"vendor/evals"))
import harness
import leakcheck
import eval_task
class TargetError(ValueError):
pass
class SplitBoundaryError(ValueError):
pass
def text_sha(text):
return hashlib.sha256(text.encode("utf-8")).hexdigest()
def load_sources(path=ROOT/"data/sources.json", manifest_path=ROOT/"data/corpus-manifest.json"):
sources=harness.read_json(path);manifest=harness.read_json(manifest_path)
if not isinstance(sources,list)or not sources or any(not isinstance(s.get("id"),str)or not s["id"].strip()for s in sources)or len({s["id"]for s in sources})!=len(sources):
raise TargetError("Source IDs must be nonempty and unique.")
if hashlib.sha256(harness.canonical(sources).encode()).hexdigest()!=manifest["sources_sha256"]:
raise TargetError("Source snapshot changed; review the corpus and its targets together.")
settings=manifest["splitter"]
passages=RecursiveCharacterTextSplitter(chunk_size=settings["chunk_size"],chunk_overlap=settings["chunk_overlap"]).split_documents([
Document(page_content=s["text"],metadata={"id":s["id"],"url":s["url"],"version":s["version"]})for s in sources])
return sources,passages,manifest
def load_questions(path=ROOT/"data/cases.json"):
return harness.load_cases(path,eval_task)
def validate_targets(cases,sources,passages):
source_by_id={s["id"]:s for s in sources};chunk_by_id={chunk_id(p):p for p in passages};families={}
for case in cases["cases"]:
tags=case.get("tags",{});version=tags.get("python_version");family=tags.get("intent_family")
if not isinstance(case["input"],str)or not case["input"].strip()or not family or not version:
raise TargetError(f"{case['id']}: question, version and intent family are required.")
if case["class"]not in {"answerable","unsupported","version_specific"}:
raise TargetError(f"{case['id']}: unknown question class.")
families.setdefault(family,set()).add(case["split"])
status,targets=case["expected"]["status"],case["expected"]["targets"]
if status=="answered"and not targets:
raise TargetError(f"{case['id']}: an answerable question needs a relevant passage target.")
if status=="not_in_docs"and targets:
raise TargetError(f"{case['id']}: an unsupported question must not name a relevant target.")
if (case["class"]=="unsupported")!=(status=="not_in_docs"):
raise TargetError(f"{case['id']}: class and answerability label disagree.")
if len({t["chunk_id"]for t in targets})!=len(targets):
raise TargetError(f"{case['id']}: duplicate passage target.")
for target in targets:
source=source_by_id.get(target["source_id"]);passage=chunk_by_id.get(target["chunk_id"])
if source is None or passage is None or passage.metadata["id"]!=target["source_id"]:
raise TargetError(f"{case['id']}: source or chunk target does not exist in this snapshot.")
if target["source_version"]!=version or source["version"]!=version or passage.metadata["version"]!=version:
raise TargetError(f"{case['id']}: source version does not match the question version.")
if target["source_text_sha256"]!=text_sha(source["text"])or target["chunk_text_sha256"]!=text_sha(passage.page_content):
raise TargetError(f"{case['id']}: target text hash is stale.")
if any(len(splits)>1 for splits in families.values()):
raise SplitBoundaryError("Related intent-family questions cross the development/held_out boundary.")
return {"cases":len(cases["cases"]),"sources":len(sources),"passages":len(passages),
"by_split":{split:sum(c["split"]==split for c in cases["cases"])for split in harness.SPLITS},
"by_class":{kind:sum(c["class"]==kind for c in cases["cases"])for kind in ["answerable","unsupported","version_specific"]}}
def validate_dataset(cases,sources,passages,locked):
if not isinstance(locked,dict)or locked.get("case_set")!=cases["name"]:
raise SplitBoundaryError("Lock belongs to a different dataset.")
counts=validate_targets(cases,sources,passages)
findings=leakcheck.check_overlap(cases)+leakcheck.check_sources(cases)+leakcheck.check_lock(locked,cases)
if findings:
raise SplitBoundaryError("Split review required: "+harness.canonical(findings))
return {"dataset_sha256":cases["sha256"],"counts":counts,"split_findings":findings,
"validation":"question/target relationships and split lock accepted; semantic relevance still needs review"}Answerable question without a passage. A supported label with an empty target list is refused. Supply the passage that supports the expected handling, or deliberately change the question’s answerability after reviewing the corpus. An unsupported label, conversely, must not claim a relevant passage.
Structural validation cannot decide whether the human selected the right passage. A test deliberately assigns the JSON-serialization passage to the UTF-8 file-writing question. Its IDs, version and hashes are valid, so the structural check accepts it. Reading the question and source reveals the wrong relevance label. Review that relationship before treating a dataset as ready for baseline measurement.
For each label, check the requested version, the specific information needed and the passage that contains it. Record why an unsupported case is outside the corpus. Do not use a high retrieval score as a substitute for a relevance annotation; the preceding pipeline already showed a read passage ranked above the answer’s write passage.
Preserve related questions and frozen labels
Development questions are available for changing the application. Held-out questions are reserved for a later comparison after those choices are fixed. The existing Evals lesson explains split design and leakage checks; this lab reuses its stored case-hash checks instead of teaching another split algorithm.
Known related questions share an intent_family. The two read_text version questions stay together in the held-out split. A derived question can also name its original case through the existing tags.source field. The adapter refuses a known family crossing the split, and the reused checks catch supplied ancestry crossing it. Undeclared paraphrases still need human review.
Edited held-out question. The driver changes one held-out question after its lock was saved. The reused lock comparison reports the changed case, and the adapter raises SplitBoundaryError. Treat this as a reviewed dataset revision and rerun comparisons on the same new dataset; silently rewriting the lock would hide the change.
Lock from another dataset. The adapter also refuses a lock carrying another dataset name. Matching case IDs and hashes is insufficient when the lock names a different dataset.
The unchanged word-overlap check can also request split review. Shared API terms do not prove that one case was copied from another. This wrapper stops when findings need review; it does not certify the absence of leakage when none are found. The held-out fixtures are public teaching data, not an unseen external test set or an access-control mechanism.
Execute the dataset checks and review a changed label
Download the complete lab. It includes the existing Evals helpers, new RAG adapter, source summaries, questions, frozen lock, tests and recorded validation output. No model server, embedding call or new weights are needed.
python -m venv .venv
.venv/bin/pip install -r requirements-lock.txt
.venv/bin/python build_dataset.py
.venv/bin/python -m unittest -v test_dataset.pyThe actual local run accepted 8 question records bound to 4 sources. It also refused every deliberate invalid-data scenario shown below. These are dataset checks, not measured retrieval or answer accuracy. The source manifest marks 3.13 as documented-only.
[
{
"case": "Wrong documentation version",
"error_type": "TargetError",
"message": "utf8-write: source version does not match the question version."
},
{
"case": "Stale target hash",
"error_type": "TargetError",
"message": "utf8-write: target text hash is stale."
},
{
"case": "Answerable question without a passage",
"error_type": "TargetError",
"message": "utf8-write: an answerable question needs a relevant passage target."
},
{
"case": "Edited held-out question",
"error_type": "SplitBoundaryError",
"message": "Split review required: [{\"case\":\"read-newline-312\",\"check\":\"lock\",\"reason\":\"edited\",\"severity\":\"block\"}]"
},
{
"case": "Lock from another dataset",
"error_type": "SplitBoundaryError",
"message": "Lock belongs to a different dataset."
}
]Try assigning the 3.13 read_text target to the 3.12 question. Expect a version rejection even though the passage describes the same method. Then restore the file and change only a held-out question’s wording: the lock should detect that edit. Neither activity needs a model call.
The recorded suite passed 23 tests. 6 isolated weakened implementations each failed a named assertion, including missing version/hash checks and ignored split locks. A valid dataset still needs semantic label review; baseline measurements come after this dataset is frozen.
test_313_source_is_explicitly_documented_only (test_dataset.DatasetTests.test_313_source_is_explicitly_documented_only) ... ok
test_actual_dataset_has_all_source_bound_targets_and_expected_classes (test_dataset.DatasetTests.test_actual_dataset_has_all_source_bound_targets_and_expected_classes) ... ok
test_answerable_case_needs_a_relevant_target (test_dataset.DatasetTests.test_answerable_case_needs_a_relevant_target) ... ok
test_blank_question_and_missing_intent_family_are_refused (test_dataset.DatasetTests.test_blank_question_and_missing_intent_family_are_refused) ... ok
test_blank_source_id_is_refused_before_snapshot_comparison (test_dataset.DatasetTests.test_blank_source_id_is_refused_before_snapshot_comparison) ... ok
test_chunk_from_other_source_cannot_be_attached_to_source_id (test_dataset.DatasetTests.test_chunk_from_other_source_cannot_be_attached_to_source_id) ... ok
test_chunk_ids_change_with_content_and_keep_versioned_source_identity (test_dataset.DatasetTests.test_chunk_ids_change_with_content_and_keep_versioned_source_identity) ... ok
test_existing_dataset_hash_changes_on_label_edit_not_json_whitespace (test_dataset.DatasetTests.test_existing_dataset_hash_changes_on_label_edit_not_json_whitespace) ... ok
test_existing_loader_rejects_duplicate_case_ids (test_dataset.DatasetTests.test_existing_loader_rejects_duplicate_case_ids) ... ok
test_existing_lock_detects_changed_labels_added_removed_or_moved_cases (test_dataset.DatasetTests.test_existing_lock_detects_changed_labels_added_removed_or_moved_cases) ... ok
test_existing_other_version_source_is_not_relevant_to_312_request (test_dataset.DatasetTests.test_existing_other_version_source_is_not_relevant_to_312_request) ... ok
test_held_out_question_edits_do_not_silently_refresh_the_lock (test_dataset.DatasetTests.test_held_out_question_edits_do_not_silently_refresh_the_lock) ... ok
test_lock_for_another_dataset_is_refused (test_dataset.DatasetTests.test_lock_for_another_dataset_is_refused) ... ok
test_paraphrase_parent_cannot_cross_the_split_boundary (test_dataset.DatasetTests.test_paraphrase_parent_cannot_cross_the_split_boundary) ... ok
test_question_class_and_answerability_must_agree (test_dataset.DatasetTests.test_question_class_and_answerability_must_agree) ... ok
test_related_intent_family_stays_in_one_split (test_dataset.DatasetTests.test_related_intent_family_stays_in_one_split) ... ok
test_semantic_relevance_is_not_proved_by_structural_validation (test_dataset.DatasetTests.test_semantic_relevance_is_not_proved_by_structural_validation) ... ok
test_source_changes_require_a_reviewed_snapshot (test_dataset.DatasetTests.test_source_changes_require_a_reviewed_snapshot) ... ok
test_stale_chunk_hash_is_refused (test_dataset.DatasetTests.test_stale_chunk_hash_is_refused) ... ok
test_stale_source_hash_is_refused (test_dataset.DatasetTests.test_stale_source_hash_is_refused) ... ok
test_unknown_chunk_is_refused_even_with_a_real_source (test_dataset.DatasetTests.test_unknown_chunk_is_refused_even_with_a_real_source) ... ok
test_unsupported_case_cannot_list_a_relevant_target (test_dataset.DatasetTests.test_unsupported_case_cannot_list_a_relevant_target) ... ok
test_wrong_declared_version_is_refused (test_dataset.DatasetTests.test_wrong_declared_version_is_refused) ... ok
----------------------------------------------------------------------
Ran 23 tests in 0.009s
OKRead the explanation
A valid reference can be a wrong human annotation. This lab deliberately demonstrates that boundary instead of claiming automatic relevance verification.
What’s next
Continue with Record a RAG Baseline.
References
Source facts were checked on 22 October 2026 against official Python 3.12 pathlib, Python 3.13 pathlib and Python JSON documentation. The added summary is original prose; links retain Python documentation licensing. The lab reuses the site’s existing Evals core and the LangChain splitter; code and authored question/source summaries are MIT. The packaged dependency pins are unchanged from the preceding local pipeline.