Build a corpus snapshot whose document revisions and chunk IDs change with the bytes, source metadata and implemented chunker configuration. The lab compares actual local text and metadata revisions without changing the published baseline.
When you need this. Use it when a changed document must produce a new corpus snapshot without confusing API versions with local file revisions.
Prerequisites. Complete the preceding ingestion module and use the supplied local files. These programs use local files and the Python standard library.
Give each identity one job
A source ID names one logical document. Its documentation version names the API described by that document. Neither is a fingerprint of the file: editing a summary can leave the source ID and API version unchanged.
A document revision binds the logical source ID to its raw byte hash and metadata hash. The metadata hash covers the path, title, documentation version, URL and supplied fact status. A chunk ID also binds the paragraph text hash, line span and complete chunker configuration. The corpus snapshot ID hashes the full canonical document and chunk records, including normalized text, metadata and chunker configuration. Documents are sorted by source ID before that hash is computed.
The supplied summaries are the same original Python API teaching text used in the preceding ingestion module. They are not newly fetched upstream documentation. The changed fixture adds a clearly labelled local teaching paragraph; its fact-status field records that local change. A separate fixture changes only a title.
Keep byte identity separate from normalized text
The loader checks pinned file bytes before decoding them. It then normalizes CRLF line endings to LF. The raw byte hash continues to identify the original file, while paragraph spans address the normalized string. This differs from the preceding source-span exercise, which preserved original line endings.
The paragraph chunker groups nonblank lines separated by blank lines. It is a small deterministic fixture policy, not a new general chunking recommendation. A blank-line boundary in the changed summary creates another chunk. The vector retrieval course remains the prerequisite for embedding and retrieval internals.
Unpinned edit. Changing a source while keeping its old manifest hash raises IngestionError. Review and explicitly pin a new fixture revision before comparing it. Hashing a changed file after the fact cannot show that its contents are accurate.
Unsupported chunker. The builder accepts its one implemented configuration. Passing another configuration raises IngestionError with unsupported_chunker; changing a label does not implement another splitting algorithm.
Compare text changes with metadata changes
The text-change fixture creates new revision and chunk IDs for the edited source. Unchanged sources retain their identifiers. The metadata-only fixture retains the same raw byte hash but changes the document revision and chunk IDs because the title is part of the recorded provenance.
Altered saved snapshot. The separate validate_snapshot function rebuilds metadata revisions and paragraph chunks from the normalized saved text and checks the complete canonical identity. A changed field with an unchanged snapshot ID raises IngestionError with invalid_snapshot. This catches inconsistent saved artifacts. It cannot reconstruct original CRLF bytes or authenticate the source author from normalized text.
This is deliberately conservative: an edit anywhere in a document gives all its chunks a new document revision. The lab does not claim stable chunk IDs across document edits, index synchronization or an optimal chunking policy.
A changed snapshot is a signal for a later ingestion or update operation. Creating this JSON alone does not remove old chunks from a database, rewrite held-out eval labels or prove a passage relevant to a question.
Execute the local corpus operation
Download the complete lab. The implementation, fixtures, tests, README and raw execution record travel together.
"""Version pinned documentation bytes, metadata, paragraphs and corpus snapshots."""
from __future__ import annotations
import argparse
import hashlib
import json
from pathlib import Path
from load_documents import IngestionError, load_documents
CHUNKER = {'name': 'paragraphs', 'version': 1, 'newline_policy': 'CRLF-to-LF', 'separator': 'blank-line'}
METADATA_KEYS = ('path', 'title', 'version', 'url', 'fact_status')
def canonical_hash(value: object) -> str:
return hashlib.sha256(json.dumps(value, sort_keys=True, separators=(',', ':'), ensure_ascii=False).encode('utf-8')).hexdigest()
def paragraphs(text: str):
"""Yield exact normalized-text slices with 1-based inclusive line spans."""
lines = text.splitlines(keepends=True)
start = None
for index, line in enumerate(lines):
if line.strip():
if start is None:
start = index
elif start is not None:
yield start + 1, index, ''.join(lines[start:index])
start = None
if start is not None:
yield start + 1, len(lines), ''.join(lines[start:])
def version_document(document: dict, config: dict) -> dict:
metadata = {key: document[key] for key in METADATA_KEYS if key in document}
metadata_sha256 = canonical_hash(metadata)
revision = canonical_hash({'source_id': document['source_id'], 'raw_sha256': document['sha256'], 'metadata_sha256': metadata_sha256})
chunks = []
for start, end, text in paragraphs(document['text']):
text_hash = hashlib.sha256(text.encode('utf-8')).hexdigest()
identity = {'document_revision': revision, 'chunker': config, 'start_line': start, 'end_line': end, 'text_sha256': text_hash}
chunks.append({'chunk_id': canonical_hash(identity), 'source_id': document['source_id'], 'document_revision': revision,
'start_line': start, 'end_line': end, 'text': text, 'text_sha256': text_hash})
return {**document, 'metadata_sha256': metadata_sha256, 'document_revision': revision, 'chunks': chunks}
def snapshot_identity(documents: list, config: dict) -> str:
# Bind normalized text, metadata and every chunk too, not only claimed revisions.
return canonical_hash({'schema_version': 1, 'chunker': config, 'documents': documents})
def build_snapshot(manifest: Path, chunker: dict | None = None) -> dict:
config = dict(CHUNKER if chunker is None else chunker)
if canonical_hash(config) != canonical_hash(CHUNKER):
raise IngestionError('unsupported_chunker')
loaded = load_documents(manifest)['documents']
documents = [version_document(document, config) for document in sorted(loaded, key=lambda d: d['source_id'])]
return {'schema_version': 1, 'execution_mode': 'offline-local-files', 'chunker': config,
'snapshot_id': snapshot_identity(documents, config), 'documents': documents}
def validate_snapshot(snapshot: dict) -> dict:
"""Reject inconsistent saved artifacts; this does not authenticate raw files.
Original byte hashes were checked while loading files. Normalized saved text
cannot reconstruct original CRLF bytes, so this validator checks its canonical
identity and internal derivations, not the original files or their provenance.
"""
try:
if not isinstance(snapshot, dict) or type(snapshot.get('schema_version')) is not int or snapshot['schema_version'] != 1:
raise ValueError()
if snapshot.get('execution_mode') != 'offline-local-files' or canonical_hash(snapshot.get('chunker')) != canonical_hash(CHUNKER):
raise ValueError()
documents = snapshot['documents']
if not isinstance(documents, list) or not documents:
raise ValueError()
seen_ids, seen_paths = set(), set()
allowed = ('source_id', 'path', 'title', 'sha256', 'version', 'url', 'fact_status', 'text')
for document in documents:
if not isinstance(document, dict):
raise ValueError()
original = {key: document[key] for key in allowed if key in document}
if any(not isinstance(original.get(key), str) or not original[key] for key in ('source_id','path','title','sha256','text')):
raise ValueError()
if any(key in original and (not isinstance(original[key],str) or not original[key]) for key in ('version','url','fact_status')):
raise ValueError()
if len(original['sha256']) != 64 or any(c not in '0123456789abcdef' for c in original['sha256']):
raise ValueError()
if len(original['text'].encode('utf-8')) > 65536 or not original['text'].strip() or '\x00' in original['text'] or '\r\n' in original['text']:
raise ValueError()
if original['source_id'] in seen_ids or original['path'] in seen_paths:
raise ValueError()
seen_ids.add(original['source_id']); seen_paths.add(original['path'])
if canonical_hash(document) != canonical_hash(version_document(original, CHUNKER)):
raise ValueError()
if documents != sorted(documents, key=lambda d:d['source_id']):
raise ValueError()
if snapshot.get('snapshot_id') != snapshot_identity(documents, CHUNKER):
raise ValueError()
except (KeyError, TypeError, ValueError, UnicodeError, OverflowError) as error:
raise IngestionError('invalid_snapshot') from error
return snapshot
def compare(before: dict, after: dict) -> dict:
previous = {d['source_id']: d for d in before['documents']}
current = {d['source_id']: d for d in after['documents']}
return {'snapshot_changed': before['snapshot_id'] != after['snapshot_id'],
'added': sorted(current.keys() - previous.keys()), 'removed': sorted(previous.keys() - current.keys()),
'changed': [{'source_id': key, 'raw_bytes_changed': previous[key]['sha256'] != current[key]['sha256'],
'metadata_changed': previous[key]['metadata_sha256'] != current[key]['metadata_sha256'],
'chunk_ids_changed': [c['chunk_id'] for c in previous[key]['chunks']] != [c['chunk_id'] for c in current[key]['chunks']]}
for key in sorted(previous.keys() & current.keys()) if previous[key]['document_revision'] != current[key]['document_revision']]}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--manifest', type=Path, default=Path(__file__).parent / 'fixtures/initial/source-manifest.json')
parser.add_argument('--output', type=Path, required=True)
args = parser.parse_args()
result = build_snapshot(args.manifest)
args.output.parent.mkdir(parents=True, exist_ok=True)
args.output.write_text(json.dumps(result, indent=2) + '\n', encoding='utf-8')
print(json.dumps({'snapshot_id': result['snapshot_id'], 'documents': len(result['documents']), 'chunks': sum(len(d['chunks']) for d in result['documents'])}, sort_keys=True))
if __name__ == '__main__': main()python3 version_documents.py --output /tmp/corpus-snapshot.json
python3 demo.py --output /tmp/corpus-version-comparison
python3 -m unittest discover -v{
"comparisons": {
"metadata-change": {
"added": [],
"changed": [
{
"chunk_ids_changed": true,
"metadata_changed": true,
"raw_bytes_changed": false,
"source_id": "path-write"
}
],
"removed": [],
"snapshot_changed": true
},
"text-change": {
"added": [],
"changed": [
{
"chunk_ids_changed": true,
"metadata_changed": true,
"raw_bytes_changed": true,
"source_id": "path-write"
}
],
"removed": [],
"snapshot_changed": true
}
},
"execution_mode": "offline-local-files"
}Compare the snapshot IDs, document revisions and chunk IDs in the complete saved result. This lab creates JSON snapshots; it does not commit rows to a database.
Inspect the complete saved result
{
"execution_mode": "offline-local-files",
"scope": "deliberate local fixture changes; no upstream refresh or model execution",
"snapshots": {
"initial": {
"schema_version": 1,
"execution_mode": "offline-local-files",
"chunker": {
"name": "paragraphs",
"version": 1,
"newline_policy": "CRLF-to-LF",
"separator": "blank-line"
},
"snapshot_id": "0615d22a3315e97c0a88c6c566c546877304f5a28a97e83f68dd36881d4bbd0e",
"documents": [
{
"source_id": "json-text",
"path": "docs/json-text.md",
"title": "json-text",
"sha256": "0f5efa9669a97f93d5e7e8fe91de80a0d954643947ac9619dba41d3904166288",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"fact_status": "original summary; primary docs checked",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"metadata_sha256": "fa80a47c53e04f67d9481651bd0b6fae72e8094b2df37fc9a375cabfd930bc10",
"document_revision": "0c4b565b2b1900787d6da9f8539c471f8cb2245c00abf887a57d460364abc57a",
"chunks": [
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"document_revision": "0c4b565b2b1900787d6da9f8539c471f8cb2245c00abf887a57d460364abc57a",
"start_line": 1,
"end_line": 1,
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"text_sha256": "0f5efa9669a97f93d5e7e8fe91de80a0d954643947ac9619dba41d3904166288"
}
]
},
{
"source_id": "path-read",
"path": "docs/path-read.md",
"title": "path-read",
"sha256": "dc097ca63486ace45279f412ea266bf754fcd2f0b1153f5d763ce905dafbc2c0",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"fact_status": "original summary; primary docs checked",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"metadata_sha256": "1f1afc1a151fe961715da50dad4c18a19fa67b1f94117123153731c63dd9a1a8",
"document_revision": "b58385b9bf597a951c16c035653d639929924e53eae3e5ada255afd6ee1592f2",
"chunks": [
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"document_revision": "b58385b9bf597a951c16c035653d639929924e53eae3e5ada255afd6ee1592f2",
"start_line": 1,
"end_line": 1,
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"text_sha256": "dc097ca63486ace45279f412ea266bf754fcd2f0b1153f5d763ce905dafbc2c0"
}
]
},
{
"source_id": "path-read-3-13",
"path": "docs/path-read-3-13.md",
"title": "path-read-3-13",
"sha256": "0dd5becb0c04458ad5f8bbf3fdef4f71034c584ef863bf748793bef7309b9f4f",
"version": "3.13",
"url": "https://docs.python.org/3.13/library/pathlib.html#pathlib.Path.read_text",
"fact_status": "documented only; Python 3.13 not executed",
"text": "In Python 3.13, Path.read_text accepts encoding, errors and newline. The newline parameter was added in Python 3.13. The method returns decoded file text and closes the file.",
"metadata_sha256": "a757e6201998dfaf780a87d8544ee4079bb5f1e5ab97ad035a77cd3afdbf0d01",
"document_revision": "d68df3e7ea1e24af3f40366a61402fc80558fefe2ef5f3eda928de0e01820db9",
"chunks": [
{
"chunk_id": "bdaeff5ae3ab88b70b12b9b6d4a0a46de45ea14520b971a7756cb03bbd384f35",
"source_id": "path-read-3-13",
"document_revision": "d68df3e7ea1e24af3f40366a61402fc80558fefe2ef5f3eda928de0e01820db9",
"start_line": 1,
"end_line": 1,
"text": "In Python 3.13, Path.read_text accepts encoding, errors and newline. The newline parameter was added in Python 3.13. The method returns decoded file text and closes the file.",
"text_sha256": "0dd5becb0c04458ad5f8bbf3fdef4f71034c584ef863bf748793bef7309b9f4f"
}
]
},
{
"source_id": "path-write",
"path": "docs/path-write.md",
"title": "path-write",
"sha256": "a48b4b9f763cd6309ebe32af19c5a814adcf810c2259683da9f328ccee46948e",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"fact_status": "original summary; primary docs checked",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"metadata_sha256": "b31d26389cf0da944c0817a580f736d8009f2b297b45ec631849c0e219818768",
"document_revision": "28c6349eaf80b629322edb310bb871dc45a04af9fad3b752bc75d12f0d7314e7",
"chunks": [
{
"chunk_id": "c66643a4609c6ee157458bf4353f7446b9c6a200a3ef453846d6a18d5b5069a7",
"source_id": "path-write",
"document_revision": "28c6349eaf80b629322edb310bb871dc45a04af9fad3b752bc75d12f0d7314e7",
"start_line": 1,
"end_line": 1,
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"text_sha256": "a48b4b9f763cd6309ebe32af19c5a814adcf810c2259683da9f328ccee46948e"
}
]
}
]
},
"text-change": {
"schema_version": 1,
"execution_mode": "offline-local-files",
"chunker": {
"name": "paragraphs",
"version": 1,
"newline_policy": "CRLF-to-LF",
"separator": "blank-line"
},
"snapshot_id": "d1ee6e652c1c6d08e6e4e10314f53153d6e0b5e5850739fd135f455196d4ec0d",
"documents": [
{
"source_id": "json-text",
"path": "docs/json-text.md",
"title": "json-text",
"sha256": "0f5efa9669a97f93d5e7e8fe91de80a0d954643947ac9619dba41d3904166288",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"fact_status": "original summary; primary docs checked",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"metadata_sha256": "fa80a47c53e04f67d9481651bd0b6fae72e8094b2df37fc9a375cabfd930bc10",
"document_revision": "0c4b565b2b1900787d6da9f8539c471f8cb2245c00abf887a57d460364abc57a",
"chunks": [
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"document_revision": "0c4b565b2b1900787d6da9f8539c471f8cb2245c00abf887a57d460364abc57a",
"start_line": 1,
"end_line": 1,
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"text_sha256": "0f5efa9669a97f93d5e7e8fe91de80a0d954643947ac9619dba41d3904166288"
}
]
},
{
"source_id": "path-read",
"path": "docs/path-read.md",
"title": "path-read",
"sha256": "dc097ca63486ace45279f412ea266bf754fcd2f0b1153f5d763ce905dafbc2c0",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"fact_status": "original summary; primary docs checked",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"metadata_sha256": "1f1afc1a151fe961715da50dad4c18a19fa67b1f94117123153731c63dd9a1a8",
"document_revision": "b58385b9bf597a951c16c035653d639929924e53eae3e5ada255afd6ee1592f2",
"chunks": [
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"document_revision": "b58385b9bf597a951c16c035653d639929924e53eae3e5ada255afd6ee1592f2",
"start_line": 1,
"end_line": 1,
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"text_sha256": "dc097ca63486ace45279f412ea266bf754fcd2f0b1153f5d763ce905dafbc2c0"
}
]
},
{
"source_id": "path-read-3-13",
"path": "docs/path-read-3-13.md",
"title": "path-read-3-13",
"sha256": "0dd5becb0c04458ad5f8bbf3fdef4f71034c584ef863bf748793bef7309b9f4f",
"version": "3.13",
"url": "https://docs.python.org/3.13/library/pathlib.html#pathlib.Path.read_text",
"fact_status": "documented only; Python 3.13 not executed",
"text": "In Python 3.13, Path.read_text accepts encoding, errors and newline. The newline parameter was added in Python 3.13. The method returns decoded file text and closes the file.",
"metadata_sha256": "a757e6201998dfaf780a87d8544ee4079bb5f1e5ab97ad035a77cd3afdbf0d01",
"document_revision": "d68df3e7ea1e24af3f40366a61402fc80558fefe2ef5f3eda928de0e01820db9",
"chunks": [
{
"chunk_id": "bdaeff5ae3ab88b70b12b9b6d4a0a46de45ea14520b971a7756cb03bbd384f35",
"source_id": "path-read-3-13",
"document_revision": "d68df3e7ea1e24af3f40366a61402fc80558fefe2ef5f3eda928de0e01820db9",
"start_line": 1,
"end_line": 1,
"text": "In Python 3.13, Path.read_text accepts encoding, errors and newline. The newline parameter was added in Python 3.13. The method returns decoded file text and closes the file.",
"text_sha256": "0dd5becb0c04458ad5f8bbf3fdef4f71034c584ef863bf748793bef7309b9f4f"
}
]
},
{
"source_id": "path-write",
"path": "docs/path-write.md",
"title": "path-write",
"sha256": "fc153a13320bf10c3bd95073064be8ca79fc49d1baf94020aab6e1f9098bcb03",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"fact_status": "deliberate local teaching revision; not an upstream documentation update",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.\n\nLocal teaching revision: callers must review overwrite behavior before saving.\n",
"metadata_sha256": "a26519682f3f5ec1043876ce74b1eafe0eff98421e486c9dd1a2c949bb19f0ad",
"document_revision": "a036f69e0484af8c5ddd938bb292a0cb63ec2f661379d871d3901a03dc987dd2",
"chunks": [
{
"chunk_id": "d27f0074d2d1a0c980e8bf541c26a83cf970e2f71e2c7f9a7d3da5442214b66b",
"source_id": "path-write",
"document_revision": "a036f69e0484af8c5ddd938bb292a0cb63ec2f661379d871d3901a03dc987dd2",
"start_line": 1,
"end_line": 1,
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.\n",
"text_sha256": "415990e67efb5d9b048c1af2ced2984f364e21bed39c04f58ed2d0b147e7e97e"
},
{
"chunk_id": "99e7deb4a60f1d2a3e3e8f33f3283b654e2ea80568aa6e2aa819d4666714c42f",
"source_id": "path-write",
"document_revision": "a036f69e0484af8c5ddd938bb292a0cb63ec2f661379d871d3901a03dc987dd2",
"start_line": 3,
"end_line": 3,
"text": "Local teaching revision: callers must review overwrite behavior before saving.\n",
"text_sha256": "f3540576a85312166e10df3c002f98b9b6503b5e88248e1bbe373485e4851ae0"
}
]
}
]
},
"metadata-change": {
"schema_version": 1,
"execution_mode": "offline-local-files",
"chunker": {
"name": "paragraphs",
"version": 1,
"newline_policy": "CRLF-to-LF",
"separator": "blank-line"
},
"snapshot_id": "970f70ddaeb7239d1c44a716174361be7b0cba4a552d011f830da98ed9ec95b5",
"documents": [
{
"source_id": "json-text",
"path": "docs/json-text.md",
"title": "json-text",
"sha256": "0f5efa9669a97f93d5e7e8fe91de80a0d954643947ac9619dba41d3904166288",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/json.html#json.dumps",
"fact_status": "original summary; primary docs checked",
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"metadata_sha256": "fa80a47c53e04f67d9481651bd0b6fae72e8094b2df37fc9a375cabfd930bc10",
"document_revision": "0c4b565b2b1900787d6da9f8539c471f8cb2245c00abf887a57d460364abc57a",
"chunks": [
{
"chunk_id": "e5455737b11664449f653280d89dfc2faaa9a14de16e9c3eb24296ca2ceea51c",
"source_id": "json-text",
"document_revision": "0c4b565b2b1900787d6da9f8539c471f8cb2245c00abf887a57d460364abc57a",
"start_line": 1,
"end_line": 1,
"text": "json.dumps(obj, ensure_ascii=True) returns a JSON string. Setting ensure_ascii=False leaves non-ASCII characters in that string. It does not write a file or select the encoding used by a file writer.",
"text_sha256": "0f5efa9669a97f93d5e7e8fe91de80a0d954643947ac9619dba41d3904166288"
}
]
},
{
"source_id": "path-read",
"path": "docs/path-read.md",
"title": "path-read",
"sha256": "dc097ca63486ace45279f412ea266bf754fcd2f0b1153f5d763ce905dafbc2c0",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.read_text",
"fact_status": "original summary; primary docs checked",
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"metadata_sha256": "1f1afc1a151fe961715da50dad4c18a19fa67b1f94117123153731c63dd9a1a8",
"document_revision": "b58385b9bf597a951c16c035653d639929924e53eae3e5ada255afd6ee1592f2",
"chunks": [
{
"chunk_id": "c737db948e8111ae00e3442d8a22c80c876e43446dc1ee176ffe8ce142a41900",
"source_id": "path-read",
"document_revision": "b58385b9bf597a951c16c035653d639929924e53eae3e5ada255afd6ee1592f2",
"start_line": 1,
"end_line": 1,
"text": "Path.read_text(encoding=None, errors=None) opens a file, returns its text as a string and closes the file. Set encoding=\"utf-8\" when reading a UTF-8 file.",
"text_sha256": "dc097ca63486ace45279f412ea266bf754fcd2f0b1153f5d763ce905dafbc2c0"
}
]
},
{
"source_id": "path-read-3-13",
"path": "docs/path-read-3-13.md",
"title": "path-read-3-13",
"sha256": "0dd5becb0c04458ad5f8bbf3fdef4f71034c584ef863bf748793bef7309b9f4f",
"version": "3.13",
"url": "https://docs.python.org/3.13/library/pathlib.html#pathlib.Path.read_text",
"fact_status": "documented only; Python 3.13 not executed",
"text": "In Python 3.13, Path.read_text accepts encoding, errors and newline. The newline parameter was added in Python 3.13. The method returns decoded file text and closes the file.",
"metadata_sha256": "a757e6201998dfaf780a87d8544ee4079bb5f1e5ab97ad035a77cd3afdbf0d01",
"document_revision": "d68df3e7ea1e24af3f40366a61402fc80558fefe2ef5f3eda928de0e01820db9",
"chunks": [
{
"chunk_id": "bdaeff5ae3ab88b70b12b9b6d4a0a46de45ea14520b971a7756cb03bbd384f35",
"source_id": "path-read-3-13",
"document_revision": "d68df3e7ea1e24af3f40366a61402fc80558fefe2ef5f3eda928de0e01820db9",
"start_line": 1,
"end_line": 1,
"text": "In Python 3.13, Path.read_text accepts encoding, errors and newline. The newline parameter was added in Python 3.13. The method returns decoded file text and closes the file.",
"text_sha256": "0dd5becb0c04458ad5f8bbf3fdef4f71034c584ef863bf748793bef7309b9f4f"
}
]
},
{
"source_id": "path-write",
"path": "docs/path-write.md",
"title": "Write UTF-8 text with pathlib",
"sha256": "a48b4b9f763cd6309ebe32af19c5a814adcf810c2259683da9f328ccee46948e",
"version": "3.12",
"url": "https://docs.python.org/3.12/library/pathlib.html#pathlib.Path.write_text",
"fact_status": "original summary; primary docs checked",
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"metadata_sha256": "fa7085e5dcd23a3cc6a63261a19b1601c12750c05110a900e3251b59c4585572",
"document_revision": "db8807f30a17947915f67ea5208f5768a4f75c2b2233ad8ff0776327a96e3706",
"chunks": [
{
"chunk_id": "12f06d42c15746b564ca36a775a7bd5b88d6376a79b47473d360cb53cc60c815",
"source_id": "path-write",
"document_revision": "db8807f30a17947915f67ea5208f5768a4f75c2b2233ad8ff0776327a96e3706",
"start_line": 1,
"end_line": 1,
"text": "Path.write_text(data, encoding=None, errors=None, newline=None) writes text to a file and closes it. Set encoding=\"utf-8\" to select UTF-8. An existing file is overwritten. The method returns the number of characters written.",
"text_sha256": "a48b4b9f763cd6309ebe32af19c5a814adcf810c2259683da9f328ccee46948e"
}
]
}
]
}
},
"comparisons": {
"text-change": {
"snapshot_changed": true,
"added": [],
"removed": [],
"changed": [
{
"source_id": "path-write",
"raw_bytes_changed": true,
"metadata_changed": true,
"chunk_ids_changed": true
}
]
},
"metadata-change": {
"snapshot_changed": true,
"added": [],
"removed": [],
"changed": [
{
"source_id": "path-write",
"raw_bytes_changed": false,
"metadata_changed": true,
"chunk_ids_changed": true
}
]
}
}
}Compare the state before and after the operation
Make a scratch copy of the fixture. Change only its title and compare the snapshots. The raw byte hash should remain the same, while the document revision and chunk IDs change. Then edit its text without changing the manifest hash and compare the explicit rejection. Do not modify the published baseline or silently update eval targets.
Read the explanation
Check the specific stored state and input contract; a completed command alone does not prove corpus consistency.
What’s next
Continue with Resume Document Ingestion.
References
APIs were checked on 23 October 2026 against official Python hashlib, Python sqlite3, SQLite transactions and SQLite FTS5 documentation as applicable. The source summaries retain their version-specific Python documentation URLs. The changed fixture is a labelled local teaching revision, not an upstream documentation update.