The tracker docs assistant already has per-stage evaluation results. Before relying on that report for another release, check that its inputs and settings still match. This lab binds the local application bundle to those saved results and then changes one part at a time.
When you need this. Use a release manifest when a prompt, documentation update, retrieval setting or application edit can make a previous evaluation irrelevant. It complements the quality gate; matching files do not mean good answers.
The six scenarios make no model calls. They read the saved Gemini recording and report from the per-stage evaluation lesson. That report still contains its answer-quality failures; this lab checks compatibility and leaves the scores as they are. Download the complete lab.
A release manifest records the content hashes and settings used to identify an application bundle. Here it lists nine files: the assistant and evaluator sources, cases, documentation, issue records, runtime settings, dependency list, model recording and evaluation report. SHA-256 hashes detect byte changes after the manifest is created. They are not signatures and do not authenticate the publisher.
Step 1: Name the bundle that the report describes
Download the complete lab, extract it and open a terminal in build-a-release-manifest-for-application-model-tokenizer-prompt-data-and-index-versions/. The README lists its runtime and recorded examples.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r tracker-requirements.txtbundle/ contains unchanged copies from the published docs-assistant lab. Its run-config.json copies the recording’s generator, embedding model, judge and strict-faithfulness settings. The model entries are names only. A hosted Gemini model name does not identify the provider’s exact weights or tokenizer, and this lab cannot pin or inspect either one.
Application source changes. Adding a comment to assistant.py causes a hash mismatch, even though the comment may not change behavior. The check does not guess whether an edit matters; it reports the changed file. Review the difference, then decide whether to create a new manifest or run an evaluation. This manifest was created after the recording already existed, so it cannot show that these exact source bytes produced that earlier run. For future releases, create the bundle when you run the evaluation and keep the two together.
verify() reads both the current application inputs and the saved report. Its file check compares the bundle with the manifest. Its recording-key check separately compares the current dataset, document content, prompt and retrieval depth with the saved record. A file mismatch blocks even when every field in that key stays the same.
Step 2: Recompute the recording key from current inputs
The earlier evaluator already hashes JSON values using sorted keys and records cases_sha, docs_sha, prompt_sha, k, model names and judge settings. current_key() uses that same serialization. It reads the literal PROMPT and K assignments with Python’s ast library, so verifying the bundle does not import LangChain, execute agent code or contact Gemini.
Documentation changes under the same filename. Appending one sentence changes both the file hash and the recomputed docs_sha. Rebuilding the manifest still fails: the old recording key describes different documentation. Rerun the original docs-assistant evaluator on the changed documents, keep its new recording and report, and then bind the new bundle. Keeping the same filename or computing a new checksum does not make the old model output valid for the changed documents.
Retrieval depth changes. Changing K = 3 to K = 4 is refused because the recorded retrieval set came from K = 3. Changing the generator name in run-config.json is refused the same way. Plan a new evaluation run for any change to K or to a recorded model name.
"""Bind one recorded tracker evaluation to its local application files.
Hashes detect differences; this is neither a signature nor a deployment check.
"""
import argparse
import ast
import hashlib
import json
from pathlib import Path
from pydantic import BaseModel, ConfigDict, Field
FILES = ('assistant.py', 'evaluate.py', 'cases.json', 'docs.json', 'issues.json',
'run-config.json', 'requirements.lock', 'recording.json', 'eval-run.json')
class Manifest(BaseModel):
model_config = ConfigDict(extra='forbid')
schema_version: int = Field(default=1, ge=1, le=1)
files: dict[str, str]
recording_key: dict
def sha(value):
# Match evaluate.py's existing recording-key serialization, including spaces.
return hashlib.sha256(json.dumps(value, sort_keys=True).encode()).hexdigest()[:12]
def read(root, name):
return json.loads((root / name).read_text())
def current_key(root):
# Read literal application settings without importing or executing the agent.
settings = {}
for node in ast.parse((root / 'assistant.py').read_text()).body:
if isinstance(node, ast.Assign) and isinstance(node.targets[0], ast.Name):
if node.targets[0].id in ('PROMPT', 'K'):
settings[node.targets[0].id] = ast.literal_eval(node.value)
cases = read(root, 'cases.json')
return {**read(root, 'run-config.json'),
'cases': f"{cases['name']}@{cases['version']}", 'cases_sha': sha(cases),
'docs_sha': sha({'docs': read(root, 'docs.json'), 'issues': read(root, 'issues.json')}),
'prompt_sha': sha(settings['PROMPT']), 'k': settings['K']}
def digests(root):
return {name: hashlib.sha256((root / name).read_bytes()).hexdigest() for name in FILES}
def verify(root, manifest):
if set(manifest.files) != set(FILES):
raise ValueError('manifest file set differs from the required application bundle')
findings = []
for name, expected in manifest.files.items():
path = root / name
if not path.is_file() or hashlib.sha256(path.read_bytes()).hexdigest() != expected:
findings.append(f'file changed or missing: {name}')
try:
key = current_key(root)
for name, other in [('manifest', manifest.recording_key),
('recording', read(root, 'recording.json')['key']),
('eval run', read(root, 'eval-run.json')['key'])]:
for field in key.keys() | other.keys():
if key.get(field) != other.get(field):
findings.append(f'{name} key differs: {field}')
cases = read(root, 'cases.json')['cases']
rows = read(root, 'eval-run.json')['rows']
expected_ids = [c['id'] for c in cases]
if sorted(r['id'] for r in rows) != sorted(expected_ids):
findings.append('eval case ids differ')
entries = read(root, 'recording.json')['entries']
if any(f'{stage}/{case_id}' not in entries for case_id in expected_ids
for stage in ('agent', 'tools', 'judge')):
findings.append('recording entry missing')
except (OSError, ValueError, KeyError, SyntaxError) as exc:
findings.append(f'cannot read application bundle: {type(exc).__name__}')
return {'decision': 'block' if findings else 'match', 'findings': sorted(findings)}
def build(root):
manifest = Manifest(files=digests(root), recording_key=current_key(root))
result = verify(root, manifest)
if result['findings']:
raise ValueError(result)
return manifest
if __name__ == '__main__':
parser = argparse.ArgumentParser()
parser.add_argument('action', choices=('build', 'verify'))
parser.add_argument('--root', type=Path, default=Path('bundle'))
parser.add_argument('--manifest', type=Path, default=Path('release-manifest.json'))
args = parser.parse_args()
if args.action == 'build':
args.manifest.write_text(build(args.root).model_dump_json(indent=2) + '\n')
else:
result = verify(args.root, Manifest.model_validate_json(args.manifest.read_text()))
print(json.dumps(result, indent=2))
raise SystemExit(result['decision'] == 'block')Step 3: Keep every evaluated case attached to the report
The file digests also cover the model recording and the evaluation report. Beyond those byte comparisons, verify() checks that report row IDs match the case set and that each case has recorded agent, tool and judge entries. A report that drops a result but keeps its old key still fails this check.
A failing case disappears from the report. The recorder removes one row and keeps the key. Verification blocks with eval case ids differ, and creating a fresh manifest is also refused. Restore the complete report or rerun the evaluation. This is a membership check: the script does not recompute the judge verdicts or validate every trajectory.
A required file or recording entry is missing. Verification blocks rather than accepting a partial bundle. The manifest must list exactly the required files; a caller cannot omit assistant.py to avoid checking it. Repair the bundle before you compare its quality results.
A compatibility decision of match can coexist with failed quality checks. The tracker report in this lab has a row with a recorded failure and still returns match; the test test_unchanged_failed_quality_scores_still_match checks this. After compatibility succeeds, compare the docs-agent stage failures with your release policy. The regression-gate lab reads issue-triage pytest reports directly. Its documentation-stage adapter maps this separate report format to named outcomes under a documentation-specific policy. Use that adapter before you gate these scores.
Step 4: Reproduce the changed-file decisions
python3 -m venv .venv
.venv/bin/python -m pip install -r tracker-requirements.txt
.venv/bin/python -m pytest -q test_release.py
.venv/bin/python record_release_checks.py --check
.venv/bin/python bind_release.py verifyunchanged bundle: match (0 findings)
documentation edited: block (4 findings)
retrieval depth changed: block (4 findings)
generator name changed: block (4 findings)
case removed from report: block (2 findings)
application source edited: block (1 findings)Run these commands from the extracted lab directory. tracker-requirements.txt installs the checker. bundle/requirements.lock is the unchanged docs-assistant dependency list that the manifest hashes; you do not install it for these checks. No API key is needed.
The saved scenarios include the unchanged bundle and five isolated edits. --check recomputes them in temporary directories and compares both saved output files without rewriting them. bind_release.py verify returns status 1 for a blocked bundle and 0 for a match. Run verify before you use the saved report in a release decision, and read its findings when it returns 1. Its build action creates a manifest only after the current recording key and membership checks succeed.
This is a local consistency check. It does not read a deployed image or attest the runtime environment. The manifest is unsigned, so someone who changes both the files and the manifest can make them agree. Use artifact signing and deployment verification tools for those jobs.
Read the explanation
The manifest file check and the recomputed recording-key check address different changes; both must match before a quality gate is meaningful.
References
Checked 2026-10-22 against the installed Pydantic and pytest APIs; exact pins are in the lab download. Python’s hashlib, ast.literal_eval, Pydantic models, and pytest temporary directories.