Bind an eval run to a release manifest

The tracker docs assistant already has per-stage evaluation results. Before relying on that report for another release, check that its inputs and settings still match. This lab binds the local application bundle to those saved results and then changes one part at a time.

When you need this. Use a release manifest when a prompt, documentation update, retrieval setting or application edit can make a previous evaluation irrelevant. It complements the quality gate; matching files do not mean good answers.

The six scenarios make no model calls. They read the unchanged Gemini recording and report from the per-stage evaluation lesson. Its recorded answer-quality failures remain unchanged. Download the complete lab.

Step 1: Name the bundle that the report describes

Download the complete lab, extract it and open a terminal in build-a-release-manifest-for-application-model-tokenizer-prompt-data-and-index-versions/. The README lists its runtime and recorded examples.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r tracker-requirements.txt

A release manifest records the content hashes and settings used to identify an application bundle. Here it lists nine files: the assistant and evaluator sources, cases, documentation, issue records, runtime settings, dependency list, model recording and evaluation report. SHA-256 hashes detect byte changes after the manifest is created. They are not signatures and do not authenticate the publisher.

bundle/ contains unchanged copies from the published docs-assistant lab. Its run-config.json copies the recording’s generator, embedding model, judge and strict-faithfulness settings. These are recorded model names. A hosted Gemini model name does not identify the provider’s exact weights or tokenizer; this lab cannot pin or inspect either one.

Application source changes. A comment added to assistant.py causes a hash mismatch even though behavior may be unchanged. Review the difference and decide whether to create a new manifest or run an evaluation. The check deliberately detects file changes rather than guessing whether an edit matters. This manifest is created around an existing recording: it does not establish that these exact source bytes produced that earlier run. For future releases, create and retain the bundle with the run.

Both the current application inputs and the saved report reach verify(). Its file check compares the bundle with the manifest. Its recording-key check separately compares the current dataset, document content, prompt and retrieval depth with the saved record. A file mismatch blocks even if the fields in that key happen to stay the same.

Step 2: Recompute the recording key from current inputs

The earlier evaluator already hashes JSON values using sorted keys and records cases_sha, docs_sha, prompt_sha, k, model names and judge settings. current_key() uses that same serialization. It reads the literal PROMPT and K assignments with Python’s ast library, so verifying the bundle does not import LangChain, execute agent code or contact Gemini.

Documentation changes under the same filename. Appending one sentence changes both the file hash and the recomputed docs_sha. Rebuilding the manifest still fails: the old recording key describes different documentation. Rerun the original docs-assistant evaluator on the changed documents and retain its new recording and report before binding the new bundle. A filename or a newly computed checksum cannot make old model output apply to changed context.

Retrieval depth changes. Changing K = 3 to K = 4 is refused because the recorded retrieval set came from another setting. The same rule applies to the recorded generator name. The script reads configured names; it does not inspect a running service or certify the model behind that name.

"""Bind one recorded tracker evaluation to its local application files.
Hashes detect differences; this is neither a signature nor a deployment check.
"""
import argparse
import ast
import hashlib
import json
from pathlib import Path

from pydantic import BaseModel, ConfigDict, Field

FILES = ('assistant.py', 'evaluate.py', 'cases.json', 'docs.json', 'issues.json',
         'run-config.json', 'requirements.lock', 'recording.json', 'eval-run.json')


class Manifest(BaseModel):
    model_config = ConfigDict(extra='forbid')
    schema_version: int = Field(default=1, ge=1, le=1)
    files: dict[str, str]
    recording_key: dict


def sha(value):
    # Match evaluate.py's existing recording-key serialization, including spaces.
    return hashlib.sha256(json.dumps(value, sort_keys=True).encode()).hexdigest()[:12]


def read(root, name):
    return json.loads((root / name).read_text())


def current_key(root):
    # Read literal application settings without importing or executing the agent.
    settings = {}
    for node in ast.parse((root / 'assistant.py').read_text()).body:
        if isinstance(node, ast.Assign) and isinstance(node.targets[0], ast.Name):
            if node.targets[0].id in ('PROMPT', 'K'):
                settings[node.targets[0].id] = ast.literal_eval(node.value)
    cases = read(root, 'cases.json')
    return {**read(root, 'run-config.json'),
            'cases': f"{cases['name']}@{cases['version']}", 'cases_sha': sha(cases),
            'docs_sha': sha({'docs': read(root, 'docs.json'), 'issues': read(root, 'issues.json')}),
            'prompt_sha': sha(settings['PROMPT']), 'k': settings['K']}


def digests(root):
    return {name: hashlib.sha256((root / name).read_bytes()).hexdigest() for name in FILES}


def verify(root, manifest):
    if set(manifest.files) != set(FILES):
        raise ValueError('manifest file set differs from the required application bundle')
    findings = []
    for name, expected in manifest.files.items():
        path = root / name
        if not path.is_file() or hashlib.sha256(path.read_bytes()).hexdigest() != expected:
            findings.append(f'file changed or missing: {name}')
    try:
        key = current_key(root)
        for name, other in [('manifest', manifest.recording_key),
                            ('recording', read(root, 'recording.json')['key']),
                            ('eval run', read(root, 'eval-run.json')['key'])]:
            for field in key.keys() | other.keys():
                if key.get(field) != other.get(field):
                    findings.append(f'{name} key differs: {field}')
        cases = read(root, 'cases.json')['cases']
        rows = read(root, 'eval-run.json')['rows']
        expected_ids = [c['id'] for c in cases]
        if sorted(r['id'] for r in rows) != sorted(expected_ids):
            findings.append('eval case ids differ')
        entries = read(root, 'recording.json')['entries']
        if any(f'{stage}/{case_id}' not in entries for case_id in expected_ids
               for stage in ('agent', 'tools', 'judge')):
            findings.append('recording entry missing')
    except (OSError, ValueError, KeyError, SyntaxError) as exc:
        findings.append(f'cannot read application bundle: {type(exc).__name__}')
    return {'decision': 'block' if findings else 'match', 'findings': sorted(findings)}


def build(root):
    manifest = Manifest(files=digests(root), recording_key=current_key(root))
    result = verify(root, manifest)
    if result['findings']:
        raise ValueError(result)
    return manifest


if __name__ == '__main__':
    parser = argparse.ArgumentParser()
    parser.add_argument('action', choices=('build', 'verify'))
    parser.add_argument('--root', type=Path, default=Path('bundle'))
    parser.add_argument('--manifest', type=Path, default=Path('release-manifest.json'))
    args = parser.parse_args()
    if args.action == 'build':
        args.manifest.write_text(build(args.root).model_dump_json(indent=2) + '\n')
    else:
        result = verify(args.root, Manifest.model_validate_json(args.manifest.read_text()))
        print(json.dumps(result, indent=2))
        raise SystemExit(result['decision'] == 'block')

Step 3: Keep every evaluated case attached to the report

The file digests also cover the model recording and the evaluation report. Beyond those byte comparisons, verify() checks that report row IDs match the case set and that each case has recorded agent, tool and judge entries. This prevents a missing result from being treated as an unchanged evaluation merely because the report kept its old key.

A failing case disappears from the report. The recorder removes one row while retaining the key. Verification blocks with eval case ids differ, and creating a fresh manifest is also refused. Restore the complete report or rerun the evaluation. This is a membership check; the script does not independently recompute the judge verdicts or validate every trajectory.

A required file or recording entry is missing. Verification blocks rather than accepting a partial bundle. The manifest must list exactly the required files; a caller cannot omit assistant.py to avoid checking it. Repair the bundle before comparing its quality results.

A compatibility decision of match can coexist with failed quality checks. The test test_unchanged_failed_quality_scores_still_match confirms that this happens with the retained tracker report. After compatibility succeeds, inspect the retained docs-agent stage failures against your release policy. The regression-gate lab consumes issue-triage pytest reports directly. Its documentation-stage adapter maps this separate report format to named outcomes under a documentation-specific policy. Use that adapter before gating these scores; a compatibility match still cannot waive its quality failures.

Step 4: Reproduce the changed-file decisions

python3 -m venv .venv
.venv/bin/python -m pip install -r tracker-requirements.txt
.venv/bin/python -m pytest -q test_release.py
.venv/bin/python record_release_checks.py --check
.venv/bin/python bind_release.py verify

unchanged bundle: match (0 findings)
documentation edited: block (4 findings)
retrieval depth changed: block (4 findings)
generator name changed: block (4 findings)
case removed from report: block (2 findings)
application source edited: block (1 findings)

Run from the extracted lab directory. The small tracker-requirements.txt installs the checker; the larger bundle/requirements.lock is the unchanged docs-assistant dependency list being hashed, not an installation requirement for these local checks. No API key is needed.

The saved scenarios include the unchanged bundle and five isolated edits. --check recomputes them in temporary directories and compares both saved output files without rewriting them. bind_release.py verify returns status 1 for a blocked bundle and 0 for a match. Its build action creates a manifest only after the current recording key and membership checks succeed.

This is a local consistency check. It does not read a deployed image, attest the runtime environment, authenticate unsigned records, identify hosted model weights or establish answer quality. Someone who changes both the files and an unsigned manifest can make them agree. Use your existing artifact signing and deployment verification tools for those separate responsibilities.

References

Checked 2026-10-22 against the installed Pydantic and pytest APIs; exact pins are in the lab download. Python’s hashlib, ast.literal_eval, Pydantic models, and pytest temporary directories.