An eval harness is the test suite for an LLM feature: fixed inputs with expected results, checks that turn each output into a pass or a fail, and a record of what produced the run. You will build one with pytest for triage(), a function that asks Gemini to label a GitHub issue for tracker, a small task-tracker CLI, set its priority and write a one-line summary.
When you need this. You have an LLM feature in a product, say a bot that labels and prioritizes new GitHub issues, and someone wants to change its prompt or move it to a cheaper model. Without a suite, you find out what broke from your users. With one, you run pytest and see which issues the change gets wrong before it ships.
Deterministic checks cover the label, the priority and the summary. A DeepEval G-Eval metric checks that each summary states only facts from the issue. The first run calls the model and records every reply; later runs replay the recording, so the same suite runs in CI with no model call and no API key. The saved live run contains 51 tests on gemini-3.8-flash. Step 5 shows its output beside the recording command.
A clean pass describes this recorded sample; it does not predict the next live run. The same suite run on the local alternative, Gemma on Ollama, fails 2 tests, and Step 5 compares the recorded replies.
The course uses DeepEval inside pytest. It supports Gemini or a local Ollama model as the judge, and provides the retrieval and tool-call metrics used in the next lesson. The lab requirements pin the package versions.
Step 1: Write the function under test
Download the complete lab, extract it and open a terminal in build-a-versioned-evaluation-harness-with-deterministic-checks/. The README lists its runtime and recorded examples.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-lock.txtWe start with the code we want to test, kept small enough to read in one screen.
"""Issue triage: Gemini labels a GitHub issue and writes a one-line summary.
Set TRIAGE_MODEL=gemma4:e2b to run the same function on Gemma through a local Ollama server."""
import os
from functools import cache
from typing import Literal
from pydantic import BaseModel, Field
MODEL = os.environ.get("TRIAGE_MODEL", "gemini-3.8-flash")
PROMPT_VERSION = "v1"
PROMPT = """You triage GitHub issues for `tracker`, an open-source task-tracker CLI.
Choose one label:
- bug: the program behaves wrongly, crashes, loses data or leaks secrets.
- feature: a request for new behavior.
- docs: the documentation is wrong or missing.
- question: the reporter asks how something works.
Set priority to p0 for data loss or a leaked secret, p1 for other bugs and p2 otherwise.
Write a summary of at most 20 words that keeps the command or file the issue names."""
class Triage(BaseModel):
label: Literal["bug", "feature", "docs", "question"]
priority: Literal["p0", "p1", "p2"]
summary: str = Field(min_length=1)
@cache
def gemini():
from google import genai
return genai.Client() # reads GOOGLE_API_KEY from the environment
def triage(issue: str, model: str = MODEL) -> str:
"""Return the model's raw JSON reply; the caller validates it against Triage."""
if model.startswith("gemini"):
from google.genai import types
config = types.GenerateContentConfig(
system_instruction=PROMPT,
response_mime_type="application/json",
response_schema=Triage,
thinking_config=types.ThinkingConfig(thinking_level="low"),
)
return gemini().models.generate_content(model=model, contents=issue, config=config).text
from ollama import chat
response = chat(
model=model,
messages=[{"role": "system", "content": PROMPT}, {"role": "user", "content": issue}],
format=Triage.model_json_schema(),
options={"temperature": 0, "seed": 7},
think=False,
)
return response.message.content
if __name__ == "__main__":
import sys
print(triage(sys.argv[1]))The Gemini branch uses the google-genai client. response_mime_type="application/json" with response_schema=Triage restricts the reply to the Pydantic model’s fields, so the label can only be one of four values. triage() returns the raw reply text rather than a Triage object: a malformed reply becomes a failed test instead of an exception inside the function under test.
Two settings are deliberate. The Gemini call leaves temperature at its default of 1.0, as the Gemini 3 guide recommends; Google warns that lower values may cause looping or degraded output. thinking_level="low" requests less reasoning effort, without setting a fixed latency or token budget. The Ollama branch can set temperature: 0 and a seed to reduce generation variation, but those settings do not establish repeatability across runtime or model changes. Replaying saved replies fixes the inputs to the checks. Step 5 replays the recording, and Step 6 measures variation across fresh calls.
Step 2: Version the dataset
Each case holds an issue, the expected label, a priority only where the priority is the point of the case, and the words the summary must keep. The dataset has 12 cases.
Inspect dataset.json
{
"name": "issue-triage",
"version": 1,
"cases": [
{
"id": "sync-crash-no-due-date",
"issue": "`tracker sync` crashes with KeyError: 'due' when a task has no due date. Traceback ends in sync/merge.py line 88. Started after upgrading to 2.4.0.",
"expected": {"label": "bug"},
"must_mention": ["sync"]
},
{
"id": "token-in-debug-log",
"issue": "Running `tracker push --debug` prints my full API token in plain text to the terminal and to ~/.tracker/debug.log. Anyone who can read the log can use my account.",
"expected": {"label": "bug", "priority": "p0"},
"must_mention": ["token"]
},
{
"id": "import-drops-tasks",
"issue": "Importing a CSV with 500 rows with `tracker import tasks.csv` reports success, but only 312 tasks exist afterwards. The rest are silently dropped and the original file is already moved to the archive folder.",
"expected": {"label": "bug", "priority": "p0"},
"must_mention": ["import"]
},
{
"id": "color-in-pipes",
"issue": "When I pipe `tracker list` into grep, the output still contains ANSI color codes, so my grep patterns fail. Colors should turn off when stdout is not a terminal.",
"expected": {"label": "bug"},
"must_mention": ["color"]
},
{
"id": "recurring-tasks",
"issue": "Please add recurring tasks, for example `tracker add \"water plants\" --every 3d`. Right now I re-create the same task by hand every few days.",
"expected": {"label": "feature"},
"must_mention": ["recurring"]
},
{
"id": "json-output",
"issue": "It would help scripting if `tracker list` had a `--json` flag that prints tasks as JSON instead of the table.",
"expected": {"label": "feature"},
"must_mention": ["json"]
},
{
"id": "calendar-export",
"issue": "Feature request: export tasks with due dates to an .ics file so they show up in my calendar app.",
"expected": {"label": "feature"},
"must_mention": ["export"]
},
{
"id": "readme-wrong-flag",
"issue": "The README says to run `tracker init --path ~/tasks`, but the flag is actually `--dir`. The example fails with 'unknown option --path'.",
"expected": {"label": "docs"},
"must_mention": ["readme"]
},
{
"id": "missing-config-docs",
"issue": "The docs do not list the keys allowed in config.toml. I had to read the source to find `default_project` and `week_start`.",
"expected": {"label": "docs"},
"must_mention": ["config"]
},
{
"id": "two-machines",
"issue": "Is it safe to run `tracker sync` from two laptops against the same remote at the same time, or should I wait for one to finish?",
"expected": {"label": "question"},
"must_mention": ["sync"]
},
{
"id": "where-is-data",
"issue": "Where does tracker store its database on disk? I want to include it in my backups.",
"expected": {"label": "question"},
"must_mention": ["backup"]
},
{
"id": "archive-vs-delete",
"issue": "What is the difference between `tracker archive` and `tracker delete`? Can archived tasks be restored?",
"expected": {"label": "question"},
"must_mention": ["archive"]
}
]
}harness.py loads the file into Pydantic models. extra="forbid" turns a misspelled field into an error, Literal limits labels and priorities to the allowed values, and a validator refuses duplicate IDs. The version field is a name for people; sha is a hash of the content, so any edit to any case gives a new value even when nobody changes the version.
"""A versioned dataset and a record/replay cassette for the triage eval."""
import hashlib
import json
from pathlib import Path
from typing import Literal
from pydantic import BaseModel, ConfigDict, Field, model_validator
HERE = Path(__file__).parent
class Expected(BaseModel):
model_config = ConfigDict(extra="forbid")
label: Literal["bug", "feature", "docs", "question"]
priority: Literal["p0", "p1", "p2"] | None = None
class Case(BaseModel):
model_config = ConfigDict(extra="forbid")
id: str = Field(pattern=r"^[a-z0-9-]+$")
issue: str = Field(min_length=1)
expected: Expected
must_mention: list[str] = []
class Dataset(BaseModel):
model_config = ConfigDict(extra="forbid")
name: str
version: int
cases: list[Case] = Field(min_length=1)
@model_validator(mode="after")
def unique_ids(self):
ids = [case.id for case in self.cases]
duplicates = sorted({i for i in ids if ids.count(i) > 1})
if duplicates:
raise ValueError(f"duplicate case id: {', '.join(duplicates)}")
return self
@property
def sha(self) -> str:
"""Content hash: any edit to any case gives a new value."""
return hashlib.sha256(self.model_dump_json().encode()).hexdigest()[:12]
def load_dataset(path: Path = HERE / "dataset.json") -> Dataset:
return Dataset.model_validate_json(Path(path).read_text())
class RecordingError(RuntimeError):
pass
class Cassette:
"""Live mode calls the model and the judge and records both replies.
Replay mode reads them back, but only for the same dataset, model and prompt."""
def __init__(self, path: Path, key: dict, live: bool):
self.path, self.key, self.live = Path(path), key, live
self.entries: dict = {}
if live:
return
if not self.path.exists():
raise RecordingError(f"no recording {self.path.name}; run pytest --live")
recorded = json.loads(self.path.read_text())
changed = [k for k in key if recorded["key"].get(k) != key[k]]
if changed:
raise RecordingError(f"recording is stale: {', '.join(changed)} changed; run pytest --live")
self.entries = recorded["entries"]
def get(self, name: str, produce):
if self.live:
self.entries[name] = produce()
elif name not in self.entries:
raise RecordingError(f"recording has no entry {name!r}; run pytest --live")
return self.entries[name]
def save(self):
body = {"key": self.key, "entries": self.entries}
self.path.write_text(json.dumps(body, indent=2, sort_keys=True) + "\n")Duplicate case ID. Loading fails and names the ID; give each case a unique ID before recording.
Unknown label in a case. Loading fails on label; use one of the dataset model’s declared labels.
Misspelled field in a case. Loading fails and names the field; correct it rather than dropping extra="forbid".
Case edited without a version change. The content hash still changes, invalidating the recording; review the edit and record again.
Step 3: Write deterministic checks
Now we turn each case into tests.
"""The triage eval: deterministic checks on every case, then one LLM-as-judge check."""
import pytest
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams
from conftest import DATASET
from triage import Triage
every_case = pytest.mark.parametrize("case", DATASET.cases, ids=lambda c: c.id)
critical = [c for c in DATASET.cases if c.expected.priority]
@every_case
def test_reply_matches_schema(raw_output):
Triage.model_validate_json(raw_output)
@every_case
def test_label(case, output):
assert output.label == case.expected.label
@pytest.mark.parametrize("case", critical, ids=lambda c: c.id)
def test_priority(case, output):
assert output.priority == case.expected.priority
@every_case
def test_summary_is_short_and_specific(case, output):
words = output.summary.split()
assert len(words) <= 20, f"{len(words)} words"
missing = [term for term in case.must_mention if term not in output.summary.lower()]
assert not missing, f"summary lacks {missing}: {output.summary!r}"
FAITHFUL = "The summary states only facts that are in the issue and keeps its main problem."
THRESHOLD = 0.7
def faithfulness(judge, issue: str, summary: str) -> dict:
metric = GEval(
name="Faithful summary",
criteria=FAITHFUL,
evaluation_params=[SingleTurnParams.INPUT, SingleTurnParams.ACTUAL_OUTPUT],
model=judge,
threshold=THRESHOLD,
async_mode=False,
)
metric.measure(LLMTestCase(input=issue, actual_output=summary))
return {"score": metric.score, "reason": metric.reason}
@every_case
def test_summary_is_faithful(case, output, cassette, judge):
verdict = cassette.get(f"judge/{case.id}", lambda: faithfulness(judge(), case.issue, output.summary))
assert verdict["score"] >= THRESHOLD, verdict["reason"]
def test_judge_rejects_an_invented_summary(cassette, judge):
"""Calibration: a judge that passes everything measures nothing."""
issue = DATASET.cases[0].issue
invented = "tracker sync is slow on Windows when the network drops."
verdict = cassette.get("judge/calibration", lambda: faithfulness(judge(), issue, invented))
assert verdict["score"] < THRESHOLD, verdict["reason"]pytest.mark.parametrize turns each case into its own test, named by its ID, so a report reads test_label[token-in-debug-log] instead of one failed loop. The first four tests are plain assertions: the reply parses against the schema, the label matches, the priority matches on the cases that set one, and the summary is at most 20 words and keeps the required terms. They make no model call and return the same verdict for the same recorded reply and dataset. Put every check you can express this way in code before you reach for a judge.
Step 4: Add an LLM-as-judge check
String matching can’t tell whether a summary invents facts. G-Eval asks a judge model to score an output against written criteria, from 0 to 1. faithfulness() builds the metric with the issue as input, the summary as the output, and a pass threshold of 0.7. The judge is gemini-3.8-flash through DeepEval’s GeminiModel. For a Gemini 3 model DeepEval leaves the temperature at Google’s default unless you set one, for the reason given in Step 1.
A judge needs its own test. test_judge_rejects_an_invented_summary gives it a summary that invents a Windows slowdown for the sync crash. The judge scored it 0, below the threshold, so this judge can fail a summary. Across the 12 real summaries its lowest score was 0.8. In this lab the judge is the same model as the system. A judge that shares the system’s blind spots can pass its mistakes, so keep the calibration case, and add one each time a person finds an error that the judge passed.
Step 5: Record once, replay in CI
Live mode calls both the system model and the judge. To make a new recording with your configured credentials:
pytest -q --live test_triage.py................................................... [100%]Running teardown with pytest sessionfinish...
51 passed in 235.39s (0:03:55)"""Fixtures for the triage eval: the dataset, the cassette and the judge model."""
import hashlib
import os
from functools import cache
from pathlib import Path
import pytest
os.environ.setdefault("DEEPEVAL_TELEMETRY_OPT_OUT", "1")
from deepeval.models import GeminiModel, OllamaModel # noqa: E402
from harness import Cassette, load_dataset # noqa: E402
from triage import MODEL, PROMPT, PROMPT_VERSION, Triage, triage # noqa: E402
DATASET = load_dataset()
JUDGE_MODEL = MODEL
KEY = {
"dataset": f"{DATASET.name}@{DATASET.version}",
"dataset_sha": DATASET.sha,
"model": MODEL,
"prompt_version": PROMPT_VERSION,
"prompt_sha": hashlib.sha256(PROMPT.encode()).hexdigest()[:12],
"judge_model": JUDGE_MODEL,
}
def pytest_addoption(parser):
parser.addoption("--live", action="store_true", help="call the model and rewrite the recording")
@pytest.fixture(scope="session")
def cassette(request):
live = request.config.getoption("--live")
name = f"{MODEL.replace(':', '-')}-{PROMPT_VERSION}.json"
tape = Cassette(Path(__file__).with_name("recordings") / name, KEY, live)
yield tape
if live:
tape.save()
@pytest.fixture
def raw_output(case, cassette):
return cassette.get(f"triage/{case.id}", lambda: triage(case.issue))
@pytest.fixture
def output(raw_output):
return Triage.model_validate_json(raw_output)
def make_judge(name: str):
"""DeepEval's judge wrapper for the model name: Gemini through the API, anything else through Ollama."""
if name.startswith("gemini"):
return GeminiModel(model=name)
return OllamaModel(model=name, base_url="http://localhost:11434", temperature=0)
@pytest.fixture(scope="session")
def judge():
"""Builds the judge on first use, so a replay needs no API key."""
return cache(lambda: make_judge(JUDGE_MODEL))
@pytest.hookimpl(optionalhook=True)
def pytest_json_modifyreport(json_report):
# Keep the run record about the eval, not about the machine that ran it.
for field in ("environment", "root"):
json_report.pop(field, None)
json_report["eval"] = KEYThe pytest_addoption hook adds --live. With it, the cassette fixture calls the model and the judge and saves every reply under recordings/. Without it, the fixture replays the saved replies, but only when the recording’s key matches the current run: the dataset name and hash, the model, the prompt version, a hash of the prompt text and the judge model. A mismatch stops the run and asks for a new recording, so this harness refuses to grade a mismatched recording.
The judge fixture returns a cached factory, not a model. GeminiModel checks for an API key when it is built, so a fixture that built it eagerly would fail every replay in a CI job that has no key. In replay mode, Cassette.get() reads the saved judge verdict without invoking the factory.
pytest-json-report writes every test’s outcome to runs/gemini-3.8-flash-v1.json, and the pytest_json_modifyreport hook adds the same key under eval. The replay gives the same outcomes in a fraction of a second:
pytest -q test_triage.py................................................... [100%]Running teardown with pytest sessionfinish...
51 passed in 0.03sInspect the run summary and key from the JSON report
{
"summary": {
"passed": 51,
"total": 51,
"collected": 51
},
"eval": {
"dataset": "issue-triage@1",
"dataset_sha": "e12a5e739a1e",
"model": "gemini-3.8-flash",
"prompt_version": "v1",
"prompt_sha": "01859a9d3df1",
"judge_model": "gemini-3.8-flash"
}
}Recording made with an older dataset. Replay stops with dataset_sha changed; run pytest --live; record the reviewed dataset instead of editing the stored key.
Recording has no entry for a test. Replay raises RecordingError; record the missing case through the live path after review.
Replay with no recording. Replay stops and asks for --live; supply the intended recording or explicitly run the live suite with credentials.
Replay differs from the live run. The test fails if any test outcome differs from runs/gemini-3.8-flash-v1.json.
What the local model got wrong. The same suite on Gemma, TRIAGE_MODEL=gemma4:e2b, failed 2 of 51 tests. recordings/gemma4-e2b-v1.json holds that run, so you can replay it without Ollama:
TRIAGE_MODEL=gemma4:e2b pytest -q test_triage.py| Test | Expected | gemma4:e2b reply | gemini-3.8-flash reply |
|---|---|---|---|
test_priority[import-drops-tasks] | p0: the import loses data | priority p1 | priority p0 |
test_summary_is_short_and_specific[where-is-data] | the summary keeps “backup” | “Where does tracker store its database on disk?” | “User asking where tracker stores its database on disk for backups.” |
The prompt says p0 for data loss, and Gemma did not treat silently dropped rows as data loss. Its second summary keeps the question but drops the reporter’s goal. The judge passed both summaries, because neither invents a fact: the faithfulness check and the required-terms check measure different things. A smaller model is a valid choice for cost or privacy, and this suite is how you find out what it costs in quality. The regression-gate lesson turns that comparison into a CI decision.
A replay that needs an API key. An earlier version of conftest.py built the judge in the fixture. Every replay without GOOGLE_API_KEY then stopped at setup, before any test ran:
deepeval.errors.DeepEvalError: Google Gemini API key is not configured. Set GOOGLE_API_KEY in your environment or pass `api_key` to GeminiModel(...).The fix is the factory in Step 5. A replay grades recorded replies, so it should need no secret; keeping the key out of pull-request jobs also keeps it away from code that a contributor controls.
Offline harness checks. test_harness.py has 9 offline tests that need no model:
pytest -q test_harness.pyInspect test_harness.py
"""Failure cases for the harness itself. No model call; runs offline."""
import json
import pytest
from pydantic import ValidationError
from harness import Cassette, Dataset, RecordingError, load_dataset
CASE = {"id": "a", "issue": "text", "expected": {"label": "bug"}}
def dataset(*cases):
return Dataset.model_validate({"name": "t", "version": 1, "cases": list(cases)})
def test_duplicate_case_id_is_rejected():
with pytest.raises(ValidationError, match="duplicate case id: a"):
dataset(CASE, CASE)
def test_unknown_label_is_rejected():
with pytest.raises(ValidationError, match="label"):
dataset({**CASE, "expected": {"label": "bugg"}})
def test_unknown_field_is_rejected():
with pytest.raises(ValidationError, match="must_mentoin"):
dataset({**CASE, "must_mentoin": ["sync"]})
def test_edited_case_changes_the_dataset_sha():
original = dataset(CASE)
edited = dataset({**CASE, "issue": "text, edited"})
assert original.version == edited.version
assert original.sha != edited.sha
def test_stale_recording_is_refused(tmp_path):
path = tmp_path / "tape.json"
tape = Cassette(path, {"dataset_sha": dataset(CASE).sha, "model": "m"}, live=True)
tape.get("triage/a", lambda: "{}")
tape.save()
edited = dataset({**CASE, "issue": "text, edited"})
with pytest.raises(RecordingError, match="dataset_sha changed"):
Cassette(path, {"dataset_sha": edited.sha, "model": "m"}, live=False)
def test_missing_recording_entry_is_an_error(tmp_path):
path = tmp_path / "tape.json"
path.write_text(json.dumps({"key": {"model": "m"}, "entries": {}}))
tape = Cassette(path, {"model": "m"}, live=False)
with pytest.raises(RecordingError, match="no entry 'triage/a'"):
tape.get("triage/a", lambda: "{}")
def test_replay_without_a_recording_asks_for_live(tmp_path):
with pytest.raises(RecordingError, match="run pytest --live"):
Cassette(tmp_path / "absent.json", {"model": "m"}, live=False)
def test_shipped_dataset_is_valid():
assert len(load_dataset().cases) > 0
def test_replay_reproduces_the_recorded_run(tmp_path):
"""Replaying the cassette gives the same outcome for every test as the live run."""
import subprocess
import sys
from pathlib import Path
here = Path(__file__).parent
out = tmp_path / "replay.json"
subprocess.run([sys.executable, "-m", "pytest", "-q", "test_triage.py", "--json-report",
f"--json-report-file={out}"], cwd=here, capture_output=True)
outcomes = lambda path: {t["nodeid"]: t["outcome"] for t in json.loads(path.read_text())["tests"]} # noqa: E731
assert outcomes(out) == outcomes(here / "runs" / "gemini-3.8-flash-v1.json")Record again whenever the model, the prompt or the SDK version changes; a replay can’t show a change in the model’s behavior, only a live run can.
Step 6: Measure variation with repeated trials
A replay checks grading against fixed replies; it does not sample the model again. Because Gemini runs at temperature 1.0, one live run is one sample. repeat.py calls triage() 5 times per case and reports two rates for the label check: pass@5, the share of cases labeled correctly at least once, and pass^5, the share labeled correctly every time. pass@k suits a task where a person picks the best of several tries; pass^k is the one that matters for an automated triage bot, which gets one try per issue.
Inspect repeat.py
"""Repeated trials: call triage() K times per case and report pass@k and pass^k for the label check."""
import json
from collections import Counter
from pathlib import Path
from harness import load_dataset
from triage import MODEL, Triage, triage
HERE = Path(__file__).parent
K = 5
def trials(cases) -> dict:
rows = {}
for case in cases:
replies = [Triage.model_validate_json(triage(case.issue)) for _ in range(K)]
rows[case.id] = {"expected": case.expected.label, "labels": [r.label for r in replies],
"priorities": [r.priority for r in replies]}
return rows
def pass_rates(rows: dict) -> dict:
right = {cid: [label == row["expected"] for label in row["labels"]] for cid, row in rows.items()}
return {"pass@1": round(sum(r[0] for r in right.values()) / len(right), 3),
f"pass@{K}": round(sum(any(r) for r in right.values()) / len(right), 3), # right at least once
f"pass^{K}": round(sum(all(r) for r in right.values()) / len(right), 3)} # right every time
if __name__ == "__main__":
rows = trials(load_dataset().cases)
result = {"model": MODEL, "k": K, "cases": rows, "rates": pass_rates(rows)}
(HERE / "runs" / "repeats.json").write_text(json.dumps(result, indent=2) + "\n")
for cid, row in rows.items():
changed = len(set(row["labels"])) > 1 or len(set(row["priorities"])) > 1
print(f"{cid:<24}{dict(Counter(row['labels']))!s:<22}{dict(Counter(row['priorities']))!s:<22}"
f"{'CHANGED' if changed else ''}")
print(result["rates"])python repeat.pysync-crash-no-due-date {'bug': 5} {'p1': 5}
token-in-debug-log {'bug': 5} {'p0': 5}
import-drops-tasks {'bug': 5} {'p0': 5}
color-in-pipes {'bug': 5} {'p1': 5}
recurring-tasks {'feature': 5} {'p2': 5}
json-output {'feature': 5} {'p2': 5}
calendar-export {'feature': 5} {'p2': 5}
readme-wrong-flag {'docs': 5} {'p2': 5}
missing-config-docs {'docs': 5} {'p2': 5}
two-machines {'question': 5} {'p2': 5}
where-is-data {'question': 5} {'p2': 5}
archive-vs-delete {'question': 5} {'p2': 5}
{'pass@1': 1.0, 'pass@5': 1.0, 'pass^5': 1.0}Labels and priorities did not change across the 5 trials on any of the 12 cases. That result covers only those two fields on these 12 cases and 5 trials: repeat.py doesn’t repeat the judged check, so it says nothing about how stable the judge’s scores are, and nothing about issues outside the dataset. Rerun repeat.py after any model or prompt change: a pass^k below pass@k is the first sign that a single green run is luck.
You edit one sentence of PROMPT in triage.py and leave PROMPT_VERSION at v1. Decide what pytest -q test_triage.py does before you answer.
Read the explanation
A recording is valid only for the dataset, model, prompt and judge that produced it. The key stores a hash of each, so an edit to any of them stops the replay instead of grading old replies.
Measure RAG and tool-call quality evaluates an agent that retrieves documents and calls tools, and reports which stage failed.
References
- pytest: parametrizing tests, fixtures and command-line options in conftest.py. Checked October 21, 2026.
- DeepEval: G-Eval and Gemini models. Checked October 21, 2026.
- Gemini API: structured output, Gemini 3 guide and models. Checked October 21, 2026.
- Ollama: structured outputs. Checked October 21, 2026.
- Pydantic: models. Checked October 21, 2026.
- pytest-json-report. Checked October 21, 2026.