Build a paired comparison report with uncertainty and slice-level results

Problem

When you need this. Use this when comparing two prompts or models on the same dataset and you need paired per-case diffs, bootstrap confidence intervals and slice breakdowns before shipping.

A prompt edit looks like an improvement: version 2 of the triage prompt passes 195 trials where version 1 passed 187. Before shipping it, three questions need answers that a pair of totals cannot give:

• Is the gain larger than the noise? The model samples at temperature 1, so runs of the same prompt with other seeds also differ.
• Where did it come from? A gain can come from a few cases while others get worse.
• Who does it help? A total can rise while one subgroup, such as Spanish messages, gets no better or gets worse.

We will build compare.py. The contract:

• Input: two run records from the harness, made with the same case set, task module, graders, splits and trial count, and the case file with its tags.
• Output: runs/comparison.json, with the mean per-case difference, 95% bootstrap intervals over three units, the cases that changed, the trial pairs by outcome, and the same difference for every slice with its case count.
• Refusal: no comparison when the harness’s compare() refuses the runs, or when the case file is another version.

The two systems are qwen3.5:0.8b with prompt v1, from the repeated-trial benchmark, and the same model with prompt v2, which adds two sentences. They run on the 44 development cases in the bundled cases-robustness.json, 5 trials each. The robustness lesson teaches related perturbation checks on a different tracker fixture.

Concepts

Paired comparison

Both systems answer the same cases, and trial t of a case uses the same seed in both runs. Each case gives one difference: its share of passing trials under v2 minus its share under v1. Matching cases controls case composition; shared seeds control the seed schedule, while sampled replies can still differ.

Bootstrap interval

Draw as many units as there are, with replacement, from the per-case differences, and take the mean; repeat 2000 times. The middle 95% of those means is the interval. It shows how much the mean would move if the units had been a different draw from the same kind of cases.

Unit of resampling

The thing that varies independently. Trials of one case share the case’s difficulty, so the unit is at least the case. Variants of one message share its meaning, so here the honest unit is the source group: a base case with its variants and translation.

Slice

The cases that share one tag value, such as every Spanish case or every typo variant. A slice has fewer cases than the whole set, which can make its estimate less stable. Interval width also depends on score variation and grouping; a constant slice can have a zero-width interval.

The two added sentences in prompt v2 are: “Messages can be in Spanish or contain typos: classify by meaning, not by exact words.” and “A password or account changed without the customer's permission always needs a human.”

An interval describes these recorded cases and seeds. It is not a promise about other messages. It answers a narrower question: could a difference this size have come from which cases and draws happened to be in the set?

Architecture

The qwen3.5:0.8b model answers both runs through bench.py + harness.py, once with prompt v1 and once with prompt v2, and the Run records hold both. compare.py · compare() first calls harness.compare(), which sends runs measured differently to Refused: RunError. It then pairs trials by case and trial number. The recorded panel shows every pair: each row is one case, its v1 trials beside its v2 trials, with the same seed in the same column, and the v2 − v1 column holds the case’s difference. Brackets group the variants of one source message. The differences feed both resample cases and resample sources, and the two intervals come out on one scale. The slice interval chart under Read the comparison gives the same intervals for every slice.

Working code

Runtime: Python 3.11 or later. Dependencies: Python standard library only. The tests replay the logs; they call no model and no network. harness.py, triage_task.py, bench.py and cases-robustness.json form the self-contained generic inbox-triage harness bundled with this report. Their schema and APIs differ from the current pytest-based evaluation course fixtures. The README lists every file and command.

"""Paired comparison of two run records: the difference, its uncertainty and its slices.

Both runs must be comparable under harness.compare(): same case set, task,
graders, splits and trials. Every case then has a share of passing trials in
each run, and the comparison is about the per-case differences:
- the mean difference, with a 95% bootstrap interval that resamples cases, because
  the trials of one case are not independent of each other;
- the interval again with whole source groups resampled, because a generated
  variant repeats its source case's meaning and is not an independent case.
  With few groups it is computed exactly, over every possible draw, together
  with the share of draws that show no gain, so it does not depend on a seed;
- the interval computed as if every trial were independent, for contrast;
- the cases that got better, worse or stayed the same, and the trial pairs
  (same case, same trial number, so the same seed) by outcome;
- the same difference and interval for every value of every case tag (a slice),
  with the number of cases behind it.
Python standard library only.
Run: python3 compare.py runs/qwen-v1.json runs/qwen-v2.json --cases cases-robustness.json --out runs/comparison.json
"""
import argparse
import math
import random
from fractions import Fraction

import harness

COMPARISON_VERSION = "1"
RESAMPLES = 2000
SEED = 20261020  # Bootstrap seed: the interval is reproducible byte for byte.
FEW_CASES = 10   # Mark sparse slices; interval width also depends on score variation.


def mean(values):
    return math.fsum(values) / len(values)


def interval(values, rng, resamples=RESAMPLES):
    """95% percentile bootstrap interval for the mean of values, resampling them with replacement."""
    means = sorted(mean([values[rng.randrange(len(values))] for _ in values]) for _ in range(resamples))
    return [round(means[math.floor(0.025 * resamples)], 4), round(means[math.ceil(0.975 * resamples) - 1], 4)]


def grouped_interval(groups, rng, resamples=RESAMPLES):
    """95% bootstrap interval for the mean over cases, resampling whole groups of cases."""
    groups = list(groups.values())
    means = []
    for _ in range(resamples):
        drawn = [value for _ in groups for value in groups[rng.randrange(len(groups))]]
        means.append(mean(drawn))
    means.sort()
    return [round(means[math.floor(0.025 * resamples)], 4), round(means[math.ceil(0.975 * resamples) - 1], 4)]


EXACT_LIMIT = 200_000  # Enumerate every draw of groups when there are at most this many.


def compositions(total, parts):
    """Every way to write total as an ordered sum of parts nonnegative counts."""
    if parts == 1:
        yield (total,)
        return
    for first in range(total + 1):
        for rest in compositions(total - first, parts - 1):
            yield (first, *rest)


def exact_grouped(groups):
    """The bootstrap over whole groups, computed exactly instead of sampled.

    Drawing G groups with replacement from G gives every count vector c with
    probability G! / (c1! ... cG!) / G^G. Each draw's mean is the mean over all
    cases in the drawn groups. Returns the 2.5% and 97.5% points of that
    distribution, defined as in interval(), and the probability that the mean
    is at or below zero: the share of draws that show no gain.
    """
    sums = [math.fsum(values) for values in groups.values()]
    sizes = [len(values) for values in groups.values()]
    count = len(sums)
    if math.comb(2 * count - 1, count - 1) > EXACT_LIMIT:
        return None
    distribution = {}
    for counts in compositions(count, count):
        weight = Fraction(math.factorial(count), count ** count)
        for c in counts:
            weight /= math.factorial(c)
        value = round(math.fsum(c * s for c, s in zip(counts, sums)) / sum(c * n for c, n in zip(counts, sizes)), 10)
        distribution[value] = distribution.get(value, 0) + weight
    cumulative, low, high = Fraction(0), None, None
    for value, weight in sorted(distribution.items()):
        cumulative += weight
        if low is None and cumulative >= Fraction(25, 1000):
            low = value
        if high is None and cumulative >= Fraction(975, 1000):
            high = value
    no_gain = sum(w for v, w in distribution.items() if v <= 0)
    return {"interval_95": [round(low, 4), round(high, 4)], "share_at_or_below_zero": round(float(no_gain), 4),
            "draws": math.comb(2 * count - 1, count - 1)}


def shares(record):
    """Per case: the share of its trials that passed."""
    trials = record["manifest"]["trials"]
    return {case: counts["pass"] / trials
            for split in record["summary"].values() for case, counts in split["per_case"].items()}


def summarize(case_ids, before, after, rng):
    differences = [after[case] - before[case] for case in case_ids]
    return {"cases": len(case_ids),
            "baseline": round(mean([before[c] for c in case_ids]), 4),
            "candidate": round(mean([after[c] for c in case_ids]), 4),
            "difference": round(mean(differences), 4),
            "interval_95": interval(differences, rng),
            "better": sum(d > 0 for d in differences), "worse": sum(d < 0 for d in differences),
            "same": sum(d == 0 for d in differences)}


def trial_pairs(baseline, candidate):
    """Outcome pairs for the same case and trial number: both pass, only one, or neither."""
    after = {(row["case_id"], row["trial"]): row["verdict"] == "pass" for row in candidate["results"]}
    pairs = {"both_pass": 0, "only_baseline": 0, "only_candidate": 0, "neither": 0}
    for row in baseline["results"]:
        b, c = row["verdict"] == "pass", after[(row["case_id"], row["trial"])]
        pairs["both_pass" if b and c else "only_baseline" if b else "only_candidate" if c else "neither"] += 1
    return pairs


def compare(baseline, candidate, case_set):
    comparison = harness.compare(baseline, candidate)  # Raises RunError unless the runs are comparable.
    if comparison["case_set"]["sha256"] != case_set["sha256"]:
        raise harness.RunError("the run records were made with another version of the case file")
    before, after = shares(baseline), shares(candidate)
    case_ids = list(before)
    rng = random.Random(SEED)
    overall = summarize(case_ids, before, after, rng)
    # The same difference, with every trial treated as an independent observation.
    passed = {(row["case_id"], row["trial"]): row["verdict"] == "pass" for row in candidate["results"]}
    trial_differences = [passed[(row["case_id"], row["trial"])] - (row["verdict"] == "pass")
                         for row in baseline["results"]]
    overall["trial_level_interval_95"] = interval(trial_differences, rng)
    tags = {case["id"]: case.get("tags", {}) for case in case_set["cases"]}
    by_source = {}
    for case in case_ids:  # A case without a source tag is its own group.
        by_source.setdefault(tags[case].get("source", case), []).append(after[case] - before[case])
    overall["source_groups"] = len(by_source)
    exact = exact_grouped(by_source)
    if exact is not None:
        overall["source_cluster_method"] = f"exact over {exact['draws']} draws"
        overall["source_cluster_interval_95"] = exact["interval_95"]
        overall["source_no_gain_share"] = exact["share_at_or_below_zero"]
    else:
        overall["source_cluster_method"] = f"bootstrap, {RESAMPLES} resamples"
        overall["source_cluster_interval_95"] = grouped_interval(by_source, rng)
        overall["source_no_gain_share"] = None
    slices = {}
    for name in sorted({tag for case in case_ids for tag in tags[case]} - {"source"}):
        for value in sorted({tags[case].get(name) for case in case_ids if name in tags[case]}):
            members = [case for case in case_ids if tags[case].get(name) == value]
            entry = summarize(members, before, after, rng)
            entry["few_cases"] = len(members) < FEW_CASES
            slices.setdefault(name, {})[value] = entry
    return {"comparison_version": COMPARISON_VERSION, "resamples": RESAMPLES, "seed": SEED,
            "case_set": comparison["case_set"], "trials": baseline["manifest"]["trials"],
            "baseline": baseline["manifest"]["system"], "candidate": candidate["manifest"]["system"],
            "overall": overall, "trial_pairs": trial_pairs(baseline, candidate), "slices": slices,
            "cases": [{"case_id": case, "baseline": round(before[case], 4), "candidate": round(after[case], 4)}
                      for case in case_ids if before[case] != after[case]]}


def main(argv=None):
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("baseline")
    parser.add_argument("candidate")
    parser.add_argument("--task", default="triage_task")
    parser.add_argument("--cases", required=True)
    parser.add_argument("--out", required=True)
    args = parser.parse_args(argv)
    case_set = harness.load_cases(args.cases, harness.load_task(args.task))
    result = compare(harness.read_json(args.baseline), harness.read_json(args.candidate), case_set)
    harness.write(args.out, result)
    o = result["overall"]
    print(f"difference {o['difference']:+} (95% interval {o['interval_95'][0]:+} to {o['interval_95'][1]:+}) over "
          f"{o['cases']} cases; better {o['better']}, worse {o['worse']}, same {o['same']}")


if __name__ == "__main__":
    try:
        main()
    except (harness.CaseError, harness.RunError) as error:
        raise SystemExit(f"error: {error}")

Read the important decisions in order:

• shares() turns each run into one number per case, the share of its trials that passed. The comparison works on these shares, not on raw trial counts.
• interval() is a percentile bootstrap with a fixed seed, so the same records always give the same interval.
• exact_grouped() computes the source-group interval exactly: with 9 groups there are only 24310 distinct draws, so it adds up every one with its probability instead of sampling. Drawing a source draws all its variants together. grouped_interval() samples instead when there are too many groups to enumerate.
• The trial-level interval treats every trial as independent. It is reported for contrast; use the source-group interval for decisions about this grouped case set.
• Slices skip the source tag, which names a group, not a property, and mark slices with fewer than 10 cases.

"""Inbox triage by a local model, sampled, with two versions of the prompt to compare.

Prompt v1 is the prompt from the repeated-trial benchmark build, unchanged.
Prompt v2 adds two sentences: classify by meaning in any language or spelling,
and flag a password or account changed without the customer's permission.

Each trial sends one message to a local Ollama model at temperature 1 with a
seed that depends only on the trial number, and asks for JSON that matches the
triage task's output fields. Input is checked before any model call, so blank
or missing text raises the same rejections as the keyword rules in the harness
build. A reply cut off at the token limit raises TruncatedReply, and a reply
that is not JSON raises json.JSONDecodeError: both become incomplete trials.
Python standard library only.
"""
import hashlib
import json
import urllib.request

from harness import Answer
from triage_task import CATEGORIES

OLLAMA = "http://127.0.0.1:11434"
# Tag -> first 12 hex digits of the digest the recordings were made with.
MODELS = {"gemma4:e2b": "b37049369adf", "qwen3.5:0.8b": "de63045f2975"}
TEMPERATURE = 1.0
NUM_PREDICT = 64
SEED_BASE = 1000  # Trial t uses seed 1000 + t, the same for every case and model.
PROMPT_V1 = (
    "Classify one customer message for an online shop's support inbox. Reply with JSON only.\n"
    "category: billing (charges, refunds, payments), delivery (orders on their way, tracking, arrival), "
    "account (signing in, passwords, account access), or other (anything else).\n"
    "needs_human: true when the message suggests that someone else used or changed the customer's account; "
    "otherwise false.")
PROMPT_V2 = PROMPT_V1 + (
    "\nMessages can be in Spanish or contain typos: classify by meaning, not by exact words. "
    "A password or account changed without the customer's permission always needs a human.")
PROMPTS = {"v1": PROMPT_V1, "v2": PROMPT_V2}
SCHEMA = {"type": "object", "required": ["category", "needs_human"],
          "properties": {"category": {"type": "string", "enum": list(CATEGORIES)},
                         "needs_human": {"type": "boolean"}}}


class TruncatedReply(RuntimeError):
    """The model stopped at the token limit, so its reply is incomplete."""


def post(path, body, timeout=120):
    request = urllib.request.Request(f"{OLLAMA}{path}", data=json.dumps(body).encode("utf-8"),
                                     headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(request, timeout=timeout) as response:
        return json.loads(response.read())


def get(path):
    with urllib.request.urlopen(f"{OLLAMA}{path}", timeout=10) as response:
        return json.loads(response.read())


class ModelTriage:
    """A harness system: name, version, mode, model, prompt and tools, plus respond(case, trial)."""

    mode = "live"
    tools = None

    def __init__(self, tag, prompt="v1"):
        self.tag, self.name, self.version = tag, f"model-triage-{tag.split(':')[0]}", prompt
        self.model = f"{tag}@{MODELS[tag]}"
        self.text = PROMPTS[prompt]
        self.prompt = f"triage-{prompt} sha256:{hashlib.sha256(self.text.encode()).hexdigest()[:16]}"
        self.server = None

    def check_model(self):
        """Refuse to run if the installed weights differ from the pinned digest."""
        digests = {m["name"]: m["digest"] for m in get("/api/tags")["models"]}
        if not digests.get(self.tag, "").startswith(MODELS[self.tag]):
            raise RuntimeError(f"{self.tag} is not installed with digest {MODELS[self.tag]}")
        self.server = get("/api/version")["version"]

    def seed_for(self, trial):
        return SEED_BASE + trial

    def chat(self, message, seed):
        return post("/api/chat", {
            "model": self.tag, "stream": False, "think": False, "format": SCHEMA,
            "options": {"temperature": TEMPERATURE, "seed": seed, "num_predict": NUM_PREDICT},
            "messages": [{"role": "system", "content": self.text}, {"role": "user", "content": message}]})

    def respond(self, case, trial):
        message = case["input"]
        if not isinstance(message, str):
            raise ValueError("message is not text")
        if not message.strip():
            raise ValueError("message is blank")
        if self.server is None:
            self.check_model()
        seed = self.seed_for(trial)
        result = self.chat(message, seed)
        metadata = {"seed": seed, "temperature": TEMPERATURE, "ollama": self.server,
                    "eval_count": result.get("eval_count"), "done_reason": result.get("done_reason")}
        if result.get("done_reason") == "length":
            raise TruncatedReply(f"reply stopped at {NUM_PREDICT} tokens")
        return Answer(json.loads(result["message"]["content"]), metadata)


qwen_v1 = ModelTriage("qwen3.5:0.8b", "v1")
qwen_v2 = ModelTriage("qwen3.5:0.8b", "v2")

Record and compare

python3 bench.py --cases cases-robustness.json --system model_triage:qwen_v1 --trials 5 --log recorded/qwen-v1.jsonl
python3 bench.py --cases cases-robustness.json --system model_triage:qwen_v2 --trials 5 --log recorded/qwen-v2.jsonl
python3 compare.py runs/qwen-v1.json runs/qwen-v2.json --cases cases-robustness.json --out runs/comparison.json

The README has the conversion and harness commands between recording and comparing.

Read the comparison

The mean per-case pass share is 0.85 with v1 and 0.8864 with v2, a difference of +0.0364. 6 cases improved, 0 got worse and 38 did not change:

Paired by case and trial, 186 trial pairs passed under both prompts and 24 under neither. Only 10 pairs disagree: 9 passed only with v2 and 1 only with v1. The whole difference lives in those pairs.

Resampling cases gives an interval from +0.0091 to +0.0682, entirely above zero. Resampling source groups, computed exactly, gives 0 to +0.0727. The changed cases come from 3 of the 9 source messages: password-changed, product-advice, charged-after-closing. A draw of source groups that misses all of them shows no gain, and the share of such draws is 0.026, just above the 2.5% that marks the interval’s lower end. That is why the lower end is 0.

The lower end cannot go below 0, because no case got worse: zero is a floor here, does not indicate harm. And the result sits on a boundary: a sampled bootstrap of the same groups puts the lower end at 0 for some seeds and just above it for others (test_sampled_lower_end_depends_on_the_seed checks 50 seeds), which is why compare.py enumerates the draws. With the honest unit, the data neither shows nor rules out a gain beyond those 3 source messages. It shows a gain on those messages and their variants.

The trial-level interval is the narrowest, because it counts 220 trials as if each were a separate case. Here it is only slightly narrower than the case-level one. It misleads most when every trial of a case agrees: then 5 trials carry the information of one case, yet count as 5.

The language: es and variant: translation slices hold the same cases: every Spanish case here is a translation variant, so they are one slice counted twice. Only 1 slice has an interval entirely above zero: language: en, the largest one. Every other slice has an interval that reaches zero, and the Spanish slice, which the new sentence was written for, has 7 cases. Claiming that v2 “fixes Spanish” from this table would rest on 1 changed Spanish case.

{
  "comparison_version": "1",
  "resamples": 2000,
  "seed": 20261020,
  "case_set": {
    "name": "inbox-triage-robustness",
    "sha256": "945fb939580c38abcadcd8df25534d2a51011800b00d7c0ed2c44587e0fa288c"
  },
  "trials": 5,
  "baseline": {
    "name": "model-triage-qwen3.5",
    "version": "v1",
    "mode": "replay",
    "model": "qwen3.5:0.8b@de63045f2975",
    "prompt": "triage-v1 sha256:053fdf51b21d891e",
    "tools": null,
    "index": null
  },
  "candidate": {
    "name": "model-triage-qwen3.5",
    "version": "v2",
    "mode": "replay",
    "model": "qwen3.5:0.8b@de63045f2975",
    "prompt": "triage-v2 sha256:d3ebc88d046c97b5",
    "tools": null,
    "index": null
  },
  "overall": {
    "cases": 44,
    "baseline": 0.85,
    "candidate": 0.8864,
    "difference": 0.0364,
    "interval_95": [
      0.0091,
      0.0682
    ],
    "better": 6,
    "worse": 0,
    "same": 38,
    "trial_level_interval_95": [
      0.0091,
      0.0636
    ],
    "source_groups": 9,
    "source_cluster_method": "exact over 24310 draws",
    "source_cluster_interval_95": [
      0.0,
      0.0727
    ],
    "source_no_gain_share": 0.026
  },
  "trial_pairs": {
    "both_pass": 186,
    "only_baseline": 1,
    "only_candidate": 9,
    "neither": 24
  },
  "slices": {
    "language": {
      "en": {
        "cases": 37,
        "baseline": 0.8595,
        "candidate": 0.8919,
        "difference": 0.0324,
        "interval_95": [
          0.0054,
          0.0649
        ],
        "better": 5,
        "worse": 0,
        "same": 32,
        "few_cases": false
      },
      "es": {
        "cases": 7,
        "baseline": 0.8,
        "candidate": 0.8571,
        "difference": 0.0571,
        "interval_95": [
          0.0,
          0.1714
        ],
        "better": 1,
        "worse": 0,
        "same": 6,
        "few_cases": true
      }
    },
    "variant": {
      "emphasis": {
        "cases": 7,
        "baseline": 0.8571,
        "candidate": 0.9143,
        "difference": 0.0571,
        "interval_95": [
          0.0,
          0.1714
        ],
        "better": 1,
        "worse": 0,
        "same": 6,
        "few_cases": true
      },
      "greeting": {
        "cases": 7,
        "baseline": 0.8571,
        "candidate": 0.8571,
        "difference": 0.0,
        "interval_95": [
          0.0,
          0.0
        ],
        "better": 0,
        "worse": 0,
        "same": 7,
        "few_cases": true
      },
      "original": {
        "cases": 9,
        "baseline": 0.8889,
        "candidate": 0.9333,
        "difference": 0.0444,
        "interval_95": [
          0.0,
          0.1111
        ],
        "better": 2,
        "worse": 0,
        "same": 7,
        "few_cases": true
      },
      "translation": {
        "cases": 7,
        "baseline": 0.8,
        "candidate": 0.8571,
        "difference": 0.0571,
        "interval_95": [
          0.0,
          0.1714
        ],
        "better": 1,
        "worse": 0,
        "same": 6,
        "few_cases": true
      },
      "typo": {
        "cases": 7,
        "baseline": 0.8286,
        "candidate": 0.8857,
        "difference": 0.0571,
        "interval_95": [
          0.0,
          0.1143
        ],
        "better": 2,
        "worse": 0,
        "same": 5,
        "few_cases": true
      },
      "uppercase": {
        "cases": 7,
        "baseline": 0.8571,
        "candidate": 0.8571,
        "difference": 0.0,
        "interval_95": [
          0.0,
          0.0
        ],
        "better": 0,
        "worse": 0,
        "same": 7,
        "few_cases": true
      }
    }
  },
  "cases": [
    {
      "case_id": "password-changed",
      "baseline": 0.8,
      "candidate": 1.0
    },
    {
      "case_id": "password-changed--typo",
      "baseline": 0.8,
      "candidate": 1.0
    },
    {
      "case_id": "product-advice",
      "baseline": 0.2,
      "candidate": 0.4
    },
    {
      "case_id": "product-advice--typo",
      "baseline": 0.0,
      "candidate": 0.2
    },
    {
      "case_id": "product-advice--emphasis",
      "baseline": 0.0,
      "candidate": 0.4
    },
    {
      "case_id": "charged-after-closing--es",
      "baseline": 0.6,
      "candidate": 1.0
    }
  ]
}

Check your understanding

Try it: suppose the candidate had fixed every variant of one source case and changed nothing else. Predict which of the two intervals, over cases or over source groups, would start above zero. Then run test_variants_of_one_source_are_resampled_together, which builds exactly that case.

Failure cases

Each behavior below has a named test in test_compare.py.

test_recorded_runs_and_comparison_match_fresh_runs converts both logs, replays them through the harness and rebuilds comparison.json, then compares everything with the saved files after removing only RUN_DESCRIPTION_FIELDS.

python3 -m unittest -v test_compare.py

The run ends with Ran 12 tests and OK on Python 3.11 and 3.15.

Deployment

The deployment unit is a directory run from the command line. Run compare.py after recording both systems with bench.py on the same case set, trial count and seed schedule. It verifies the case set, task, graders, splits and trial count; the included model wrappers define the shared seed schedule, which the comparison does not independently validate. Keep comparison.json with the logs and the run records: a reviewer can rebuild it from the logs without the model.

python3 -m unittest test_compare.py
python3 compare.py runs/qwen-v1.json runs/qwen-v2.json --cases cases-robustness.json --out runs/comparison.json

compare.py prints the difference, its case-level interval and the case counts on one line, and exits with status 1 and an error: line when it refuses.

Scaling

No throughput or capacity was measured. The bootstrap is the only cost that grows quickly:

To measure, time compare.py on the recorded runs while varying RESAMPLES.

Key takeaways

• Pair the runs: same cases, same seeds, one difference per case.
• An interval needs the right unit. Trials of a case and variants of a message are not independent.
• Report which cases changed and the trial pairs that disagree; a difference lives in a few rows.
• A slice’s result needs its case count. A small slice cannot carry a claim on its own.

Best practices

• Choose the unit of resampling from how the cases were made, before looking at the result.
• Add independent source cases before adding trials or variants when the interval is too wide.
• Report a gain on a slice only with its interval, and keep held-out cases for the final check.

References

Checked October 20, 2026:

• Efron and Tibshirani, An Introduction to the Bootstrap (1993) — the bootstrap and percentile intervals.
• Anthropic: Demystifying evals for AI agents — trials, nondeterminism and comparing agent versions.
• Python: random.Random and math.fsum — the seeded generator and exact float sums.
• Qwen: Qwen3.5-0.8B model card — Apache 2.0.
• Ollama: Chat API — format, seed and temperature.

For trials, pass counts and the benchmark log, start with Build a repeated-trial benchmark.