The tracker issue-triage bot needs cases that were not used to tune its prompt. This lab keeps each original issue and its generated variants together, then checks for overlap and later edits before an evaluation.
When you need this. Use these checks when adding paraphrases, changing expected labels, or copying examples into a prompt. They help find known contamination paths; they cannot establish what a model saw in training.
This is one deterministic run with fictional issues, not a new model run. Download the complete lab.
Step 1: Group related issues before splitting
Download the complete lab, extract it and open a terminal in build-held-out-dataset-and-leakage-checks/. The README lists its runtime and recorded examples.
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -r tracker-requirements.txtA development split is available while you change the application. A held-out split is reserved for a later check. Leakage occurs when information from that held-out split influences the application being measured. A new spelling or a louder rendering of the same issue is not an independent example.
tracker-dataset.json is the unchanged 12-case dataset from the evaluation harness. The recorder applies the robustness lesson’s existing shouting() transformation to two issues. It records the original ID as source_id, giving 14 inputs in 12 groups. GroupShuffleSplit assigns whole groups with random_state=7; this run puts 10 inputs in development and four in held-out. Its test_size applies to groups, so do not expect precisely one third of individual rows in held-out.
A variant crosses the split. The recorded color-in-pipes-derived case is deliberately put in development while its original is held-out. The source-ID check blocks it even if the words have changed. Fix the split by moving the complete family together, not by deleting its ancestry field. Group IDs record known relationships; untagged paraphrases still need review. This lab accepts one original plus direct variants; the Pydantic model refuses missing sources and chains of variants.
"""Typed tracker cases; derived from the course's existing Pydantic case schema."""
import hashlib
import json
from typing import Literal
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
class Expected(BaseModel):
model_config=ConfigDict(extra='forbid')
label:Literal['bug','feature','docs','question']
priority:Literal['p0','p1','p2']|None=None
class SplitCase(BaseModel):
model_config=ConfigDict(extra='forbid')
id:str=Field(pattern=r'^[a-z0-9-]+$')
issue:str=Field(min_length=1)
expected:Expected
must_mention:list[str]=[]
source_id:str|None=None
split:Literal['development','held_out']
@field_validator('issue')
@classmethod
def nonblank(cls,value):
if not any(c.isalnum() for c in value):raise ValueError('issue must contain letters or digits')
return value
class SplitDataset(BaseModel):
model_config=ConfigDict(extra='forbid')
name:str
version:int
cases:list[SplitCase]=Field(min_length=1)
@model_validator(mode='after')
def valid_sources(self):
by_id={c.id:c for c in self.cases}
if len(by_id)!=len(self.cases):raise ValueError('duplicate case ID')
for c in self.cases:
if c.source_id is not None:
parent=by_id.get(c.source_id)
if parent is None or parent.source_id is not None:
raise ValueError('source_id must name an original case')
return self
class FrozenSplit(BaseModel):
model_config=ConfigDict(extra='forbid')
schema_version:Literal[1]=1
dataset_name:str
held_out:dict[str,str]
def canonical(value):
return json.dumps(value,sort_keys=True,ensure_ascii=False,separators=(',',':'))
def case_hash(case:SplitCase):
return hashlib.sha256(canonical(case.model_dump()).encode()).hexdigest()
def freeze(dataset:SplitDataset):
return FrozenSplit(dataset_name=dataset.name,held_out={c.id:case_hash(c) for c in dataset.cases if c.split=='held_out'})The group assignment happens before overlap checks. The held-out file also supplies the hashes saved when the split is frozen. Both the current cases and those saved hashes reach check(); prompt wording is an optional additional input. The output can block a comparison, request review, or report no findings.
Step 2: Separate word overlap from a split violation
A Jaccard similarity is the number of shared unique words divided by the number in their union. CountVectorizer(binary=True) encodes word presence; jaccard_score() computes the overlap. Normalization lowercases text and retains Unicode letters and digits. Repeated words do not increase this score.
An exact input appears in both splits. If normalized inputs match, the checker blocks the comparison. Punctuation or letter case cannot make those copies independent. Remove the duplicate from one split and freeze a deliberately reviewed revision.
A near duplicate might change meaning. Similarity of at least 0.6 requests review. It does not establish leakage: “give read access” and “give write access” can share most words while requiring different answers. Conversely, a semantic paraphrase can share too few words to be flagged. The tests keep both limitations visible. Review the source and intended behavior before regrouping or retaining the cases. The threshold is a lab heuristic, not a calibrated detector.
"""Check tracker eval splits. Reuses the legacy leakage rules with scoped decisions."""
import json
import re
from pathlib import Path
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.metrics import jaccard_score
from sklearn.model_selection import GroupShuffleSplit
from split_models import SplitCase,SplitDataset,FrozenSplit,case_hash
NEAR_DUPLICATE=0.6
COPIED_WORDS=8
def normalize(text):
# Legacy rule: preserve Unicode letters/digits, remove punctuation and case.
return ' '.join(re.findall(r'[^\W_]+',text.lower()))
def word_similarity(left,right):
words=[normalize(left),normalize(right)]
if not words[0] or not words[1]:return 0.0
binary=CountVectorizer(binary=True,token_pattern=r'(?u)\b\w+\b').fit_transform(words).toarray()
return float(jaccard_score(binary[0],binary[1],zero_division=0))
def assign_groups(cases:list[dict],seed=7):
groups=[c.get('source_id') or c['id'] for c in cases]
split=GroupShuffleSplit(n_splits=1,test_size=1/3,random_state=seed)
dev,held=next(split.split(cases,groups=groups))
held_ids=set(held.tolist())
return [dict(c,split='held_out' if i in held_ids else 'development') for i,c in enumerate(cases)]
def check(dataset:SplitDataset,prompt:str='',locked:FrozenSplit|None=None):
rows=[]
def finding(check,severity,**details):rows.append(dict(check=check,severity=severity,**details))
dev=[c for c in dataset.cases if c.split=='development']
held=[c for c in dataset.cases if c.split=='held_out']
for left in dev:
for right in held:
score=word_similarity(left.issue,right.issue)
if normalize(left.issue) and normalize(left.issue)==normalize(right.issue):
finding('duplicate input','block',development=left.id,held_out=right.id)
elif score>=NEAR_DUPLICATE:
finding('near duplicate','review',development=left.id,held_out=right.id,similarity=round(score,4))
by_id={c.id:c for c in dataset.cases}
for c in dataset.cases:
if c.source_id and by_id[c.source_id].split!=c.split:
finding('source crosses split','block',case=c.id,source=c.source_id)
prompt_words=normalize(prompt).split()
for c in held:
words=normalize(c.issue).split()
copied=[' '.join(words[i:i+COPIED_WORDS]) for i in range(len(words)-COPIED_WORDS+1)
if any(prompt_words[j:j+COPIED_WORDS]==words[i:i+COPIED_WORDS]
for j in range(len(prompt_words)-COPIED_WORDS+1))]
if copied:finding('held-out excerpt in prompt','review',case=c.id,excerpt=copied[0])
if locked is not None:
if locked.dataset_name!=dataset.name:raise ValueError('lock belongs to another dataset')
for id,digest in locked.held_out.items():
c=by_id.get(id)
if c is None:finding('frozen split changed','block',case=id,reason='removed')
elif c.split!='held_out':finding('frozen split changed','block',case=id,reason='moved')
elif case_hash(c)!=digest:finding('frozen split changed','block',case=id,reason='edited')
for c in held:
if c.id not in locked.held_out:finding('frozen split changed','block',case=c.id,reason='added')
severity={r['severity'] for r in rows}
return {'dataset':dataset.name,'version':dataset.version,
'cases':{'development':len(dev),'held_out':len(held)},
'decision':'block' if 'block' in severity else 'review' if rows else 'pass','findings':rows}
if __name__=='__main__':
import argparse
parser=argparse.ArgumentParser(description=__doc__)
parser.add_argument('cases',type=Path)
parser.add_argument('--lock',type=Path)
parser.add_argument('--prompt',type=Path)
args=parser.parse_args()
data=SplitDataset.model_validate_json(args.cases.read_text())
locked=FrozenSplit.model_validate_json(args.lock.read_text()) if args.lock else None
report=check(data,args.prompt.read_text() if args.prompt else '',locked)
print(json.dumps(report,indent=2));raise SystemExit(1 if report['decision']=='block' else 0)Step 3: Freeze cases and inspect prompt excerpts
A frozen split stores a SHA-256 hash of each held-out case’s serialized fields, including input, expected label and group metadata. Comparing those hashes detects edits, additions, removals and moves. The lock belongs to a named dataset; a lock for another name is refused. Freezing is a local versioning decision, not an access-control mechanism.
A held-out label changes. The recorder changes one expected label after writing the lock; the result blocks with frozen split changed: edited. Create a reviewed dataset revision and rerun the baseline and candidate on that same revision. Rewriting the lock alone does not make earlier measurements comparable.
Held-out wording appears in the prompt. Eight consecutive normalized words request review. In the recorded prompt scenario, the excerpt is when i pipe tracker list into grep the. Check whether someone copied an evaluation case into a few-shot example; move that case out of the held-out set if it influenced tuning. Common product language can overlap without copying, so the finding is not an automatic block. This check reads only the supplied prompt string; it does not inspect a model provider’s training data or the team’s editing history.
Step 4: Reproduce the decisions before measuring the bot
python3 -m venv .venv
.venv/bin/python -m pip install -r tracker-requirements.txt
.venv/bin/python -m pytest -q test_tracker_splits.py
.venv/bin/python record_tracker_splits.py --check
.venv/bin/python tracker_leakcheck.py tracker-splits.json --lock tracker-heldout.lock.jsongrouped split: pass (0 findings)
variant moved across split: block (2 findings)
held-out label edited: block (1 findings)
prompt contains held-out wording: review (1 findings)Run these commands from the extracted lab directory. --check regenerates the three saved outputs in memory and refuses a mismatch; omitting it rewrites those files. The test suite covers group assignments, duplicates, the word-overlap blind spot, changed expected labels, membership changes, prompt excerpts and invalid inputs. No API key or model is needed.
Blank inputs or unknown sources. Pydantic refuses empty, whitespace-only or punctuation-only issues and unknown sources before splitting. Repair the case definition rather than interpreting an empty comparison as a clean result. A block decision exits with status 1; review exits with status 0 and still requires a person to decide. If CI should stop for review, make that policy explicit in the calling workflow.
The grouped run has no findings. That means these checks did not find their specific conditions; it does not establish an unbiased dataset. Keep split membership separate from whether the model passed each issue’s checks. The regression gate still compares actual results on matching datasets.
Read the explanation
Exact normalized copies and explicit cross-split ancestry block; near word overlap alone needs a person to decide.
What’s next
Bind an eval run to a release manifest so the report identifies the application files and settings it evaluated.
References
Checked 2026-10-22 against the installed scikit-learn, Pydantic and pytest APIs; exact pins are in the lab download. Data leakage, GroupShuffleSplit, Jaccard score, and pytest parameterization.