Define what makes a rollback explanation useful and supported. Apply a task-specific rubric to authored answer contrasts, bind each judgement to the reviewed text and keep unfinished review visible in coverage.
Problem
When you need this. Reviewers disagree about whether a documentation answer is correct enough to use.
An answer gives the correct rollback command and then promises zero downtime. Another gives the same command but omits the health check. Both can look useful, yet they fail different requirements. Comparing each answer with one gold sentence would also reject a correct paraphrase.
A rubric states the criteria for an acceptable answer and how judgements produce a decision. You will define one for a specific docuctl support question, then apply authored judgement fixtures through the existing evaluation runner. The program checks revision bindings and applies the decision rule; it does not decide the meaning of free text automatically.
The eight contrast answers are authored examples. Three meet the rubric, three do not, and two remain pending. Their differences make the acceptance boundaries visible.
Concepts
A rubric separates the requirements of a good answer into criteria that a reviewer can apply. For this task, mentioning the command alone is insufficient: prerequisites, database boundaries and verification instructions also matter.
An annotation records a reviewer’s judgement and supporting excerpts for a particular answer revision. The lab applies supplied annotations; it does not ask a model to grade new answers. A pending judgement has not supplied enough information for a final quality decision and must not disappear from the report.
Architecture
The task and guide define rubric criteria. Candidate answers and annotations are loaded with their revision bindings. The evaluator checks those bindings, retains pending reviews and applies the rubric’s decision rule to completed judgements. The existing harness reports execution coverage separately from review coverage and answer outcomes.
Working code
Start with the reader’s task and its source
The reader asks for a staging rollback to release r41: explain the command, its prerequisites, the database boundary and post-rollback verification. No execution is requested. The source is a short fictional docuctl 3 guide:
Fictional docuctl 3 guide: staging rollback to r41
Use `docuctl releases rollback r41 --environment staging` to select release r41.
Before the command, obtain release approval and confirm that the active environment is staging.
The command changes the deployed service version. It does not restore database contents.
After the rollback, check service health.
An assistant may explain or propose this command; it does not execute it in this example.Write the acceptance requirements before inspecting candidate answers. Here, the required points are the current command, approval and environment confirmation, the database limitation and the health check. A fifth criterion requires every material claim to stay within the source’s support.
The last criterion catches additions rather than omissions. “It guarantees zero downtime” is unsupported by this guide even when the answer includes every required point. “Rollback does not restore database contents” is a supported limitation, not a failure merely because it contains words about restoring a database.
Download the complete lab and enter ai-system-answer-rubric/:
python3.12 -m venv .venv
source .venv/bin/activateThe ZIP contains the guide, rubric, eight answers, example judgement annotations, tests and the unchanged standard-library harness from the earlier evaluation lesson. It requires no package installation or model key.
Make each criterion specific enough to judge
Each criterion has an ID, a rule and a reference quote. It asks whether a concrete requirement was met; it does not assign points for sounding confident or using a preferred sentence structure.
{
"id": "staging-rollback-explanation",
"version": 1,
"task": "Explain a staging rollback to r41, including the command, prerequisites, database boundary and post-rollback verification. Do not execute it.",
"acceptance": "First require a judgement for every criterion with no uncertainty; otherwise pending. Then reject any does_not_meet and accept only when all meet.",
"ratings": [
"meets",
"does_not_meet",
"uncertain"
],
"criteria": [
{
"id": "command",
"rule": "Give the exact docuctl 3 staging command for release r41.",
"reference_quote": "Use `docuctl releases rollback r41 --environment staging` to select release r41."
},
{
"id": "preconditions",
"rule": "Require release approval and confirm staging before the command.",
"reference_quote": "Before the command, obtain release approval and confirm that the active environment is staging."
},
{
"id": "database_scope",
"rule": "Explain that rollback does not restore database contents.",
"reference_quote": "The command changes the deployed service version. It does not restore database contents."
},
{
"id": "verification",
"rule": "Tell the reader to check service health after rollback.",
"reference_quote": "After the rollback, check service health."
},
{
"id": "support",
"rule": "Every material claim must be supported by this source; do not invent guarantees.",
"reference_quote": "The command changes the deployed service version. It does not restore database contents."
}
]
}The fixture accepts three ratings: meets, does_not_meet and uncertain. A reviewer should supply a reason, use the criterion’s source reference and identify a relevant answer span where useful. An omission may have no span to quote; its reason must identify the absent requirement.
This rubric uses hard requirements. All complete judgements must meet for acceptance; a supported command cannot compensate for an unsupported guarantee. There is no weighted score whose average could hide a missing database warning. A different task may need a different rubric, so changing the question also requires reviewing the criteria.
Consider these contrasts before running the code:
| Answer pattern | Judgement | Reason |
|---|---|---|
| Correct paraphrase | Accepted | All required meaning is retained. |
| Zero-downtime guarantee added | Rejected | The source does not support the guarantee. |
| Health check omitted | Rejected | A required task point is absent. |
| “Your data will be safe” added | Pending | The wording may imply an undocumented database or durability guarantee. |
The uncertain example needs adjudication or a clearer rubric. The software must not turn that ambiguity into a confident semantic verdict.
Bind a judgement to the answer it actually describes
An annotation includes hashes for the answer, guide and rubric. Its case also records the annotation revision. If the answer changes, the old annotation no longer applies. Updating a judgement is an explicit fixture revision, not a silent correction to an earlier result.
rubric.py validates those bindings and the source excerpts. It checks that the reference quote is the one assigned to the criterion and that a supplied answer span occurs in the answer. These are provenance checks. A matching quote does not establish that a claim follows from it, and a stored meets label can still be wrong.
"""Validate authored judgement bindings and apply a task-specific decision rule.
No semantic judgement is inferred from text. A quote match checks provenance,
not entailment. Uncertain/missing judgements need review before a quality verdict.
"""
import hashlib
import json
RATINGS = {'meets', 'does_not_meet', 'uncertain'}
def fingerprint(value):
return hashlib.sha256(json.dumps(value,sort_keys=True,ensure_ascii=False,separators=(',',':')).encode()).hexdigest()
def validate_rubric(rubric, source):
if (not isinstance(rubric,dict) or set(rubric) != {'id','version','task','acceptance','ratings','criteria'}
or type(rubric['version']) is not int or rubric['version'] < 1
or not all(isinstance(rubric[key],str) and rubric[key] for key in ('id','task','acceptance'))
or not isinstance(rubric['ratings'],list) or not all(isinstance(x,str) for x in rubric['ratings'])
or set(rubric['ratings']) != RATINGS or not isinstance(rubric['criteria'],list)
or not rubric['criteria']):
raise ValueError('invalid rubric contract')
seen={}
for criterion in rubric['criteria']:
if (not isinstance(criterion,dict) or set(criterion) != {'id','rule','reference_quote'} or not all(isinstance(criterion[key],str) and criterion[key] for key in ('id','rule','reference_quote'))
or criterion['id'] in seen
or not criterion['reference_quote'] or criterion['reference_quote'] not in source):
raise ValueError('invalid criterion or reference quote')
seen[criterion['id']]=criterion
return seen
def assess(case, review, criteria, source, source_sha256, rubric_sha256):
answer=case['input']['answer']
if (set(answer) != {'text','citation'} or not all(isinstance(v,str) and v for v in answer.values())):
raise ValueError('answer needs nonempty text and citation')
expected=case['expected']
if expected['source_sha256'] != source_sha256 or expected['rubric_sha256'] != rubric_sha256:
raise ValueError('case uses a different source or rubric version')
if expected['review_sha256'] != fingerprint(review):
raise ValueError('case uses a different judgement revision')
if review is None:
return {'status':'pending','reason':'no authored review fixture','ratings':{}}
if (set(review) != {'answer_id','answer_sha256','source_sha256','rubric_sha256','judgements'}
or review['answer_id'] != case['id'] or review['answer_sha256'] != fingerprint(answer)
or review['source_sha256'] != source_sha256 or review['rubric_sha256'] != rubric_sha256):
raise ValueError('judgement is bound to different answer/source/rubric content')
judgements=review['judgements']
if not isinstance(judgements,dict) or not set(judgements) <= set(criteria):
raise ValueError('unknown criterion in judgement')
for criterion, judgement in judgements.items():
if (not isinstance(judgement,dict) or set(judgement) != {'rating','reason','reference_quote','answer_span'}
or not isinstance(judgement['rating'],str) or judgement['rating'] not in RATINGS
or not isinstance(judgement['reason'],str) or not judgement['reason']
or not isinstance(judgement['reference_quote'],str) or not judgement['reference_quote']
or judgement['reference_quote'] != criteria[criterion]['reference_quote']
or not isinstance(judgement['answer_span'],str)
or judgement['answer_span'] not in answer['text']):
raise ValueError('invalid judgement rating, reason or bound excerpt')
ratings={key:value['rating'] for key,value in judgements.items()}
if set(judgements) != set(criteria) or 'uncertain' in ratings.values():
return {'status':'pending','reason':'missing or uncertain criterion judgement','ratings':ratings}
failed=[key for key,value in ratings.items() if value=='does_not_meet']
return {'status':'rejected' if failed else 'accepted','reason':', '.join(failed),'ratings':ratings}The annotations here illustrate judgements. Real use requires reviewed annotations and a way to resolve disagreements.
Inspect the authored judgement annotations
{
"origin": "Authored example judgements, not a human-rating study or model judge run.",
"reviews": [
{
"answer_id": "complete",
"answer_sha256": "8702dd02bbdecc22c4bf563cf65a8799b03390dbe6916007e45e1ea89248d144",
"source_sha256": "2ce5d06d1dc27d2c278ac24bf446442d8276124ce33a629c1ca58fdb59d5fbd2",
"rubric_sha256": "3a8dcd22e9f421ec51f4eae59bee09d2ac2f67b1bc5ad98ff1f465e7d6535228",
"judgements": {
"command": {
"rating": "meets",
"reason": "The answer supplies the v3 rollback syntax with release r41 and staging.",
"reference_quote": "Use `docuctl releases rollback r41 --environment staging` to select release r41.",
"answer_span": ""
},
"preconditions": {
"rating": "meets",
"reason": "Release approval and staging confirmation precede the command.",
"reference_quote": "Before the command, obtain release approval and confirm that the active environment is staging.",
"answer_span": ""
},
"database_scope": {
"rating": "meets",
"reason": "The answer explicitly excludes restoring database contents.",
"reference_quote": "The command changes the deployed service version. It does not restore database contents.",
"answer_span": ""
},
"verification": {
"rating": "meets",
"reason": "The health check is described after the rollback.",
"reference_quote": "After the rollback, check service health.",
"answer_span": ""
},
"support": {
"rating": "meets",
"reason": "The substantive claims stay within the documented guide.",
"reference_quote": "The command changes the deployed service version. It does not restore database contents.",
"answer_span": ""
}
}
},
{
"answer_id": "paraphrase",
"answer_sha256": "350a2d02cee1f26c843248d58a75a0f67d2fa3159df379b50e73575a3f5971f7",
"source_sha256": "2ce5d06d1dc27d2c278ac24bf446442d8276124ce33a629c1ca58fdb59d5fbd2",
"rubric_sha256": "3a8dcd22e9f421ec51f4eae59bee09d2ac2f67b1bc5ad98ff1f465e7d6535228",
"judgements": {
"command": {
"rating": "meets",
"reason": "The answer supplies the v3 rollback syntax with release r41 and staging.",
"reference_quote": "Use `docuctl releases rollback r41 --environment staging` to select release r41.",
"answer_span": ""
},
"preconditions": {
"rating": "meets",
"reason": "Release approval and staging confirmation precede the command.",
"reference_quote": "Before the command, obtain release approval and confirm that the active environment is staging.",
"answer_span": ""
},
"database_scope": {
"rating": "meets",
"reason": "The answer explicitly excludes restoring database contents.",
"reference_quote": "The command changes the deployed service version. It does not restore database contents.",
"answer_span": ""
},
"verification": {
"rating": "meets",
"reason": "The health check is described after the rollback.",
"reference_quote": "After the rollback, check service health.",
"answer_span": ""
},
"support": {
"rating": "meets",
"reason": "The substantive claims stay within the documented guide.",
"reference_quote": "The command changes the deployed service version. It does not restore database contents.",
"answer_span": ""
}
}
},
{
"answer_id": "negation",
"answer_sha256": "331e665c38699c6800453691f2d46691964e3f222f3c5a84b6042db7c6a19ec3",
"source_sha256": "2ce5d06d1dc27d2c278ac24bf446442d8276124ce33a629c1ca58fdb59d5fbd2",
"rubric_sha256": "3a8dcd22e9f421ec51f4eae59bee09d2ac2f67b1bc5ad98ff1f465e7d6535228",
"judgements": {
"command": {
"rating": "meets",
"reason": "The answer supplies the v3 rollback syntax with release r41 and staging.",
"reference_quote": "Use `docuctl releases rollback r41 --environment staging` to select release r41.",
"answer_span": ""
},
"preconditions": {
"rating": "meets",
"reason": "Release approval and staging confirmation precede the command.",
"reference_quote": "Before the command, obtain release approval and confirm that the active environment is staging.",
"answer_span": ""
},
"database_scope": {
"rating": "meets",
"reason": "The answer explicitly excludes restoring database contents.",
"reference_quote": "The command changes the deployed service version. It does not restore database contents.",
"answer_span": ""
},
"verification": {
"rating": "meets",
"reason": "The health check is described after the rollback.",
"reference_quote": "After the rollback, check service health.",
"answer_span": ""
},
"support": {
"rating": "meets",
"reason": "The substantive claims stay within the documented guide.",
"reference_quote": "The command changes the deployed service version. It does not restore database contents.",
"answer_span": ""
}
}
},
{
"answer_id": "unsupported-guarantee",
"answer_sha256": "c25f2a6ed4056bd378a4e557c644a30c51143f7a7f3bb89827c2193766b2b43d",
"source_sha256": "2ce5d06d1dc27d2c278ac24bf446442d8276124ce33a629c1ca58fdb59d5fbd2",
"rubric_sha256": "3a8dcd22e9f421ec51f4eae59bee09d2ac2f67b1bc5ad98ff1f465e7d6535228",
"judgements": {
"command": {
"rating": "meets",
"reason": "The answer supplies the v3 rollback syntax with release r41 and staging.",
"reference_quote": "Use `docuctl releases rollback r41 --environment staging` to select release r41.",
"answer_span": ""
},
"preconditions": {
"rating": "meets",
"reason": "Release approval and staging confirmation precede the command.",
"reference_quote": "Before the command, obtain release approval and confirm that the active environment is staging.",
"answer_span": ""
},
"database_scope": {
"rating": "meets",
"reason": "The answer explicitly excludes restoring database contents.",
"reference_quote": "The command changes the deployed service version. It does not restore database contents.",
"answer_span": ""
},
"verification": {
"rating": "meets",
"reason": "The health check is described after the rollback.",
"reference_quote": "After the rollback, check service health.",
"answer_span": ""
},
"support": {
"rating": "does_not_meet",
"reason": "The source makes no zero-downtime guarantee.",
"reference_quote": "The command changes the deployed service version. It does not restore database contents.",
"answer_span": "It guarantees zero downtime."
}
}
},
{
"answer_id": "missing-verification",
"answer_sha256": "c0584a06f84334da8a50d0351cad0f82db4dc547d18067007e2655b99562c186",
"source_sha256": "2ce5d06d1dc27d2c278ac24bf446442d8276124ce33a629c1ca58fdb59d5fbd2",
"rubric_sha256": "3a8dcd22e9f421ec51f4eae59bee09d2ac2f67b1bc5ad98ff1f465e7d6535228",
"judgements": {
"command": {
"rating": "meets",
"reason": "The answer supplies the v3 rollback syntax with release r41 and staging.",
"reference_quote": "Use `docuctl releases rollback r41 --environment staging` to select release r41.",
"answer_span": ""
},
"preconditions": {
"rating": "meets",
"reason": "Release approval and staging confirmation precede the command.",
"reference_quote": "Before the command, obtain release approval and confirm that the active environment is staging.",
"answer_span": ""
},
"database_scope": {
"rating": "meets",
"reason": "The answer explicitly excludes restoring database contents.",
"reference_quote": "The command changes the deployed service version. It does not restore database contents.",
"answer_span": ""
},
"verification": {
"rating": "does_not_meet",
"reason": "The answer gives no post-rollback health check.",
"reference_quote": "After the rollback, check service health.",
"answer_span": ""
},
"support": {
"rating": "meets",
"reason": "The substantive claims stay within the documented guide.",
"reference_quote": "The command changes the deployed service version. It does not restore database contents.",
"answer_span": ""
}
}
},
{
"answer_id": "wrong-command",
"answer_sha256": "c70ddb59af93775964a5a28df3ab9909bc75f496c019e697180f9ccb6148c5fb",
"source_sha256": "2ce5d06d1dc27d2c278ac24bf446442d8276124ce33a629c1ca58fdb59d5fbd2",
"rubric_sha256": "3a8dcd22e9f421ec51f4eae59bee09d2ac2f67b1bc5ad98ff1f465e7d6535228",
"judgements": {
"command": {
"rating": "does_not_meet",
"reason": "The command uses the older syntax instead of the supplied version 3 command.",
"reference_quote": "Use `docuctl releases rollback r41 --environment staging` to select release r41.",
"answer_span": "docuctl rollback r41 --env staging"
},
"preconditions": {
"rating": "meets",
"reason": "Release approval and staging confirmation precede the command.",
"reference_quote": "Before the command, obtain release approval and confirm that the active environment is staging.",
"answer_span": ""
},
"database_scope": {
"rating": "meets",
"reason": "The answer explicitly excludes restoring database contents.",
"reference_quote": "The command changes the deployed service version. It does not restore database contents.",
"answer_span": ""
},
"verification": {
"rating": "meets",
"reason": "The health check is described after the rollback.",
"reference_quote": "After the rollback, check service health.",
"answer_span": ""
},
"support": {
"rating": "does_not_meet",
"reason": "The proposed command is not supported by this version of the guide.",
"reference_quote": "The command changes the deployed service version. It does not restore database contents.",
"answer_span": "docuctl rollback r41 --env staging"
}
}
},
{
"answer_id": "uncertain-wording",
"answer_sha256": "20b9c00e53127cc98f4d2f31991fefcb9ba3d5dd54df72c7f4a5ba231078fc3c",
"source_sha256": "2ce5d06d1dc27d2c278ac24bf446442d8276124ce33a629c1ca58fdb59d5fbd2",
"rubric_sha256": "3a8dcd22e9f421ec51f4eae59bee09d2ac2f67b1bc5ad98ff1f465e7d6535228",
"judgements": {
"command": {
"rating": "meets",
"reason": "The answer supplies the v3 rollback syntax with release r41 and staging.",
"reference_quote": "Use `docuctl releases rollback r41 --environment staging` to select release r41.",
"answer_span": ""
},
"preconditions": {
"rating": "meets",
"reason": "Release approval and staging confirmation precede the command.",
"reference_quote": "Before the command, obtain release approval and confirm that the active environment is staging.",
"answer_span": ""
},
"database_scope": {
"rating": "meets",
"reason": "The answer explicitly excludes restoring database contents.",
"reference_quote": "The command changes the deployed service version. It does not restore database contents.",
"answer_span": ""
},
"verification": {
"rating": "meets",
"reason": "The health check is described after the rollback.",
"reference_quote": "After the rollback, check service health.",
"answer_span": ""
},
"support": {
"rating": "uncertain",
"reason": "Data will be safe is ambiguous: it could imply a database or durability guarantee absent from the guide.",
"reference_quote": "The command changes the deployed service version. It does not restore database contents.",
"answer_span": "Your data will be safe."
}
}
}
]
}Keep pending review visible before scoring completed judgements
The decision policy first requires a determinate judgement for every criterion. A missing criterion, missing review or uncertain rating keeps the answer pending. Once review is complete, any does_not_meet rejects the answer; all meets accepts it.
That ordering is deliberate. If one criterion already fails while another is uncertain, this experiment still records a pending review and retains the known failure in its ratings. It does not include that unfinished assessment in the completed-review totals.
Pending review is different from missing application output. Both pending examples have answer text. The adapter therefore keeps them in coverage and sends only the six completed decisions through the unchanged runner. It does not simulate a missing model response to obtain the runner’s incomplete status.
Inspect the existing-runner adapter and coverage selection
"""Adapter for the unchanged runner: grade completed rubric decisions only."""
NAME, VERSION = 'docuctl-explanation-rubric', '1'
OUTPUT_FIELDS = {'answer_id':str,'acceptable':bool}
def validate_expected(expected):
if set(expected) != {'acceptable','source_sha256','rubric_sha256','review_sha256'} or expected['acceptable'] is not True:
raise ValueError('case must require rubric acceptance and all revision bindings')
for name in ('source_sha256','rubric_sha256','review_sha256'):
if not isinstance(expected[name],str) or len(expected[name])!=64 or any(c not in '0123456789abcdef' for c in expected[name]):
raise ValueError('expected revision must be a SHA-256 digest')
def decision_contract(case, output):
return (isinstance(output,dict) and set(output)==set(OUTPUT_FIELDS)
and type(output['answer_id']) is str and type(output['acceptable']) is bool
and output['answer_id']==case['id'], 'invalid or misassigned rubric decision')
def rubric_acceptance(case, output):
return output['acceptable'] is True, 'one or more required rubric criteria did not meet'
GRADERS=({'name':'decision-contract','version':VERSION,'kind':'contract','check':decision_contract},
{'name':'rubric-acceptance','version':VERSION,'kind':'outcome','check':rubric_acceptance})"""Apply the existing runner to completed authored rubric judgements, not models."""
import argparse
import copy
import hashlib
import json
from pathlib import Path
import tempfile
from vendor import harness
import task
from rubric import validate_rubric, assess
ROOT=Path(__file__).resolve().parent
def load(root=ROOT):
root=Path(root)
rubric_raw=(root/'rubric.json').read_bytes();source_raw=(root/'source.txt').read_bytes()
rubric=json.loads(rubric_raw);source=source_raw.decode()
criteria=validate_rubric(rubric,source)
cases=harness.load_cases(root/'cases.json',harness.load_task('task'))
for case in cases['cases']:
if (not isinstance(case['input'],dict) or set(case['input'])!={'question','answer'}
or case['input']['question']!=rubric['task']):
raise ValueError('case question does not match the rubric task')
annotations=json.loads((root/'annotations.json').read_text())
reviews={}
ids={case['id'] for case in cases['cases']}
for review in annotations['reviews']:
id=review['answer_id']
if id not in ids or id in reviews:raise ValueError('unknown or duplicate review identity')
reviews[id]=review
decisions={case['id']:assess(case,reviews.get(case['id']),criteria,source,
hashlib.sha256(source_raw).hexdigest(),hashlib.sha256(rubric_raw).hexdigest())
for case in cases['cases']}
return cases,decisions
class FixtureJudgements:
name='authored-rubric-adapter';version='1';mode='authored_annotation_fixture'
model=prompt=tools=None
def __init__(self,decisions):self.decisions=decisions
def respond(self,case,trial):
decision=self.decisions[case['id']]
if decision['status']=='pending':raise ValueError('pending review must not enter the scored subset')
return harness.Answer({'answer_id':case['id'],'acceptable':decision['status']=='accepted'},
{'origin':'authored judgement fixture','ratings':decision['ratings'],
'answer':case['input']['answer'],'reason':decision['reason']})
def record(root=ROOT):
full,decisions=load(root)
selected=[case for case in full['cases'] if decisions[case['id']]['status']!='pending']
report=None
if selected:
subset={'name':full['name']+'-completed-judgements','task':full['task'],'cases':selected}
with tempfile.TemporaryDirectory() as tmp:
path=Path(tmp)/'cases.json';path.write_text(json.dumps(subset))
cases=harness.load_cases(path,harness.load_task('task'))
report=harness.run(FixtureJudgements(decisions),cases,task)
pending=[{'id':id,'reason':row['reason']} for id,row in decisions.items() if row['status']=='pending']
return {'mode':'authored answer/judgement fixtures; local decision execution; no model or human study',
'full_case_set_sha256':full['sha256'], 'coverage':{'candidates':len(full['cases']),
'graded':len(selected),'pending':len(pending),'pending_cases':pending},
'decisions':decisions,'report':report,
'files':{name:hashlib.sha256((Path(root)/name).read_bytes()).hexdigest()
for name in ('source.txt','rubric.json','cases.json','annotations.json')}}
def stable_result(result):
stable=copy.deepcopy(result)
if stable['report']:
stable['report']['manifest'].pop('started_at',None)
stable['report']['manifest'].pop('environment',None)
for row in stable['report']['results']:row.pop('latency_s',None)
return stable
def main():
p=argparse.ArgumentParser();p.add_argument('--check',action='store_true');p.add_argument('--save',type=Path)
args=p.parse_args();result=record()
if args.check and stable_result(result)!=stable_result(json.loads((ROOT/'result.json').read_text())):
raise SystemExit('rubric decisions or revision bindings differ from the bundled record')
if args.save:args.save.write_text(json.dumps(result,indent=2)+'\n')
coverage=result['coverage'];summary=result['report']['summary']['development'] if result['report'] else {}
print('Authored answer and judgement fixtures; no model responses or human-rating study')
print(f"Coverage: {coverage['graded']} graded of {coverage['candidates']} candidates; {coverage['pending']} pending")
print(f"Completed rubric decisions: accepted={summary.get('pass',0)}; rejected={summary.get('fail',0)}")
for id,decision in result['decisions'].items():print(f"{id}: {decision['status']}"+(f" ({decision['reason']})" if decision['reason'] else ''))
print('Pending judgements remain outside the scored subset and visible in coverage.')
print('Quote matching validates excerpts; semantic ratings are supplied judgements.')
if __name__=='__main__':main()The result stores both identities: the full eight-case set and the harness’s actual six-case subset. Per-trial metadata retains the answer, ratings and judgement origin.
Run the contrasts and inspect coverage with the outcomes
python evaluate_rubric.py
python evaluate_rubric.py --check
python -m unittest -v test_rubric.pyAuthored answer and judgement fixtures; no model responses or human-rating study
Coverage: 6 graded of 8 candidates; 2 pending
Completed rubric decisions: accepted=3; rejected=3
complete: accepted
paraphrase: accepted
negation: accepted
unsupported-guarantee: rejected (support)
missing-verification: rejected (verification)
wrong-command: rejected (command, support)
uncertain-wording: pending (missing or uncertain criterion judgement)
not-reviewed: pending (no authored review fixture)
Pending judgements remain outside the scored subset and visible in coverage.
Quote matching validates excerpts; semantic ratings are supplied judgements.Three accepted and three rejected decisions describe the six completed assessments. Two of eight candidates still await review. Reporting only the accepted share among completed assessments would hide that coverage gap.
The intentionally bad answers fail the rubric while the implementation tests pass. Tests check the unsupported guarantee, missing verification, old command syntax, pending coverage and changed answer/source/rubric bindings. Removing the pending guard must not turn an unfinished review into acceptance.
The full result retains all eight decision records, including the pending reasons. Both candidate and judgement fixtures are development material. Use the existing held-out dataset lesson before interpreting a later system comparison as a result beyond these examples.
Design exercise: a stakeholder asks to remove the database warning to shorten answers. Identify the task requirement this changes, the rubric and annotation revisions needed, and the contrast case that must change. Explain why deleting the failed example while keeping the old rubric would conceal the change rather than justify it.
Failure cases
An answer can contain every requested point and still fail because it invents a zero-downtime promise. An answer that omits verification fails a different requirement. If review is uncertain or stale, the program keeps it pending rather than manufacturing a quality verdict.
Check the input and failure boundaries
| Trigger | Required behavior |
|---|---|
| Unsupported promise despite required content | Reject an unsupported guarantee even when other task points are present. |
| Missing verification | Reject an answer that omits the required post-rollback check. |
| Old command syntax | Retain the task failure instead of accepting a plausible older command. |
| Uncertain or missing review | Keep the judgement pending. |
| Pending reviews disappear from coverage | Report unfinished review alongside completed and executed cases. |
| Stale answer judgement | Refuse reuse when the answer text has changed. |
| Changed source or rubric | Refuse a judgement bound to another source or criterion revision. |
| Unacknowledged label edit | Require an explicit expectation revision for changed annotations. |
| Missing criterion | Keep the review pending when a required criterion is absent. |
| Mismatched source quote | Refuse an excerpt that does not match the criterion and supplied source. |
| Question differs from rubric task | Refuse the old task rubric for a changed question. |
| Zero completed reviews mistaken for quality | Report zero review coverage and no quality summary. |
Deployment
Keep rubric, source and annotation revisions with the answers they describe. Before using the rubric in a release process, review new task cases and decide how pending reviews affect that release. A later automated judge must be calibrated against reviewed examples; this annotation evaluator does not implement that judge.
Scaling
The program can process more annotation records, but the limiting work is obtaining consistent judgements. Sample disagreements by criterion and preserve unresolved cases before expanding review throughput. Faster execution of saved labels does not measure reviewer agreement or automated-judge quality.
Key takeaways
- A rubric expresses task requirements beyond command inclusion.
- Judgements apply to specific answer, source and rubric revisions.
- Execution coverage and completed-review coverage are separate quantities.
Read the explanation
A missing or uncertain judgement keeps the answer pending and outside the scored subset. It is not silently counted as a pass or failure.
Best practices
- Write contrast answers that isolate one criterion at a time.
- Require a source excerpt for a supported or unsupported judgement.
- Revise task requirements explicitly rather than deleting inconvenient failed examples.
References
- The lab’s task-specific rubric defines the criteria, source excerpts and decision rule used in this exercise.
- The supplied judgement annotations retain the reviewed answer bindings and unresolved judgements.
- Create an evaluation set and a failure taxonomy introduces the reused runner and its pass, fail and incomplete outcomes.