Join predictions to their original observations, keep unfinished outcomes pending and report quality by the bundle that produced them.
Problem
When you need this. The return target covers the next 24 hours, followed by two hours of reporting delay. Immediately scoring a missing return as label zero would make unfinished outcomes look like model errors.
This workshop reuses the retained prediction IDs and the same separately authored finalized labels. It evaluates them at two fixture times: one before label readiness and one after. The monitoring job admits each label once it is ready and leaves earlier predictions unchanged.
Concepts
A delayed outcome is an observation whose target is not yet final. Pending is a state of knowledge, not a negative label. The join therefore counts pending and unknown observations separately from scored predictions.
The join key is the original observation ID. The join also checks that the user and prediction time match, validates the label and refuses duplicate predictions or labels. The reporting-delay check requires each label to become ready at least 26 hours after prediction time.
Results are grouped by bundle ID because a rollout can produce predictions from different versions. An overall score that mixes versions can hide which release produced an error.
The demonstration uses the same 48 synthetic validation observations that appeared earlier. Changing the as_of time tests when finalized labels are admitted. The eventual 29/48 comes from those same rows, so it is not a new, independent estimate.
Architecture
Working code
Download the shared application and tests, unzip it and enter mlops-return-workflow/. Keep the same directory throughout the course. Create the environment once. Install the pinned platform dependencies for this workshop.
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-platform.txtOn PowerShell, activate with .venv\Scripts\Activate.ps1.
Continue from Monitor prediction requests and input failures; keep the same lab directory and outputs.
Admit a label only after its declared readiness time
The join counts pending observations instead of scoring them. It checks identity before it compares the stored decision with the finalized label.
def evaluate_delayed(predictions,fixture,as_of):
w.availability.validate(fixture['observations'],[],fixture['labels'])
now=w.availability.instant(as_of);labels={x['id']:x for x in fixture['labels']}
if len(labels)!=len(fixture['labels']):raise ValueError('duplicate label identity')
observations={o['id']:o for o in fixture['observations']}
ids=set();scored=[];pending=unknown=0
for p in predictions:
if p['id'] in ids:raise ValueError('duplicate prediction identity')
ids.add(p['id']);o=observations.get(p['id']);label=labels.get(p['id'])
if o is None or label is None:unknown+=1;continue
if (p['entity']!=o['entity'] or w.availability.instant(p['predict_at'])!=w.availability.instant(o['predict_at'])):raise ValueError('prediction and observation differ')
if w.partition(p['predict_at'])=='test':raise ValueError('this workshop preserves the final test')
if now<w.availability.instant(p['predict_at']):raise ValueError('prediction lies after evaluation time')
if now<w.availability.instant(label['ready_at']):pending+=1;continue
# Validate label schema and its full 26-hour readiness contract.
w.availability.validate([o],[],[label])
from datetime import timedelta
if w.availability.instant(label['ready_at'])<w.availability.instant(o['predict_at'])+timedelta(hours=26):
raise ValueError('label violates the declared reporting delay')
if type(p['predicted_return']) is not int or p['predicted_return'] not in [0,1]:raise ValueError('invalid prediction decision')
probability=p['return_probability']
if type(probability) not in (int,float) or not math.isfinite(probability) or not 0<=probability<=1:raise ValueError('invalid prediction probability')
scored.append(dict(correct=int(p['predicted_return']==label['value']),bundle_id=p['bundle_id']))
grouped={key:{'rows':sum(s['bundle_id']==key for s in scored),'correct':sum(s['correct'] for s in scored if s['bundle_id']==key)} for key in sorted({s['bundle_id'] for s in scored})}
return {'as_of':as_of,'received':len(predictions),'pending':pending,'unknown':unknown,'scored':len(scored),
'correct':sum(s['correct'] for s in scored),'accuracy':sum(s['correct'] for s in scored)/len(scored) if scored else None,
'by_bundle':grouped,'final_test_used':False,'population':'the same synthetic validation observations; not a new held-out estimate'}At January 11, 13:00 UTC, the first day’s 24 predictions are still pending: their 26-hour readiness time is 14:00. An empty scored set has accuracy null, not zero.
Revisit the same predictions after finalization
The second call uses January 13, 14:00 UTC, after all validation labels are ready. No model is retrained and no prediction is rewritten.
def delayed_report(root):
predictions=w.strict_json(need(root,'observed-predictions.json'))['predictions'];fixture=w.strict_json(w.HERE/'scenario.json')
first=[p for p in predictions if p['predict_at'].startswith('2026-01-10')]
early=operations.evaluate_delayed(first,fixture,'2026-01-11T13:00:00Z')
final=operations.evaluate_delayed(predictions,fixture,'2026-01-13T14:00:00Z')
return {'before_reporting_delay':early,'after_labels_finalize':final}python course.py delayed-evaluation --workspace work/course{
"before_reporting_delay": {
"as_of": "2026-01-11T13:00:00Z",
"received": 24,
"pending": 24,
"unknown": 0,
"scored": 0,
"correct": 0,
"accuracy": null,
"by_bundle": {},
"final_test_used": false,
"population": "the same synthetic validation observations; not a new held-out estimate"
},
"after_labels_finalize": {
"as_of": "2026-01-13T14:00:00Z",
"received": 48,
"pending": 0,
"unknown": 0,
"scored": 48,
"correct": 29,
"accuracy": 0.6041666666666666,
"by_bundle": {
"c93a6048839f362f0ad902ff6b4480f293cc2af0ac593c2c06ffba954b60c90a": {
"rows": 48,
"correct": 29
}
},
"final_test_used": false,
"population": "the same synthetic validation observations; not a new held-out estimate"
}
}The later result scores 48 observations, 29 of them correct, and groups them under the bundle that produced them. Both as_of values are fixture times passed to local code; the exercise does not wait several days.
The output above is from a saved local run in recorded-platform/; your commands use work/. Read the actual command records. Use a fresh work directory for another attempt.
Failure cases
test_labels_are_pending_before_ready_and_finalized_later checks three results: the empty scored set at 13:00, the 48 scored rows with 29 correct after finalization and the refusal of duplicate predictions. The equivalent-UTC test also accepts Z and +00:00 representations of the same prediction time.
The function refuses predictions from the reserved final test window. It counts an unknown observation ID as unknown, not as label zero. The fixture assumes a complete collector of finalized outcomes. Real monitoring needs the source system to guarantee that.
python -m unittest -v test_platform.pyRunning and scaling the workshop
A scheduled evaluator can rerun this join as outcomes finalize and store each as_of report. This example runs the two specified times locally and does not create a cloud schedule.
Use time and release cohorts appropriate to the consumer. Track sample sizes and label coverage beside accuracy; a score on a small or selectively finalized subset can mislead. Before you interpret real retention effects, add human review and product experiments.
Key takeaways
- Pending outcomes are not negative outcomes.
- Join by observation identity and retain the producing release.
- A repeated report over the same observations is not a new validation sample.
Best practices
Report scored, pending and unknown counts together.
Preserve original predictions when labels or later reports arrive.
Read accuracy together with scored and pending. In the early report, null accuracy with 24 pending rows means the labels are not ready yet; it says nothing about the model.
Pass as_of explicitly, as delayed_report does. A rerun with the same time and fixture then admits the same labels and returns the same counts.
Check your understanding
Why does the early report return null accuracy instead of zero?
Show the explanation
No labels have become final, so there is no denominator for accuracy. Zero would falsely imply that completed outcomes were scored and all predictions were wrong.
Read the explanation
Retain the prediction as pending until its finalized label is available. The scored denominator includes only eligible outcomes.