aiml

Lesson 19 of 21Module 6

Evaluate predictions after return labels become final

Join predictions to their original observations, keep unfinished outcomes pending and report quality by the bundle that produced them.

Prerequisites

Complete the preceding course lessons in the shared user-return lab; keep their work/ outputs. Python functions and local CLI commands.

Preparation and term definitions

By the end, you can

  • Keep unfinished outcomes pending rather than assigning zero.
  • Join finalized labels to the correct observation and release.
  • Distinguish repeated reports from independent measurements of model quality.
Runtime
Python 3.12+; local CPU
Dependencies
Pinned MLflow, Prefect and FastAPI dependencies in requirements-platform.txt

Lab code, tests and fixtures (ZIP, 1.6 MB)Quiz check your understanding

Progress is saved in this browser.

Join predictions to their original observations, keep unfinished outcomes pending and report quality by the bundle that produced them.

Problem

When you need this. The return target covers the next 24 hours, followed by two hours of reporting delay. Immediately scoring a missing return as label zero would make unfinished outcomes look like model errors.

This workshop reuses the retained prediction IDs and the same separately authored finalized labels. It evaluates them at two fixture times: one before label readiness and one after. The monitoring job admits each label once it is ready and leaves earlier predictions unchanged.

Concepts

A delayed outcome is an observation whose target is not yet final. Pending is a state of knowledge, not a negative label. The join therefore counts pending and unknown observations separately from scored predictions.

The join key is the original observation ID. The join also checks that the user and prediction time match, validates the label and refuses duplicate predictions or labels. The reporting-delay check requires each label to become ready at least 26 hours after prediction time.

Results are grouped by bundle ID because a rollout can produce predictions from different versions. An overall score that mixes versions can hide which release produced an error.

The demonstration uses the same 48 synthetic validation observations that appeared earlier. Changing the as_of time tests when finalized labels are admitted. The eventual 29/48 comes from those same rows, so it is not a new, independent estimate.

Architecture

Prediction + labelsame observation IDCheck ready_at26-hour minimumScore final labelsretain bundle groupPending outcomeno guessed zero labelPrediction + labelsame observation IDCheck ready_at26-hour minimumScore final labelsretain bundle groupPending outcomeno guessed zero label
The join admits finalized outcomes and keeps unfinished labels pending. The actual before/after reports use explicit fixture times.

Working code

Download the shared application and tests, unzip it and enter mlops-return-workflow/. Keep the same directory throughout the course. Create the environment once. Install the pinned platform dependencies for this workshop.

Create the local environmentTerminal · bash
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-platform.txt

On PowerShell, activate with .venv\Scripts\Activate.ps1.

Continue from Monitor prediction requests and input failures; keep the same lab directory and outputs.

Admit a label only after its declared readiness time

The join counts pending observations instead of scoring them. It checks identity before it compares the stored decision with the finalized label.

Source excerpt: evaluate_delayedSource · pythonDownload
def evaluate_delayed(predictions,fixture,as_of):
    w.availability.validate(fixture['observations'],[],fixture['labels'])
    now=w.availability.instant(as_of);labels={x['id']:x for x in fixture['labels']}
    if len(labels)!=len(fixture['labels']):raise ValueError('duplicate label identity')
    observations={o['id']:o for o in fixture['observations']}
    ids=set();scored=[];pending=unknown=0
    for p in predictions:
        if p['id'] in ids:raise ValueError('duplicate prediction identity')
        ids.add(p['id']);o=observations.get(p['id']);label=labels.get(p['id'])
        if o is None or label is None:unknown+=1;continue
        if (p['entity']!=o['entity'] or w.availability.instant(p['predict_at'])!=w.availability.instant(o['predict_at'])):raise ValueError('prediction and observation differ')
        if w.partition(p['predict_at'])=='test':raise ValueError('this workshop preserves the final test')
        if now<w.availability.instant(p['predict_at']):raise ValueError('prediction lies after evaluation time')
        if now<w.availability.instant(label['ready_at']):pending+=1;continue
        # Validate label schema and its full 26-hour readiness contract.
        w.availability.validate([o],[],[label])
        from datetime import timedelta
        if w.availability.instant(label['ready_at'])<w.availability.instant(o['predict_at'])+timedelta(hours=26):
            raise ValueError('label violates the declared reporting delay')
        if type(p['predicted_return']) is not int or p['predicted_return'] not in [0,1]:raise ValueError('invalid prediction decision')
        probability=p['return_probability']
        if type(probability) not in (int,float) or not math.isfinite(probability) or not 0<=probability<=1:raise ValueError('invalid prediction probability')
        scored.append(dict(correct=int(p['predicted_return']==label['value']),bundle_id=p['bundle_id']))
    grouped={key:{'rows':sum(s['bundle_id']==key for s in scored),'correct':sum(s['correct'] for s in scored if s['bundle_id']==key)} for key in sorted({s['bundle_id'] for s in scored})}
    return {'as_of':as_of,'received':len(predictions),'pending':pending,'unknown':unknown,'scored':len(scored),
            'correct':sum(s['correct'] for s in scored),'accuracy':sum(s['correct'] for s in scored)/len(scored) if scored else None,
            'by_bundle':grouped,'final_test_used':False,'population':'the same synthetic validation observations; not a new held-out estimate'}

At January 11, 13:00 UTC, the first day’s 24 predictions are still pending: their 26-hour readiness time is 14:00. An empty scored set has accuracy null, not zero.

Revisit the same predictions after finalization

The second call uses January 13, 14:00 UTC, after all validation labels are ready. No model is retrained and no prediction is rewritten.

Source excerpt: delayed_reportSource · pythonDownload
def delayed_report(root):
    predictions=w.strict_json(need(root,'observed-predictions.json'))['predictions'];fixture=w.strict_json(w.HERE/'scenario.json')
    first=[p for p in predictions if p['predict_at'].startswith('2026-01-10')]
    early=operations.evaluate_delayed(first,fixture,'2026-01-11T13:00:00Z')
    final=operations.evaluate_delayed(predictions,fixture,'2026-01-13T14:00:00Z')
    return {'before_reporting_delay':early,'after_labels_finalize':final}
Revisit the same predictions after finalizationTerminal · bash
python course.py delayed-evaluation --workspace work/course
Actual local application outputRecorded output · jsonDownload
{
  "before_reporting_delay": {
    "as_of": "2026-01-11T13:00:00Z",
    "received": 24,
    "pending": 24,
    "unknown": 0,
    "scored": 0,
    "correct": 0,
    "accuracy": null,
    "by_bundle": {},
    "final_test_used": false,
    "population": "the same synthetic validation observations; not a new held-out estimate"
  },
  "after_labels_finalize": {
    "as_of": "2026-01-13T14:00:00Z",
    "received": 48,
    "pending": 0,
    "unknown": 0,
    "scored": 48,
    "correct": 29,
    "accuracy": 0.6041666666666666,
    "by_bundle": {
      "c93a6048839f362f0ad902ff6b4480f293cc2af0ac593c2c06ffba954b60c90a": {
        "rows": 48,
        "correct": 29
      }
    },
    "final_test_used": false,
    "population": "the same synthetic validation observations; not a new held-out estimate"
  }
}

The later result scores 48 observations, 29 of them correct, and groups them under the bundle that produced them. Both as_of values are fixture times passed to local code; the exercise does not wait several days.

The output above is from a saved local run in recorded-platform/; your commands use work/. Read the actual command records. Use a fresh work directory for another attempt.

Failure cases

test_labels_are_pending_before_ready_and_finalized_later checks three results: the empty scored set at 13:00, the 48 scored rows with 29 correct after finalization and the refusal of duplicate predictions. The equivalent-UTC test also accepts Z and +00:00 representations of the same prediction time.

The function refuses predictions from the reserved final test window. It counts an unknown observation ID as unknown, not as label zero. The fixture assumes a complete collector of finalized outcomes. Real monitoring needs the source system to guarantee that.

Run the application failure checksTerminal · bash
python -m unittest -v test_platform.py

Running and scaling the workshop

A scheduled evaluator can rerun this join as outcomes finalize and store each as_of report. This example runs the two specified times locally and does not create a cloud schedule.

Use time and release cohorts appropriate to the consumer. Track sample sizes and label coverage beside accuracy; a score on a small or selectively finalized subset can mislead. Before you interpret real retention effects, add human review and product experiments.

Key takeaways

  • Pending outcomes are not negative outcomes.
  • Join by observation identity and retain the producing release.
  • A repeated report over the same observations is not a new validation sample.

Best practices

Report scored, pending and unknown counts together.

Preserve original predictions when labels or later reports arrive.

Read accuracy together with scored and pending. In the early report, null accuracy with 24 pending rows means the labels are not ready yet; it says nothing about the model.

Pass as_of explicitly, as delayed_report does. A rerun with the same time and fixture then admits the same labels and returns the same counts.

Check your understanding

Why does the early report return null accuracy instead of zero?

Show the explanation

No labels have become final, so there is no denominator for accuracy. Zero would falsely imply that completed outcomes were scored and all predictions were wrong.

A prediction’s finalized label has not arrived. How should outcome evaluation treat it?

Choose one answer, then check your reasoning.

Read the explanation

Retain the prediction as pending until its finalized label is available. The scored denominator includes only eligible outcomes.

References

Written by Arjun Prabhulal.

Module 6

Operate and improve it