aiml

Lesson 2 of 21Module 1

Choose evaluation dates for the return model

Match the evaluation to future predictions for existing users, keep the declared date windows and exclude later outcomes from earlier decisions.

Prerequisites

Python functions, dictionaries, UTC timestamps and basic supervised learning. Start with the course application brief.

Preparation and term definitions

By the end, you can

  • Choose evaluation dates for later observations of existing users.
  • Keep outcome windows and label readiness separate from feature cutoffs.
  • Carry the same split assignments into training and evaluation.
Runtime
Python 3.12+; local CPU
Dependencies
Python standard library and SQLite

Lab code, tests and fixtures (ZIP, 1.6 MB)Quiz check your understanding

Progress is saved in this browser.

Match the evaluation to tomorrow’s predictions for existing users, and keep future outcomes out of earlier decisions.

Problem

When you need this. The analyst will score users already known to the application. The useful evaluation question is whether a model fitted on earlier observations can predict their later returns. Randomly mixing dates would answer a different question: performance on a shuffled sample of this historical dataset.

Use the same 240 observations as the manual workflow. Each user has several prediction dates, and the target always covers the following 24 hours. This workshop checks that the declared date windows produce the baseline’s 144 training, 48 validation and 48 test examples. It also checks when the last training label becomes available.

Concepts

An observation is one prediction opportunity: a user ID and prediction time. It is different from an event, such as a visit received by the application. The same user may legitimately appear in multiple date windows when the intended task concerns future observations for existing users.

An entity split instead keeps each user ID in one split only. Use it when the model must score users it has not seen. It is not automatically a stronger evaluation: it changes which population the score describes. The original split workshop shows that comparison; this application chooses the existing-user target.

Training covers January 3–8, validation January 10–11, and final test January 13–14. January 9 and 12 leave gaps between the 24-hour outcome windows. A gap alone is not enough: the training labels must also be ready before the first validation prediction. The code compares the latest training label’s ready time with that first validation prediction. Size each gap from the label’s ready time, not only from the end of the outcome window.

Use validation for decisions such as comparing thresholds. Do not look at final test outcomes while you make those choices. This workshop reports the test window’s size and dates but does not score predictions against its labels.

Architecture

Observationsuser + prediction timeAssign by datedeclared UTC windowsTrainingJanuary 3–8ValidationJanuary 10–11Final testJanuary 13–14Observationsuser + prediction timeAssign by datedeclared UTC windowsTrainingJanuary 3–8ValidationJanuary 10–11Final testJanuary 13–14
The split follows prediction dates. Shared users are intentional; the target is later observations for existing users.

Working code

Download the shared application and tests, unzip it and enter mlops-return-workflow/. Keep the same directory throughout the course. Create the environment once. This workshop uses the standard library.

Create the local environmentTerminal · bash
python3.12 -m venv .venv
source .venv/bin/activate

On PowerShell, activate with .venv\Scripts\Activate.ps1.

The opening commands read scenario.json; no fitted model is needed.

Assign observations by prediction date

The adapter reuses the published split algorithm with this project’s date boundaries. It then checks every assignment against the manual workflow’s partition function. If the two definitions assign any observation differently, that assertion fails instead of silently producing a different training cohort.

Source excerpt: split_reportSource · pythonDownload
def split_report(source):
    fixture = w.strict_json(source)
    # Reuse the published partition algorithm with this application's date windows.
    for name, stamp in {'START':'2026-01-03', 'TRAIN_END':'2026-01-09',
                        'VALIDATION_START':'2026-01-10', 'VALIDATION_END':'2026-01-12',
                        'TEST_START':'2026-01-13', 'END':'2026-01-15'}.items():
        setattr(partitions, name, datetime.fromisoformat(stamp).replace(tzinfo=timezone.utc))
    rows = [dict(id=o['id'], entity=o['entity'], observed_at=o['predict_at']) for o in fixture['observations']]
    parts = partitions.partition(rows, 'future')
    assigned = {row['id']:name for name, items in parts.items() for row in items}
    assert all(assigned[o['id']]==w.partition(o['predict_at']) for o in fixture['observations'])
    report = partitions.describe(parts)
    report['target'] = 'later observations for existing users'
    report['maximum_training_label_ready_at'] = max(l['ready_at'] for l in fixture['labels'] if assigned[l['id']]=='training')
    report['earliest_validation_prediction'] = report['ranges']['validation']['first']
    report['training_labels_ready_before_validation'] = (w.availability.instant(report['maximum_training_label_ready_at']) < w.availability.instant(report['earliest_validation_prediction']))
    return report
Assign observations by prediction dateTerminal · bash
python adapt.py splits
Actual local application outputRecorded output · jsonDownload
{
  "rows": {
    "training": 144,
    "validation": 48,
    "test": 48,
    "excluded": 0
  },
  "entities": {
    "training": 24,
    "validation": 24,
    "test": 24
  },
  "train_validation_shared_entities": 24,
  "train_test_shared_entities": 24,
  "ranges": {
    "training": {
      "first": "2026-01-03T12:00:00Z",
      "last": "2026-01-08T12:00:00Z"
    },
    "validation": {
      "first": "2026-01-10T12:00:00Z",
      "last": "2026-01-11T12:00:00Z"
    },
    "test": {
      "first": "2026-01-13T12:00:00Z",
      "last": "2026-01-14T12:00:00Z"
    }
  },
  "target": "later observations for existing users",
  "maximum_training_label_ready_at": "2026-01-09T14:00:00Z",
  "earliest_validation_prediction": "2026-01-10T12:00:00Z",
  "training_labels_ready_before_validation": true
}

Read the shared-entity counts alongside the row counts. All 24 users are present on both sides of the training/validation boundary. That is expected for this prediction population; it would be a defect if the stated target were unseen users. Write the target population into the report, as the target field does here, so a reader can tell whether shared users are expected.

The output above is from a saved local Python run in recorded-adaptations/; your commands use work/ and leave the recorded outputs unchanged. Read the actual command records. Use a fresh work directory for another attempt.

Failure cases

Run test_time_windows_match_manual_cohort_and_labels_are_ready. It checks all three partition sizes and the training-label readiness comparison. The assertion that compares the two partition implementations also runs inside python adapt.py splits.

The underlying partition validator refuses duplicate observation IDs and timestamps without a UTC offset. A valid split still cannot prove that a feature was available at prediction time; the earlier availability workshop checks event arrival and label delay.

Run the application failure checksTerminal · bash
python -m unittest -v test_workflow.py test_foundations.py

Running and scaling the workshop

This is a local inspection command. It does not fit a model or choose a deployment threshold. Its output is the split policy used by the shared application.

For a larger event history, put the same date conditions in the storage query. Then test rows that fall on each boundary. A faster query with different cutoff rules is a different experiment.

Key takeaways

  • Choose the evaluation population before selecting a split.
  • Keep the prediction date, outcome window and label-readiness time distinct.
  • The later training and evaluation commands must consume these same assignments.

Best practices

Store the boundaries with the training recipe. Change the policy explicitly when the deployment population changes.

Check boundary examples and label delay. Non-overlapping date windows alone do not show that leakage is absent.

Check your understanding

Should a user appearing in both training and validation always fail this evaluation?

Show the explanation

No. This application predicts later returns for existing users. Shared users match that target. If the application must generalize to unseen users, define a different entity-aware evaluation and report it separately.

The application predicts later returns for existing users. What does its date split test?

Choose one answer, then check your reasoning.

Read the explanation

The date windows evaluate future observations for users already present in the application population. The same users can appear in each period.

References

Written by Arjun Prabhulal.

Module 1

Define the application