Match the evaluation to tomorrow’s predictions for existing users, and keep future outcomes out of earlier decisions.
Problem
When you need this. The analyst will score users already known to the application. The useful evaluation question is whether a model fitted on earlier observations can predict their later returns. Randomly mixing dates would answer a different question: performance on a shuffled sample of this historical dataset.
Use the same 240 observations as the manual workflow. Each user has several prediction dates, and the target always covers the following 24 hours. This workshop checks that the declared date windows produce the baseline’s 144 training, 48 validation and 48 test examples. It also checks when the last training label becomes available.
Concepts
An observation is one prediction opportunity: a user ID and prediction time. It is different from an event, such as a visit received by the application. The same user may legitimately appear in multiple date windows when the intended task concerns future observations for existing users.
An entity split instead keeps each user ID in one split only. Use it when the model must score users it has not seen. It is not automatically a stronger evaluation: it changes which population the score describes. The original split workshop shows that comparison; this application chooses the existing-user target.
Training covers January 3–8, validation January 10–11, and final test January 13–14. January 9 and 12 leave gaps between the 24-hour outcome windows. A gap alone is not enough: the training labels must also be ready before the first validation prediction. The code compares the latest training label’s ready time with that first validation prediction. Size each gap from the label’s ready time, not only from the end of the outcome window.
Use validation for decisions such as comparing thresholds. Do not look at final test outcomes while you make those choices. This workshop reports the test window’s size and dates but does not score predictions against its labels.
Architecture
Working code
Download the shared application and tests, unzip it and enter mlops-return-workflow/. Keep the same directory throughout the course. Create the environment once. This workshop uses the standard library.
python3.12 -m venv .venv
source .venv/bin/activateOn PowerShell, activate with .venv\Scripts\Activate.ps1.
The opening commands read scenario.json; no fitted model is needed.
Assign observations by prediction date
The adapter reuses the published split algorithm with this project’s date boundaries. It then checks every assignment against the manual workflow’s partition function. If the two definitions assign any observation differently, that assertion fails instead of silently producing a different training cohort.
def split_report(source):
fixture = w.strict_json(source)
# Reuse the published partition algorithm with this application's date windows.
for name, stamp in {'START':'2026-01-03', 'TRAIN_END':'2026-01-09',
'VALIDATION_START':'2026-01-10', 'VALIDATION_END':'2026-01-12',
'TEST_START':'2026-01-13', 'END':'2026-01-15'}.items():
setattr(partitions, name, datetime.fromisoformat(stamp).replace(tzinfo=timezone.utc))
rows = [dict(id=o['id'], entity=o['entity'], observed_at=o['predict_at']) for o in fixture['observations']]
parts = partitions.partition(rows, 'future')
assigned = {row['id']:name for name, items in parts.items() for row in items}
assert all(assigned[o['id']]==w.partition(o['predict_at']) for o in fixture['observations'])
report = partitions.describe(parts)
report['target'] = 'later observations for existing users'
report['maximum_training_label_ready_at'] = max(l['ready_at'] for l in fixture['labels'] if assigned[l['id']]=='training')
report['earliest_validation_prediction'] = report['ranges']['validation']['first']
report['training_labels_ready_before_validation'] = (w.availability.instant(report['maximum_training_label_ready_at']) < w.availability.instant(report['earliest_validation_prediction']))
return reportpython adapt.py splits{
"rows": {
"training": 144,
"validation": 48,
"test": 48,
"excluded": 0
},
"entities": {
"training": 24,
"validation": 24,
"test": 24
},
"train_validation_shared_entities": 24,
"train_test_shared_entities": 24,
"ranges": {
"training": {
"first": "2026-01-03T12:00:00Z",
"last": "2026-01-08T12:00:00Z"
},
"validation": {
"first": "2026-01-10T12:00:00Z",
"last": "2026-01-11T12:00:00Z"
},
"test": {
"first": "2026-01-13T12:00:00Z",
"last": "2026-01-14T12:00:00Z"
}
},
"target": "later observations for existing users",
"maximum_training_label_ready_at": "2026-01-09T14:00:00Z",
"earliest_validation_prediction": "2026-01-10T12:00:00Z",
"training_labels_ready_before_validation": true
}Read the shared-entity counts alongside the row counts. All 24 users are present on both sides of the training/validation boundary. That is expected for this prediction population; it would be a defect if the stated target were unseen users. Write the target population into the report, as the target field does here, so a reader can tell whether shared users are expected.
The output above is from a saved local Python run in recorded-adaptations/; your commands use work/ and leave the recorded outputs unchanged. Read the actual command records. Use a fresh work directory for another attempt.
Failure cases
Run test_time_windows_match_manual_cohort_and_labels_are_ready. It checks all three partition sizes and the training-label readiness comparison. The assertion that compares the two partition implementations also runs inside python adapt.py splits.
The underlying partition validator refuses duplicate observation IDs and timestamps without a UTC offset. A valid split still cannot prove that a feature was available at prediction time; the earlier availability workshop checks event arrival and label delay.
python -m unittest -v test_workflow.py test_foundations.pyRunning and scaling the workshop
This is a local inspection command. It does not fit a model or choose a deployment threshold. Its output is the split policy used by the shared application.
For a larger event history, put the same date conditions in the storage query. Then test rows that fall on each boundary. A faster query with different cutoff rules is a different experiment.
Key takeaways
- Choose the evaluation population before selecting a split.
- Keep the prediction date, outcome window and label-readiness time distinct.
- The later training and evaluation commands must consume these same assignments.
Best practices
Store the boundaries with the training recipe. Change the policy explicitly when the deployment population changes.
Check boundary examples and label delay. Non-overlapping date windows alone do not show that leakage is absent.
Check your understanding
Should a user appearing in both training and validation always fail this evaluation?
Show the explanation
No. This application predicts later returns for existing users. Shared users match that target. If the application must generalize to unseen users, define a different entity-aware evaluation and report it separately.
Read the explanation
The date windows evaluate future observations for users already present in the application population. The same users can appear in each period.