aiml

Building MLOps

Take one user-return prediction application from a manual baseline through reproducible training, local release controls and outcome monitoring.

What is MLOps?

MLOps is the practice of developing, releasing and monitoring machine-learning applications through repeatable data and model workflows. It connects training code and data to versioned model artifacts, release checks and observations from use.

Follow the supplied user-return prediction application: turn dated events into training data, compare a candidate with a baseline, then trace the selected model through serving and outcome monitoring. The release and failure checks run locally; the course does not claim an operating production fleet.

The application

At noon UTC, estimate whether an existing user returns in the following 24 hours.

A product analyst reviews daily return estimates to plan a retention experiment. Predictions do not send messages or take actions on users.

Input: The number of events received before prediction in the preceding 24 hours. The user ID identifies the observation; it is not a model feature.

Output: A return probability and a binary prediction using the declared threshold.

Outcome: A finalized return label, ready after the outcome window and a two-hour reporting delay in this fictional fixture.

Before you start

Build the data, training, release and monitoring workflow around one supervised user-return model. Each stage keeps the same application, feature definition and saved artifacts.

Python and SQL prerequisites
Read Python functions and dictionaries, run a script and unit test, and follow a SQL SELECT with filtering and joins. The current labs run on a local CPU.
Machine-learning prerequisites
Understand that features are inputs, labels are target outcomes, and fitting adjusts a model using training examples. Review the split and metric examples below if these ideas need a refresher.

Introduced in the course. Prediction-time data, delayed labels, historical features, run tracking, pipeline retries, model registration, release controls and separate service/data/outcome monitoring are explained before use.

Review three MLOps examplesEvaluation boundaries, prediction metrics and data availability
  1. Keep evaluation data separate

    Training and selection must not use the final test outcomes. For time-dependent tasks, split boundaries must also match when predictions will be made.

    Training → fit parameters
    Validation → choose settings
    Test → evaluate the chosen model
  2. Read a confusion matrix

    Accuracy counts all correct decisions. Recall measures the fraction of actual positives found. These illustrative counts are a refresher, not results from a course lab.

    Illustrative counts: TP=8, FN=2, FP=1, TN=9
    Accuracy = (8+9)/20 = 85%
    Recall = 8/(8+2) = 80%
  3. Check what was available

    An event can happen before prediction and still arrive too late to be used. A historical feature must respect both event time and arrival time.

    Event:      09:00 UTC
    Prediction: 09:10 UTC
    Arrival:    09:20 UTC
    Exclude this event from that prediction.
Course termsShort definitions linked to their worked implementation
Feature, label and prediction

A feature is an input available when a prediction is made. A label records the outcome the model should learn to predict. A prediction is the model’s estimated outcome.

Open the worked lesson →
Training, validation and test sets

Training data fits parameters. Validation data guides settings and selection. Test data evaluates the chosen model after those decisions.

Open the worked lesson →
Metric and decision cost

A metric summarizes prediction behavior. A decision-cost policy assigns explicit consequences to errors; equal accuracy can conceal different costs.

Open the worked lesson →
Leakage and evaluation split

Leakage gives training or evaluation information that would not be available in the intended prediction setting. Time and entity boundaries help define which examples may be compared.

Open the worked lesson →
Event time, arrival time and label delay

Event time is when something happened; arrival time is when the system received it. A label may become known only after an outcome window and further reporting delay.

Open the worked lesson →
Schema validation and quarantine

Validation checks records against explicit field, type and consistency rules. Quarantine retains rejected records and reasons so they can be investigated.

Open the worked lesson →
Point-in-time feature

A feature reconstructed from values available at the prediction timestamp. Later events and later-published corrections cannot silently alter what the historical model could have known.

Open the worked lesson →
Freshness and training-serving parity

Freshness measures whether an input is recent enough for its contract. Parity compares training and serving calculations under the same definitions, inputs and time rules.

Open the worked lesson →
Data and feature version

A version identifies the dataset, schema and feature definition used together. Content hashes detect changed bytes; they do not establish whether the data is appropriate.

Open the worked lesson →
Training recipe and run

A recipe records supported inputs, configuration and dependencies. A run is one execution attempt with its own identity, logs, status and resulting files.

Open the worked lesson →
Artifact and lineage

An artifact is a retained result, such as model parameters or an evaluation report. Lineage links that result to its inputs and execution. A report cannot replace the reloadable model.

Open the worked lesson →
Model registry and release

A model registry records model versions and supporting records. A release selects a compatible version for a serving environment under explicit promotion and rollback rules.

Explore model registry and release →
Monitoring, drift and retraining

Monitoring compares incoming data and delayed outcomes with the operating contract. A changed distribution is drift; it is a signal to investigate before deciding whether to retrain.

Explore monitoring, drift and retraining →

The pipeline this course builds

At noon UTC, estimate whether an existing user returns in the following 24 hours. A product analyst reviews daily return estimates to plan a retention experiment. Predictions do not send messages or take actions on users.

Current state: One local application connects event data, historical features, actual fitting, MLflow tracking and registration, Prefect execution, batch/HTTP predictions and release controls.

The fixture is synthetic. Gates and costs are illustrative; local measurements do not establish product usefulness, service capacity or a cloud deployment.

One local application · data to predictions and delayed outcomes

Dataset + featuresvalidated and versionedTrain + evaluaterecord inputs and outcomesModel registryimmutable model versionReview + releasepromotion and rollback rulesServe predictionsmatching feature contractMonitor outcomesfreshness, errors and labelstraining inputcandidateselected versionrolloutobservationsDataset + featuresvalidated and versionedTrain + evaluaterecord inputs and outcomesModel registryimmutable model versionReview + releasepromotion and rollback rulesServe predictionsmatching feature contractMonitor outcomesfreshness, errors and labels
Monitoring supplies measurements for a retraining or rollback decision. It does not automatically authorize a new model. The workshops retain the same feature contract and model bundle throughout.

What you'll learn

  • Define prediction time, data availability and delayed labels; train and reload a baseline under explicit evaluation rules.
  • Version historical features and datasets, then track real local training with MLflow.
  • Automate the same workflow with Prefect, test failure recovery and gate candidates.
  • Register, serve, stage and roll back compatible model bundles locally.
  • Monitor request failures, delayed outcomes and input drift before choosing a response.

Lessons

Module 1 · 2 lessons · about 50 minDefine the application

Set the prediction cutoff, available data, outcome and evaluation population.

  1. Use only data available when the prediction is madeRead

    Separate event time, arrival time and label readiness so a historical return prediction uses only available inputs and finalized outcomes.

    25 min readLabQuiz

  2. Choose evaluation dates for the return modelRead

    Match the evaluation to future predictions for existing users, keep the declared date windows and exclude later outcomes from earlier decisions.

    25 min readLabQuiz

Module 2 · 2 lessons · about 50 minRun a manual workflow

Fit, evaluate, save and reload one user-return model before adding operational tools.

  1. Train, save and reload the user-return modelRead

    Prepare historical user-return examples, fit and evaluate a baseline, then reload its saved scaler and model to predict in a fresh process.

    25 min readLabQuiz

  2. Choose a decision threshold using explicit error costsRead

    Compare three thresholds on the same validation predictions while keeping the fitted model and final test set unchanged.

    25 min readLabQuiz

Module 3 · 5 lessons · about 2 hoursMake the workflow reproducible

Validate and version the same inputs and feature contract; track the actual training run.

  1. Validate incoming events before preparing training dataRead

    Preserve original events, quarantine invalid records and block the entire training batch when field, identity or temporal checks fail.

    25 min readLabQuiz

  2. Reconstruct the feature used by an earlier predictionRead

    Store feature publication history and reconstruct the count available to an earlier return prediction, without applying later corrections.

    25 min readLabQuiz

  3. Compare historical and current feature calculationsRead

    Compare the same feature version, cutoff and arrival policy, then distinguish a calculation mismatch from stale or unavailable feature data.

    25 min readLabQuiz

  4. Version the dataset and feature contract togetherRead

    Bind data, schemas and feature definitions to explicit versions in SQLite; refuse identity conflicts and roll back a partial binding attempt.

    25 min readLabQuiz

  5. Track a training run and reload its modelRead

    Connect the dataset, source, fitting settings and validation result to one MLflow run, then retrieve the model another process can use.

    30 min readLabQuiz

Module 4 · 4 lessons · about 2 hoursAutomate and test the workflow

Connect the existing steps in Prefect, recover bounded failures and enforce an explicit quality gate.

  1. Connect preparation, training and evaluation with PrefectRead

    Automate the manual return-model workflow while keeping the accepted events, historical features and fitted model connected.

    30 min readLabQuiz

  2. Retry a failed step without repeating the experimentRead

    Recover a failed write of the completion record without repeating training; reuse a completed attempt only when its inputs and artifacts still match.

    30 min readLabQuiz

  3. Test pipeline changes before producing a candidateRead

    Run unit and integration checks through one local CI command, retain discovered test counts and diagnostics, and propagate a failed result.

    30 min readLabQuiz

  4. Evaluate a candidate against a declared release policyRead

    Recompute validation from the saved model, bind the decision to a policy and keep failed candidates available for inspection.

    30 min readLabQuiz

Module 5 · 4 lessons · about 2 hoursRelease the model

Select a compatible registered bundle, produce predictions and exercise local rollout and rollback.

  1. Register and select an explicit model versionRead

    Package the fitted return model with its contract and gate decision, then select a checked version through a local MLflow alias.

    30 min readLabQuiz

  2. Produce daily predictions and expose an optional HTTP interfaceRead

    Use one validated predictor for the analyst’s batch and a loopback HTTP endpoint, with the selected bundle attached to every prediction.

    30 min readLabQuiz

  3. Stage a candidate with an explicit traffic decisionRead

    Route a repeatable subset of observation IDs to a candidate bundle, retain routing generations and reject stale configuration updates.

    30 min readLabQuiz

  4. Roll back to a compatible model bundleRead

    Observe a controlled candidate failure, restore a checked earlier routing generation and confirm predictions use the stable bundle.

    30 min readLabQuiz

Module 6 · 4 lessons · about 2 hoursOperate and improve it

Separate service health, delayed outcomes and changed inputs before attempting recovery or retraining.

  1. Monitor prediction requests and input failuresRead

    Collect bounded request counts, validation failures and local latency measurements; keep these service signals separate from prediction quality.

    30 min readLabQuiz

  2. Evaluate predictions after return labels become finalRead

    Join predictions to their original observations, keep unfinished outcomes pending and report quality by the bundle that produced them.

    30 min readLabQuiz

  3. Investigate changed inputs before choosing a responseRead

    Compare a retained feature distribution, separate invalid data from a genuine change and consult finalized outcomes before proposing retraining.

    30 min readLabQuiz

  4. Recover the input path and evaluate a new candidateRead

    Return to the valid source after the controlled data fault, run the checked pipeline again and keep traffic unchanged until a release decision.

    30 min readLabQuiz