Building MLOps
Take one user-return prediction application from a manual baseline through reproducible training, local release controls and outcome monitoring.
What is MLOps?
MLOps is the practice of developing, releasing and monitoring machine-learning applications through repeatable data and model workflows. It connects training code and data to versioned model artifacts, release checks and observations from use.
Follow the supplied user-return prediction application: turn dated events into training data, compare a candidate with a baseline, then trace the selected model through serving and outcome monitoring. The release and failure checks run locally; the course does not claim an operating production fleet.
The application
At noon UTC, estimate whether an existing user returns in the following 24 hours.
A product analyst reviews daily return estimates to plan a retention experiment. Predictions do not send messages or take actions on users.
Input: The number of events received before prediction in the preceding 24 hours. The user ID identifies the observation; it is not a model feature.
Output: A return probability and a binary prediction using the declared threshold.
Outcome: A finalized return label, ready after the outcome window and a two-hour reporting delay in this fictional fixture.
Before you start
Build the data, training, release and monitoring workflow around one supervised user-return model. Each stage keeps the same application, feature definition and saved artifacts.
- Python and SQL prerequisites
- Read Python functions and dictionaries, run a script and unit test, and follow a SQL SELECT with filtering and joins. The current labs run on a local CPU.
- Machine-learning prerequisites
- Understand that features are inputs, labels are target outcomes, and fitting adjusts a model using training examples. Review the split and metric examples below if these ideas need a refresher.
Introduced in the course. Prediction-time data, delayed labels, historical features, run tracking, pipeline retries, model registration, release controls and separate service/data/outcome monitoring are explained before use.
Review three MLOps examplesEvaluation boundaries, prediction metrics and data availability
Keep evaluation data separate
Training and selection must not use the final test outcomes. For time-dependent tasks, split boundaries must also match when predictions will be made.
Training → fit parameters Validation → choose settings Test → evaluate the chosen modelRead a confusion matrix
Accuracy counts all correct decisions. Recall measures the fraction of actual positives found. These illustrative counts are a refresher, not results from a course lab.
Illustrative counts: TP=8, FN=2, FP=1, TN=9 Accuracy = (8+9)/20 = 85% Recall = 8/(8+2) = 80%Check what was available
An event can happen before prediction and still arrive too late to be used. A historical feature must respect both event time and arrival time.
Event: 09:00 UTC Prediction: 09:10 UTC Arrival: 09:20 UTC Exclude this event from that prediction.
Course termsShort definitions linked to their worked implementation
- Feature, label and prediction
A feature is an input available when a prediction is made. A label records the outcome the model should learn to predict. A prediction is the model’s estimated outcome.
Open the worked lesson →- Training, validation and test sets
Training data fits parameters. Validation data guides settings and selection. Test data evaluates the chosen model after those decisions.
Open the worked lesson →- Metric and decision cost
A metric summarizes prediction behavior. A decision-cost policy assigns explicit consequences to errors; equal accuracy can conceal different costs.
Open the worked lesson →- Leakage and evaluation split
Leakage gives training or evaluation information that would not be available in the intended prediction setting. Time and entity boundaries help define which examples may be compared.
Open the worked lesson →- Event time, arrival time and label delay
Event time is when something happened; arrival time is when the system received it. A label may become known only after an outcome window and further reporting delay.
Open the worked lesson →- Schema validation and quarantine
Validation checks records against explicit field, type and consistency rules. Quarantine retains rejected records and reasons so they can be investigated.
Open the worked lesson →- Point-in-time feature
A feature reconstructed from values available at the prediction timestamp. Later events and later-published corrections cannot silently alter what the historical model could have known.
Open the worked lesson →- Freshness and training-serving parity
Freshness measures whether an input is recent enough for its contract. Parity compares training and serving calculations under the same definitions, inputs and time rules.
Open the worked lesson →- Data and feature version
A version identifies the dataset, schema and feature definition used together. Content hashes detect changed bytes; they do not establish whether the data is appropriate.
Open the worked lesson →- Training recipe and run
A recipe records supported inputs, configuration and dependencies. A run is one execution attempt with its own identity, logs, status and resulting files.
Open the worked lesson →- Artifact and lineage
An artifact is a retained result, such as model parameters or an evaluation report. Lineage links that result to its inputs and execution. A report cannot replace the reloadable model.
Open the worked lesson →- Model registry and release
A model registry records model versions and supporting records. A release selects a compatible version for a serving environment under explicit promotion and rollback rules.
Explore model registry and release →- Monitoring, drift and retraining
Monitoring compares incoming data and delayed outcomes with the operating contract. A changed distribution is drift; it is a signal to investigate before deciding whether to retrain.
Explore monitoring, drift and retraining →
The pipeline this course builds
At noon UTC, estimate whether an existing user returns in the following 24 hours. A product analyst reviews daily return estimates to plan a retention experiment. Predictions do not send messages or take actions on users.
Current state: One local application connects event data, historical features, actual fitting, MLflow tracking and registration, Prefect execution, batch/HTTP predictions and release controls.
The fixture is synthetic. Gates and costs are illustrative; local measurements do not establish product usefulness, service capacity or a cloud deployment.
One local application · data to predictions and delayed outcomes
What you'll learn
- Define prediction time, data availability and delayed labels; train and reload a baseline under explicit evaluation rules.
- Version historical features and datasets, then track real local training with MLflow.
- Automate the same workflow with Prefect, test failure recovery and gate candidates.
- Register, serve, stage and roll back compatible model bundles locally.
- Monitor request failures, delayed outcomes and input drift before choosing a response.
Lessons
Module 1 · 2 lessons · about 50 minDefine the application
Set the prediction cutoff, available data, outcome and evaluation population.
Use only data available when the prediction is madeRead
Separate event time, arrival time and label readiness so a historical return prediction uses only available inputs and finalized outcomes.
Choose evaluation dates for the return modelRead
Match the evaluation to future predictions for existing users, keep the declared date windows and exclude later outcomes from earlier decisions.
Module 2 · 2 lessons · about 50 minRun a manual workflow
Fit, evaluate, save and reload one user-return model before adding operational tools.
Train, save and reload the user-return modelRead
Prepare historical user-return examples, fit and evaluate a baseline, then reload its saved scaler and model to predict in a fresh process.
Choose a decision threshold using explicit error costsRead
Compare three thresholds on the same validation predictions while keeping the fitted model and final test set unchanged.
Module 3 · 5 lessons · about 2 hoursMake the workflow reproducible
Validate and version the same inputs and feature contract; track the actual training run.
Validate incoming events before preparing training dataRead
Preserve original events, quarantine invalid records and block the entire training batch when field, identity or temporal checks fail.
Reconstruct the feature used by an earlier predictionRead
Store feature publication history and reconstruct the count available to an earlier return prediction, without applying later corrections.
Compare historical and current feature calculationsRead
Compare the same feature version, cutoff and arrival policy, then distinguish a calculation mismatch from stale or unavailable feature data.
Version the dataset and feature contract togetherRead
Bind data, schemas and feature definitions to explicit versions in SQLite; refuse identity conflicts and roll back a partial binding attempt.
Track a training run and reload its modelRead
Connect the dataset, source, fitting settings and validation result to one MLflow run, then retrieve the model another process can use.
Module 4 · 4 lessons · about 2 hoursAutomate and test the workflow
Connect the existing steps in Prefect, recover bounded failures and enforce an explicit quality gate.
Connect preparation, training and evaluation with PrefectRead
Automate the manual return-model workflow while keeping the accepted events, historical features and fitted model connected.
Retry a failed step without repeating the experimentRead
Recover a failed write of the completion record without repeating training; reuse a completed attempt only when its inputs and artifacts still match.
Test pipeline changes before producing a candidateRead
Run unit and integration checks through one local CI command, retain discovered test counts and diagnostics, and propagate a failed result.
Evaluate a candidate against a declared release policyRead
Recompute validation from the saved model, bind the decision to a policy and keep failed candidates available for inspection.
Module 5 · 4 lessons · about 2 hoursRelease the model
Select a compatible registered bundle, produce predictions and exercise local rollout and rollback.
Register and select an explicit model versionRead
Package the fitted return model with its contract and gate decision, then select a checked version through a local MLflow alias.
Produce daily predictions and expose an optional HTTP interfaceRead
Use one validated predictor for the analyst’s batch and a loopback HTTP endpoint, with the selected bundle attached to every prediction.
Stage a candidate with an explicit traffic decisionRead
Route a repeatable subset of observation IDs to a candidate bundle, retain routing generations and reject stale configuration updates.
Roll back to a compatible model bundleRead
Observe a controlled candidate failure, restore a checked earlier routing generation and confirm predictions use the stable bundle.
Module 6 · 4 lessons · about 2 hoursOperate and improve it
Separate service health, delayed outcomes and changed inputs before attempting recovery or retraining.
Monitor prediction requests and input failuresRead
Collect bounded request counts, validation failures and local latency measurements; keep these service signals separate from prediction quality.
Evaluate predictions after return labels become finalRead
Join predictions to their original observations, keep unfinished outcomes pending and report quality by the bundle that produced them.
Investigate changed inputs before choosing a responseRead
Compare a retained feature distribution, separate invalid data from a genuine change and consult finalized outcomes before proposing retraining.
Recover the input path and evaluate a new candidateRead
Return to the valid source after the controlled data fault, run the checked pipeline again and keep traffic unchanged until a release decision.