Building MLOps
Build reproducible data and model artifacts, serve a selected classifier, and exercise local release and monitoring controls.
- Prerequisites
- Python, SQL, basic supervised learning and familiarity with training and validation datasets.
- Duration
- about 8 hours of reading
- Cost
- Free, no sign-up
Before you start
This course follows the data, training and release work around a supervised model. The current labs establish the foundations; the integrated event-prediction pipeline is still being built.
- Python and SQL prerequisites
- Read Python functions and dictionaries, run a script and unit test, and follow a SQL SELECT with filtering and joins. The current labs run on a local CPU.
- Machine-learning prerequisites
- Understand that features are inputs, labels are target outcomes, and fitting adjusts a model using training examples. Review the split and metric examples below if these ideas need a refresher.
Introduced in the course. Data validation, point-in-time features, training-run records and versioned contracts come first. Pipeline automation, model registration, serving and monitoring follow in later stages.
Review three MLOps examplesEvaluation boundaries, prediction metrics and data availability
Keep evaluation data separate
Training and selection must not use the final test outcomes. For time-dependent tasks, split boundaries must also match when predictions will be made.
Training → fit parameters Validation → choose settings Test → evaluate the chosen modelRead a confusion matrix
Accuracy counts all correct decisions. Recall measures the fraction of actual positives found. These illustrative counts are a refresher, not results from a course lab.
Illustrative counts: TP=8, FN=2, FP=1, TN=9 Accuracy = (8+9)/20 = 85% Recall = 8/(8+2) = 80%Check what was available
An event can happen before prediction and still arrive too late to be used. A historical feature must respect both event time and arrival time.
Event: 09:00 UTC Prediction: 09:10 UTC Arrival: 09:20 UTC Exclude this event from that prediction.
Course termsShort definitions linked to their worked implementation
- Feature, label and prediction
A feature is an input available when a prediction is made. A label records the outcome the model should learn to predict. A prediction is the model’s estimated outcome.
Open the worked lesson →- Training, validation and test sets
Training data fits parameters. Validation data guides settings and selection. Test data evaluates the chosen model after those decisions.
Open the worked lesson →- Metric and decision cost
A metric summarizes prediction behavior. A decision-cost policy assigns explicit consequences to errors; equal accuracy can conceal different costs.
Open the worked lesson →- Leakage and evaluation split
Leakage gives training or evaluation information that would not be available in the intended prediction setting. Time and entity boundaries help define which examples may be compared.
Open the worked lesson →- Event time, arrival time and label delay
Event time is when something happened; arrival time is when the system received it. A label may become known only after an outcome window and further reporting delay.
Open the worked lesson →- Schema validation and quarantine
Validation checks records against explicit field, type and consistency rules. Quarantine retains rejected records and reasons so they can be investigated.
Open the worked lesson →- Point-in-time feature
A feature reconstructed from values available at the prediction timestamp. Later events and later-published corrections cannot silently alter what the historical model could have known.
Open the worked lesson →- Freshness and training-serving parity
Freshness measures whether an input is recent enough for its contract. Parity compares training and serving calculations under the same definitions, inputs and time rules.
Open the worked lesson →- Data and feature version
A version identifies the dataset, schema and feature definition used together. Content hashes detect changed bytes; they do not establish whether the data is appropriate.
Open the worked lesson →- Training recipe and run
A recipe records supported inputs, configuration and dependencies. A run is one execution attempt with its own identity, logs, status and resulting files.
Open the worked lesson →- Artifact and lineage
An artifact is a retained result, such as an evaluation report or a model file. Lineage connects results to their inputs and execution. The current Rice report is an evaluation artifact, not a resumable or servable checkpoint.
Open the worked lesson →- Model registry and release
A model registry records model versions and supporting records. A release selects a compatible version for a serving environment under explicit promotion and rollback rules.
Planned stages 5–7: model registration, serving and release.
- Monitoring, drift and retraining
Monitoring compares incoming data and delayed outcomes with the operating contract. A changed distribution is drift; it is a signal to investigate before deciding whether to retrain.
Planned stage 8: monitor and maintain.
The pipeline this course builds
The target project predicts whether a user returns after an event. Its pipeline will connect event ingestion, historical features, training, evaluation, serving and delayed-outcome monitoring.
Current state: The available lessons are tested foundation exercises. The Rice classifier introduces baseline evaluation and training-run records; the event exercises introduce time, labels and features. Their artifacts are not yet wired into one running pipeline.
Integration work must connect the event features and labels to one trainer before the later pipeline, registry and serving stages can be called complete. The stages below show that intended dependency order.
Target design · integration and later stages remain to be built
What you'll learn
- Define a supervised baseline, error costs and leakage-resistant evaluation splits.
- Validate availability, quarantine invalid data and build versioned point-in-time features.
- Capture training configuration, execution records and full-precision inference parameters.
- Gate and register a candidate, then serve its selected version through batch and HTTP.
- Exercise local shadow, canary and rollback decisions; monitor input signals and delayed outcomes.
Lessons
Module 1 · 4 lessonsDefine the prediction task
Specify what a prediction means, when it is made, and how its errors will be judged.
Train a baseline modelRead
Establish a reproducible reference with fixed data splits and training-only preprocessing.
Measure the cost of prediction errorsRead
Compare the same predictions under explicit penalties for different mistakes.
Choose evaluation splitsRead
Match time and entity boundaries to the prediction setting you intend to serve.
Handle late data and delayed labelsRead
Construct examples from information available at prediction time, then wait for outcomes.
Module 2 · 4 lessonsPrepare data and features
Turn input events into validated, versioned feature snapshots that respect prediction time.
Validate incoming dataRead
Reject invalid records and preserve the invalid records needed to repair a batch.
Build historical feature snapshotsRead
Join each prediction to feature values that were available at that time.
Check features before servingRead
Refuse stale values and compare training and serving calculations on the same snapshot.
Version the data and feature contractRead
Bind dataset bytes, schema and feature logic to explicit versions.
Module 3 · 2 lessonsRun and track training
Execute an existing trainer with a captured setup; keep attempts, artifacts and continuation lineage distinct.
Capture the training configurationRead
Record the trainer’s actual settings, inputs, dependencies and runtime.
Record and select a training runRead
Keep each attempt’s inputs, logs and evaluation report together, then verify the selected result.
Remaining planned lessons in this stage
- Register a resumed training job as a continuation of its parent run
- Bind the dataset, code, features and model in one lineage record
These lessons do not yet have published articles or labs.
Module 4 · 4 lessonsValidate, register and serve a candidate
Continue the Rice baseline with an exact inference bundle, an explicit quality gate and a shared prediction contract.
Export a full-precision inference bundleRead
Keep the trained scaler and classifier together without rounding serving parameters. A local Rice-classifier lab with explicit failure checks.
Gate a model candidate on validation resultsRead
Make a quality decision about the exact exported candidate using explicit count-based checks. A local Rice-classifier lab with explicit failure checks.
Register and select a content-addressed modelRead
Bind a local model selection to separately retained expected bytes. A local Rice-classifier lab with explicit failure checks.
Serve a versioned predictor through batch and HTTPRead
Reuse one validated preprocessing and prediction function across two adapters. A local Rice-classifier lab with explicit failure checks.
Module 5 · 3 lessonsExercise a local model release
Observe a controlled candidate through shadow and sticky canary routing, then refuse incompatible rollback targets.
Shadow predictions without changing live responsesRead
Route a request to the live model while recording a separate candidate prediction. A local executable Rice rollout and monitoring continuation.
Route a sticky canary and compare cohort outcomesRead
Assign repeated caller IDs to a stable canary cohort with an explicit salt and allocation. A local executable Rice rollout and monitoring continuation.
Roll back a model with feature-contract checksRead
Select a prior model by its retained digest, refuse incompatible feature contracts, and verify the local serving selection over actual HTTP.
Module 6 · 2 lessonsMonitor inputs and delayed outcomes
Compute separate data signals and preserve label coverage before preparing a retraining review.
Distinguish schema, freshness and distribution signalsRead
Validate event fields and feature values before considering freshness. A local executable Rice rollout and monitoring continuation.
Join delayed outcomes and review retrainingRead
Join finalized labels to predictions by identity rather than list position. A local executable Rice rollout and monitoring continuation.
Planned modules and capstone
The lessons above are available now. The following topics are planned; they have no lesson pages or labs yet.
Module 7: Automate the training workflowPlanned
- Build an idempotent training pipeline
- Partition work and recover failed stages
- Backfill features without leaking future information
- Schedule training with explicit dependencies and budgets
Module 8: Extend evaluation and promotionPlanned
- Evaluate calibration, thresholds and business costs
- Compare candidates under a versioned evaluation policy
- Use a remote model registry with authenticated promotion
- Define promotion rules and review an incomplete candidate
Module 9: Serve predictionsPlanned
- Choose batch, online and asynchronous prediction
- Harden serving with request deadlines and admission limits
- Package a model and test resource limits
- Handle timeouts, missing features and fallback behavior
Module 10: Automate release coordinationPlanned
- Test training and serving changes in CI
- Coordinate multi-process release state
- Verify deployment and producer compatibility during rollback
Module 11: Extend operational monitoring and recoveryPlanned
- Validate statistical drift detectors across request slices
- Account for delayed-label uncertainty across model cohorts
- Exercise dependency recovery and the event-based return-prediction pipeline
Final capstone: Proposed capstone: operate an event-based return-prediction classifier from a versioned event snapshot through point-in-time features, a registered model, gated release and delayed-outcome monitoring.Planned
This project is planned. No project lab is available yet.