Returning to older model weights can still fail if the application now supplies a different feature contract. A rollback target needs both the expected bytes and a compatible interpretation of the incoming measurements.
When you need this. Use a checked rollback when an older model must be selected while the serving feature contract may have changed.
This rollback unit contains the complete selected bundle: scaler, coefficients, ordered features, contract and model digest. The application declares which serving contract it expects. A digest mismatch identifies changed bytes; a compatible digest alone does not make a model’s input layout usable.
1. Follow the decision boundary
Router.rollback() loads and validates the candidate selection before taking the state lock. It checks the retained digest, supported bundle shape and feature ordering, then compares the target contract with the expected serving contract. Only after all checks pass does it replace the complete local selection. A failed check leaves the old reference intact.
The lab reuses an exact copy of the recorded Rice predictor, bundle and dataset from registry and serving. It performs no training and uses no cloud service. Monitoring fixtures are explicitly authored; the rollout parameter variant is labeled in the result. Keep the complete extracted mlops-rollout-monitoring/ directory together; Python’s standard library is sufficient.
Read the implementation for this activity
"""Local routing around a recorded Rice predictor; no cloud deployment."""
import copy
import hashlib
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer
from pathlib import Path
from predictor import decode, encode, digest, validate, predict, CONTRACT
HERE=Path(__file__).resolve().parent
SALT='rice-canary-fixture-v1'
def selected(data, expected, contract=CONTRACT):
if digest(data)!=expected:raise ValueError('selected model digest mismatch')
bundle=validate(decode(data))
if bundle['contract']!=contract:raise ValueError('serving feature contract incompatible')
return {'bundle':bundle,'version':expected}
def bundles():
raw=(HERE/'baseline-model.json').read_bytes();live=selected(raw,digest(raw))
altered=copy.deepcopy(live['bundle']);altered['bias']+=1.0
# Deliberate parameter fixture, not an independently trained candidate.
altered['fixture_variant']='recorded baseline bias plus 1.0; no training'
raw=encode(altered)
return live,selected(raw,digest(raw))
def identity(value):
if not isinstance(value,str) or not 1<=len(value)<=80:raise ValueError('bounded nonempty identity required')
return value
def cohort(user, basis_points=2000):
identity(user)
if type(basis_points) is not int or not 0<=basis_points<=10000:raise ValueError('invalid canary allocation')
bucket=int.from_bytes(hashlib.sha256((SALT+':'+user).encode()).digest()[:8],'big')%10000
return 'canary' if bucket<basis_points else 'control'
class Router:
def __init__(self,live,candidate):
self.live,self.candidate=live,candidate
self.active=live
self.audit=[]
self.lock=threading.Lock()
def infer(self,request,mode='shadow',basis_points=2000):
if not isinstance(request,dict) or set(request)!={'id','user','input'}:raise ValueError('routing envelope mismatch')
identity(request['id']);identity(request['user'])
if mode not in ('shadow','canary','active'):raise ValueError('unknown route mode')
arm=cohort(request['user'],basis_points) if mode=='canary' else 'control'
with self.lock:chosen=self.active if mode=='active' else self.candidate if arm=='canary' else self.live
answer=predict(chosen['bundle'],request['input'],chosen['version'])
entry={'id':request['id'],'user':request['user'],'arm':arm,'served':answer}
if mode=='shadow':
try:entry['shadow']=predict(self.candidate['bundle'],request['input'],self.candidate['version'])
except (ValueError,TypeError,OverflowError,KeyError) as error:entry['shadow_error']=type(error).__name__
with self.lock:self.audit.append(entry)
return answer
def rollback(self,data,expected,contract=CONTRACT):
candidate=selected(data,expected,contract)
# Swap the complete local selection after every check has succeeded.
with self.lock:
previous=self.active['version'];self.active=candidate
return {'previous':previous,'selected':expected,'contract':contract}
def server(router,mode='shadow',basis_points=2000):
class Handler(BaseHTTPRequestHandler):
def log_message(self,*_):pass
def reply(self,status,value):
data=encode(value);self.send_response(status);self.send_header('Content-Type','application/json')
self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data)
def do_POST(self):
if self.path!='/predict':self.reply(404,{'error':'unknown route'});return
try:
size=int(self.headers.get('Content-Length','0'))
if not 0<size<=4096:self.reply(413,{'error':'body size'});return
if self.headers.get_content_type()!='application/json':self.reply(415,{'error':'content type'});return
self.reply(200,router.infer(decode(self.rfile.read(size)),mode,basis_points))
except (ValueError,TypeError,OverflowError,KeyError):self.reply(422,{'error':'invalid request'})
return HTTPServer(('127.0.0.1',0),Handler)Download the complete lab, extract it and open mlops-rollout-monitoring/. The candidate is an intentionally altered parameter fixture used to demonstrate routing. It has not been approved by the earlier quality gate for a real release.
2. Run the focused activity
Run python exercise.py rollback. The exercise begins with the bias-parameter fixture active and sends one actual HTTP request. It attempts three invalid rollbacks, confirms the active selection is unchanged, then selects the original recorded bundle and sends another request. Both return HTTP 200; the served model digest and probability change. The label happens to remain Cammeo for this input.
python exercise.py rollback{
"mode": "local in-process model-selection swap and actual HTTP requests",
"before": {
"status": 200,
"body": {
"contract": "rice-measurements-v1",
"label": "Cammeo",
"positive_probability": 0.7921192512749718,
"version": "22c31b9297103bf62c29b1560bf4b0236a29316bebba13b7e9531675fe76dfdb"
}
},
"after": {
"status": 200,
"body": {
"contract": "rice-measurements-v1",
"label": "Cammeo",
"positive_probability": 0.5836432257994117,
"version": "a4692350c5e6d1f231eea8c5eb3354c24276870e73a60922a10abb07abffa5f6"
}
},
"refusals": {
"wrong_digest": "selected model digest mismatch",
"feature_order": "unsupported bundle contract",
"serving_contract": "serving feature contract incompatible"
},
"failed_checks_preserved_active_selection": true,
"transition": {
"previous": "22c31b9297103bf62c29b1560bf4b0236a29316bebba13b7e9531675fe76dfdb",
"selected": "a4692350c5e6d1f231eea8c5eb3354c24276870e73a60922a10abb07abffa5f6",
"contract": "rice-measurements-v1"
},
"scope": "Model selection and its feature contract change together in one process; no cloud or producer rollback."
}The raw result, execution metadata and locally generated actual-output report retain the computed counts and source identities. Opening the report does not repeat execution. --record deliberately replaces that stage’s saved files; ordinary execution leaves them unchanged.

3. Inspect refusals before acting
| Failure condition | Required behavior |
|---|---|
| Incompatible rollback target | Reject an incompatible target before replacing the complete active selection. |
| Invalid routing envelope | Reject an invalid request envelope before invoking either selected model. |
The saved refusals distinguish selected model digest mismatch, unsupported bundle contract after reversing ordered feature names, and serving feature contract incompatible for a v2 expectation. test_rollback_refuses_incompatibility_without_changing_selection checks all three and the subsequent successful HTTP selection change.
python -m unittest -v test_continuation.pyIn a temporary copy, change the expected serving contract without changing the target model. Explain why retaining the right digest still cannot authorize an incompatible rollback. Name which external producer would also need coordination in a real system.
How the result should guide the next step
This is a single-process reference swap, not a distributed rollback. It does not rewind external feature producers, shared data, schema migrations or cloud revisions. Requests that already captured the old selection can finish with it; the lock prevents mixing selection fields, not all in-flight work. The local file directory is trusted.
Check your understanding
Read the explanation
This implementation checks compatibility before swapping the local selection reference.
References
Official primary sources checked October 24, 2026. The local records supply this lesson’s numbers and refusals.
- Google ML Crash Course: monitoring: distinguish data checks, distributions and live model quality.
- Python hashlib: deterministic byte digests used by this local implementation.
- Python HTTP server: local server APIs and production limitations.
- Cloud Deploy canary concepts: subset rollout terminology. This lab does not run Cloud Deploy or reproduce its orchestration.