aiml
SearchMy learning
Menu

Lesson 17 of 19Module 5

Roll back a model with feature-contract checks

Select a prior model by its retained digest, refuse incompatible feature contracts, and verify the local serving selection over actual HTTP.

Prerequisites

Register and select a content-addressed model; sticky canary routing and feature contracts.

Preparation and term definitions

By the end, you can

  • Select a prior model by its separately retained digest.
  • Validate model dimensions, feature ordering and serving-contract compatibility before rollback.
  • Preserve the active selection on refusal and verify the changed digest through HTTP after a local swap.
Runtime
Python 3.12+
Dependencies
Standard library only

Lab code, tests and fixtures (ZIP, 3.1 MB)Quiz check your understanding

Progress is saved in this browser.

Returning to older model weights can still fail if the application now supplies a different feature contract. A rollback target needs both the expected bytes and a compatible interpretation of the incoming measurements.

When you need this. Use a checked rollback when an older model must be selected while the serving feature contract may have changed.

This rollback unit contains the complete selected bundle: scaler, coefficients, ordered features, contract and model digest. The application declares which serving contract it expects. A digest mismatch identifies changed bytes; a compatible digest alone does not make a model’s input layout usable.

Prior bytes + digestDigest + feature contractSwap entire selectionRefuse; keep active modelcompatibleinvalidPrior bytes + digestCheck compatibilitySwap entire selectionRefuse; keep activecompatible
Branches follow the implemented local decision. Excluded work and failed checks remain visible rather than being counted as successful output.

1. Follow the decision boundary

Router.rollback() loads and validates the candidate selection before taking the state lock. It checks the retained digest, supported bundle shape and feature ordering, then compares the target contract with the expected serving contract. Only after all checks pass does it replace the complete local selection. A failed check leaves the old reference intact.

The lab reuses an exact copy of the recorded Rice predictor, bundle and dataset from registry and serving. It performs no training and uses no cloud service. Monitoring fixtures are explicitly authored; the rollout parameter variant is labeled in the result. Keep the complete extracted mlops-rollout-monitoring/ directory together; Python’s standard library is sufficient.

Read the implementation for this activity
rollout.pySource · pythonDownload
"""Local routing around a recorded Rice predictor; no cloud deployment."""
import copy
import hashlib
import threading
from http.server import BaseHTTPRequestHandler, HTTPServer
from pathlib import Path
from predictor import decode, encode, digest, validate, predict, CONTRACT
HERE=Path(__file__).resolve().parent
SALT='rice-canary-fixture-v1'

def selected(data, expected, contract=CONTRACT):
    if digest(data)!=expected:raise ValueError('selected model digest mismatch')
    bundle=validate(decode(data))
    if bundle['contract']!=contract:raise ValueError('serving feature contract incompatible')
    return {'bundle':bundle,'version':expected}

def bundles():
    raw=(HERE/'baseline-model.json').read_bytes();live=selected(raw,digest(raw))
    altered=copy.deepcopy(live['bundle']);altered['bias']+=1.0
    # Deliberate parameter fixture, not an independently trained candidate.
    altered['fixture_variant']='recorded baseline bias plus 1.0; no training'
    raw=encode(altered)
    return live,selected(raw,digest(raw))

def identity(value):
    if not isinstance(value,str) or not 1<=len(value)<=80:raise ValueError('bounded nonempty identity required')
    return value

def cohort(user, basis_points=2000):
    identity(user)
    if type(basis_points) is not int or not 0<=basis_points<=10000:raise ValueError('invalid canary allocation')
    bucket=int.from_bytes(hashlib.sha256((SALT+':'+user).encode()).digest()[:8],'big')%10000
    return 'canary' if bucket<basis_points else 'control'

class Router:
    def __init__(self,live,candidate):
        self.live,self.candidate=live,candidate
        self.active=live
        self.audit=[]
        self.lock=threading.Lock()
    def infer(self,request,mode='shadow',basis_points=2000):
        if not isinstance(request,dict) or set(request)!={'id','user','input'}:raise ValueError('routing envelope mismatch')
        identity(request['id']);identity(request['user'])
        if mode not in ('shadow','canary','active'):raise ValueError('unknown route mode')
        arm=cohort(request['user'],basis_points) if mode=='canary' else 'control'
        with self.lock:chosen=self.active if mode=='active' else self.candidate if arm=='canary' else self.live
        answer=predict(chosen['bundle'],request['input'],chosen['version'])
        entry={'id':request['id'],'user':request['user'],'arm':arm,'served':answer}
        if mode=='shadow':
            try:entry['shadow']=predict(self.candidate['bundle'],request['input'],self.candidate['version'])
            except (ValueError,TypeError,OverflowError,KeyError) as error:entry['shadow_error']=type(error).__name__
        with self.lock:self.audit.append(entry)
        return answer
    def rollback(self,data,expected,contract=CONTRACT):
        candidate=selected(data,expected,contract)
        # Swap the complete local selection after every check has succeeded.
        with self.lock:
            previous=self.active['version'];self.active=candidate
        return {'previous':previous,'selected':expected,'contract':contract}

def server(router,mode='shadow',basis_points=2000):
    class Handler(BaseHTTPRequestHandler):
        def log_message(self,*_):pass
        def reply(self,status,value):
            data=encode(value);self.send_response(status);self.send_header('Content-Type','application/json')
            self.send_header('Content-Length',str(len(data)));self.end_headers();self.wfile.write(data)
        def do_POST(self):
            if self.path!='/predict':self.reply(404,{'error':'unknown route'});return
            try:
                size=int(self.headers.get('Content-Length','0'))
                if not 0<size<=4096:self.reply(413,{'error':'body size'});return
                if self.headers.get_content_type()!='application/json':self.reply(415,{'error':'content type'});return
                self.reply(200,router.infer(decode(self.rfile.read(size)),mode,basis_points))
            except (ValueError,TypeError,OverflowError,KeyError):self.reply(422,{'error':'invalid request'})
    return HTTPServer(('127.0.0.1',0),Handler)

Download the complete lab, extract it and open mlops-rollout-monitoring/. The candidate is an intentionally altered parameter fixture used to demonstrate routing. It has not been approved by the earlier quality gate for a real release.

2. Run the focused activity

Run python exercise.py rollback. The exercise begins with the bias-parameter fixture active and sends one actual HTTP request. It attempts three invalid rollbacks, confirms the active selection is unchanged, then selects the original recorded bundle and sends another request. Both return HTTP 200; the served model digest and probability change. The label happens to remain Cammeo for this input.

Execute the rollback activityTerminal · bash
python exercise.py rollback
Actual saved rollback resultRecorded output · textDownload
{
  "mode": "local in-process model-selection swap and actual HTTP requests",
  "before": {
    "status": 200,
    "body": {
      "contract": "rice-measurements-v1",
      "label": "Cammeo",
      "positive_probability": 0.7921192512749718,
      "version": "22c31b9297103bf62c29b1560bf4b0236a29316bebba13b7e9531675fe76dfdb"
    }
  },
  "after": {
    "status": 200,
    "body": {
      "contract": "rice-measurements-v1",
      "label": "Cammeo",
      "positive_probability": 0.5836432257994117,
      "version": "a4692350c5e6d1f231eea8c5eb3354c24276870e73a60922a10abb07abffa5f6"
    }
  },
  "refusals": {
    "wrong_digest": "selected model digest mismatch",
    "feature_order": "unsupported bundle contract",
    "serving_contract": "serving feature contract incompatible"
  },
  "failed_checks_preserved_active_selection": true,
  "transition": {
    "previous": "22c31b9297103bf62c29b1560bf4b0236a29316bebba13b7e9531675fe76dfdb",
    "selected": "a4692350c5e6d1f231eea8c5eb3354c24276870e73a60922a10abb07abffa5f6",
    "contract": "rice-measurements-v1"
  },
  "scope": "Model selection and its feature contract change together in one process; no cloud or producer rollback."
}

The raw result, execution metadata and locally generated actual-output report retain the computed counts and source identities. Opening the report does not repeat execution. --record deliberately replaces that stage’s saved files; ordinary execution leaves them unchanged.

Actual local rollback exercise report, retaining the observed outcomes and failure cases.
Actual local execution. Routing uses real loopback HTTP and an explicit parameter fixture. Full-resolution capture · Raw result · Capture record.

3. Inspect refusals before acting

Failure condition Required behavior
Incompatible rollback target Reject an incompatible target before replacing the complete active selection.
Invalid routing envelope Reject an invalid request envelope before invoking either selected model.

The saved refusals distinguish selected model digest mismatch, unsupported bundle contract after reversing ordered feature names, and serving feature contract incompatible for a v2 expectation. test_rollback_refuses_incompatibility_without_changing_selection checks all three and the subsequent successful HTTP selection change.

Check the local failure boundariesTerminal · bash
python -m unittest -v test_continuation.py

In a temporary copy, change the expected serving contract without changing the target model. Explain why retaining the right digest still cannot authorize an incompatible rollback. Name which external producer would also need coordination in a real system.

How the result should guide the next step

This is a single-process reference swap, not a distributed rollback. It does not rewind external feature producers, shared data, schema migrations or cloud revisions. Requests that already captured the old selection can finish with it; the lock prevents mixing selection fields, not all in-flight work. The local file directory is trusted.

Check your understanding

The prior bundle has the expected digest but its feature contract is incompatible. What should happen to the active selection?

Choose one answer, then check your reasoning.

Read the explanation

This implementation checks compatibility before swapping the local selection reference.

References

Official primary sources checked October 24, 2026. The local records supply this lesson’s numbers and refusals.

Written by Arjun Prabhulal.

Module 5

Exercise a local model release