Recover the input path and evaluate a new candidate

Return to the valid source after the controlled data fault, run the checked pipeline again and keep traffic unchanged until a release decision.

Problem

When you need this. The investigation identified an invalid input without showing that new fitting would improve the model. Recovery starts by choosing a valid source and preserving the failed example, not by training on the broken input.

This final exercise uses the original valid scenario in a fresh pipeline workspace, fits an actual candidate and applies the unchanged release policy. It then checks that the restored routing configuration has not changed.

Concepts

Recovery restores the application’s intended behavior. Depending on the cause, that can mean repairing ingestion, restoring artifact availability or selecting a compatible release. Retraining is one possible follow-up; it is not the universal repair.

A new training run creates a new execution identity even when it fits identical parameters. This exercise uses the same fixture, recipe and splits, so it claims no model improvement. The purpose is to check that a recovered input path can still produce a loadable, evaluated candidate.

The same policy remains in force. Passing it permits the local rollout exercise, while actual traffic selection is still separate. The new candidate does not automatically move the routing head.

The sequence now closes the six-stage lifecycle: an application requirement became a manual model, a reproducible run, a checked pipeline, a selected release and an operated prediction path. The original observation, feature and model contracts remain visible through those handoffs.

Architecture

Working code

Download the shared application and tests, unzip it and enter mlops-return-workflow/. Keep the same directory throughout the course. Create the environment once. Install the pinned platform dependencies for this workshop.

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-platform.txt

On PowerShell, activate with .venv\Scripts\Activate.ps1.

Continue from Investigate changed inputs before choosing a response; keep the same lab directory and outputs.

Recover through the existing pipeline

Use a fresh workspace to keep the failed and recovered attempts separate. The source remains the original valid authored scenario; the deliberately invalid drift input is not included in training.

def recovery_report(root):
    need(root,'drift.json');routing=rollout.Routing(root/'routing.sqlite')
    try:before=routing.current()
    finally:routing.close()
    # Operator chooses the original valid source after the invalid-input drill.
    # This is a fresh actual fit, not approval to switch traffic.
    result=pipeline.run(w.HERE/'scenario.json',root/'recovery-training')
    w.save(root/'recovery-state.json',result);_,source=release.candidate_files(root/'recovery-state.json')
    decision=release.gate(source/'outputs/cohort.json',source/'outputs/candidate.json')
    w.save(root/'recovery-gate.json',decision)
    routing=rollout.Routing(root/'routing.sqlite')
    try:after=routing.current()
    finally:routing.close()
    if before!=after:raise ValueError('recovery unexpectedly changed traffic')
    return {'source':'original valid authored scenario; invalid drill inputs excluded','new_training_run':result['run_id'],
            'gate_passed':decision['passed'],'policy_unchanged':decision['policy']==release.POLICY,
            'traffic_unchanged':before==after,'automatic_promotion':False,'final_test_used':False}

HANDLERS=dict(zip(TOPICS,[pipeline_report,retry_report,ci_report,gate_report,registry_report,serving_report,rollout_report,rollback_report,monitoring_report,delayed_report,drift_report,recovery_report]))

python course.py recovery --workspace work/course

{
  "source": "original valid authored scenario; invalid drill inputs excluded",
  "new_training_run": "110b82203fdf48cd9178430d283fca4b",
  "gate_passed": true,
  "policy_unchanged": true,
  "traffic_unchanged": true,
  "automatic_promotion": false,
  "final_test_used": false
}

The command performs actual CPU fitting through Prefect and MLflow. The candidate is evaluated with the same release policy, and the reserved final test is unused.

Retain the current serving decision

The exercise compares the complete routing configuration before and after the new fit. Any candidate rollout would require the explicit staging operation from the release stage.

def investigate(reference,current,quality):
    try:drift=distribution(reference,current)
    except ValueError as e:return {'decision':'repair data before retraining','reason':str(e),'retrain':False}
    if quality['scored']==0:return {'decision':'wait for finalized outcomes','drift':drift,'retrain':False}
    if quality['accuracy']<0.60:return {'decision':'investigate data and candidate errors; evaluate any new model with the same gate','drift':drift,'retrain':False}
    return {'decision':'investigate distribution change' if drift['investigate'] else 'continue observing','drift':drift,'retrain':False}

The earlier diagnosis does not authorize promotion. The recovered candidate and its gate report remain available alongside the unchanged stable release.

Recorded commands retain outputs in recorded-platform/; your commands use work/. Read the actual command records. Use a fresh work directory for another attempt; original outputs are preserved.

Failure cases

The command fails if the new pipeline or gate computation raises an error; it retains the attempted workspace. It never changes the policy to rescue a failed candidate.

The reusable integration suite covers invalid input, incomplete attempts, artifact changes, failed gates and rollback history. The final course command additionally checks that the actual recovery run leaves traffic unchanged.

python -m unittest -v test_platform.py

Running and scaling the workshop

Keep recovery-state.json and recovery-gate.json with the original command log. Compare the new run identity, model identity and policy identity before considering another rollout.

A real incident needs source-system repair records, an owner-approved recovery decision and representative post-repair observations. This bounded synthetic exercise measures no incident recovery time, retention benefit or production reliability.

Key takeaways

• Repair the diagnosed failure before deciding whether fitting is useful.
• A new run must satisfy the same checks.
• A passing candidate does not automatically change traffic.

Best practices

Preserve the failed attempt and the recovered run.

Confirm the stable prediction path while a new candidate awaits review.

References

• Google: MLOps lifecycle and automation
• Prefect: flows and tasks
• MLflow: tracking