Connect preparation, training and evaluation with Prefect

Automate the manual return-model workflow while keeping the accepted events, historical features and fitted model connected.

Problem

When you need this. The manual commands establish that the model can be fitted and reloaded. An automated run must preserve those dependencies: fitting cannot begin after an invalid event batch, and evaluation must use the cohort that produced the candidate.

The pipeline reads the same scenario, validates its event records, materializes the historical count for each observation and passes that prepared cohort to the tracked trainer. Its output is a candidate and validation result, ready for evaluation against a release policy.

Concepts

A flow coordinates work; a task is a separately tracked unit inside it. Prefect records task and flow states. Calling a decorated task and waiting for its result makes the next operation depend on that result; naming three tasks alone would not connect their data.

The validation task retains accepted and rejected bytes. Only a ready batch produces accepted-scenario.json. The history task reads that accepted source, joins the prediction cutoffs to their historical counts and saves cohort.json. It checks those counts against the direct assembler before handing the file to training.

Tracking receives the prepared file explicitly. It verifies that the supplied cohort matches the copied scenario, then fits only its training window. MLflow retains the model and validation artifacts. Prefect manages execution state; it does not grade the model.

Architecture

Working code

Download the shared application and tests, unzip it and enter mlops-return-workflow/. Keep the same directory throughout the course. Create the environment once. Install the pinned platform dependencies for this workshop.

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-platform.txt

On PowerShell, activate with .venv\Scripts\Activate.ps1.

Continue from Track a training run and reload its model; keep the same lab directory and outputs.

Validate events before constructing features

The task must consume the validator’s accepted records. Otherwise validation would only produce a report while fitting still read unchecked input.

The clean fixture contains 240 observations. Each joined historical count matches the original direct calculation. An invalid record blocks training and remains in the quarantine output.

Run the connected tasks

The flow passes each concrete file to its consumer. Automatic Prefect caching is disabled; the following workshop introduces explicit result reuse.

python course.py pipeline --workspace work/course

{
  "status": "FINISHED",
  "training_rows": 144,
  "validation_accuracy": 0.6041666666666666,
  "run_id": "4cdcd1ccf1f14fb4a52cc5674c0b2a9e",
  "model_sha256": "576847afc0d4b064e9f3653f2d25b994b638ae12dcb53fcf02beae45a0799b43",
  "cohort_consumed_from_history": "4e808d638866651009546e045fa211836690b0a4210e7bec816e4bc4165834be",
  "receipt_attempts": 1,
  "final_test_used": false
}

The actual local run trains on 144 rows and reports validation accuracy 0.6042. Its model and cohort digests identify these particular files. The final test window stays unused.

Recorded commands retain outputs in recorded-platform/; your commands use work/. Read the actual command records. Use a fresh work directory for another attempt; original outputs are preserved.

Failure cases

test_invalid_event_batch_cannot_train supplies a malformed event, checks the failed flow and confirms that no candidate.json exists. The quarantine record remains available for diagnosis.

test_completed_cohort_tampering_is_refused changes a saved count. An identical rerun must reject the altered result rather than reuse it.

python -m unittest -v test_platform.py

Running and scaling the workshop

Calling course.py starts a local Prefect server for this exercise. Server analytics are disabled and no Prefect Cloud account is used. Retain the command log when inspecting a task failure.

Tasks run sequentially because later tasks consume earlier outputs. Parallelize independent partitions only after defining their identities and merge rule. This small fixture supplies no throughput or worker-capacity measurement.

Key takeaways

• An automated pipeline must carry artifacts between stages.
• Data validation and model-quality evaluation are separate decisions.
• A successful flow produces a candidate, not permission to serve it.

Best practices

Make downstream inputs explicit and preserve the failed batch.

Keep orchestration outside the reusable prediction mathematics. The same trainer remains usable from the manual commands.

References

• Prefect: flows and tasks
• MLflow: tracking