Investigate changed inputs before choosing a response

Compare a retained feature distribution, separate invalid data from a genuine change and consult finalized outcomes before proposing retraining.

Problem

When you need this. The number of recent events may change because user behavior changed, an event collector broke or a feature calculation changed. A distribution alert alone cannot tell which cause occurred.

This workshop compares the retained validation-request counts with themselves and with a controlled all-zero sample. It also supplies a boolean as an invalid count. The investigation code distinguishes a distribution difference from a contract failure and combines the valid comparison with the delayed-quality report.

Concepts

Distribution drift means that the observed input distribution differs from a chosen reference. It does not necessarily mean that the model has become less accurate. Data quality failures instead violate the agreed representation or time rules.

The example fixes four count bins before comparing samples: zero, one to two, three to five and six to 1000. It computes the fraction in each bin, then total variation: half the sum of absolute differences between corresponding fractions. Identical histograms have distance zero; disjoint histograms have distance one.

The 0.2 investigation threshold is an illustrative rule for this exercise. It is not a calibrated statistical significance test and does not account for sampling uncertainty. Bin boundaries and sample coverage affect the result.

The reference consists of the retained 48 validation-request features. The all-zero sample is deliberately constructed to test detection; it is not a newly collected production batch. No outcome labels are invented for that shifted sample.

Architecture

Working code

Download the shared application and tests, unzip it and enter mlops-return-workflow/. Keep the same directory throughout the course. Create the environment once. Install the pinned platform dependencies for this workshop.

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements-platform.txt

On PowerShell, activate with .venv\Scripts\Activate.ps1.

Continue from Evaluate predictions after return labels become final; keep the same lab directory and outputs.

Calculate a defined distribution difference

The function validates counts before binning, preserves exact proportions and returns the reference and current fractions with their sample sizes.

def distribution(reference,current):
    # Fixed bins: zero, one-to-two, three-to-five, six or more known events.
    def hist(values):
        if not values:raise ValueError('empty feature sample')
        counts=[0,0,0,0]
        for value in values:
            w.count(value);counts[0 if value==0 else 1 if value<3 else 2 if value<6 else 3]+=1
        return [v/len(values) for v in counts]
    a,b=hist(reference),hist(current);distance=sum(abs(x-y) for x,y in zip(a,b))/2
    return {'bins':['0','1–2','3–5','6–1000'],'reference_rows':len(reference),'current_rows':len(current),
            'reference_fraction':a,'current_fraction':b,'total_variation':distance,'investigation_threshold':0.2,
            'investigate':distance>0.2,'retraining_authorized':False,'quality_measured':False}

The unchanged sample has total variation zero. The controlled shift exceeds the illustrative investigation threshold. These are feature-distribution observations, not new accuracy measurements.

Choose an investigation, not automatic retraining

The decision checks malformed data first, waits when no outcomes are finalized and otherwise distinguishes changed inputs from a degraded-quality report.

def investigate(reference,current,quality):
    try:drift=distribution(reference,current)
    except ValueError as e:return {'decision':'repair data before retraining','reason':str(e),'retrain':False}
    if quality['scored']==0:return {'decision':'wait for finalized outcomes','drift':drift,'retrain':False}
    if quality['accuracy']<0.60:return {'decision':'investigate data and candidate errors; evaluate any new model with the same gate','drift':drift,'retrain':False}
    return {'decision':'investigate distribution change' if drift['investigate'] else 'continue observing','drift':drift,'retrain':False}

python course.py drift --workspace work/course

{
  "unchanged": {
    "bins": [
      "0",
      "1\u20132",
      "3\u20135",
      "6\u20131000"
    ],
    "reference_rows": 48,
    "current_rows": 48,
    "reference_fraction": [
      0.10416666666666667,
      0.16666666666666666,
      0.4583333333333333,
      0.2708333333333333
    ],
    "current_fraction": [
      0.10416666666666667,
      0.16666666666666666,
      0.4583333333333333,
      0.2708333333333333
    ],
    "total_variation": 0.0,
    "investigation_threshold": 0.2,
    "investigate": false,
    "retraining_authorized": false,
    "quality_measured": false
  },
  "controlled_shift": {
    "bins": [
      "0",
      "1\u20132",
      "3\u20135",
      "6\u20131000"
    ],
    "reference_rows": 48,
    "current_rows": 48,
    "reference_fraction": [
      0.10416666666666667,
      0.16666666666666666,
      0.4583333333333333,
      0.2708333333333333
    ],
    "current_fraction": [
      1.0,
      0.0,
      0.0,
      0.0
    ],
    "total_variation": 0.8958333333333334,
    "investigation_threshold": 0.2,
    "investigate": true,
    "retraining_authorized": false,
    "quality_measured": false
  },
  "investigation": {
    "decision": "investigate distribution change",
    "drift": {
      "bins": [
        "0",
        "1\u20132",
        "3\u20135",
        "6\u20131000"
      ],
      "reference_rows": 48,
      "current_rows": 48,
      "reference_fraction": [
        0.10416666666666667,
        0.16666666666666666,
        0.4583333333333333,
        0.2708333333333333
      ],
      "current_fraction": [
        1.0,
        0.0,
        0.0,
        0.0
      ],
      "total_variation": 0.8958333333333334,
      "investigation_threshold": 0.2,
      "investigate": true,
      "retraining_authorized": false,
      "quality_measured": false
    },
    "retrain": false
  },
  "invalid_input": {
    "decision": "repair data before retraining",
    "reason": "event count must be an integer between 0 and 1000",
    "retrain": false
  }
}

The changed sample prompts investigation; a boolean count prompts data repair. Both results explicitly leave retraining unauthorized. The retained quality report still describes the original validation predictions.

Recorded commands retain outputs in recorded-platform/; your commands use work/. Read the actual command records. Use a fresh work directory for another attempt; original outputs are preserved.

Failure cases

test_drift_is_distribution_only_and_invalid_data_is_repair checks identical samples, a controlled shift, the absence of a quality claim and refusal of a boolean feature.

An empty sample is invalid for this comparison. Do not draw a zero-drift conclusion from missing data. Likewise, a stable histogram can hide a label relationship change that only outcome evaluation reveals.

python -m unittest -v test_platform.py

Running and scaling the workshop

This is an interpretable four-bin local comparison. It uses no external drift-reporting service or Evidently integration. The histogram and policy are explicit in the downloaded code.

For a real application, choose reference periods, minimum sample sizes and alert thresholds from operating needs. Inspect collector health and feature versions before training on an apparent change. Keep any proposed model behind the existing candidate gate.

Key takeaways

• A changed input distribution is a reason to investigate.
• Invalid feature values require data repair.
• Prediction quality needs outcomes from the population being discussed.

Best practices

Retain reference sample identity and fixed bin definitions.

Keep investigation, retraining and promotion as separate decisions.

References

• Google: MLOps lifecycle and automation