Measure the cost of prediction errors

Problem

When you need this. A classifier makes two kinds of mistake when sorting Cammeo and Osmancik rice. Counting both as one error is useful for accuracy, but a product decision may treat their consequences differently. Before choosing a model or changing its decision threshold, make those consequences explicit.

You will add error-cost accounting to the reproducible baseline. Both classifiers and their decision rules stay fixed. On the same 590 validation cases, the majority rule makes 248 errors and logistic regression makes 43. The lab assigns each error a declared scenario cost and retains a ledger that explains the total.

Concepts

Accuracy counts correct predictions. A confusion matrix counts each actual/predicted class pair. With Cammeo as the positive class, predicting Osmancik for Cammeo is a false negative; the reverse error is a false positive. A cost matrix then assigns a declared consequence to each supported true/predicted pair.

Costs express a decision policy, not a property learned by the classifier. Changing them can change which fixed predictor is preferred without changing a single prediction. To interpret the resulting total, the reference membership and one-to-one prediction alignment must stay fixed.

Architecture

The driver fits the unchanged baseline on training rows and creates its validation references and predictions. The figure starts at costs.score(): it validates IDs and labels, aligns predictions with references, then applies the supplied cost matrix. Each charged error remains in a ledger that reconciles to the total. Fitting happens before this scoring flow. The cost calculation does not retrain the model or change its decision threshold.

Working code

Define what each wrong decision costs

Assume a fictional sorting process: labeling Cammeo as Osmancik incurs five cost units; the reverse mistake incurs one. Correct decisions have zero error cost. These are teaching assumptions, not prices or measured business impact. A real application needs cost estimates from its process owner to justify its values.

{
  "unit": "illustrative sorting cost units",
  "assumption": "A fictional quality-control process assigns five units to Cammeo labelled Osmancik and one unit to the reverse error. These are declared scenario inputs, not measured prices or business impact.",
  "matrix": {
    "Cammeo": {"Cammeo": 0, "Osmancik": 5},
    "Osmancik": {"Cammeo": 1, "Osmancik": 0}
  }
}

Rows represent the actual class and columns the prediction. Keeping that direction explicit prevents an easy mistake: reversing the two axes would charge the expensive error at the cheap rate. The cost unit is shared across both classifiers and every case in this experiment.

Download the complete lab and enter mlops-decision-contract/:

python3.12 -m venv .venv
source .venv/bin/activate
python experiment.py
python -m unittest -v test_costs.py

The ZIP includes the dataset, unchanged baseline trainer, scorer, tests and recorded result. It uses the Python standard library and local CPU training. No package installation, model API or data download is needed.

Require one prediction for every reference case

Prediction files can arrive in a different order from the reference labels. Scoring them with a positional zip() would pair unrelated rows; a shorter prediction list could also silently omit difficult cases. Either problem can make a result look better without improving the classifier.

score() indexes predictions by case ID, checks exact coverage and reconstructs reference order before calling the baseline’s existing evaluator. An ID is the digest of the row key used by the baseline. Its purpose here is alignment, not anonymization of an otherwise available dataset.

"""Apply a declared decision-cost contract to ID-aligned predictions."""
import math
from baseline import LABELS, evaluate


def validate_costs(costs):
    if not isinstance(costs, dict) or set(costs) != set(LABELS):
        raise ValueError('cost matrix needs exactly the known actual labels')
    for actual, row in costs.items():
        if not isinstance(row, dict) or set(row) != set(LABELS):
            raise ValueError('each cost row needs exactly the known predictions')
        for predicted, value in row.items():
            if type(value) not in (int, float) or not math.isfinite(value) or value < 0:
                raise ValueError('costs must be finite nonnegative numbers')
            if actual == predicted and value != 0:
                raise ValueError('this error-cost contract requires zero diagonal costs')
    return costs


def score(cases, predictions, costs):
    costs = validate_costs(costs)
    if not cases:
        raise ValueError('cannot score an empty reference set')
    expected = {}
    for case in cases:
        if not isinstance(case['id'], str) or not case['id'] or case['id'] in expected:
            raise ValueError('reference IDs must be nonempty and unique')
        if case['label'] not in LABELS:
            raise ValueError('unknown reference label')
        expected[case['id']] = case
    received = {}
    for prediction in predictions:
        id, label = prediction['id'], prediction['label']
        if not isinstance(id, str) or not id:
            raise ValueError('prediction IDs must be nonempty strings')
        if id in received:
            raise ValueError('duplicate prediction ID')
        if id not in expected or label not in LABELS:
            raise ValueError('unknown prediction ID or label')
        received[id] = label
    if set(received) != set(expected):
        raise ValueError('every reference case needs one prediction')
    # Reuse the baseline's evaluator, but align by ID before its ordered input.
    ordered = [received[case['id']] for case in cases]
    metrics = evaluate(ordered, cases)
    errors = [{'id': case['id'], 'actual': case['label'], 'predicted': predicted,
               'cost': costs[case['label']][predicted]}
              for case, predicted in zip(cases, ordered) if predicted != case['label']]
    total_cost = math.fsum(error['cost'] for error in errors)
    return {'metrics': metrics, 'errors': errors, 'total_cost': total_cost,
            'cost_per_case': round(total_cost / len(cases), 6)}

The scorer refuses duplicate reference IDs, duplicate predictions, unknown IDs or labels, and missing predictions. test_prediction_order_cannot_change_the_ledger reverses the predictions and requires the same result. test_missing_prediction_cannot_be_reported_as_lower_cost requires an error instead of a partial score.

The cost validator also requires every known class pair. Negative values, Boolean values, infinity and NaN are invalid. A missing entry must not quietly default to zero. This contract treats only incorrect decisions as costs and requires a zero diagonal; an application that charges for correct decisions or manual review needs a different, explicit action-cost contract.

Sum the charged mistakes without changing the model

For each mismatched label, the ledger records the case ID, actual class, prediction and matrix entry. The total is the sum of those entries. Dividing by all reference cases gives cost per case, including the correctly classified cases with zero error cost.

validation: 590 ID-aligned cases; fixed majority and logistic classifiers
majority: accuracy=0.5797; errors=248; cost=1240; cost/case=2.101695
logistic: accuracy=0.9271; errors=43; cost=143; cost/case=0.242373
Cost unit: illustrative sorting cost units
Costs are declared scenario assumptions. No monetary impact or threshold improvement was measured.

The majority rule predicts Osmancik throughout this split. Its 248 Cammeo errors each cost five units, producing 1,240 units. Logistic regression produces errors in both directions; inspect the actual counts rather than multiplying all 43 mistakes by one rate:

The learned classifier has fewer errors and lower assumed cost on these validation rows. The cost difference is not a measured financial saving. It also excludes review, operating and deployment costs, and it says nothing about a later population with a different class mix or error pattern.

"""Compare fixed classifiers under a declared cost contract; no policy selection."""
import argparse
import hashlib
import json
from pathlib import Path
from baseline import (load_rows, split_rows, fit_scaler, train_logistic,
                      predict_logistic, majority_label, DATA_SHA256, SALT)
from costs import score

ROOT = Path(__file__).resolve().parent


def record():
    parts = split_rows(load_rows())
    scaler = fit_scaler(parts['training'])
    model = train_logistic(parts['training'], scaler)
    majority = majority_label(parts['training'])
    validation = parts['validation']
    cases = [{'id': hashlib.sha256(row['key'].encode()).hexdigest(), 'label': row['label']}
             for row in validation]
    fixed = {'majority': [majority] * len(cases),
             'logistic': [predict_logistic(model, scaler, row) for row in validation]}
    contract = json.loads((ROOT / 'costs.json').read_text())
    results = {}
    for name, labels in fixed.items():
        predictions = [{'id': case['id'], 'label': label} for case, label in zip(cases, labels)]
        results[name] = score(cases, predictions, contract['matrix'])
    return {'data_sha256': DATA_SHA256, 'split_salt': SALT, 'split': 'validation',
            'cases': len(cases), 'cost_contract': contract, 'results': results}


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--save', type=Path, help='Write a new local result JSON')
    parser.add_argument('--check', action='store_true', help='Compare with the bundled result')
    args = parser.parse_args()
    result = record()
    if args.check and result != json.loads((ROOT / 'result.json').read_text()):
        raise SystemExit('current result differs from the bundled record')
    if args.save:
        args.save.write_text(json.dumps(result, indent=2) + '\n')
    print(f"validation: {result['cases']} ID-aligned cases; fixed majority and logistic classifiers")
    for name, row in result['results'].items():
        print(f"{name}: accuracy={row['metrics']['accuracy']:.4f}; errors={len(row['errors'])}; "
              f"cost={row['total_cost']:g}; cost/case={row['cost_per_case']:.6f}")
    print('Cost unit:', result['cost_contract']['unit'])
    print('Costs are declared scenario assumptions. No monetary impact or threshold improvement was measured.')


if __name__ == '__main__':
    main()

The driver fits preprocessing and logistic regression on training rows, then scores validation predictions. It does not search thresholds, change training settings or select a cost-minimizing model. The test set remains outside this comparison.

Inspect why equal accuracy can imply different consequences

The unit test test_accuracy_tie_can_hide_a_cost_difference uses two reference cases, one of each class. Always predicting Cammeo makes one cheap mistake; always predicting Osmancik makes one expensive mistake. Both achieve 50% accuracy, but their totals are one and five cost units.

That example explains why a metric must follow the application decision. Accuracy remains a useful description of errors; the matrix adds a stated interpretation of their consequences. It does not make the underlying predictions more accurate.

To compare your local execution with the bundled record or save a separate result:

python experiment.py --check
python experiment.py --save my-result.json

The complete result includes data identity, split salt, declared costs, confusion matrices and every charged error. --check compares the recomputed result with that record; a mismatch needs investigation before interpreting a changed cost.

Decide which measurements are still missing

These are static, labeled validation cases. Before using a similar result to choose an operating policy, define the target population, label availability, review options and acceptable uncertainty. Keep the final evaluation separate from the cases used to adjust thresholds. The later calibration and threshold lesson will own that selection step.

Design exercise: make a separate copy of the cost contract and set both error costs to one. Predict the new totals before running the program. Explain which measurements remain unchanged and why a different total alone cannot support a claim that the model improved. Then describe what additional label and action would be needed to account for a human-review option.

Failure cases

Shuffling predictions must not change the ledger. Removing one prediction must stop scoring instead of shrinking the denominator. These tests protect the meaning of the cost comparison before any numerical policy is applied.

Deployment

Store the reference-set identity, predictions and cost policy beside a model-selection decision. Have the application owner review whether those costs represent the intended consequences. The ledger can support a later release policy, but it does not measure business losses or supply a deployment gate by itself.

Scaling

The ledger grows with scored examples. Aggregate by error type when reviewing large reports, while retaining stable case identities for investigation. Operational decisions also need class prevalence and impact data; multiplying this small validation ledger by a traffic estimate would not supply those measurements.

Key takeaways

• Equal error counts can have different consequences under a declared policy.
• Cost changes do not improve the saved predictions.
• A valid comparison requires exactly one prediction per reference case.

Best practices

• Review the complete class-pair cost matrix before scoring.
• Join by case identity rather than position.
• Report error counts with scenario costs so policy assumptions remain visible.

References

• Google Machine Learning Crash Course, accuracy, precision and recall: choose metrics in relation to the task and error consequences.
• Google Machine Learning Crash Course, thresholds: changing a decision threshold changes the balance of classification errors.
• UCI Rice dataset: dataset source, feature definitions and CC BY 4.0 license.

Sources checked October 24, 2026. The error counts are from the bundled local execution; the cost matrix is an authored scenario input.