aiml
SearchMy learning
Menu

Lesson 20 of 20Module 6

Size serving capacity from accepted benchmark results

Size serving replicas from accepted workload-matched results. Convert request demand to peak capacity and include reserves and stated cost assumptions.

Prerequisites

Python, basic causal attention, and the preceding inference lessons.

By the end, you can

  • Size serving capacity from accepted benchmark results.
  • Explain the bottleneck, measurement boundary, and tradeoffs in this implementation.
  • Separate accepted benchmark results from demand forecasts and reserve policy.
Runtime
Python 3.12+
Dependencies
Standard library only

Lab code, tests and fixtures (ZIP, 108 KB)Quiz check your understanding

Progress is saved in this browser.

When you need this. A service owner has a traffic forecast and wants a replica budget. User count alone is insufficient: request rate, prompt/output mix, concurrency, quality requirements, and latency targets determine useful capacity.

What you run. Real capacity-planning code with labeled test inputs. No production fleet, quality pass, market price, or percentage savings is claimed.

This lesson builds on budget cache transfer before splitting inference pools.

Require accepted benchmark results

Start with a workload-matched benchmark that meets the service’s latency and task-quality criteria. Its accepted sustainable request rate becomes a planning input. Convert forecast daily requests into a peak rate, divide by accepted per-replica capacity, round up, and include an explicit reserve.

This arithmetic assumes replicas scale independently. Shared storage, routing, admission, network, and model loading can invalidate that assumption. The calculator therefore requires a benchmark record reference and an explicit acceptance decision before returning a plan. It cannot validate the referenced benchmark itself.

Accepted benchmarkPeak and reserveReplica cost plancapacityreplica planAccepted benchmarkPeak and reserveReplica cost plancapacityreplica plan
Follow the named data through the downloadable program. The article identifies whether the result is measured execution, a controlled fixture, or a planning estimate.

CapacityEvidence separates measured input from the forecast. plan() rejects missing or unaccepted benchmark records and includes reserve replicas in hourly cost. The default output uses clearly labeled unit-test inputs so its arithmetic is reproducible; it is not a million-user fleet estimate.

Calculate a capacity plan

Download the lab from this page, extract it, and run these commands from its directory. The default command reads saved run records or runs the explicitly described local algorithm; it never downloads a model or calls a paid API.

Run the local analysis and checksTerminal · bash
python3 lab.py
python3 -m unittest -v test_lab.py
Read lab.py
lab.pySource · pythonDownload
"""Capacity and cost arithmetic with explicit evidence requirements. No invented latency or quality scores."""

from dataclasses import dataclass, asdict
import json
import math


@dataclass(frozen=True)
class CapacityEvidence:
    sustainable_requests_per_second: float
    hourly_replica_cost: float
    workload: str
    latency_and_quality_accepted: bool
    record_reference: str

    def validate(self):
        if self.sustainable_requests_per_second <= 0 or self.hourly_replica_cost < 0:
            raise ValueError("Invalid rate or price")
        if not self.latency_and_quality_accepted or not self.record_reference:
            raise ValueError("Supply accepted benchmark evidence before sizing")


def plan(daily_requests, peak_to_average, evidence, reserve_replicas=1):
    evidence.validate()
    if daily_requests <= 0 or peak_to_average < 1 or reserve_replicas < 0:
        raise ValueError("Invalid traffic or reserve")
    peak_rps = daily_requests / 86400 * peak_to_average
    active = math.ceil(peak_rps / evidence.sustainable_requests_per_second)
    return {
        "daily_requests_assumed": daily_requests,
        "peak_to_average_assumed": peak_to_average,
        "peak_rps": round(peak_rps, 6),
        "active_replicas": active,
        "reserve_replicas": reserve_replicas,
        "total_replicas": active + reserve_replicas,
        "peak_hour_cost": round(
            (active + reserve_replicas) * evidence.hourly_replica_cost, 2
        ),
        "evidence": asdict(evidence),
        "limits": "Arithmetic assumes independent replicas and identical workload. Load-test routing, shared dependencies, cold starts and a lost replica before deployment.",
    }


def main():
    # Explicit controlled fixture: demonstrates arithmetic, not a real fleet recommendation.
    fixture = CapacityEvidence(
        2.5,
        1.0,
        "controlled calculator test workload",
        True,
        "unit-test fixture; not measured serving capacity",
    )
    print(
        json.dumps(
            {
                "execution_mode": "capacity arithmetic over labeled test inputs",
                "base": plan(100000, 3, fixture),
                "higher_peak": plan(100000, 5, fixture),
                "limits": "No million-user deployment or percentage cost saving is claimed. Replace the test inputs with accepted, workload-matched measurements.",
            },
            indent=2,
        )
    )


if __name__ == "__main__":
    main()

The higher-peak plan requires more active replicas. Reserve adds capacity and cost separately. The input acceptance flag is an explicit caller decision; the calculator cannot prove that a referenced benchmark is sound.

QuantityRecorded value
Base active replicas2
Base reserve replicas1
Higher-peak active replicas3
Base peak-hour cost (test currency inputs)3
Inspect the complete local result
Recorded local resultRecorded output · jsonDownload
{
  "execution_mode": "capacity arithmetic over labeled test inputs",
  "base": {
    "daily_requests_assumed": 100000,
    "peak_to_average_assumed": 3,
    "peak_rps": 3.472222,
    "active_replicas": 2,
    "reserve_replicas": 1,
    "total_replicas": 3,
    "peak_hour_cost": 3.0,
    "evidence": {
      "sustainable_requests_per_second": 2.5,
      "hourly_replica_cost": 1.0,
      "workload": "controlled calculator test workload",
      "latency_and_quality_accepted": true,
      "record_reference": "unit-test fixture; not measured serving capacity"
    },
    "limits": "Arithmetic assumes independent replicas and identical workload. Load-test routing, shared dependencies, cold starts and a lost replica before deployment."
  },
  "higher_peak": {
    "daily_requests_assumed": 100000,
    "peak_to_average_assumed": 5,
    "peak_rps": 5.787037,
    "active_replicas": 3,
    "reserve_replicas": 1,
    "total_replicas": 4,
    "peak_hour_cost": 4.0,
    "evidence": {
      "sustainable_requests_per_second": 2.5,
      "hourly_replica_cost": 1.0,
      "workload": "controlled calculator test workload",
      "latency_and_quality_accepted": true,
      "record_reference": "unit-test fixture; not measured serving capacity"
    },
    "limits": "Arithmetic assumes independent replicas and identical workload. Load-test routing, shared dependencies, cold starts and a lost replica before deployment."
  },
  "limits": "No million-user deployment or percentage cost saving is claimed. Replace the test inputs with accepted, workload-matched measurements."
}
Saved output report for size serving capacity from accepted benchmark results; execution mode and values come from the linked raw result.

This image is an application-generated report of the saved program result, not a new inference run. Open the full-resolution report or read the raw result.

Budget peak demand and reserve

Extra reserve protects capacity when a replica is unavailable but increases idle cost. A peak-sized fleet may waste off-peak capacity, while aggressive scaling can miss bursts during cold starts. Optimization effects interact, so multiplying advertised savings percentages is not a valid cost model.

Unaccepted capacity. Capacity marked unaccepted is refused; a traffic forecast cannot substitute for a benchmark that meets the service’s criteria.

Missing benchmark record. A missing benchmark reference is refused. A supplied reference is still the caller’s responsibility to validate; the calculator only performs arithmetic.

Read the executable checks
Executable checksSource · pythonDownload
import unittest
from lab import CapacityEvidence, plan


class CapacityTests(unittest.TestCase):
    def evidence(self, accepted=True, reference="run.json"):
        return CapacityEvidence(2, 3, "same workload", accepted, reference)

    def test_peak_rate_and_reserve_are_counted(self):
        result = plan(86400, 3, self.evidence(), 1)
        self.assertEqual(result["active_replicas"], 2)
        self.assertEqual(result["total_replicas"], 3)
        self.assertEqual(result["peak_hour_cost"], 9)

    def test_unaccepted_capacity_is_rejected(self):
        with self.assertRaisesRegex(ValueError, "accepted benchmark"):
            plan(86400, 3, self.evidence(False))

    def test_missing_evidence_is_rejected(self):
        with self.assertRaises(ValueError):
            plan(86400, 3, self.evidence(reference=""))

    def test_higher_peak_requires_more_capacity(self):
        self.assertGreater(
            plan(86400, 5, self.evidence())["active_replicas"],
            plan(86400, 2, self.evidence())["active_replicas"],
        )


if __name__ == "__main__":
    unittest.main()


class GuardBoundaryTests(unittest.TestCase):
    def test_rejects_invalid_boundary_inputs_with_specific_reason(self):
        import lab
        cases = [('CapacityEvidence(0,1,"test",True,"record").validate()', 'rate or price'), ('plan(0,1,CapacityEvidence(1,1,"test",True,"record"))', 'traffic or reserve')]
        for code, reason in cases:
            with self.subTest(code=code):
                with self.assertRaisesRegex((ValueError, MemoryError, KeyError, RuntimeError), reason):
                    exec(code, vars(lab).copy())

Try a change and explain the result

Replace the test inputs with an accepted benchmark record for a defined workload. Raise the peak ratio and reserve separately. Identify which inputs require measurement and which are business assumptions before presenting the cost.

What input should drive per-replica capacity in a serving plan?

Choose one answer, then check your reasoning.

Read the explanation

Forecasts determine demand; accepted measurements determine useful capacity. Their assumptions must stay separate.

Recheck combined changes under load

Include unsuccessful requests, input/output tokens, task-quality acceptance, and the full cost denominator. Re-benchmark combined optimizations rather than stacking their individual claimed gains. Validate a replica-loss and traffic-burst scenario before release.

References

Sources checked October 24, 2026. Runtime-specific commands and results apply to the recorded versions; linked rolling documentation can change. The application scenarios are teaching examples, not claims about an author’s production incident.

Written by Arjun Prabhulal.

Module 6

Run at production scale