When you need this. A service owner has a traffic forecast and wants a replica budget. User count alone is insufficient: request rate, prompt/output mix, concurrency, quality requirements, and latency targets determine useful capacity.
What you run. Real capacity-planning code with labeled test inputs. No production fleet, quality pass, market price, or percentage savings is claimed.
This lesson builds on budget cache transfer before splitting inference pools.
Require accepted benchmark results
Start with a workload-matched benchmark that meets the service’s latency and task-quality criteria. Its accepted sustainable request rate becomes a planning input. Convert forecast daily requests into a peak rate, divide by accepted per-replica capacity, round up, and include an explicit reserve.
This arithmetic assumes replicas scale independently. Shared storage, routing, admission, network, and model loading can invalidate that assumption. The calculator therefore requires a benchmark record reference and an explicit acceptance decision before returning a plan. It cannot validate the referenced benchmark itself.
CapacityEvidence separates measured input from the forecast. plan() rejects missing or unaccepted benchmark records and includes reserve replicas in hourly cost. The default output uses clearly labeled unit-test inputs so its arithmetic is reproducible; it is not a million-user fleet estimate.
Calculate a capacity plan
Download the lab from this page, extract it, and run these commands from its directory. The default command reads saved run records or runs the explicitly described local algorithm; it never downloads a model or calls a paid API.
python3 lab.py
python3 -m unittest -v test_lab.pyRead lab.py
"""Capacity and cost arithmetic with explicit evidence requirements. No invented latency or quality scores."""
from dataclasses import dataclass, asdict
import json
import math
@dataclass(frozen=True)
class CapacityEvidence:
sustainable_requests_per_second: float
hourly_replica_cost: float
workload: str
latency_and_quality_accepted: bool
record_reference: str
def validate(self):
if self.sustainable_requests_per_second <= 0 or self.hourly_replica_cost < 0:
raise ValueError("Invalid rate or price")
if not self.latency_and_quality_accepted or not self.record_reference:
raise ValueError("Supply accepted benchmark evidence before sizing")
def plan(daily_requests, peak_to_average, evidence, reserve_replicas=1):
evidence.validate()
if daily_requests <= 0 or peak_to_average < 1 or reserve_replicas < 0:
raise ValueError("Invalid traffic or reserve")
peak_rps = daily_requests / 86400 * peak_to_average
active = math.ceil(peak_rps / evidence.sustainable_requests_per_second)
return {
"daily_requests_assumed": daily_requests,
"peak_to_average_assumed": peak_to_average,
"peak_rps": round(peak_rps, 6),
"active_replicas": active,
"reserve_replicas": reserve_replicas,
"total_replicas": active + reserve_replicas,
"peak_hour_cost": round(
(active + reserve_replicas) * evidence.hourly_replica_cost, 2
),
"evidence": asdict(evidence),
"limits": "Arithmetic assumes independent replicas and identical workload. Load-test routing, shared dependencies, cold starts and a lost replica before deployment.",
}
def main():
# Explicit controlled fixture: demonstrates arithmetic, not a real fleet recommendation.
fixture = CapacityEvidence(
2.5,
1.0,
"controlled calculator test workload",
True,
"unit-test fixture; not measured serving capacity",
)
print(
json.dumps(
{
"execution_mode": "capacity arithmetic over labeled test inputs",
"base": plan(100000, 3, fixture),
"higher_peak": plan(100000, 5, fixture),
"limits": "No million-user deployment or percentage cost saving is claimed. Replace the test inputs with accepted, workload-matched measurements.",
},
indent=2,
)
)
if __name__ == "__main__":
main()The higher-peak plan requires more active replicas. Reserve adds capacity and cost separately. The input acceptance flag is an explicit caller decision; the calculator cannot prove that a referenced benchmark is sound.
| Quantity | Recorded value |
|---|---|
| Base active replicas | 2 |
| Base reserve replicas | 1 |
| Higher-peak active replicas | 3 |
| Base peak-hour cost (test currency inputs) | 3 |
Inspect the complete local result
{
"execution_mode": "capacity arithmetic over labeled test inputs",
"base": {
"daily_requests_assumed": 100000,
"peak_to_average_assumed": 3,
"peak_rps": 3.472222,
"active_replicas": 2,
"reserve_replicas": 1,
"total_replicas": 3,
"peak_hour_cost": 3.0,
"evidence": {
"sustainable_requests_per_second": 2.5,
"hourly_replica_cost": 1.0,
"workload": "controlled calculator test workload",
"latency_and_quality_accepted": true,
"record_reference": "unit-test fixture; not measured serving capacity"
},
"limits": "Arithmetic assumes independent replicas and identical workload. Load-test routing, shared dependencies, cold starts and a lost replica before deployment."
},
"higher_peak": {
"daily_requests_assumed": 100000,
"peak_to_average_assumed": 5,
"peak_rps": 5.787037,
"active_replicas": 3,
"reserve_replicas": 1,
"total_replicas": 4,
"peak_hour_cost": 4.0,
"evidence": {
"sustainable_requests_per_second": 2.5,
"hourly_replica_cost": 1.0,
"workload": "controlled calculator test workload",
"latency_and_quality_accepted": true,
"record_reference": "unit-test fixture; not measured serving capacity"
},
"limits": "Arithmetic assumes independent replicas and identical workload. Load-test routing, shared dependencies, cold starts and a lost replica before deployment."
},
"limits": "No million-user deployment or percentage cost saving is claimed. Replace the test inputs with accepted, workload-matched measurements."
}
This image is an application-generated report of the saved program result, not a new inference run. Open the full-resolution report or read the raw result.
Budget peak demand and reserve
Extra reserve protects capacity when a replica is unavailable but increases idle cost. A peak-sized fleet may waste off-peak capacity, while aggressive scaling can miss bursts during cold starts. Optimization effects interact, so multiplying advertised savings percentages is not a valid cost model.
Unaccepted capacity. Capacity marked unaccepted is refused; a traffic forecast cannot substitute for a benchmark that meets the service’s criteria.
Missing benchmark record. A missing benchmark reference is refused. A supplied reference is still the caller’s responsibility to validate; the calculator only performs arithmetic.
Read the executable checks
import unittest
from lab import CapacityEvidence, plan
class CapacityTests(unittest.TestCase):
def evidence(self, accepted=True, reference="run.json"):
return CapacityEvidence(2, 3, "same workload", accepted, reference)
def test_peak_rate_and_reserve_are_counted(self):
result = plan(86400, 3, self.evidence(), 1)
self.assertEqual(result["active_replicas"], 2)
self.assertEqual(result["total_replicas"], 3)
self.assertEqual(result["peak_hour_cost"], 9)
def test_unaccepted_capacity_is_rejected(self):
with self.assertRaisesRegex(ValueError, "accepted benchmark"):
plan(86400, 3, self.evidence(False))
def test_missing_evidence_is_rejected(self):
with self.assertRaises(ValueError):
plan(86400, 3, self.evidence(reference=""))
def test_higher_peak_requires_more_capacity(self):
self.assertGreater(
plan(86400, 5, self.evidence())["active_replicas"],
plan(86400, 2, self.evidence())["active_replicas"],
)
if __name__ == "__main__":
unittest.main()
class GuardBoundaryTests(unittest.TestCase):
def test_rejects_invalid_boundary_inputs_with_specific_reason(self):
import lab
cases = [('CapacityEvidence(0,1,"test",True,"record").validate()', 'rate or price'), ('plan(0,1,CapacityEvidence(1,1,"test",True,"record"))', 'traffic or reserve')]
for code, reason in cases:
with self.subTest(code=code):
with self.assertRaisesRegex((ValueError, MemoryError, KeyError, RuntimeError), reason):
exec(code, vars(lab).copy())Try a change and explain the result
Replace the test inputs with an accepted benchmark record for a defined workload. Raise the peak ratio and reserve separately. Identify which inputs require measurement and which are business assumptions before presenting the cost.
Read the explanation
Forecasts determine demand; accepted measurements determine useful capacity. Their assumptions must stay separate.
Recheck combined changes under load
Include unsuccessful requests, input/output tokens, task-quality acceptance, and the full cost denominator. Re-benchmark combined optimizations rather than stacking their individual claimed gains. Validate a replica-loss and traffic-burst scenario before release.
References
Sources checked October 24, 2026. Runtime-specific commands and results apply to the recorded versions; linked rolling documentation can change. The application scenarios are teaching examples, not claims about an author’s production incident.