When you need this. A code assistant needs responsive streams during the day and high throughput for background code summaries at night. The fastest single request and the cheapest bulk configuration need not be the same.
What you run. Constraint and Pareto-selection code over an assumed multi-device cost model; linked real static-batch timings use a single local model.
This lesson builds on check speculative decoding correctness and its cost.
Filter infeasible operating points
An operating point includes active batch, context distribution, precision, and parallelism. A Pareto frontier retains configurations where no alternative improves one objective without worsening another. The lab removes configurations that do not fit device memory, builds the frontier from the rest, and then applies the latency cap.
Tensor parallelism splits parts of a model across devices. The devices then exchange partial results through collective communication; the lab models two all-reduces per layer. A larger batch reads each weight once for more requests, but it also adds KV cache reads for every active request. The calculator reports each of these terms. Its effective bandwidth and communication coefficients are assumed values, not measurements.
pareto_frontier() compares feasible configurations and select_pareto_config() applies the latency cap.
The supplied sweep applies a 25 ms TPOT cap to configurations with different batches and device counts. Follow one candidate through its memory check before its latency and throughput comparison. A point that exceeds capacity cannot become a useful choice merely by reporting more tokens per second. In the saved run, fp8-tp2-b128-nvlink reports 2003.3 tokens/s per device, more than the selected point, but it needs 81.45 GB per device against the assumed 80 GB.
Select a feasible operating point
Download the lab from this page, extract it, and run these commands from its directory. lab.py runs the cost model on the CPU with the Python standard library. It does not download a model or call a paid API.
python3 lab.py
python3 -m unittest -v test_lab.pyRead lab.py
"""Throughput-latency Pareto optimizer for LLM decode serving configurations."""
from __future__ import annotations
from dataclasses import asdict, dataclass
import json
@dataclass(frozen=True)
class ServingConfig:
name: str
precision: str # "bf16" or "fp8"
tp_degree: int # 1, 2, 4, 8
batch_size: int
interconnect: str = "nvlink" # "nvlink" or "pcie"
context_len: int = 4096
params_b: float = 70.0
num_layers: int = 80
hidden_dim: int = 8192
num_kv_heads: int = 8
head_dim: int = 128
gpu_mem_gb: float = 80.0
@dataclass(frozen=True)
class DecodeStepProfile:
name: str
precision: str
tp_degree: int
batch_size: int
interconnect: str
fits_hbm: bool
memory_per_gpu_gb: float
weight_read_ms: float
kv_read_ms: float
compute_ms: float
allreduce_ms: float
tpot_ms: float
total_tokens_per_s: float
tokens_per_s_per_gpu: float
bottleneck: str
def compute_decode_step_profile(config: ServingConfig) -> DecodeStepProfile:
if config.tp_degree not in (1, 2, 4, 8):
raise ValueError(f"Unsupported tp_degree: {config.tp_degree}")
if config.batch_size < 1:
raise ValueError("batch_size must be >= 1")
if config.precision not in ("bf16", "fp8"):
raise ValueError(f"Unsupported precision: {config.precision}")
if config.interconnect not in ("nvlink", "pcie"):
raise ValueError(f"Unsupported interconnect: {config.interconnect}")
elem_bytes = 2 if config.precision == "bf16" else 1
eff_tflops = 650.0 if config.precision == "bf16" else 1300.0
hbm_eff_gb_s = 2680.0 # H100 SXM effective HBM3 stream bandwidth
weight_bytes_per_gpu = (config.params_b * 1e9 * elem_bytes) / config.tp_degree
kv_bytes_per_token = (
2 * config.num_layers * config.num_kv_heads * config.head_dim * elem_bytes
)
kv_bytes_per_gpu = (
config.batch_size * config.context_len * kv_bytes_per_token
) / config.tp_degree
activation_reserve_gb = 3.5
memory_per_gpu_gb = (
(weight_bytes_per_gpu + kv_bytes_per_gpu) / 1e9
) + activation_reserve_gb
fits_hbm = memory_per_gpu_gb <= config.gpu_mem_gb
weight_read_ms = (weight_bytes_per_gpu / (hbm_eff_gb_s * 1e9)) * 1000.0
kv_read_ms = (kv_bytes_per_gpu / (hbm_eff_gb_s * 1e9)) * 1000.0
mem_ms = weight_read_ms + kv_read_ms
linear_flops = 2.0 * config.params_b * 1e9 * config.batch_size
attn_flops = (
4.0
* config.num_layers
* config.batch_size
* config.context_len
* config.hidden_dim
)
flops_per_gpu = (linear_flops + attn_flops) / config.tp_degree
compute_ms = (flops_per_gpu / (eff_tflops * 1e12)) * 1000.0
kernel_ms = max(mem_ms, compute_ms) + 0.15 * min(mem_ms, compute_ms)
if config.tp_degree == 1:
allreduce_ms = 0.0
else:
if config.interconnect == "nvlink":
bw_gb_s = 400.0
sync_us = 3.5
else:
bw_gb_s = 16.0
sync_us = 25.0
# 2 all-reduces per transformer layer on BF16 (2-byte) hidden activations
payload_bytes = config.batch_size * config.hidden_dim * 2
ring_bytes_per_ar = (
2.0 * ((config.tp_degree - 1) / config.tp_degree) * payload_bytes
)
total_ar_bytes = 2 * config.num_layers * ring_bytes_per_ar
ar_transfer_ms = (total_ar_bytes / (bw_gb_s * 1e9)) * 1000.0
ar_sync_ms = (2 * config.num_layers * sync_us) / 1000.0
allreduce_ms = ar_transfer_ms + ar_sync_ms
fixed_launch_ms = 0.35
tpot_ms = kernel_ms + allreduce_ms + fixed_launch_ms
total_tokens_per_s = config.batch_size / (tpot_ms / 1000.0)
tokens_per_s_per_gpu = total_tokens_per_s / config.tp_degree
if allreduce_ms >= max(weight_read_ms, kv_read_ms, compute_ms):
bottleneck = "allreduce_comm"
elif compute_ms >= mem_ms:
bottleneck = "tensor_compute"
elif kv_read_ms > weight_read_ms:
bottleneck = "kv_bandwidth"
else:
bottleneck = "weight_bandwidth"
return DecodeStepProfile(
name=config.name,
precision=config.precision,
tp_degree=config.tp_degree,
batch_size=config.batch_size,
interconnect=config.interconnect,
fits_hbm=fits_hbm,
memory_per_gpu_gb=round(memory_per_gpu_gb, 2),
weight_read_ms=round(weight_read_ms, 2),
kv_read_ms=round(kv_read_ms, 2),
compute_ms=round(compute_ms, 2),
allreduce_ms=round(allreduce_ms, 2),
tpot_ms=round(tpot_ms, 2),
total_tokens_per_s=round(total_tokens_per_s, 1),
tokens_per_s_per_gpu=round(tokens_per_s_per_gpu, 1),
bottleneck=bottleneck,
)
def pareto_frontier(profiles: list[DecodeStepProfile]) -> list[DecodeStepProfile]:
feasible = [p for p in profiles if p.fits_hbm]
frontier: list[DecodeStepProfile] = []
for candidate in feasible:
dominated = False
for other in feasible:
if other is candidate:
continue
if (
other.tpot_ms <= candidate.tpot_ms
and other.tokens_per_s_per_gpu >= candidate.tokens_per_s_per_gpu
and (
other.tpot_ms < candidate.tpot_ms
or other.tokens_per_s_per_gpu > candidate.tokens_per_s_per_gpu
)
):
dominated = True
break
if not dominated:
frontier.append(candidate)
return sorted(frontier, key=lambda p: (p.tpot_ms, -p.tokens_per_s_per_gpu))
def select_pareto_config(
configs: list[ServingConfig], sla_tpot_ms: float = 25.0
) -> dict[str, object]:
profiles = [compute_decode_step_profile(c) for c in configs]
frontier = pareto_frontier(profiles)
sla_matches = [p for p in frontier if p.tpot_ms <= sla_tpot_ms]
if not sla_matches:
raise ValueError(
f"No HBM-feasible configuration satisfies tpot_ms <= {sla_tpot_ms}"
)
best = max(sla_matches, key=lambda p: (p.tokens_per_s_per_gpu, -p.tpot_ms))
return {
"sla_tpot_ms": sla_tpot_ms,
"selected": asdict(best),
"pareto_frontier": [asdict(p) for p in frontier],
"all_profiles": [asdict(p) for p in profiles],
}
def build_candidate_sweep() -> list[ServingConfig]:
candidates: list[ServingConfig] = []
for precision in ("bf16", "fp8"):
for tp in (1, 2, 4, 8):
for batch in (1, 8, 16, 32, 64, 96, 128):
candidates.append(
ServingConfig(
name=f"{precision}-tp{tp}-b{batch}-nvlink",
precision=precision,
tp_degree=tp,
batch_size=batch,
interconnect="nvlink",
)
)
return candidates
def main() -> None:
sweep = build_candidate_sweep()
decision = select_pareto_config(sweep, sla_tpot_ms=25.0)
pcie_comparison = compute_decode_step_profile(
ServingConfig(
name="fp8-tp4-b64-pcie",
precision="fp8",
tp_degree=4,
batch_size=64,
interconnect="pcie",
)
)
naive_low_latency = compute_decode_step_profile(
ServingConfig(
name="bf16-tp8-b1-nvlink",
precision="bf16",
tp_degree=8,
batch_size=1,
interconnect="nvlink",
)
)
tp2_b64 = compute_decode_step_profile(
ServingConfig(
name="fp8-tp2-b64-nvlink",
precision="fp8",
tp_degree=2,
batch_size=64,
interconnect="nvlink",
)
)
tp2_b128 = compute_decode_step_profile(
ServingConfig(
name="fp8-tp2-b128-nvlink",
precision="fp8",
tp_degree=2,
batch_size=128,
interconnect="nvlink",
)
)
report = {
"model": "70B-GQA-80L",
"context_len": 4096,
"sla_tpot_ms": decision["sla_tpot_ms"],
"selected": decision["selected"],
"naive_low_latency_baseline": asdict(naive_low_latency),
"pcie_tp4_b64_comparison": asdict(pcie_comparison),
"tp2_batch_scaling": [asdict(tp2_b64), asdict(tp2_b128)],
"pareto_frontier": decision["pareto_frontier"],
}
print(
json.dumps(
{
"execution_mode": "CPU algorithm with controlled inputs; any latency/capacity coefficients are assumptions",
**report,
},
indent=2,
)
)
if __name__ == "__main__":
main()The selector removes infeasible points before it compares objectives. From the points that meet the cap, it picks the highest tokens per second per device: fp8-tp4-b128-nvlink in the saved run. Its multi-device times come from the assumed cost model, so do not compare them with the measured static-batch sweep in the first lab.
Inspect the complete local result
{
"execution_mode": "CPU algorithm with controlled inputs; any latency/capacity coefficients are assumptions",
"model": "70B-GQA-80L",
"context_len": 4096,
"sla_tpot_ms": 25.0,
"selected": {
"name": "fp8-tp4-b128-nvlink",
"precision": "fp8",
"tp_degree": 4,
"batch_size": 128,
"interconnect": "nvlink",
"fits_hbm": true,
"memory_per_gpu_gb": 42.47,
"weight_read_ms": 6.53,
"kv_read_ms": 8.01,
"compute_ms": 3.71,
"allreduce_ms": 1.82,
"tpot_ms": 17.27,
"total_tokens_per_s": 7412.7,
"tokens_per_s_per_gpu": 1853.2,
"bottleneck": "kv_bandwidth"
},
"naive_low_latency_baseline": {
"name": "bf16-tp8-b1-nvlink",
"precision": "bf16",
"tp_degree": 8,
"batch_size": 1,
"interconnect": "nvlink",
"fits_hbm": true,
"memory_per_gpu_gb": 21.17,
"weight_read_ms": 6.53,
"kv_read_ms": 0.06,
"compute_ms": 0.03,
"allreduce_ms": 0.57,
"tpot_ms": 7.52,
"total_tokens_per_s": 133.0,
"tokens_per_s_per_gpu": 16.6,
"bottleneck": "weight_bandwidth"
},
"pcie_tp4_b64_comparison": {
"name": "fp8-tp4-b64-pcie",
"precision": "fp8",
"tp_degree": 4,
"batch_size": 64,
"interconnect": "pcie",
"fits_hbm": true,
"memory_per_gpu_gb": 31.74,
"weight_read_ms": 6.53,
"kv_read_ms": 4.01,
"compute_ms": 1.86,
"allreduce_ms": 19.73,
"tpot_ms": 30.89,
"total_tokens_per_s": 2071.6,
"tokens_per_s_per_gpu": 517.9,
"bottleneck": "allreduce_comm"
},
"tp2_batch_scaling": [
{
"name": "fp8-tp2-b64-nvlink",
"precision": "fp8",
"tp_degree": 2,
"batch_size": 64,
"interconnect": "nvlink",
"fits_hbm": true,
"memory_per_gpu_gb": 59.97,
"weight_read_ms": 13.06,
"kv_read_ms": 8.01,
"compute_ms": 3.71,
"allreduce_ms": 0.98,
"tpot_ms": 22.96,
"total_tokens_per_s": 2787.6,
"tokens_per_s_per_gpu": 1393.8,
"bottleneck": "weight_bandwidth"
},
{
"name": "fp8-tp2-b128-nvlink",
"precision": "fp8",
"tp_degree": 2,
"batch_size": 128,
"interconnect": "nvlink",
"fits_hbm": false,
"memory_per_gpu_gb": 81.45,
"weight_read_ms": 13.06,
"kv_read_ms": 16.03,
"compute_ms": 7.42,
"allreduce_ms": 1.4,
"tpot_ms": 31.95,
"total_tokens_per_s": 4006.5,
"tokens_per_s_per_gpu": 2003.3,
"bottleneck": "kv_bandwidth"
}
],
"pareto_frontier": [
{
"name": "fp8-tp8-b1-nvlink",
"precision": "fp8",
"tp_degree": 8,
"batch_size": 1,
"interconnect": "nvlink",
"fits_hbm": true,
"memory_per_gpu_gb": 12.33,
"weight_read_ms": 3.26,
"kv_read_ms": 0.03,
"compute_ms": 0.01,
"allreduce_ms": 0.57,
"tpot_ms": 4.22,
"total_tokens_per_s": 237.0,
"tokens_per_s_per_gpu": 29.6,
"bottleneck": "weight_bandwidth"
},
{
"name": "fp8-tp8-b8-nvlink",
"precision": "fp8",
"tp_degree": 8,
"batch_size": 8,
"interconnect": "nvlink",
"fits_hbm": true,
"memory_per_gpu_gb": 12.92,
"weight_read_ms": 3.26,
"kv_read_ms": 0.25,
"compute_ms": 0.12,
"allreduce_ms": 0.65,
"tpot_ms": 4.53,
"total_tokens_per_s": 1764.3,
"tokens_per_s_per_gpu": 220.5,
"bottleneck": "weight_bandwidth"
},
{
"name": "fp8-tp8-b16-nvlink",
"precision": "fp8",
"tp_degree": 8,
"batch_size": 16,
"interconnect": "nvlink",
"fits_hbm": true,
"memory_per_gpu_gb": 13.59,
"weight_read_ms": 3.26,
"kv_read_ms": 0.5,
"compute_ms": 0.23,
"allreduce_ms": 0.74,
"tpot_ms": 4.89,
"total_tokens_per_s": 3269.3,
"tokens_per_s_per_gpu": 408.7,
"bottleneck": "weight_bandwidth"
},
{
"name": "fp8-tp8-b32-nvlink",
"precision": "fp8",
"tp_degree": 8,
"batch_size": 32,
"interconnect": "nvlink",
"fits_hbm": true,
"memory_per_gpu_gb": 14.93,
"weight_read_ms": 3.26,
"kv_read_ms": 1.0,
"compute_ms": 0.46,
"allreduce_ms": 0.93,
"tpot_ms": 5.61,
"total_tokens_per_s": 5700.9,
"tokens_per_s_per_gpu": 712.6,
"bottleneck": "weight_bandwidth"
},
{
"name": "fp8-tp8-b64-nvlink",
"precision": "fp8",
"tp_degree": 8,
"batch_size": 64,
"interconnect": "nvlink",
"fits_hbm": true,
"memory_per_gpu_gb": 17.62,
"weight_read_ms": 3.26,
"kv_read_ms": 2.0,
"compute_ms": 0.93,
"allreduce_ms": 1.29,
"tpot_ms": 7.05,
"total_tokens_per_s": 9076.3,
"tokens_per_s_per_gpu": 1134.5,
"bottleneck": "weight_bandwidth"
},
{
"name": "fp8-tp8-b96-nvlink",
"precision": "fp8",
"tp_degree": 8,
"batch_size": 96,
"interconnect": "nvlink",
"fits_hbm": true,
"memory_per_gpu_gb": 20.3,
"weight_read_ms": 3.26,
"kv_read_ms": 3.0,
"compute_ms": 1.39,
"allreduce_ms": 1.66,
"tpot_ms": 8.49,
"total_tokens_per_s": 11308.1,
"tokens_per_s_per_gpu": 1413.5,
"bottleneck": "weight_bandwidth"
},
{
"name": "fp8-tp8-b128-nvlink",
"precision": "fp8",
"tp_degree": 8,
"batch_size": 128,
"interconnect": "nvlink",
"fits_hbm": true,
"memory_per_gpu_gb": 22.99,
"weight_read_ms": 3.26,
"kv_read_ms": 4.01,
"compute_ms": 1.86,
"allreduce_ms": 2.03,
"tpot_ms": 9.93,
"total_tokens_per_s": 12893.2,
"tokens_per_s_per_gpu": 1611.6,
"bottleneck": "kv_bandwidth"
},
{
"name": "fp8-tp4-b96-nvlink",
"precision": "fp8",
"tp_degree": 4,
"batch_size": 96,
"interconnect": "nvlink",
"fits_hbm": true,
"memory_per_gpu_gb": 37.11,
"weight_read_ms": 6.53,
"kv_read_ms": 6.01,
"compute_ms": 2.78,
"allreduce_ms": 1.5,
"tpot_ms": 14.81,
"total_tokens_per_s": 6481.8,
"tokens_per_s_per_gpu": 1620.4,
"bottleneck": "weight_bandwidth"
},
{
"name": "fp8-tp4-b128-nvlink",
"precision": "fp8",
"tp_degree": 4,
"batch_size": 128,
"interconnect": "nvlink",
"fits_hbm": true,
"memory_per_gpu_gb": 42.47,
"weight_read_ms": 6.53,
"kv_read_ms": 8.01,
"compute_ms": 3.71,
"allreduce_ms": 1.82,
"tpot_ms": 17.27,
"total_tokens_per_s": 7412.7,
"tokens_per_s_per_gpu": 1853.2,
"bottleneck": "kv_bandwidth"
}
]
}
This report shows the saved output of python3 lab.py, a CPU calculation with assumed coefficients. Open the full-resolution report or read the raw result.
Read the latency-throughput frontier
Lower TPOT can consume more devices per request and reduce throughput per device. Larger batches may raise throughput while worsening latency or exceeding memory. In the saved run, bf16-tp8-b1-nvlink reaches 7.52 ms TPOT but only 16.6 tokens/s per device. Do not judge a configuration by single-request decode time or aggregate tokens per second alone. When configurations use different device counts, compare tokens per second per device: fp8-tp8-b128-nvlink has the highest aggregate rate on the frontier, but the selected tp4 point does more work per device.
Communication cost. A slower assumed interconnect raises collective communication time and can dominate the tensor-parallel step estimate. Over PCIe, fp8-tp4-b64-pcie spends 19.73 ms of its 30.89 ms TPOT in all-reduce and misses the 25 ms cap. Check the interconnect before you choose a tensor-parallel degree.
Latency and memory feasibility. An oversized batch can break the latency cap, the memory limit, or both. At tp2, going from batch 64 to batch 128 raises TPOT from 22.96 ms to 31.95 ms and memory to 81.45 GB per device. The selector removes such points instead of offering them as high-throughput choices.
Read the executable checks
"""Unit and failure-boundary tests for the decode latency-throughput Pareto optimizer."""
from __future__ import annotations
import unittest
from lab import (
ServingConfig,
build_candidate_sweep,
compute_decode_step_profile,
pareto_frontier,
select_pareto_config,
)
class TestLatencyThroughputPareto(unittest.TestCase):
def test_selected_config_satisfies_sla_and_beats_small_batch(self) -> None:
sweep = build_candidate_sweep()
result = select_pareto_config(sweep, sla_tpot_ms=25.0)
selected = result["selected"]
self.assertTrue(selected["fits_hbm"])
self.assertLessEqual(selected["tpot_ms"], 25.0)
self.assertEqual(selected["precision"], "fp8")
self.assertGreater(selected["tokens_per_s_per_gpu"], 600.0)
def test_bf16_single_gpu_70b_rejected_by_hbm_capacity(self) -> None:
profile = compute_decode_step_profile(
ServingConfig(
name="bf16-tp1-b16",
precision="bf16",
tp_degree=1,
batch_size=16,
)
)
self.assertFalse(profile.fits_hbm)
self.assertGreater(profile.memory_per_gpu_gb, 80.0)
def test_tensor_parallel_over_pcie_adds_allreduce_bottleneck(self) -> None:
nvlink_profile = compute_decode_step_profile(
ServingConfig(
name="fp8-tp4-b64-nvlink",
precision="fp8",
tp_degree=4,
batch_size=64,
interconnect="nvlink",
)
)
pcie_profile = compute_decode_step_profile(
ServingConfig(
name="fp8-tp4-b64-pcie",
precision="fp8",
tp_degree=4,
batch_size=64,
interconnect="pcie",
)
)
self.assertLessEqual(nvlink_profile.tpot_ms, 25.0)
self.assertGreater(pcie_profile.tpot_ms, 25.0)
self.assertEqual(pcie_profile.bottleneck, "allreduce_comm")
self.assertGreater(pcie_profile.allreduce_ms, nvlink_profile.allreduce_ms * 5.0)
def test_oversized_batch_pushes_tpot_past_sla(self) -> None:
b64 = compute_decode_step_profile(
ServingConfig(
name="fp8-tp2-b64-nvlink",
precision="fp8",
tp_degree=2,
batch_size=64,
interconnect="nvlink",
)
)
b128 = compute_decode_step_profile(
ServingConfig(
name="fp8-tp2-b128-nvlink",
precision="fp8",
tp_degree=2,
batch_size=128,
interconnect="nvlink",
)
)
self.assertLessEqual(b64.tpot_ms, 25.0)
self.assertGreater(b128.tpot_ms, 25.0)
self.assertGreater(b128.kv_read_ms, b64.kv_read_ms)
def test_pareto_frontier_excludes_dominated_points(self) -> None:
profiles = [compute_decode_step_profile(c) for c in build_candidate_sweep()]
frontier = pareto_frontier(profiles)
for p in frontier:
self.assertTrue(p.fits_hbm)
for other in frontier:
if other is p:
continue
self.assertFalse(
other.tpot_ms <= p.tpot_ms
and other.tokens_per_s_per_gpu >= p.tokens_per_s_per_gpu
and (
other.tpot_ms < p.tpot_ms
or other.tokens_per_s_per_gpu > p.tokens_per_s_per_gpu
)
)
if __name__ == "__main__":
unittest.main()
class GuardBoundaryTests(unittest.TestCase):
def test_rejects_invalid_boundary_inputs_with_specific_reason(self):
import lab
cases = [('compute_decode_step_profile(ServingConfig("x","bf16",3,1))', 'Unsupported tp'), ('compute_decode_step_profile(ServingConfig("x","bf16",1,0))', 'batch_size'), ('compute_decode_step_profile(ServingConfig("x","bad",1,1))', 'Unsupported precision'), ('compute_decode_step_profile(ServingConfig("x","bf16",1,1,"bad"))', 'Unsupported interconnect'), ('select_pareto_config([])', 'No HBM-feasible')]
for code, reason in cases:
with self.subTest(code=code):
with self.assertRaisesRegex((ValueError, MemoryError, KeyError, RuntimeError), reason):
exec(code, vars(lab).copy())Try a change and explain the result
Find a configuration that is dominated and explain the two quantities that eliminate it. Tighten the latency target and observe which candidates disappear. Relate this to the measured batch sweep without claiming its results predict tensor-parallel performance.
Read the explanation
Latency, throughput, memory fit, and cost define different constraints. There is no single best operating point.
Choose targets for the application
Set the latency cap from the application’s service goals, such as responsive daytime streams versus overnight summaries. Use the cost model to shortlist candidates, then benchmark the shortlist on the actual context distribution; this lab fixes context at 4,096 tokens. Check memory fit and reject unsupported precision/sharding combinations before you compare cost.
References
Sources checked October 24, 2026. Runtime-specific commands and results apply to the recorded versions; linked rolling documentation can change.