aiml

Lesson 13 of 20Module 4

Choose a latency-throughput operating point

Choose a feasible latency and throughput operating point. Inspect memory limits, communication assumptions and efficiency per serving device.

Prerequisites

Python, basic causal attention, and the preceding inference lessons.

By the end, you can

  • Choose a latency-throughput operating point.
  • Explain why capacity feasibility must be checked before computing a latency-throughput frontier.
  • Remove infeasible configurations before selecting a latency-throughput tradeoff.
Runtime
Python 3.12+
Dependencies
Standard library only

Lab code, tests and fixtures (ZIP, 101 KB)Quiz check your understanding

Progress is saved in this browser.

When you need this. A code assistant needs responsive streams during the day and high throughput for background code summaries at night. The fastest single request and the cheapest bulk configuration need not be the same.

What you run. Constraint and Pareto-selection code over an assumed multi-device cost model; linked real static-batch timings use a single local model.

This lesson builds on check speculative decoding correctness and its cost.

Filter infeasible operating points

An operating point includes active batch, context distribution, precision, and parallelism. A Pareto frontier retains configurations where no alternative improves one objective without worsening another. The lab removes configurations that do not fit device memory, builds the frontier from the rest, and then applies the latency cap.

Tensor parallelism splits parts of a model across devices. The devices then exchange partial results through collective communication; the lab models two all-reduces per layer. A larger batch reads each weight once for more requests, but it also adds KV cache reads for every active request. The calculator reports each of these terms. Its effective bandwidth and communication coefficients are assumed values, not measurements.

Candidate settingsFeasibility filterPareto frontiersettingsfeasible setCandidate settingsFeasibility filterPareto frontiersettingsfeasible set
Follow the named data through the downloadable program. The article identifies whether the result is measured execution, a controlled fixture, or a planning estimate.

pareto_frontier() compares feasible configurations and select_pareto_config() applies the latency cap.

The supplied sweep applies a 25 ms TPOT cap to configurations with different batches and device counts. Follow one candidate through its memory check before its latency and throughput comparison. A point that exceeds capacity cannot become a useful choice merely by reporting more tokens per second. In the saved run, fp8-tp2-b128-nvlink reports 2003.3 tokens/s per device, more than the selected point, but it needs 81.45 GB per device against the assumed 80 GB.

Select a feasible operating point

Download the lab from this page, extract it, and run these commands from its directory. lab.py runs the cost model on the CPU with the Python standard library. It does not download a model or call a paid API.

Run the local analysis and checksTerminal · bash
python3 lab.py
python3 -m unittest -v test_lab.py
Read lab.py
lab.pySource · pythonDownload
"""Throughput-latency Pareto optimizer for LLM decode serving configurations."""

from __future__ import annotations

from dataclasses import asdict, dataclass
import json


@dataclass(frozen=True)
class ServingConfig:
    name: str
    precision: str  # "bf16" or "fp8"
    tp_degree: int  # 1, 2, 4, 8
    batch_size: int
    interconnect: str = "nvlink"  # "nvlink" or "pcie"
    context_len: int = 4096
    params_b: float = 70.0
    num_layers: int = 80
    hidden_dim: int = 8192
    num_kv_heads: int = 8
    head_dim: int = 128
    gpu_mem_gb: float = 80.0


@dataclass(frozen=True)
class DecodeStepProfile:
    name: str
    precision: str
    tp_degree: int
    batch_size: int
    interconnect: str
    fits_hbm: bool
    memory_per_gpu_gb: float
    weight_read_ms: float
    kv_read_ms: float
    compute_ms: float
    allreduce_ms: float
    tpot_ms: float
    total_tokens_per_s: float
    tokens_per_s_per_gpu: float
    bottleneck: str


def compute_decode_step_profile(config: ServingConfig) -> DecodeStepProfile:
    if config.tp_degree not in (1, 2, 4, 8):
        raise ValueError(f"Unsupported tp_degree: {config.tp_degree}")
    if config.batch_size < 1:
        raise ValueError("batch_size must be >= 1")
    if config.precision not in ("bf16", "fp8"):
        raise ValueError(f"Unsupported precision: {config.precision}")
    if config.interconnect not in ("nvlink", "pcie"):
        raise ValueError(f"Unsupported interconnect: {config.interconnect}")

    elem_bytes = 2 if config.precision == "bf16" else 1
    eff_tflops = 650.0 if config.precision == "bf16" else 1300.0
    hbm_eff_gb_s = 2680.0  # H100 SXM effective HBM3 stream bandwidth

    weight_bytes_per_gpu = (config.params_b * 1e9 * elem_bytes) / config.tp_degree
    kv_bytes_per_token = (
        2 * config.num_layers * config.num_kv_heads * config.head_dim * elem_bytes
    )
    kv_bytes_per_gpu = (
        config.batch_size * config.context_len * kv_bytes_per_token
    ) / config.tp_degree
    activation_reserve_gb = 3.5

    memory_per_gpu_gb = (
        (weight_bytes_per_gpu + kv_bytes_per_gpu) / 1e9
    ) + activation_reserve_gb
    fits_hbm = memory_per_gpu_gb <= config.gpu_mem_gb

    weight_read_ms = (weight_bytes_per_gpu / (hbm_eff_gb_s * 1e9)) * 1000.0
    kv_read_ms = (kv_bytes_per_gpu / (hbm_eff_gb_s * 1e9)) * 1000.0
    mem_ms = weight_read_ms + kv_read_ms

    linear_flops = 2.0 * config.params_b * 1e9 * config.batch_size
    attn_flops = (
        4.0
        * config.num_layers
        * config.batch_size
        * config.context_len
        * config.hidden_dim
    )
    flops_per_gpu = (linear_flops + attn_flops) / config.tp_degree
    compute_ms = (flops_per_gpu / (eff_tflops * 1e12)) * 1000.0

    kernel_ms = max(mem_ms, compute_ms) + 0.15 * min(mem_ms, compute_ms)

    if config.tp_degree == 1:
        allreduce_ms = 0.0
    else:
        if config.interconnect == "nvlink":
            bw_gb_s = 400.0
            sync_us = 3.5
        else:
            bw_gb_s = 16.0
            sync_us = 25.0
        # 2 all-reduces per transformer layer on BF16 (2-byte) hidden activations
        payload_bytes = config.batch_size * config.hidden_dim * 2
        ring_bytes_per_ar = (
            2.0 * ((config.tp_degree - 1) / config.tp_degree) * payload_bytes
        )
        total_ar_bytes = 2 * config.num_layers * ring_bytes_per_ar
        ar_transfer_ms = (total_ar_bytes / (bw_gb_s * 1e9)) * 1000.0
        ar_sync_ms = (2 * config.num_layers * sync_us) / 1000.0
        allreduce_ms = ar_transfer_ms + ar_sync_ms

    fixed_launch_ms = 0.35
    tpot_ms = kernel_ms + allreduce_ms + fixed_launch_ms

    total_tokens_per_s = config.batch_size / (tpot_ms / 1000.0)
    tokens_per_s_per_gpu = total_tokens_per_s / config.tp_degree

    if allreduce_ms >= max(weight_read_ms, kv_read_ms, compute_ms):
        bottleneck = "allreduce_comm"
    elif compute_ms >= mem_ms:
        bottleneck = "tensor_compute"
    elif kv_read_ms > weight_read_ms:
        bottleneck = "kv_bandwidth"
    else:
        bottleneck = "weight_bandwidth"

    return DecodeStepProfile(
        name=config.name,
        precision=config.precision,
        tp_degree=config.tp_degree,
        batch_size=config.batch_size,
        interconnect=config.interconnect,
        fits_hbm=fits_hbm,
        memory_per_gpu_gb=round(memory_per_gpu_gb, 2),
        weight_read_ms=round(weight_read_ms, 2),
        kv_read_ms=round(kv_read_ms, 2),
        compute_ms=round(compute_ms, 2),
        allreduce_ms=round(allreduce_ms, 2),
        tpot_ms=round(tpot_ms, 2),
        total_tokens_per_s=round(total_tokens_per_s, 1),
        tokens_per_s_per_gpu=round(tokens_per_s_per_gpu, 1),
        bottleneck=bottleneck,
    )


def pareto_frontier(profiles: list[DecodeStepProfile]) -> list[DecodeStepProfile]:
    feasible = [p for p in profiles if p.fits_hbm]
    frontier: list[DecodeStepProfile] = []
    for candidate in feasible:
        dominated = False
        for other in feasible:
            if other is candidate:
                continue
            if (
                other.tpot_ms <= candidate.tpot_ms
                and other.tokens_per_s_per_gpu >= candidate.tokens_per_s_per_gpu
                and (
                    other.tpot_ms < candidate.tpot_ms
                    or other.tokens_per_s_per_gpu > candidate.tokens_per_s_per_gpu
                )
            ):
                dominated = True
                break
        if not dominated:
            frontier.append(candidate)
    return sorted(frontier, key=lambda p: (p.tpot_ms, -p.tokens_per_s_per_gpu))


def select_pareto_config(
    configs: list[ServingConfig], sla_tpot_ms: float = 25.0
) -> dict[str, object]:
    profiles = [compute_decode_step_profile(c) for c in configs]
    frontier = pareto_frontier(profiles)
    sla_matches = [p for p in frontier if p.tpot_ms <= sla_tpot_ms]
    if not sla_matches:
        raise ValueError(
            f"No HBM-feasible configuration satisfies tpot_ms <= {sla_tpot_ms}"
        )
    best = max(sla_matches, key=lambda p: (p.tokens_per_s_per_gpu, -p.tpot_ms))
    return {
        "sla_tpot_ms": sla_tpot_ms,
        "selected": asdict(best),
        "pareto_frontier": [asdict(p) for p in frontier],
        "all_profiles": [asdict(p) for p in profiles],
    }


def build_candidate_sweep() -> list[ServingConfig]:
    candidates: list[ServingConfig] = []
    for precision in ("bf16", "fp8"):
        for tp in (1, 2, 4, 8):
            for batch in (1, 8, 16, 32, 64, 96, 128):
                candidates.append(
                    ServingConfig(
                        name=f"{precision}-tp{tp}-b{batch}-nvlink",
                        precision=precision,
                        tp_degree=tp,
                        batch_size=batch,
                        interconnect="nvlink",
                    )
                )
    return candidates


def main() -> None:
    sweep = build_candidate_sweep()
    decision = select_pareto_config(sweep, sla_tpot_ms=25.0)
    pcie_comparison = compute_decode_step_profile(
        ServingConfig(
            name="fp8-tp4-b64-pcie",
            precision="fp8",
            tp_degree=4,
            batch_size=64,
            interconnect="pcie",
        )
    )
    naive_low_latency = compute_decode_step_profile(
        ServingConfig(
            name="bf16-tp8-b1-nvlink",
            precision="bf16",
            tp_degree=8,
            batch_size=1,
            interconnect="nvlink",
        )
    )
    tp2_b64 = compute_decode_step_profile(
        ServingConfig(
            name="fp8-tp2-b64-nvlink",
            precision="fp8",
            tp_degree=2,
            batch_size=64,
            interconnect="nvlink",
        )
    )
    tp2_b128 = compute_decode_step_profile(
        ServingConfig(
            name="fp8-tp2-b128-nvlink",
            precision="fp8",
            tp_degree=2,
            batch_size=128,
            interconnect="nvlink",
        )
    )
    report = {
        "model": "70B-GQA-80L",
        "context_len": 4096,
        "sla_tpot_ms": decision["sla_tpot_ms"],
        "selected": decision["selected"],
        "naive_low_latency_baseline": asdict(naive_low_latency),
        "pcie_tp4_b64_comparison": asdict(pcie_comparison),
        "tp2_batch_scaling": [asdict(tp2_b64), asdict(tp2_b128)],
        "pareto_frontier": decision["pareto_frontier"],
    }
    print(
        json.dumps(
            {
                "execution_mode": "CPU algorithm with controlled inputs; any latency/capacity coefficients are assumptions",
                **report,
            },
            indent=2,
        )
    )


if __name__ == "__main__":
    main()

The selector removes infeasible points before it compares objectives. From the points that meet the cap, it picks the highest tokens per second per device: fp8-tp4-b128-nvlink in the saved run. Its multi-device times come from the assumed cost model, so do not compare them with the measured static-batch sweep in the first lab.

Inspect the complete local result
Recorded local resultRecorded output · jsonDownload
{
  "execution_mode": "CPU algorithm with controlled inputs; any latency/capacity coefficients are assumptions",
  "model": "70B-GQA-80L",
  "context_len": 4096,
  "sla_tpot_ms": 25.0,
  "selected": {
    "name": "fp8-tp4-b128-nvlink",
    "precision": "fp8",
    "tp_degree": 4,
    "batch_size": 128,
    "interconnect": "nvlink",
    "fits_hbm": true,
    "memory_per_gpu_gb": 42.47,
    "weight_read_ms": 6.53,
    "kv_read_ms": 8.01,
    "compute_ms": 3.71,
    "allreduce_ms": 1.82,
    "tpot_ms": 17.27,
    "total_tokens_per_s": 7412.7,
    "tokens_per_s_per_gpu": 1853.2,
    "bottleneck": "kv_bandwidth"
  },
  "naive_low_latency_baseline": {
    "name": "bf16-tp8-b1-nvlink",
    "precision": "bf16",
    "tp_degree": 8,
    "batch_size": 1,
    "interconnect": "nvlink",
    "fits_hbm": true,
    "memory_per_gpu_gb": 21.17,
    "weight_read_ms": 6.53,
    "kv_read_ms": 0.06,
    "compute_ms": 0.03,
    "allreduce_ms": 0.57,
    "tpot_ms": 7.52,
    "total_tokens_per_s": 133.0,
    "tokens_per_s_per_gpu": 16.6,
    "bottleneck": "weight_bandwidth"
  },
  "pcie_tp4_b64_comparison": {
    "name": "fp8-tp4-b64-pcie",
    "precision": "fp8",
    "tp_degree": 4,
    "batch_size": 64,
    "interconnect": "pcie",
    "fits_hbm": true,
    "memory_per_gpu_gb": 31.74,
    "weight_read_ms": 6.53,
    "kv_read_ms": 4.01,
    "compute_ms": 1.86,
    "allreduce_ms": 19.73,
    "tpot_ms": 30.89,
    "total_tokens_per_s": 2071.6,
    "tokens_per_s_per_gpu": 517.9,
    "bottleneck": "allreduce_comm"
  },
  "tp2_batch_scaling": [
    {
      "name": "fp8-tp2-b64-nvlink",
      "precision": "fp8",
      "tp_degree": 2,
      "batch_size": 64,
      "interconnect": "nvlink",
      "fits_hbm": true,
      "memory_per_gpu_gb": 59.97,
      "weight_read_ms": 13.06,
      "kv_read_ms": 8.01,
      "compute_ms": 3.71,
      "allreduce_ms": 0.98,
      "tpot_ms": 22.96,
      "total_tokens_per_s": 2787.6,
      "tokens_per_s_per_gpu": 1393.8,
      "bottleneck": "weight_bandwidth"
    },
    {
      "name": "fp8-tp2-b128-nvlink",
      "precision": "fp8",
      "tp_degree": 2,
      "batch_size": 128,
      "interconnect": "nvlink",
      "fits_hbm": false,
      "memory_per_gpu_gb": 81.45,
      "weight_read_ms": 13.06,
      "kv_read_ms": 16.03,
      "compute_ms": 7.42,
      "allreduce_ms": 1.4,
      "tpot_ms": 31.95,
      "total_tokens_per_s": 4006.5,
      "tokens_per_s_per_gpu": 2003.3,
      "bottleneck": "kv_bandwidth"
    }
  ],
  "pareto_frontier": [
    {
      "name": "fp8-tp8-b1-nvlink",
      "precision": "fp8",
      "tp_degree": 8,
      "batch_size": 1,
      "interconnect": "nvlink",
      "fits_hbm": true,
      "memory_per_gpu_gb": 12.33,
      "weight_read_ms": 3.26,
      "kv_read_ms": 0.03,
      "compute_ms": 0.01,
      "allreduce_ms": 0.57,
      "tpot_ms": 4.22,
      "total_tokens_per_s": 237.0,
      "tokens_per_s_per_gpu": 29.6,
      "bottleneck": "weight_bandwidth"
    },
    {
      "name": "fp8-tp8-b8-nvlink",
      "precision": "fp8",
      "tp_degree": 8,
      "batch_size": 8,
      "interconnect": "nvlink",
      "fits_hbm": true,
      "memory_per_gpu_gb": 12.92,
      "weight_read_ms": 3.26,
      "kv_read_ms": 0.25,
      "compute_ms": 0.12,
      "allreduce_ms": 0.65,
      "tpot_ms": 4.53,
      "total_tokens_per_s": 1764.3,
      "tokens_per_s_per_gpu": 220.5,
      "bottleneck": "weight_bandwidth"
    },
    {
      "name": "fp8-tp8-b16-nvlink",
      "precision": "fp8",
      "tp_degree": 8,
      "batch_size": 16,
      "interconnect": "nvlink",
      "fits_hbm": true,
      "memory_per_gpu_gb": 13.59,
      "weight_read_ms": 3.26,
      "kv_read_ms": 0.5,
      "compute_ms": 0.23,
      "allreduce_ms": 0.74,
      "tpot_ms": 4.89,
      "total_tokens_per_s": 3269.3,
      "tokens_per_s_per_gpu": 408.7,
      "bottleneck": "weight_bandwidth"
    },
    {
      "name": "fp8-tp8-b32-nvlink",
      "precision": "fp8",
      "tp_degree": 8,
      "batch_size": 32,
      "interconnect": "nvlink",
      "fits_hbm": true,
      "memory_per_gpu_gb": 14.93,
      "weight_read_ms": 3.26,
      "kv_read_ms": 1.0,
      "compute_ms": 0.46,
      "allreduce_ms": 0.93,
      "tpot_ms": 5.61,
      "total_tokens_per_s": 5700.9,
      "tokens_per_s_per_gpu": 712.6,
      "bottleneck": "weight_bandwidth"
    },
    {
      "name": "fp8-tp8-b64-nvlink",
      "precision": "fp8",
      "tp_degree": 8,
      "batch_size": 64,
      "interconnect": "nvlink",
      "fits_hbm": true,
      "memory_per_gpu_gb": 17.62,
      "weight_read_ms": 3.26,
      "kv_read_ms": 2.0,
      "compute_ms": 0.93,
      "allreduce_ms": 1.29,
      "tpot_ms": 7.05,
      "total_tokens_per_s": 9076.3,
      "tokens_per_s_per_gpu": 1134.5,
      "bottleneck": "weight_bandwidth"
    },
    {
      "name": "fp8-tp8-b96-nvlink",
      "precision": "fp8",
      "tp_degree": 8,
      "batch_size": 96,
      "interconnect": "nvlink",
      "fits_hbm": true,
      "memory_per_gpu_gb": 20.3,
      "weight_read_ms": 3.26,
      "kv_read_ms": 3.0,
      "compute_ms": 1.39,
      "allreduce_ms": 1.66,
      "tpot_ms": 8.49,
      "total_tokens_per_s": 11308.1,
      "tokens_per_s_per_gpu": 1413.5,
      "bottleneck": "weight_bandwidth"
    },
    {
      "name": "fp8-tp8-b128-nvlink",
      "precision": "fp8",
      "tp_degree": 8,
      "batch_size": 128,
      "interconnect": "nvlink",
      "fits_hbm": true,
      "memory_per_gpu_gb": 22.99,
      "weight_read_ms": 3.26,
      "kv_read_ms": 4.01,
      "compute_ms": 1.86,
      "allreduce_ms": 2.03,
      "tpot_ms": 9.93,
      "total_tokens_per_s": 12893.2,
      "tokens_per_s_per_gpu": 1611.6,
      "bottleneck": "kv_bandwidth"
    },
    {
      "name": "fp8-tp4-b96-nvlink",
      "precision": "fp8",
      "tp_degree": 4,
      "batch_size": 96,
      "interconnect": "nvlink",
      "fits_hbm": true,
      "memory_per_gpu_gb": 37.11,
      "weight_read_ms": 6.53,
      "kv_read_ms": 6.01,
      "compute_ms": 2.78,
      "allreduce_ms": 1.5,
      "tpot_ms": 14.81,
      "total_tokens_per_s": 6481.8,
      "tokens_per_s_per_gpu": 1620.4,
      "bottleneck": "weight_bandwidth"
    },
    {
      "name": "fp8-tp4-b128-nvlink",
      "precision": "fp8",
      "tp_degree": 4,
      "batch_size": 128,
      "interconnect": "nvlink",
      "fits_hbm": true,
      "memory_per_gpu_gb": 42.47,
      "weight_read_ms": 6.53,
      "kv_read_ms": 8.01,
      "compute_ms": 3.71,
      "allreduce_ms": 1.82,
      "tpot_ms": 17.27,
      "total_tokens_per_s": 7412.7,
      "tokens_per_s_per_gpu": 1853.2,
      "bottleneck": "kv_bandwidth"
    }
  ]
}
Saved output report for choose a latency-throughput operating point; execution mode and values come from the linked raw result.

This report shows the saved output of python3 lab.py, a CPU calculation with assumed coefficients. Open the full-resolution report or read the raw result.

Read the latency-throughput frontier

Lower TPOT can consume more devices per request and reduce throughput per device. Larger batches may raise throughput while worsening latency or exceeding memory. In the saved run, bf16-tp8-b1-nvlink reaches 7.52 ms TPOT but only 16.6 tokens/s per device. Do not judge a configuration by single-request decode time or aggregate tokens per second alone. When configurations use different device counts, compare tokens per second per device: fp8-tp8-b128-nvlink has the highest aggregate rate on the frontier, but the selected tp4 point does more work per device.

Communication cost. A slower assumed interconnect raises collective communication time and can dominate the tensor-parallel step estimate. Over PCIe, fp8-tp4-b64-pcie spends 19.73 ms of its 30.89 ms TPOT in all-reduce and misses the 25 ms cap. Check the interconnect before you choose a tensor-parallel degree.

Latency and memory feasibility. An oversized batch can break the latency cap, the memory limit, or both. At tp2, going from batch 64 to batch 128 raises TPOT from 22.96 ms to 31.95 ms and memory to 81.45 GB per device. The selector removes such points instead of offering them as high-throughput choices.

Read the executable checks
Executable checksSource · pythonDownload
"""Unit and failure-boundary tests for the decode latency-throughput Pareto optimizer."""

from __future__ import annotations

import unittest

from lab import (
    ServingConfig,
    build_candidate_sweep,
    compute_decode_step_profile,
    pareto_frontier,
    select_pareto_config,
)


class TestLatencyThroughputPareto(unittest.TestCase):
    def test_selected_config_satisfies_sla_and_beats_small_batch(self) -> None:
        sweep = build_candidate_sweep()
        result = select_pareto_config(sweep, sla_tpot_ms=25.0)
        selected = result["selected"]
        self.assertTrue(selected["fits_hbm"])
        self.assertLessEqual(selected["tpot_ms"], 25.0)
        self.assertEqual(selected["precision"], "fp8")
        self.assertGreater(selected["tokens_per_s_per_gpu"], 600.0)

    def test_bf16_single_gpu_70b_rejected_by_hbm_capacity(self) -> None:
        profile = compute_decode_step_profile(
            ServingConfig(
                name="bf16-tp1-b16",
                precision="bf16",
                tp_degree=1,
                batch_size=16,
            )
        )
        self.assertFalse(profile.fits_hbm)
        self.assertGreater(profile.memory_per_gpu_gb, 80.0)

    def test_tensor_parallel_over_pcie_adds_allreduce_bottleneck(self) -> None:
        nvlink_profile = compute_decode_step_profile(
            ServingConfig(
                name="fp8-tp4-b64-nvlink",
                precision="fp8",
                tp_degree=4,
                batch_size=64,
                interconnect="nvlink",
            )
        )
        pcie_profile = compute_decode_step_profile(
            ServingConfig(
                name="fp8-tp4-b64-pcie",
                precision="fp8",
                tp_degree=4,
                batch_size=64,
                interconnect="pcie",
            )
        )
        self.assertLessEqual(nvlink_profile.tpot_ms, 25.0)
        self.assertGreater(pcie_profile.tpot_ms, 25.0)
        self.assertEqual(pcie_profile.bottleneck, "allreduce_comm")
        self.assertGreater(pcie_profile.allreduce_ms, nvlink_profile.allreduce_ms * 5.0)

    def test_oversized_batch_pushes_tpot_past_sla(self) -> None:
        b64 = compute_decode_step_profile(
            ServingConfig(
                name="fp8-tp2-b64-nvlink",
                precision="fp8",
                tp_degree=2,
                batch_size=64,
                interconnect="nvlink",
            )
        )
        b128 = compute_decode_step_profile(
            ServingConfig(
                name="fp8-tp2-b128-nvlink",
                precision="fp8",
                tp_degree=2,
                batch_size=128,
                interconnect="nvlink",
            )
        )
        self.assertLessEqual(b64.tpot_ms, 25.0)
        self.assertGreater(b128.tpot_ms, 25.0)
        self.assertGreater(b128.kv_read_ms, b64.kv_read_ms)

    def test_pareto_frontier_excludes_dominated_points(self) -> None:
        profiles = [compute_decode_step_profile(c) for c in build_candidate_sweep()]
        frontier = pareto_frontier(profiles)
        for p in frontier:
            self.assertTrue(p.fits_hbm)
            for other in frontier:
                if other is p:
                    continue
                self.assertFalse(
                    other.tpot_ms <= p.tpot_ms
                    and other.tokens_per_s_per_gpu >= p.tokens_per_s_per_gpu
                    and (
                        other.tpot_ms < p.tpot_ms
                        or other.tokens_per_s_per_gpu > p.tokens_per_s_per_gpu
                    )
                )


if __name__ == "__main__":
    unittest.main()


class GuardBoundaryTests(unittest.TestCase):
    def test_rejects_invalid_boundary_inputs_with_specific_reason(self):
        import lab
        cases = [('compute_decode_step_profile(ServingConfig("x","bf16",3,1))', 'Unsupported tp'), ('compute_decode_step_profile(ServingConfig("x","bf16",1,0))', 'batch_size'), ('compute_decode_step_profile(ServingConfig("x","bad",1,1))', 'Unsupported precision'), ('compute_decode_step_profile(ServingConfig("x","bf16",1,1,"bad"))', 'Unsupported interconnect'), ('select_pareto_config([])', 'No HBM-feasible')]
        for code, reason in cases:
            with self.subTest(code=code):
                with self.assertRaisesRegex((ValueError, MemoryError, KeyError, RuntimeError), reason):
                    exec(code, vars(lab).copy())

Try a change and explain the result

Find a configuration that is dominated and explain the two quantities that eliminate it. Tighten the latency target and observe which candidates disappear. Relate this to the measured batch sweep without claiming its results predict tensor-parallel performance.

A configuration lowers TPOT but uses more devices and lowers throughput per device. Is it always better?

Choose one answer, then check your reasoning.

Read the explanation

Latency, throughput, memory fit, and cost define different constraints. There is no single best operating point.

Choose targets for the application

Set the latency cap from the application’s service goals, such as responsive daytime streams versus overnight summaries. Use the cost model to shortlist candidates, then benchmark the shortlist on the actual context distribution; this lab fixes context at 4,096 tokens. Check memory fit and reject unsupported precision/sharding combinations before you compare cost.

References

Sources checked October 24, 2026. Runtime-specific commands and results apply to the recorded versions; linked rolling documentation can change.

Written by Arjun Prabhulal.

Module 4

Compare inference optimizations and their costs