When you need this. A code assistant accepts a request immediately, then appears idle before text arrives. Adding replicas will not help if the delay is inside tokenization or prompt processing. Build a trace that separates the wait for the first token from the pace of later tokens.
What you run. Fresh local inference with a pinned Apache-2.0 model. The default command analyzes its committed record offline; the separate live runner executes the model.
Read the request timeline
Tokenization converts text into the model’s input IDs. Prefill evaluates that prompt and creates the attention key/value cache (KV cache). The final prompt position supplies logits for the first output token. Decode then feeds back one selected token at a time, reusing cached keys and values instead of recomputing the entire prefix.
Time to first token (TTFT) ends when the first output token is available at the measurement boundary. Time per output token (TPOT) averages intervals after that first token. A client-side trace also includes transport and queueing; this local trace starts before tokenization and has neither. A one-token response has no decode interval, so its TPOT is undefined.
LocalDecoder.run() performs real forward passes, records each selected token, and synchronizes the device before recording timestamps. It explicitly sets greedy decoding, an output cap, the attention backend, and whether to retain the cache. summarize() rejects inconsistent timing records before calculating medians.
Run and read the trace
Download the lab from this page, extract it, and run these commands from its directory. The default command reads saved run records or runs the explicitly described local algorithm; it never downloads a model or calls a paid API.
python3 lab.py
python3 -m unittest -v test_lab.pyRead lab.py
"""Summarize the committed real local-inference record without rerunning a model."""
import json
import math
import statistics
from pathlib import Path
def summarize(samples):
if not samples:
raise ValueError("At least one completed generation is required")
for sample in samples:
values = [sample["ttft_ms"], sample["e2e_ms"], *sample["itl_ms"]]
if not all(math.isfinite(v) and v >= 0 for v in values):
raise ValueError("Timing values must be finite and nonnegative")
if sample["generated_tokens"] != len(sample["generated_token_ids"]):
raise ValueError("Token count mismatch")
if sample["ttft_ms"] < 0 or sample["e2e_ms"] < sample["ttft_ms"]:
raise ValueError("Invalid token timing")
if len(sample["itl_ms"]) != max(0, sample["generated_tokens"] - 1):
raise ValueError("Missing token intervals")
if not math.isclose(
math.fsum(sample["itl_ms"]),
sample["e2e_ms"] - sample["ttft_ms"],
abs_tol=0.002 * max(1, len(sample["itl_ms"])),
):
raise ValueError("Intervals do not account for end-to-end timing")
return {
"samples": len(samples),
"median_ttft_ms": round(statistics.median(x["ttft_ms"] for x in samples), 3),
"median_tpot_ms": (
round(
statistics.median(
statistics.fmean(x["itl_ms"]) for x in samples if x["itl_ms"]
),
3,
)
if any(x["itl_ms"] for x in samples)
else None
),
}
def main():
record = json.loads(Path("live-run.json").read_text())
print(
json.dumps(
{
"execution_mode": "offline analysis of fresh local inference record",
"model": record["model"],
"generation": record["generation"],
"comparisons": {
run["name"]: summarize(run["samples"]) for run in record["runs"]
},
"batch_sweep": record["batch_sweep"],
"prefix_reuse": record["prefix_reuse"],
},
indent=2,
)
)
if __name__ == "__main__":
main()Read live_inference.py
"""Real local text generation and measured inference experiments. Never downloads implicitly."""
from __future__ import annotations
import argparse
import hashlib
import json
import math
import platform
import statistics
import time
from pathlib import Path
MODEL_ID = "Qwen/Qwen2.5-0.5B-Instruct"
MODEL_REVISION = "7ae557604adf67be50417f59c2c2f167def9a775"
def percentile(values, q):
return sorted(values)[max(0, math.ceil(len(values) * q) - 1)] if values else None
class LocalDecoder:
def __init__(self, model_path, device):
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
self.torch = torch
self.device = device
self.tokenizer = AutoTokenizer.from_pretrained(
model_path, local_files_only=True
)
self.tokenizer.padding_side = "left"
self.tokenizer.pad_token = self.tokenizer.eos_token
self.model = (
AutoModelForCausalLM.from_pretrained(
model_path,
local_files_only=True,
dtype=torch.float32,
attn_implementation="eager",
trust_remote_code=False,
)
.eval()
.to(device)
)
def sync(self):
if self.device == "mps":
self.torch.mps.synchronize()
elif self.device.startswith("cuda"):
self.torch.cuda.synchronize()
def encode(self, text):
if not text.strip():
raise ValueError("Prompt must not be empty")
return self.tokenizer.apply_chat_template(
[{"role": "user", "content": text}],
tokenize=True,
add_generation_prompt=True,
return_tensors="pt",
return_dict=True,
)["input_ids"].to(self.device)
def _forward(self, ids, cache=None, use_cache=True):
with self.torch.inference_mode():
return self.model(input_ids=ids, past_key_values=cache, use_cache=use_cache)
def run(
self,
text,
max_new_tokens=24,
use_cache=True,
chunk_size=None,
cancel_after=None,
):
if max_new_tokens < 1:
raise ValueError("Output limit must be positive")
if chunk_size is not None and chunk_size < 1:
raise ValueError("Chunk size must be positive")
self.sync()
started = time.perf_counter()
ids = self.encode(text)
self.sync()
tokenized = time.perf_counter()
cache = None
if chunk_size is not None and not use_cache:
raise ValueError("Chunking needs a cache")
for chunk in ids.split(chunk_size or ids.shape[-1], dim=-1):
result = self._forward(chunk, cache, use_cache)
cache = result.past_key_values if use_cache else None
emitted = []
arrivals = []
stop = "length"
context = ids
eos_ids = self.model.generation_config.eos_token_id
eos_ids = set(eos_ids if isinstance(eos_ids, list) else [eos_ids])
for _ in range(max_new_tokens):
token = result.logits[:, -1].argmax(-1, keepdim=True)
self.sync()
token_id = int(token.item())
arrivals.append(time.perf_counter() - started)
emitted.append(token_id)
if token_id in eos_ids:
stop = "eos"
break
if cancel_after is not None and len(emitted) >= cancel_after:
stop = "cancelled"
break
if len(emitted) == max_new_tokens:
break
context = self.torch.cat((context, token), dim=-1)
result = self._forward(token if use_cache else context, cache, use_cache)
cache = result.past_key_values if use_cache else None
cache_length = int(cache.get_seq_length()) if cache is not None else 0
# Release the request-owned cache on every stop, including cancellation.
del cache, result
self.sync()
intervals = [1000 * (b - a) for a, b in zip(arrivals, arrivals[1:])]
return {
"prompt": text,
"prompt_tokens": ids.shape[-1],
"generated_token_ids": emitted,
"generated_tokens": len(emitted),
"text": self.tokenizer.decode(emitted, skip_special_tokens=True),
"use_cache": use_cache,
"chunk_size": chunk_size,
"stop_reason": stop,
"tokenization_ms": round(1000 * (tokenized - started), 3),
"ttft_ms": round(arrivals[0] * 1000, 3),
"tpot_ms": round(statistics.fmean(intervals), 3) if intervals else None,
"itl_ms": [round(x, 3) for x in intervals],
"e2e_ms": round(arrivals[-1] * 1000, 3),
"cache_tokens_before_release": cache_length,
}
def batch(self, texts, max_new_tokens=24):
prompts = [
self.tokenizer.apply_chat_template(
[{"role": "user", "content": t}],
tokenize=False,
add_generation_prompt=True,
)
for t in texts
]
encoded = self.tokenizer(prompts, padding=True, return_tensors="pt").to(
self.device
)
self.sync()
start = time.perf_counter()
with self.torch.inference_mode():
out = self.model.generate(
**encoded,
max_new_tokens=max_new_tokens,
do_sample=False,
use_cache=True,
)
self.sync()
elapsed = time.perf_counter() - start
ids = out[:, encoded["input_ids"].shape[-1] :].tolist()
eos = self.model.generation_config.eos_token_id
eos = set(eos if isinstance(eos, list) else [eos])
trimmed = []
for row in ids:
n = next((i + 1 for i, x in enumerate(row) if x in eos), len(row))
trimmed.append(row[:n])
return {
"batch_size": len(texts),
"elapsed_ms": round(elapsed * 1000, 3),
"output_tokens": sum(map(len, trimmed)),
"output_tokens_per_s": round(sum(map(len, trimmed)) / elapsed, 3),
"texts": [
self.tokenizer.decode(row, skip_special_tokens=True) for row in trimmed
],
}
def prefix_reuse(self, prefix, suffix):
# Tokenize the full sequence once: concatenated text tokenization need not equal separate tokenizations.
full = self.encode(prefix + suffix)
prefix_length = max(1, full.shape[-1] // 2)
self.sync()
start = time.perf_counter()
cold = self._forward(full)
self.sync()
cold_ms = (time.perf_counter() - start) * 1000
cached = self._forward(full[:, :prefix_length]).past_key_values
self.sync()
start = time.perf_counter()
warm = self._forward(full[:, prefix_length:], cached)
self.sync()
warm_ms = (time.perf_counter() - start) * 1000
error = float((cold.logits[:, -1] - warm.logits[:, -1]).abs().max().item())
return {
"total_tokens": full.shape[-1],
"cached_prefix_tokens": prefix_length,
"cold_prefill_ms": round(cold_ms, 3),
"suffix_prefill_ms": round(warm_ms, 3),
"max_abs_last_logit_diff": error,
"same_greedy_next_token": int(cold.logits[:, -1].argmax())
== int(warm.logits[:, -1].argmax()),
}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--model-path", required=True)
parser.add_argument("--device", choices=["cpu", "mps"], default="cpu")
parser.add_argument("--output", required=True)
args = parser.parse_args()
import torch, transformers
decoder = LocalDecoder(args.model_path, args.device)
warmup = decoder.run("Explain a cache in one sentence.", max_new_tokens=4)
short = "Explain why a code assistant should stop generating when its user cancels."
long = (
"A retrieval service indexes source code and returns relevant functions. " * 20
) + short
runs = []
for name, text, options in [
("short_cached", short, {}),
("long_cached", long, {}),
("short_without_cache", short, {"use_cache": False}),
("long_chunked", long, {"chunk_size": 64}),
("cancelled", short, {"cancel_after": 4}),
]:
samples = [decoder.run(text, **options) for _ in range(3)]
runs.append(
{
"name": name,
"samples": samples,
"median_ttft_ms": round(
statistics.median(s["ttft_ms"] for s in samples), 3
),
"median_tpot_ms": round(
statistics.median(
s["tpot_ms"] for s in samples if s["tpot_ms"] is not None
),
3,
),
}
)
batches = [decoder.batch([short] * n) for n in [1, 2, 4]]
config = decoder.model.config
parameters = sum(p.numel() for p in decoder.model.parameters())
report = {
"execution_mode": "fresh local inference; no simulated latency",
"environment": {
"python": platform.python_version(),
"platform": platform.platform(),
"torch": torch.__version__,
"transformers": transformers.__version__,
"device": args.device,
},
"model": {
"id": MODEL_ID,
"revision": MODEL_REVISION,
"license": "Apache-2.0",
"dtype": "float32",
"attention_backend": "eager",
"parameters": parameters,
"weight_bytes": sum(
p.numel() * p.element_size() for p in decoder.model.parameters()
),
"layers": config.num_hidden_layers,
"kv_heads": config.num_key_value_heads,
"head_dim": config.hidden_size // config.num_attention_heads,
"kv_bytes_per_token": 2
* config.num_hidden_layers
* config.num_key_value_heads
* (config.hidden_size // config.num_attention_heads)
* 4,
},
"generation": {
"strategy": "greedy argmax",
"max_new_tokens": 24,
"do_sample": False,
"warmup_tokens": 4,
"thinking": "not a configurable reasoning mode in this model",
},
"source_sha256": hashlib.sha256(Path(__file__).read_bytes()).hexdigest(),
"warmup": warmup,
"runs": runs,
"batch_sweep": batches,
"prefix_reuse": decoder.prefix_reuse(long, " Give one practical example."),
"limits": "Three repeated development prompts per configuration; no held-out quality evaluation, HTTP latency, CUDA profiling, or production load test.",
}
Path(args.output).write_text(json.dumps(report, indent=2) + "\n")
print(
json.dumps(
{
"output": Path(args.output).name,
"runs": len(runs),
"batch_sizes": [x["batch_size"] for x in batches],
},
indent=2,
)
)
if __name__ == "__main__":
main()The cached and uncached short-prompt runs use the same prompt and output cap. Compare their decode medians; this isolates a useful cache question without claiming that the small model represents a production fleet. The long-prompt comparison changes the input workload, so it answers a different question.
| Quantity | Recorded value |
|---|---|
| Short prompt TTFT (ms) | 36.399 |
| Short prompt TPOT (ms) | 24.093 |
| Uncached short prompt TPOT (ms) | 39.635 |
| Long prompt TTFT (ms) | 86.29 |
| Chunked long prompt TTFT (ms) | 192.01 |
Inspect the complete local result
{
"execution_mode": "offline analysis of fresh local inference record",
"model": {
"id": "Qwen/Qwen2.5-0.5B-Instruct",
"revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"license": "Apache-2.0",
"dtype": "float32",
"attention_backend": "eager",
"parameters": 494032768,
"weight_bytes": 1976131072,
"layers": 24,
"kv_heads": 2,
"head_dim": 64,
"kv_bytes_per_token": 24576
},
"generation": {
"strategy": "greedy argmax",
"max_new_tokens": 24,
"do_sample": false,
"warmup_tokens": 4,
"thinking": "not a configurable reasoning mode in this model"
},
"comparisons": {
"short_cached": {
"samples": 3,
"median_ttft_ms": 36.399,
"median_tpot_ms": 24.093
},
"long_cached": {
"samples": 3,
"median_ttft_ms": 86.29,
"median_tpot_ms": 25.288
},
"short_without_cache": {
"samples": 3,
"median_ttft_ms": 38.318,
"median_tpot_ms": 39.635
},
"long_chunked": {
"samples": 3,
"median_ttft_ms": 192.01,
"median_tpot_ms": 28.243
},
"cancelled": {
"samples": 3,
"median_ttft_ms": 34.266,
"median_tpot_ms": 22.446
}
},
"batch_sweep": [
{
"batch_size": 1,
"elapsed_ms": 1144.114,
"output_tokens": 24,
"output_tokens_per_s": 20.977,
"texts": [
"When a code assistant encounters the user's cancellation of an operation, it is crucial to understand that this action signifies the end"
]
},
{
"batch_size": 2,
"elapsed_ms": 1490.669,
"output_tokens": 48,
"output_tokens_per_s": 32.2,
"texts": [
"When a code assistant encounters the user's cancellation of an operation, it is crucial to understand that this action signifies the end",
"When a code assistant encounters the user's cancellation of an operation, it is crucial to understand that this action signifies the end"
]
},
{
"batch_size": 4,
"elapsed_ms": 1243.557,
"output_tokens": 96,
"output_tokens_per_s": 77.198,
"texts": [
"When a code assistant encounters the user's cancellation of an operation, it is crucial to understand that this action signifies the end",
"When a code assistant encounters the user's cancellation of an operation, it is crucial to understand that this action signifies the end",
"When a code assistant encounters the user's cancellation of an operation, it is crucial to understand that this action signifies the end",
"When a code assistant encounters the user's cancellation of an operation, it is crucial to understand that this action signifies the end"
]
}
],
"prefix_reuse": {
"total_tokens": 268,
"cached_prefix_tokens": 134,
"cold_prefill_ms": 90.296,
"suffix_prefill_ms": 60.948,
"max_abs_last_logit_diff": 4.506111145019531e-05,
"same_greedy_next_token": true
}
}Inspect the actual execution record
{
"execution_mode": "fresh local inference; no simulated latency",
"environment": {
"python": "3.12.8",
"platform": "macOS-27.0.1-arm64-arm-64bit",
"torch": "2.14.1",
"transformers": "5.20.2",
"device": "mps"
},
"model": {
"id": "Qwen/Qwen2.5-0.5B-Instruct",
"revision": "7ae557604adf67be50417f59c2c2f167def9a775",
"license": "Apache-2.0",
"dtype": "float32",
"attention_backend": "eager",
"parameters": 494032768,
"weight_bytes": 1976131072,
"layers": 24,
"kv_heads": 2,
"head_dim": 64,
"kv_bytes_per_token": 24576
},
"generation": {
"strategy": "greedy argmax",
"max_new_tokens": 24,
"do_sample": false,
"warmup_tokens": 4,
"thinking": "not a configurable reasoning mode in this model"
},
"source_sha256": "754d9bf81d04694541e53d5c0472f984c9f046886070c33b376ca071c81c8086",
"warmup": {
"prompt": "Explain a cache in one sentence.",
"prompt_tokens": 37,
"generated_token_ids": [
32,
6500,
374,
264
],
"generated_tokens": 4,
"text": "A cache is a",
"use_cache": true,
"chunk_size": null,
"stop_reason": "length",
"tokenization_ms": 6.358,
"ttft_ms": 138.267,
"tpot_ms": 25.619,
"itl_ms": [
27.98,
23.916,
24.96
],
"e2e_ms": 215.123,
"cache_tokens_before_release": 40
},
"runs": [
{
"name": "short_cached",
"samples": [
{
"prompt": "Explain why a code assistant should stop generating when its user cancels.",
"prompt_tokens": 44,
"generated_token_ids": [
4498,
264,
2038,
17847,
33906,
264,
1196,
879,
702,
33446,
862,
1681,
11,
432,
374,
16587,
311,
3535,
279,
2266,
323,
279,
6993,
315
],
"generated_tokens": 24,
"text": "When a code assistant encounters a user who has canceled their request, it is crucial to understand the context and the nature of",
"use_cache": true,
"chunk_size": null,
"stop_reason": "length",
"tokenization_ms": 0.42,
"ttft_ms": 36.971,
"tpot_ms": 37.841,
"itl_ms": [
24.717,
24.26,
24.26,
23.673,
23.945,
24.013,
23.746,
23.935,
24.656,
24.232,
24.202,
23.957,
24.223,
24.584,
24.632,
24.522,
25.532,
25.876,
25.605,
201.281,
148.208,
27.729,
28.559
],
"e2e_ms": 907.318,
"cache_tokens_before_release": 67
},
{
"prompt": "Explain why a code assistant should stop generating when its user cancels.",
"prompt_tokens": 44,
"generated_token_ids": [
4498,
264,
2038,
17847,
33906,
264,
1196,
879,
702,
33446,
862,
1681,
11,
432,
374,
16587,
311,
3535,
279,
2266,
323,
279,
6993,
315
],
"generated_tokens": 24,
"text": "When a code assistant encounters a user who has canceled their request, it is crucial to understand the context and the nature of",
"use_cache": true,
"chunk_size": null,
"stop_reason": "length",
"tokenization_ms": 0.508,
"ttft_ms": 36.399,
"tpot_ms": 24.093,
"itl_ms": [
24.36,
24.582,
23.627,
23.961,
24.925,
24.601,
24.032,
24.172,
24.462,
24.122,
24.799,
24.095,
23.782,
23.391,
23.578,
24.265,
23.98,
24.264,
23.391,
22.59,
24.304,
24.778,
24.084
],
"e2e_ms": 590.543,
"cache_tokens_before_release": 67
},
{
"prompt": "Explain why a code assistant should stop generating when its user cancels.",
"prompt_tokens": 44,
"generated_token_ids": [
4498,
264,
2038,
17847,
33906,
264,
1196,
879,
702,
33446,
862,
1681,
11,
432,
374,
16587,
311,
3535,
279,
2266,
323,
279,
6993,
315
],
"generated_tokens": 24,
"text": "When a code assistant encounters a user who has canceled their request, it is crucial to understand the context and the nature of",
"use_cache": true,
"chunk_size": null,
"stop_reason": "length",
"tokenization_ms": 0.538,
"ttft_ms": 36.34,
"tpot_ms": 23.944,
"itl_ms": [
23.965,
23.525,
23.857,
23.826,
22.808,
23.116,
22.953,
23.944,
23.659,
23.274,
23.324,
23.85,
23.623,
24.623,
24.886,
25.403,
24.198,
24.893,
24.711,
23.619,
24.964,
24.146,
23.548
],
"e2e_ms": 587.054,
"cache_tokens_before_release": 67
}
],
"median_ttft_ms": 36.399,
"median_tpot_ms": 24.093
},
{
"name": "long_cached",
"samples": [
{
"prompt": "A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. Explain why a code assistant should stop generating when its user cancels.",
"prompt_tokens": 263,
"generated_token_ids": [
4498,
264,
2038,
17847,
33906,
264,
1196,
879,
702,
33446,
279,
10435,
11,
432,
594,
2989,
311,
3535,
279,
2266,
323,
279,
7966,
4815
],
"generated_tokens": 24,
"text": "When a code assistant encounters a user who has canceled the conversation, it's important to understand the context and the reasons behind",
"use_cache": true,
"chunk_size": null,
"stop_reason": "length",
"tokenization_ms": 0.758,
"ttft_ms": 273.572,
"tpot_ms": 26.82,
"itl_ms": [
27.356,
27.6,
26.837,
26.249,
27.147,
27.567,
27.574,
26.044,
26.353,
26.533,
26.771,
26.266,
25.951,
26.33,
26.194,
26.868,
26.187,
27.931,
27.892,
26.783,
27.818,
25.98,
26.633
],
"e2e_ms": 890.437,
"cache_tokens_before_release": 286
},
{
"prompt": "A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. Explain why a code assistant should stop generating when its user cancels.",
"prompt_tokens": 263,
"generated_token_ids": [
4498,
264,
2038,
17847,
33906,
264,
1196,
879,
702,
33446,
279,
10435,
11,
432,
594,
2989,
311,
3535,
279,
2266,
323,
279,
7966,
4815
],
"generated_tokens": 24,
"text": "When a code assistant encounters a user who has canceled the conversation, it's important to understand the context and the reasons behind",
"use_cache": true,
"chunk_size": null,
"stop_reason": "length",
"tokenization_ms": 0.85,
"ttft_ms": 86.29,
"tpot_ms": 25.288,
"itl_ms": [
26.506,
25.52,
26.769,
24.864,
25.298,
26.576,
25.433,
23.891,
23.809,
23.687,
26.404,
26.394,
25.43,
25.29,
25.059,
24.146,
24.421,
24.663,
26.637,
25.673,
25.497,
24.925,
24.726
],
"e2e_ms": 667.908,
"cache_tokens_before_release": 286
},
{
"prompt": "A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. Explain why a code assistant should stop generating when its user cancels.",
"prompt_tokens": 263,
"generated_token_ids": [
4498,
264,
2038,
17847,
33906,
264,
1196,
879,
702,
33446,
279,
10435,
11,
432,
594,
2989,
311,
3535,
279,
2266,
323,
279,
7966,
4815
],
"generated_tokens": 24,
"text": "When a code assistant encounters a user who has canceled the conversation, it's important to understand the context and the reasons behind",
"use_cache": true,
"chunk_size": null,
"stop_reason": "length",
"tokenization_ms": 0.694,
"ttft_ms": 84.662,
"tpot_ms": 24.334,
"itl_ms": [
25.428,
25.259,
24.846,
24.946,
25.267,
23.982,
24.173,
24.283,
24.333,
23.722,
23.755,
23.511,
24.927,
24.42,
24.321,
24.326,
24.096,
24.343,
23.912,
24.211,
24.209,
23.855,
23.554
],
"e2e_ms": 644.34,
"cache_tokens_before_release": 286
}
],
"median_ttft_ms": 86.29,
"median_tpot_ms": 25.288
},
{
"name": "short_without_cache",
"samples": [
{
"prompt": "Explain why a code assistant should stop generating when its user cancels.",
"prompt_tokens": 44,
"generated_token_ids": [
4498,
264,
2038,
17847,
33906,
264,
1196,
879,
702,
33446,
862,
1681,
11,
432,
374,
16587,
311,
3535,
279,
2266,
323,
279,
6993,
315
],
"generated_tokens": 24,
"text": "When a code assistant encounters a user who has canceled their request, it is crucial to understand the context and the nature of",
"use_cache": false,
"chunk_size": null,
"stop_reason": "length",
"tokenization_ms": 0.37,
"ttft_ms": 66.297,
"tpot_ms": 67.008,
"itl_ms": [
79.186,
134.713,
39.694,
226.116,
201.879,
41.316,
41.58,
41.005,
40.472,
41.45,
81.137,
42.528,
40.705,
40.632,
41.435,
41.415,
42.948,
40.251,
41.818,
41.779,
109.786,
44.586,
44.763
],
"e2e_ms": 1607.491,
"cache_tokens_before_release": 0
},
{
"prompt": "Explain why a code assistant should stop generating when its user cancels.",
"prompt_tokens": 44,
"generated_token_ids": [
4498,
264,
2038,
17847,
33906,
264,
1196,
879,
702,
33446,
862,
1681,
11,
432,
374,
16587,
311,
3535,
279,
2266,
323,
279,
6993,
315
],
"generated_tokens": 24,
"text": "When a code assistant encounters a user who has canceled their request, it is crucial to understand the context and the nature of",
"use_cache": false,
"chunk_size": null,
"stop_reason": "length",
"tokenization_ms": 0.388,
"ttft_ms": 37.483,
"tpot_ms": 39.405,
"itl_ms": [
36.29,
37.237,
37.341,
39.518,
38.716,
39.582,
38.704,
39.39,
39.481,
39.121,
39.194,
40.456,
40.368,
39.749,
40.942,
39.908,
39.032,
38.143,
39.892,
36.976,
41.664,
41.7,
42.915
],
"e2e_ms": 943.802,
"cache_tokens_before_release": 0
},
{
"prompt": "Explain why a code assistant should stop generating when its user cancels.",
"prompt_tokens": 44,
"generated_token_ids": [
4498,
264,
2038,
17847,
33906,
264,
1196,
879,
702,
33446,
862,
1681,
11,
432,
374,
16587,
311,
3535,
279,
2266,
323,
279,
6993,
315
],
"generated_tokens": 24,
"text": "When a code assistant encounters a user who has canceled their request, it is crucial to understand the context and the nature of",
"use_cache": false,
"chunk_size": null,
"stop_reason": "length",
"tokenization_ms": 0.417,
"ttft_ms": 38.318,
"tpot_ms": 39.635,
"itl_ms": [
36.313,
37.162,
37.48,
40.273,
39.284,
39.079,
39.382,
38.926,
40.223,
39.338,
39.432,
39.342,
39.309,
39.563,
40.44,
40.655,
40.428,
39.193,
40.016,
38.449,
41.942,
42.91,
42.457
],
"e2e_ms": 949.915,
"cache_tokens_before_release": 0
}
],
"median_ttft_ms": 38.318,
"median_tpot_ms": 39.635
},
{
"name": "long_chunked",
"samples": [
{
"prompt": "A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. Explain why a code assistant should stop generating when its user cancels.",
"prompt_tokens": 263,
"generated_token_ids": [
4498,
264,
2038,
17847,
33906,
264,
1196,
879,
702,
33446,
279,
10435,
11,
432,
594,
2989,
311,
3535,
279,
2266,
323,
279,
7966,
4815
],
"generated_tokens": 24,
"text": "When a code assistant encounters a user who has canceled the conversation, it's important to understand the context and the reasons behind",
"use_cache": true,
"chunk_size": 64,
"stop_reason": "length",
"tokenization_ms": 0.835,
"ttft_ms": 702.895,
"tpot_ms": 28.833,
"itl_ms": [
38.292,
66.814,
41.939,
32.362,
26.243,
25.407,
24.949,
25.175,
26.761,
25.854,
25.866,
25.026,
25.473,
25.808,
25.566,
26.479,
25.683,
24.899,
25.496,
24.559,
24.987,
24.635,
24.89
],
"e2e_ms": 1366.057,
"cache_tokens_before_release": 286
},
{
"prompt": "A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. Explain why a code assistant should stop generating when its user cancels.",
"prompt_tokens": 263,
"generated_token_ids": [
4498,
264,
2038,
17847,
33906,
264,
1196,
879,
702,
33446,
279,
10435,
11,
432,
594,
2989,
311,
3535,
279,
2266,
323,
279,
7966,
4815
],
"generated_tokens": 24,
"text": "When a code assistant encounters a user who has canceled the conversation, it's important to understand the context and the reasons behind",
"use_cache": true,
"chunk_size": 64,
"stop_reason": "length",
"tokenization_ms": 0.797,
"ttft_ms": 180.806,
"tpot_ms": 28.243,
"itl_ms": [
24.767,
25.971,
28.707,
28.913,
29.586,
32.806,
33.812,
32.961,
28.86,
30.909,
27.018,
25.968,
25.662,
26.752,
26.08,
25.695,
26.85,
25.259,
26.738,
29.582,
31.276,
28.427,
26.987
],
"e2e_ms": 830.391,
"cache_tokens_before_release": 286
},
{
"prompt": "A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. A retrieval service indexes source code and returns relevant functions. Explain why a code assistant should stop generating when its user cancels.",
"prompt_tokens": 263,
"generated_token_ids": [
4498,
264,
2038,
17847,
33906,
264,
1196,
879,
702,
33446,
279,
10435,
11,
432,
594,
2989,
311,
3535,
279,
2266,
323,
279,
7966,
4815
],
"generated_tokens": 24,
"text": "When a code assistant encounters a user who has canceled the conversation, it's important to understand the context and the reasons behind",
"use_cache": true,
"chunk_size": 64,
"stop_reason": "length",
"tokenization_ms": 0.763,
"ttft_ms": 192.01,
"tpot_ms": 25.814,
"itl_ms": [
25.66,
25.885,
25.866,
26.904,
26.668,
26.756,
26.449,
27.375,
25.687,
25.242,
25.962,
25.179,
24.328,
24.858,
24.397,
24.939,
24.364,
24.276,
26.038,
27.134,
27.045,
26.886,
25.831
],
"e2e_ms": 785.739,
"cache_tokens_before_release": 286
}
],
"median_ttft_ms": 192.01,
"median_tpot_ms": 28.243
},
{
"name": "cancelled",
"samples": [
{
"prompt": "Explain why a code assistant should stop generating when its user cancels.",
"prompt_tokens": 44,
"generated_token_ids": [
4498,
264,
2038,
17847
],
"generated_tokens": 4,
"text": "When a code assistant",
"use_cache": true,
"chunk_size": null,
"stop_reason": "cancelled",
"tokenization_ms": 0.454,
"ttft_ms": 39.63,
"tpot_ms": 27.403,
"itl_ms": [
28.558,
28.742,
24.91
],
"e2e_ms": 121.84,
"cache_tokens_before_release": 47
},
{
"prompt": "Explain why a code assistant should stop generating when its user cancels.",
"prompt_tokens": 44,
"generated_token_ids": [
4498,
264,
2038,
17847
],
"generated_tokens": 4,
"text": "When a code assistant",
"use_cache": true,
"chunk_size": null,
"stop_reason": "cancelled",
"tokenization_ms": 0.387,
"ttft_ms": 33.383,
"tpot_ms": 22.446,
"itl_ms": [
21.98,
21.693,
23.664
],
"e2e_ms": 100.721,
"cache_tokens_before_release": 47
},
{
"prompt": "Explain why a code assistant should stop generating when its user cancels.",
"prompt_tokens": 44,
"generated_token_ids": [
4498,
264,
2038,
17847
],
"generated_tokens": 4,
"text": "When a code assistant",
"use_cache": true,
"chunk_size": null,
"stop_reason": "cancelled",
"tokenization_ms": 0.456,
"ttft_ms": 34.266,
"tpot_ms": 20.906,
"itl_ms": [
20.964,
21.509,
20.245
],
"e2e_ms": 96.985,
"cache_tokens_before_release": 47
}
],
"median_ttft_ms": 34.266,
"median_tpot_ms": 22.446
}
],
"batch_sweep": [
{
"batch_size": 1,
"elapsed_ms": 1144.114,
"output_tokens": 24,
"output_tokens_per_s": 20.977,
"texts": [
"When a code assistant encounters the user's cancellation of an operation, it is crucial to understand that this action signifies the end"
]
},
{
"batch_size": 2,
"elapsed_ms": 1490.669,
"output_tokens": 48,
"output_tokens_per_s": 32.2,
"texts": [
"When a code assistant encounters the user's cancellation of an operation, it is crucial to understand that this action signifies the end",
"When a code assistant encounters the user's cancellation of an operation, it is crucial to understand that this action signifies the end"
]
},
{
"batch_size": 4,
"elapsed_ms": 1243.557,
"output_tokens": 96,
"output_tokens_per_s": 77.198,
"texts": [
"When a code assistant encounters the user's cancellation of an operation, it is crucial to understand that this action signifies the end",
"When a code assistant encounters the user's cancellation of an operation, it is crucial to understand that this action signifies the end",
"When a code assistant encounters the user's cancellation of an operation, it is crucial to understand that this action signifies the end",
"When a code assistant encounters the user's cancellation of an operation, it is crucial to understand that this action signifies the end"
]
}
],
"prefix_reuse": {
"total_tokens": 268,
"cached_prefix_tokens": 134,
"cold_prefill_ms": 90.296,
"suffix_prefill_ms": 60.948,
"max_abs_last_logit_diff": 4.506111145019531e-05,
"same_greedy_next_token": true
},
"limits": "Three repeated development prompts per configuration; no held-out quality evaluation, HTTP latency, CUDA profiling, or production load test."
}
This image is an application-generated report of the saved program result, not a new inference run. Open the full-resolution report or read the raw result.
Separate first-token delay from decode pace
A longer prompt adds prefill work and cache state. Keeping KV state saves repeated computation but consumes memory for every active request. Disabling the cache changes both compute and allocation behavior; it is not a deployment recommendation. Device synchronization makes these stage timings interpretable but can slow the execution being measured.
Missing token intervals. A response with missing decode intervals is rejected instead of silently averaging an incomplete trace.
Invalid token timing. An end timestamp before the first token is rejected. Negative and nonfinite timing values also fail validation.
Token count mismatch. A declared output count that differs from the saved token IDs is rejected. A valid one-token response has no TPOT.
Read the executable checks
import unittest
from lab import summarize
class TraceTests(unittest.TestCase):
def sample(self):
return {
"generated_tokens": 3,
"generated_token_ids": [4, 5, 6],
"ttft_ms": 10,
"e2e_ms": 30,
"itl_ms": [8, 12],
}
def test_average_decode_interval_excludes_prefill(self):
result = summarize([self.sample()])
self.assertEqual(result["median_ttft_ms"], 10)
self.assertEqual(result["median_tpot_ms"], 10)
def test_rejects_missing_token_interval(self):
sample = self.sample()
sample["itl_ms"] = [8]
with self.assertRaisesRegex(ValueError, "Missing token intervals"):
summarize([sample])
def test_rejects_end_before_first_token(self):
sample = self.sample()
sample["e2e_ms"] = 2
with self.assertRaisesRegex(ValueError, "Invalid token timing"):
summarize([sample])
def test_rejects_inconsistent_generated_count(self):
sample = self.sample()
sample["generated_tokens"] = 2
with self.assertRaisesRegex(ValueError, "Token count mismatch"):
summarize([sample])
class TraceBoundaryTests(unittest.TestCase):
def test_one_token_has_no_tpot(self):
self.assertIsNone(
summarize(
[
{
"generated_tokens": 1,
"generated_token_ids": [4],
"ttft_ms": 10,
"e2e_ms": 10,
"itl_ms": [],
}
]
)["median_tpot_ms"]
)
def test_negative_and_nonfinite_intervals_are_rejected(self):
for interval in [-1, float("nan"), float("inf")]:
with self.assertRaises(ValueError):
summarize(
[
{
"generated_tokens": 2,
"generated_token_ids": [4, 5],
"ttft_ms": 10,
"e2e_ms": 20,
"itl_ms": [interval],
}
]
)
if __name__ == "__main__":
unittest.main()
class GuardBoundaryTests(unittest.TestCase):
def test_rejects_invalid_boundary_inputs_with_specific_reason(self):
import lab
cases = [('summarize([])', 'At least one'), ('summarize([dict(generated_tokens=1,generated_token_ids=[1],ttft_ms=float("nan"),e2e_ms=0,itl_ms=[])])', 'Timing values'), ('summarize([dict(generated_tokens=2,generated_token_ids=[1,2],ttft_ms=10,e2e_ms=20,itl_ms=[9])])', 'Intervals do not')]
for code, reason in cases:
with self.subTest(code=code):
with self.assertRaisesRegex((ValueError, MemoryError, KeyError, RuntimeError), reason):
exec(code, vars(lab).copy())The live runner’s prefix_reuse record retains the final-logit comparison, while batch_sweep records separate static-batch generation calls. Those calls use a different execution loop from the stage trace and should not be compared as if only batch size changed.
Run fresh local inference
The optional download below is about 1 GB and uses the official model at its pinned revision. Use --device cpu on a CPU host. The recorded run used the Metal backend; it is not a CUDA result. The model card identifies the Apache-2.0 license.
python3 -m venv .venv
.venv/bin/python -m pip install -r requirements-live.txt
.venv/bin/python download_model.py --destination ./model
.venv/bin/python live_inference.py --model-path ./model --device mps --output new-live-run.jsonCompare your result
Compare the cached and uncached runs on the same short prompt. Then compare the two prompt lengths. Explain why TTFT and TPOT should be reported separately; use the saved token IDs and intervals to support your answer.
Read the explanation
TTFT and decode intervals have different boundaries. Locate the delay before changing the system.
Measure at a consistent boundary
Use the same measurement boundary, prompt, generation settings, warmup policy, and output length when comparing changes. Keep cold start separate from warm request latency. A plausible generated explanation is not a task-quality evaluation.
References
Sources checked October 24, 2026. Runtime-specific commands and results apply to the recorded versions; linked rolling documentation can change. The application scenarios are teaching examples, not claims about an author’s production incident.