LLM Inference Engineering
Implement and inspect inference request handling, cache ownership, scheduling, measurement and capacity decisions using real local runs and clearly bounded experiments.
- Prerequisites
- Python functions, dataclasses, HTTP and JSON; basic causal attention. Offline labs run without a GPU. Live text experiments require the pinned model and runtime; Metal and CPU commands are identified separately.
- Duration
- about 8 hours of reading
- Cost
- Free, no sign-up
What you'll learn
- Measure actual prefill, decode, streaming and cancellation behavior with a pinned open model.
- Implement cache allocation and request scheduling, then reason about memory, queueing and latency.
- Evaluate a quantized candidate, profile host work and compare compiled normalization on real tensors.
- Use workload measurements to choose operating points, admission policies, rollout gates and capacity plans.
Lessons
Module 1 · 3 lessonsServe and stream a model
Trace a real inference request from prompt to first token. Stream model output and stop work when the client disconnects. Benchmark streaming inference under load.
Trace a real inference request from prompt to first tokenRead
Trace tokenization, prefill and decode with a pinned open model. Compare actual cached and uncached token intervals from a local inference run.
Stream model output and stop work when the client disconnectsRead
Build a local streaming HTTP server with bounded admission and disconnect cleanup. Test SSE framing and iterator release over real sockets.
Benchmark streaming inference under loadRead
Measure offered load, completed streams, rejections and first-token latency. Use a real HTTP client and retain errors in the benchmark record.
Module 2 · 4 lessonsFit the model and KV cache in GPU memory
Budget model weights, KV cache, and runtime memory. Choose hardware from the workload bottleneck. Implement paged cache allocation and copy-on-write. Explain tiled attention and verify its output.
Budget model weights, KV cache, and runtime memoryRead
Calculate weight and attention-cache payloads with consistent byte units. Compare grouped-query cache sizes and budget runtime memory separately.
Choose hardware from the workload bottleneckRead
Use a roofline estimate to distinguish compute and bandwidth limits. Compare memory fit and explain why hardware choice depends on workload.
Implement paged cache allocation and copy-on-writeRead
Implement a paged cache allocator with reference counts and copy-on-write. Inspect ownership and fragmentation using concrete token markers.
Explain tiled attention and verify its outputRead
Implement tiled attention with online softmax on the CPU. Verify numerical agreement and distinguish storage savings from measured GPU runtime.
Module 3 · 3 lessonsBatch requests and cache shared prompts
Schedule requests between decode iterations. Balance long-prompt prefill against streaming latency. Reuse prompt prefixes without overloading a warm replica.
Schedule requests between decode iterationsRead
Compare fixed and changing batch membership with a request scheduler. Inspect actual local model interleaving and its latency tradeoffs.
Balance long-prompt prefill against streaming latencyRead
Schedule prompt chunks around active decode work. Compare modeled token budgets with a real cached-forward experiment and explain added overhead.
Reuse prompt prefixes without overloading a warm replicaRead
Implement radix-prefix matching, pinning and eviction. Compare cache affinity with queue waiting and inspect real reuse of model cache tensors.
Module 4 · 3 lessonsMake inference faster and cheaper
Quantize a model and evaluate the regression. Check speculative decoding correctness and its cost. Choose a latency-throughput operating point.
Quantize a model and evaluate the regressionRead
Convert real model linear layers to INT8 and compare development likelihood. Inspect the observed regression before accepting a serving candidate.
Check speculative decoding correctness and its costRead
Implement speculative token acceptance and residual resampling. Verify the target distribution and separate correctness from assumed speedups.
Choose a latency-throughput operating pointRead
Choose a feasible latency and throughput operating point. Inspect memory limits, communication assumptions and efficiency per serving device.
Module 5 · 3 lessonsDebug slow GPUs and tail latency
Investigate host overhead before changing GPU settings. Compare eager and compiled normalization on real tensors. Investigate inference tail latency with request records.
Investigate host overhead before changing GPU settingsRead
Profile host operators around an actual Metal model forward. Read recorded events without treating CPU timings as GPU kernel measurements.
Compare eager and compiled normalization on real tensorsRead
Compare eager and compiled normalization on real Metal tensors. Check numerical agreement and synchronized timings across several input shapes.
Investigate inference tail latency with request recordsRead
Aggregate request records to investigate tail latency. Keep timing boundaries, preemptions and admission outcomes distinct when testing hypotheses.
Module 6 · 4 lessonsRun at production scale
Protect interactive requests from noisy neighbors. Plan cold starts and gate model rollouts. Budget cache transfer before splitting inference pools. Size serving capacity from accepted benchmark results.
Protect interactive requests from noisy neighborsRead
Implement tenant queues, token budgets and deficit round-robin scheduling. Inspect admission limits and the latency cost of protecting interactive work.
Plan cold starts and gate model rolloutsRead
Model replica startup and finite standby capacity, then evaluate canary gates. Distinguish requested capacity from ready capacity and compare tradeoffs.
Budget cache transfer before splitting inference poolsRead
Calculate cache-transfer payloads before separating prefill and decode. Explore bandwidth and expert-placement assumptions without claiming fleet benchmarks.
Size serving capacity from accepted benchmark resultsRead
Size serving replicas from accepted workload-matched results. Convert request demand to peak capacity and include reserves and stated cost assumptions.