LLM Inference Engineering
Implement and inspect inference request handling, cache ownership, scheduling, measurement and capacity decisions using real local runs and clearly bounded experiments.
What is LLM inference?
LLM inference runs a trained language model on an input to produce token scores or generated text. Tokenization encodes the prompt, prefill evaluates it and creates attention-cache state, and decode produces later tokens. Queueing, memory ownership and client streaming add application boundaries around those model operations. Changing one boundary can move the bottleneck to another.
Follow a local model request from the prompt to its first token, then compare cached decoding, HTTP streaming and offered load. Later labs isolate cache allocation, scheduling and capacity planning. Each lesson distinguishes actual model measurements, controlled fixtures and assumed cost models; the course does not claim a deployed GPU fleet.
What you'll learn
- Measure actual prefill, decode, streaming and cancellation behavior with a pinned open model.
- Implement cache allocation and request scheduling, then reason about memory, queueing and latency.
- Evaluate a quantized candidate, profile host work and compare compiled normalization on real tensors.
- Separate workload measurements from assumed cost models when comparing operating points, admission policies, rollout gates and capacity plans.
Lessons
Module 1 · 3 lessons · about 1 hourServe and stream a model
Trace a real inference request from prompt to first token. Stream model output and stop work when the client disconnects. Benchmark streaming inference under load.
Trace a real inference request from prompt to first tokenRead
Trace tokenization, prefill and decode with a pinned open model. Compare actual cached and uncached token intervals from a local inference run.
Stream model output and stop work when the client disconnectsRead
Build a local streaming HTTP server with bounded admission and disconnect cleanup. Test SSE framing and iterator release over real sockets.
Benchmark streaming inference under loadRead
Measure offered load, completed streams, rejections and first-token latency. Use a real HTTP client and retain errors in the benchmark record.
Module 2 · 4 lessons · about 2 hoursBudget weights, cache and attention memory
Budget model weights, KV cache, and runtime memory. Choose hardware from the workload bottleneck. Implement paged cache allocation and copy-on-write. Explain tiled attention and verify its output.
Budget model weights, KV cache, and runtime memoryRead
Calculate weight and attention-cache payloads with consistent byte units. Compare grouped-query cache sizes and budget runtime memory separately.
Choose hardware from the workload bottleneckRead
Use a roofline estimate to distinguish compute and bandwidth limits. Compare memory fit and explain why hardware choice depends on workload.
Implement paged cache allocation and copy-on-writeRead
Implement a paged cache allocator with reference counts and copy-on-write. Inspect ownership and fragmentation using concrete token markers.
Explain tiled attention and verify its outputRead
Implement tiled attention with online softmax on the CPU. Verify numerical agreement and distinguish storage savings from measured GPU runtime.
Module 3 · 3 lessons · about 1 hourBatch requests and cache shared prompts
Schedule requests between decode iterations. Balance long-prompt prefill against streaming latency. Reuse prompt prefixes without overloading a warm replica.
Schedule requests between decode iterationsRead
Compare fixed and changing batch membership with a request scheduler. Inspect actual local model interleaving and its latency tradeoffs.
Balance long-prompt prefill against streaming latencyRead
Schedule prompt chunks around active decode work. Compare modeled token budgets with a real cached-forward experiment and explain added overhead.
Reuse prompt prefixes without overloading a warm replicaRead
Implement radix-prefix matching, pinning and eviction. Compare cache affinity with queue waiting and inspect real reuse of model cache tensors.
Module 4 · 3 lessons · about 1 hourCompare inference optimizations and their costs
Quantize a model and evaluate the regression. Check speculative decoding correctness and its cost. Choose a latency-throughput operating point.
Quantize a model and evaluate the regressionRead
Convert real model linear layers to INT8 and compare development likelihood. Inspect the observed regression before accepting a serving candidate.
Check speculative decoding correctness and its costRead
Implement speculative token acceptance and residual resampling. Verify the target distribution and separate correctness from assumed speedups.
Choose a latency-throughput operating pointRead
Choose a feasible latency and throughput operating point. Inspect memory limits, communication assumptions and efficiency per serving device.
Module 5 · 3 lessons · about 1 hourProfile inference work and tail latency
Investigate host overhead before changing GPU settings. Compare eager and compiled normalization on real tensors. Investigate inference tail latency with request records.
Investigate host overhead before changing GPU settingsRead
Profile host operators around an actual Metal model forward. Read recorded events without treating CPU timings as GPU kernel measurements.
Compare eager and compiled normalization on real tensorsRead
Compare eager and compiled normalization on real Metal tensors. Check numerical agreement and synchronized timings across several input shapes.
Investigate inference tail latency with request recordsRead
Aggregate request records to investigate tail latency. Keep timing boundaries, preemptions and admission outcomes distinct when testing hypotheses.
Module 6 · 4 lessons · about 2 hoursPlan capacity and model rollouts
Protect interactive requests from noisy neighbors. Plan cold starts and gate model rollouts. Budget cache transfer before splitting inference pools. Size serving capacity from accepted benchmark results.
Protect interactive requests from noisy neighborsRead
Implement tenant queues, token budgets and deficit round-robin scheduling. Inspect admission limits and the latency cost of protecting interactive work.
Plan cold starts and gate model rolloutsRead
Model replica startup and finite standby capacity, then evaluate canary gates. Distinguish requested capacity from ready capacity and compare tradeoffs.
Budget cache transfer before splitting inference poolsRead
Calculate cache-transfer payloads before separating prefill and decode. Explore bandwidth and expert-placement assumptions without claiming fleet benchmarks.
Size serving capacity from accepted benchmark resultsRead
Size serving replicas from accepted workload-matched results. Convert request demand to peak capacity and include reserves and stated cost assumptions.