Building AI Systems
Choose, measure and improve an AI system against explicit quality, latency, cost and failure requirements.
- Prerequisites
- Python, basic HTTP and JSON, and AI Fundamentals. The adaptation modules add introductory linear algebra and gradients.
- Duration
- about 9 hours of reading
- Cost
- Free, no sign-up
What you can build now
The seventeen available lessons are separate experiments, not one integrated deployment. Their tasks include docuctl support, classification, policy boundaries, request tracing and inference measurements. Each article identifies its own data and execution mode. Follow the five modules in order, then use the planned integration work to connect the components into the course project.
What you'll learn
- Define task acceptance, resource budgets and separate response-quality checks.
- Compare model interfaces and candidates with reproducible evaluation contracts.
- Apply access boundaries, output guards, tracing and monitoring.
- Inspect inference work and low-rank updates in bounded local experiments.
- Separate preference-pair and sampled-reward objectives; inspect packed weight error and decoding admission schedules.
Lessons
Module 1 · 3 lessonsDefine the task and its acceptance contract
Choose an architecture from task requirementsRead
Build a version-aware documentation lookup and an approval-bound action proposal. Use their contracts to decide where application code ends and a model could help.
Create an evaluation set and a failure taxonomyRead
Turn the documentation assistant's version, environment, role and action requirements into cases. Reuse the evaluation runner and distinguish failed contracts from incomplete execution.
Set task acceptance criteria and request resource budgetsRead
Require both a correct task outcome and an explicit request resource policy. Reuse application graders and capacity admission, and distinguish preflight limits from post-execution checks.
Module 2 · 5 lessonsEvaluate answers and compare system changes
Define a task-specific answer rubric using the existing eval runnerRead
Write source-based criteria for a documentation answer, bind example judgements to exact revisions and keep pending review separate from scored outcomes.
Separate source support, action correctness and format validityRead
Diagnose a structured answer without letting valid JSON, a matching citation or a correct command stand in for the other requirements.
Measure quality by request slice and asymmetric failure costRead
Inspect request-group coverage and declared failure penalties without hiding critical failures, incomplete execution or empty groups behind an overall pass count.
Build a repeated-trial benchmark that records failures and incomplete runsRead
Run every evaluation case many times against sampled local models, log each trial as it finishes, resume a stopped run without losing work, and report per-case pass counts, flaky cases, pass@k and pass^k.
Build a paired comparison report with uncertainty and slice-level resultsRead
Compare two prompt versions on the same cases and seeds, report the per-case difference with bootstrap intervals over cases, source groups and trials, and read the slices before trusting a gain.
Module 3 · 2 lessonsAdapt model interfaces and select a candidate
Build a portable model adapter with contract testsRead
Put one chat contract in front of two wire formats, Ollama's native API and its OpenAI-compatible API, and run the same contract tests against recorded traffic from both before calling either one supported.
Build a model selection harnessRead
Write the requirements for one small task, check two installed open models against them, run labeled development and held-out cases, and let a recorded rule choose the model.
Module 4 · 5 lessonsControl access, guard output and inspect requests
Build authentication and authorization boundaries for users and toolsRead
Let an agent call tools with the signed-in user's authority, not its own. Mint a short-lived token for each call that names the user, one scope and the agent, and make every tool service check it again.
Build a policy boundary for untrusted notes, tool outputs, and agent handoffsRead
Let an agent read notes, tool results and other agents' handoffs without letting their text choose its actions. Check sensitive arguments against the account's own records, hold refunds once untrusted text is in the conversation, and test it with recorded agent loops.
Build a guardrails proxy with streaming-output testsRead
Put a proxy between a streaming model and the client that redacts or blocks email addresses and card numbers even when they arrive split across chunks, and test it with recorded streams and every chunk size.
Build end-to-end request tracing with source provenanceRead
Trace one support answer across a gateway and a retrieval service with W3C traceparent, give every citation a provenance verdict, and follow each failed evaluation row to the step that lost its source.
Build LLM monitoring for latency, cost and errorsRead
Turn every request of a simulated model service into window metrics for latency, tokens, cost, errors and queue depth, estimate percentiles from histogram buckets, and test alert rules that go pending, fire and resolve.
Module 5 · 4 lessonsCompare inference work and adaptation choices
Profile KV-cache work across serving workloadsRead
Implement one causal attention layer, reuse previous keys and values, and compare equivalent outputs, counted work and local CPU timings.
Compare full updates and low-rank adaptationRead
Train full and low-rank updates of a small text classifier, compare held-out predictions and count the parameters each method changes.
Compare packed weight quantization and output errorRead
Pack trained matrices into eight-bit values or paired four-bit codes. Compare numeric payload sizes and output errors using dequantized CPU inference.
Admit decoding requests between token stepsRead
Compute last-position logits for unequal-length prefixes in one padded tensor batch. Execute a focused local CPU experiment and inspect its raw result.
Module 6 · 2 lessonsInspect preference and reward policy updates
Separate fixed-pair DPO from online sampled group-relative updates in bounded CPU experiments.
Inspect a direct preference optimization updateRead
Align completion-token scores with their preceding prompt positions. Execute a focused local CPU experiment and inspect its raw result.
Run group-relative policy updates on sampled completionsRead
Sample completions from a tiny CPU policy, compute group-relative rewards, and inspect clipped updates with fixed old scores and a frozen reference.
Planned modules and capstone
The lessons above are available now. The following topics are planned; they have no lesson pages or labs yet.
Module 7: Integrate the reviewed components into docuctlPlanned
- Choose source correction, retrieval or adaptation from observed failures
- Apply the Context & Memory course components to the docuctl request
- Connect the reviewed identity and tool boundaries to the application
- Compare a candidate under unchanged quality and resource budgets
- Measure dependency recovery and hand off a versioned release decision
Module 8: Measure inference and serving under loadPlanned
- Locate prefill, decoding and application work in one request
- Budget weights, activations and KV state for a workload
- Compare MHA, MQA and grouped-query serving requirements
- Measure real serving queueing, first-token latency and throughput
- Test prefix-cache isolation and deployment-scale quantized quality
Module 9: Adapt a language modelPlanned
- Prepare supervised language-model examples and a training objective
- Fine-tune a pinned language model with LoRA
- Run an actual QLoRA experiment and measure its memory tradeoff
Module 10: Evaluate preference learning and reasoningPlanned
- Build and audit human preference pairs
- Train and evaluate a reward model
- Evaluate pretrained-model preference and reward updates on held-out tasks
- Evaluate reasoning strategies under a fixed request budget
Final capstone: Build a versioned docuctl support system, compare its baseline and candidate on held-out tasks, inject dependency failures, and write a release decision from the results.Planned
This project is planned. No project lab is available yet.