Build a Small Language Model From Scratch
Build and inspect a small one-block transformer and its complete local training and generation path.
- Prerequisites
- Python basics, elementary algebra and probabilities. The preparation guide below reviews the numerical ideas used in Lesson 1.
- Duration
- about 5 hours of reading
- Cost
- Free, no sign-up
Before you start
The examples use a bundled synthetic story corpus and Python’s standard library. Follow the same tokenizer and model through training, checkpointing, generation and a local HTTP service.
- Python prerequisites
- Read loops and nested lists, write functions, work with dictionaries and JSON, and run a unit test. The labs use Python’s standard library and run on a CPU.
- Mathematics prerequisites
- Follow basic algebra, multiply and add vector entries, and read probabilities. The examples below introduce the derivatives and logarithms used in the first training step.
Introduced in the course. Embeddings, loss, gradients, attention, transformer blocks, checkpoints, inference and cached generation are explained before you implement them. You do not need prior experience with an ML framework.
Review three numerical examplesDot products, gradient updates and prediction loss
Dot product
Multiply matching entries, then add. Attention uses this operation to score a query against a key.
[1, 2] · [3, 4] = 1×3 + 2×4 = 11Gradient update
The derivative describes local sensitivity. Subtracting the learning rate times that derivative gives one gradient-descent update; a rate that is too large can overshoot.
L(w) = w²; at w = 2, dL/dw = 4 new w = 2 − 0.1×4 = 1.6Prediction loss
Giving less probability to the correct token produces a larger loss. Lesson 1 connects that loss to parameter gradients.
−ln(0.5) ≈ 0.693 −ln(0.1) ≈ 2.303
Course termsShort definitions linked to their worked implementation
- Token, token ID and vocabulary
A token is one text unit the model represents. Its token ID is an integer used to select a vocabulary entry. A vocabulary is the fixed set of entries the model can predict.
Explore token, token id and vocabulary →- Vector, matrix, tensor and shape
A vector is a one-dimensional array of numbers; a matrix has rows and columns. Tensor is the general array term. Its shape lists the length of each axis, such as (3, 2) for three rows of two values.
Explore vector, matrix, tensor and shape →- Dot product and matrix multiplication
A dot product multiplies matching vector entries and sums them into one score. Matrix multiplication applies compatible dot products across rows and columns. Matching inner dimensions is required.
Explore dot product and matrix multiplication →- Embedding and parameter
An embedding maps a token ID to a trainable vector. A parameter is a numeric value training adjusts, including embedding entries, weights and biases.
Explore embedding and parameter →- Logit, softmax and probability distribution
A logit is a raw prediction score. Softmax exponentiates a set of scores and normalizes them into nonnegative probabilities summing to one.
Explore logit, softmax and probability distribution →- Loss and cross-entropy
A loss is a numerical penalty used to guide training. For the observed next token here, cross-entropy is the negative natural logarithm of the probability assigned to that token, averaged across scored examples.
Explore loss and cross-entropy →- Derivative, gradient and chain rule
A derivative measures how a quantity changes locally with one input. A gradient collects derivatives with respect to several parameters. The chain rule combines local derivatives through a sequence of operations.
Explore derivative, gradient and chain rule →- Forward pass, backward pass and optimizer
The forward pass computes predictions and loss. The backward pass computes parameter gradients. An optimizer uses those gradients and its update rule to change the parameters.
Explore forward pass, backward pass and optimizer →- Learning rate, batch, step and epoch
The learning rate controls update size. A batch is the group of examples contributing to one update; that update is a step. An epoch is a pass through the training examples.
Explore learning rate, batch, step and epoch →- Training, development and test sets
Training examples determine parameter updates. Development examples guide settings and checkpoint selection. Test examples are reserved for evaluation after those choices.
Explore training, development and test sets →- Causal attention, query, key and value
Each position computes a query that scores allowed keys, then mixes their value vectors. A causal mask prevents it from reading later positions that would reveal the prediction target.
Explore causal attention, query, key and value →- Transformer block
The layer built here combines attention with a per-position feed-forward network, residual additions and normalization. Position embeddings supply position information to the model.
Explore transformer block →- Checkpoint
A resumable checkpoint preserves parameters and the optimizer, data-order and input identities needed for the next training step. Selected model parameters alone serve a different purpose.
Explore checkpoint →- Training, inference and generation
Training changes model parameters using examples. Inference applies fixed parameters to an input to produce predictions. Text generation repeats inference and token selection, adding each chosen token to the context.
Explore training, inference and generation →- KV cache
A key-value cache retains earlier attention keys and values for one sequence. It avoids recomputing those vectors during inference while keeping the model parameters fixed.
Explore kv cache →
From training inputs to generated text
What you'll learn
- Implement and inspect a training step, tokenizer, causal attention and transformer block.
- Train on bundled synthetic stories and bind resumable checkpoints to their inputs.
- Generate text with sampling, compare cached generation and validate a local HTTP interface.
- Port the one-block transformer to PyTorch and compare outputs and gradients within explicit numerical tolerances.
Lessons
Module 1 · 5 lessonsTokenization, baselines and transformer blocks
Build one training step by handRead
Follow the forward pass, check every gradient and apply one update.
Build a byte-pair encoding tokenizerRead
Learn reusable text pieces and save the tokenizer used throughout the course.
Build a next-token prediction baselineRead
Compare uniform, unigram and bigram predictions on fixed data splits.
Build causal self-attention and inspect intermediate outputsRead
Mix information from earlier tokens while preventing access to future positions.
Build a transformer block with positional informationRead
Combine attention, position embeddings and a feed-forward network into one model.
Module 2 · 6 lessonsTraining, checkpoints and inference
Train and retain the model, generate tokens, cache computation and serve it over HTTP. Then compare its handwritten equations and gradients with a CPU tensor implementation.
Build a small language-model training pipelineRead
Train the model in batches and monitor development loss.
Build checkpoints and resumable trainingRead
Resume from complete training state and select parameters by development loss.
Build text generation with sampling controlsRead
Run inference repeatedly, choosing tokens with temperature, top-k sampling and stop rules.
Build a KV cache for LLM generationRead
Reuse cached keys and values during inference while matching the uncached generator.
Build a local server for the trained modelRead
Serve the inference loop over HTTP and compare responses with direct generation.
Compare a handwritten transformer with PyTorchRead
Port the existing one-block transformer to CPU tensors, then compare logits, loss and every parameter gradient with explicit attention and PyTorch SDPA.