Compare full weight updates with a low-rank adapter on a small intent classifier. Inspect which parameters change, keep held-out inputs separate and judge the update on the same fixed evaluation cases.
Problem
When you need this. You need to inspect which parameters change under a full update and a low-rank update.
Suppose a support-routing model recognizes “rollback” but sends “restore the previous release” to the deployment queue. You could add a rule, change its inputs, or train an update. Before choosing a large training job, make the proposed change observable on a small model.
This lab compares full weight updates with low-rank adaptation, or LoRA, on a small intent classifier. The classifier maps text features to deployment, rollback or credentials. You can inspect every weight and run every experiment on CPU. The from-scratch training step supplies the gradient calculation; this lesson compares the adaptation choices built on it. The course’s later language-model labs will use larger models; this pilot isolates the update mechanism.
Concepts
Full adaptation changes the base weight matrix. LoRA freezes that matrix and learns two factors whose product supplies an additive update: W = W_base + B × A. The adapter rank controls the number of trainable values and restricts the rank of the update, rather than reducing the number of output classes.
Quantization represents stored weights with lower-precision values. QLoRA combines a quantized frozen base with low-rank adaptation. This CPU classifier implements the low-rank update only; it does not implement quantized language-model training. The earlier training-step lesson supplies the gradient mechanism used here.
Architecture
Text becomes a fixed vocabulary feature vector. The baseline scores it with hand-written weights. Full adaptation changes those weights; the adapter path freezes them and adds B × A. Each candidate is evaluated on the same held-out feature combinations, with duplicate text and duplicate feature vectors checked before training.
Working code
Fix the task and the evaluation split
features() turns the presence of ten vocabulary words into a numeric vector. A three-row weight matrix maps that vector to class scores. Softmax converts the scores into probabilities. The initial weights are hand-written: they recognize a few obvious words but miss several support phrases.
train.json contains labeled adaptation examples. heldout.json contains fixed evaluation examples that training never reads. The split checker rejects both duplicate text and duplicate numeric feature vectors across training and evaluation. The 9 evaluation rows use different feature combinations from the training rows. Their labels were fixed before this run; they remain a small synthetic test, rather than a measurement of real support traffic.
The baseline, vocabulary and held-out labels are fixed before the training run. Changing the held-out examples to make a candidate win would invalidate the comparison. Real evaluation also needs deduplication beyond exact strings and cases from the target distribution.
Download and extract the lab. In ai-system-adaptation, run:
python3.12 -m venv .venv
source .venv/bin/activate
python adapt.py
python -m unittest -v test_adapt.pyFollow the matrices that change
Full adaptation updates the original weight matrix. With three output classes and ten input features, this matrix has thirty trainable values.
The LoRA path freezes that matrix and adds an update factored into two smaller matrices: W = W_base + B × A. At rank r, A has r × 10 values and B has 3 × r. Their product has the original matrix’s shape, while restricting the rank of its change.
B starts at zero, so the adapted model initially produces exactly the baseline’s outputs. A starts with small seeded random values. If both matrices started at zero, their product’s gradients would not start useful learning in this implementation.
"""Full and LoRA updates of a small softmax text classifier, on CPU without an SDK."""
from pathlib import Path
import copy
import hashlib
import json
import math
import random
VOCAB=['deploy','release','rollback','restore','token','credential','rotate','stage','revert','key']
LABELS=['deployment','rollback','credentials']
D=len(VOCAB)
C=len(LABELS)
def features(text):
import re
words=set(re.findall(r'[a-z]+',text.lower()))
return [float(word in words) for word in VOCAB]
def softmax(values):
top=max(values);raw=[math.exp(v-top) for v in values];total=math.fsum(raw)
return [v/total for v in raw]
def dot(a,b): return math.fsum(x*y for x,y in zip(a,b))
def base_weights():
w=[[0.0]*D for _ in LABELS]
for word,label in [('deploy','deployment'),('release','deployment'),('rollback','rollback'),('token','credentials')]:
w[LABELS.index(label)][VOCAB.index(word)]=1.5
return w
class Model:
def __init__(self,base,rank=None):
if rank is not None and not 1<=rank<=min(C,D): raise ValueError('rank must be between 1 and 3')
self.base=copy.deepcopy(base);self.rank=rank;self.w=copy.deepcopy(base)
rng=random.Random(31)
self.a=[[rng.uniform(-.1,.1) for _ in range(D)] for _ in range(rank or 0)]
self.b=[[0.0]*(rank or 0) for _ in range(C)]
def effective(self):
if self.rank is None: return self.w
return [[self.base[c][d]+math.fsum(self.b[c][r]*self.a[r][d] for r in range(self.rank))
for d in range(D)] for c in range(C)]
def predict(self,x): return softmax([dot(row,x) for row in self.effective()])
def train(self,rows,epochs=200,lr=.12):
for _ in range(epochs):
for row in rows:
x=features(row['text']);y=LABELS.index(row['label']);p=self.predict(x)
grad=[prob-(c==y) for c,prob in enumerate(p)]
if self.rank is None:
for c in range(C):
for d in range(D): self.w[c][d]-=lr*grad[c]*x[d]
else:
hidden=[dot(a,x) for a in self.a]
# Both gradients use the same pre-update A and B.
back=[math.fsum(self.b[c][r]*grad[c] for c in range(C)) for r in range(self.rank)]
for c in range(C):
for r in range(self.rank): self.b[c][r]-=lr*grad[c]*hidden[r]
for r in range(self.rank):
for d in range(D): self.a[r][d]-=lr*back[r]*x[d]
@property
def trainable(self): return C*D if self.rank is None else self.rank*(C+D)
def evaluate(model,rows):
correct=0;loss=0.0;predictions=[]
for row in rows:
p=model.predict(features(row['text']));target=LABELS.index(row['label']);guess=max(range(C),key=lambda i:p[i])
correct+=guess==target;loss-=math.log(max(p[target],1e-15))
predictions.append({'id':row['id'],'expected':row['label'],'predicted':LABELS[guess]})
return {'correct':correct,'total':len(rows),'mean_cross_entropy':round(loss/len(rows),6),'predictions':predictions}
def validate_split(train,test):
if not train or not test: raise ValueError('both splits required')
if set(r['text'].strip().lower() for r in train)&set(r['text'].strip().lower() for r in test):
raise ValueError('duplicate text across splits')
if {tuple(features(r['text'])) for r in train}&{tuple(features(r['text'])) for r in test}:
raise ValueError('duplicate model input across splits')
for row in train+test:
if row['label'] not in LABELS: raise ValueError('unknown label')
if not any(features(row['text'])): raise ValueError('text has no known feature')
def experiment():
root=Path(__file__).parent
train=json.loads((root/'train.json').read_text());test=json.loads((root/'heldout.json').read_text())
validate_split(train,test);base=base_weights();baseline=Model(base);results={}
for name,rank in [('full',None),('lora_rank_1',1),('lora_rank_2',2)]:
model=Model(base,rank);model.train(train)
results[name]={'trainable_parameters':model.trainable,'train':evaluate(model,train),'heldout':evaluate(model,test),
'base_unchanged':model.base==base}
return {'task':'small bag-of-words intent classifier; not a pretrained LLM','seed':31,'epochs':200,'learning_rate':.12,
'base_sha256':hashlib.sha256(json.dumps(base).encode()).hexdigest(),
'train_sha256':hashlib.sha256((root/'train.json').read_bytes()).hexdigest(),
'heldout_sha256':hashlib.sha256((root/'heldout.json').read_bytes()).hexdigest(),
'evaluation_feature_duplicates':sum(tuple(features(row['text'])) in {tuple(features(t['text'])) for t in train} for row in test),
'baseline':evaluate(baseline,test),'results':results}
if __name__ == '__main__':print(json.dumps(experiment(),indent=2,sort_keys=True))Read the update loop carefully. It computes gradients for A and B from the same pre-update parameters, then changes both. Updating B first and using the new B to calculate A’s gradient would implement a different optimization step.
Compare the observed results
All candidates use the supplied training rows, iteration count and learning rate. That makes the configuration explicit; equal epochs do not mean equal compute or equally tuned optimization.
| Method | Trainable values | Held-out correct | Training loss |
|---|---|---|---|
| full | 30 | 9 / 9 | 0.016239 |
| lora rank 1 | 13 | 9 / 9 | 0.06405 |
| lora rank 2 | 26 | 9 / 9 | 0.001077 |
The initial classifier answers 4 of 9 held-out cases correctly. Inspect the individual predictions after training, rather than reading a single aggregate score. In particular, follow the cases containing “restore,” “revert,” “credential” and “key.”
Cross-entropy measures the probability assigned to the target class, not just whether it ranks first. Two candidates can predict the same labels while assigning different probabilities. Lower loss on these training rows is not a reason to skip held-out evaluation.
Inspect predictions, split hashes and training settings
{
"base_sha256": "71330800b6de065580e9d7341399d451547f9a7e1eb4220ceba9ed00c4c50e6b",
"baseline": {
"correct": 4,
"mean_cross_entropy": 0.710581,
"predictions": [
{
"expected": "deployment",
"id": "heldout-1",
"predicted": "deployment"
},
{
"expected": "deployment",
"id": "heldout-2",
"predicted": "deployment"
},
{
"expected": "deployment",
"id": "heldout-3",
"predicted": "deployment"
},
{
"expected": "rollback",
"id": "heldout-4",
"predicted": "deployment"
},
{
"expected": "rollback",
"id": "heldout-5",
"predicted": "deployment"
},
{
"expected": "rollback",
"id": "heldout-6",
"predicted": "deployment"
},
{
"expected": "credentials",
"id": "heldout-7",
"predicted": "deployment"
},
{
"expected": "credentials",
"id": "heldout-8",
"predicted": "deployment"
},
{
"expected": "credentials",
"id": "heldout-9",
"predicted": "credentials"
}
],
"total": 9
},
"epochs": 200,
"evaluation_feature_duplicates": 0,
"heldout_sha256": "46bac3b951adb5f015d8bc748d547a6832fd22bf5db4abfed71678de29d5a289",
"learning_rate": 0.12,
"results": {
"full": {
"base_unchanged": true,
"heldout": {
"correct": 9,
"mean_cross_entropy": 0.086381,
"predictions": [
{
"expected": "deployment",
"id": "heldout-1",
"predicted": "deployment"
},
{
"expected": "deployment",
"id": "heldout-2",
"predicted": "deployment"
},
{
"expected": "deployment",
"id": "heldout-3",
"predicted": "deployment"
},
{
"expected": "rollback",
"id": "heldout-4",
"predicted": "rollback"
},
{
"expected": "rollback",
"id": "heldout-5",
"predicted": "rollback"
},
{
"expected": "rollback",
"id": "heldout-6",
"predicted": "rollback"
},
{
"expected": "credentials",
"id": "heldout-7",
"predicted": "credentials"
},
{
"expected": "credentials",
"id": "heldout-8",
"predicted": "credentials"
},
{
"expected": "credentials",
"id": "heldout-9",
"predicted": "credentials"
}
],
"total": 9
},
"train": {
"correct": 12,
"mean_cross_entropy": 0.016239,
"predictions": [
{
"expected": "deployment",
"id": "train-1",
"predicted": "deployment"
},
{
"expected": "deployment",
"id": "train-2",
"predicted": "deployment"
},
{
"expected": "deployment",
"id": "train-3",
"predicted": "deployment"
},
{
"expected": "rollback",
"id": "train-4",
"predicted": "rollback"
},
{
"expected": "rollback",
"id": "train-5",
"predicted": "rollback"
},
{
"expected": "rollback",
"id": "train-6",
"predicted": "rollback"
},
{
"expected": "rollback",
"id": "train-7",
"predicted": "rollback"
},
{
"expected": "credentials",
"id": "train-8",
"predicted": "credentials"
},
{
"expected": "credentials",
"id": "train-9",
"predicted": "credentials"
},
{
"expected": "credentials",
"id": "train-10",
"predicted": "credentials"
},
{
"expected": "credentials",
"id": "train-11",
"predicted": "credentials"
},
{
"expected": "credentials",
"id": "train-12",
"predicted": "credentials"
}
],
"total": 12
},
"trainable_parameters": 30
},
"lora_rank_1": {
"base_unchanged": true,
"heldout": {
"correct": 9,
"mean_cross_entropy": 0.115805,
"predictions": [
{
"expected": "deployment",
"id": "heldout-1",
"predicted": "deployment"
},
{
"expected": "deployment",
"id": "heldout-2",
"predicted": "deployment"
},
{
"expected": "deployment",
"id": "heldout-3",
"predicted": "deployment"
},
{
"expected": "rollback",
"id": "heldout-4",
"predicted": "rollback"
},
{
"expected": "rollback",
"id": "heldout-5",
"predicted": "rollback"
},
{
"expected": "rollback",
"id": "heldout-6",
"predicted": "rollback"
},
{
"expected": "credentials",
"id": "heldout-7",
"predicted": "credentials"
},
{
"expected": "credentials",
"id": "heldout-8",
"predicted": "credentials"
},
{
"expected": "credentials",
"id": "heldout-9",
"predicted": "credentials"
}
],
"total": 9
},
"train": {
"correct": 12,
"mean_cross_entropy": 0.06405,
"predictions": [
{
"expected": "deployment",
"id": "train-1",
"predicted": "deployment"
},
{
"expected": "deployment",
"id": "train-2",
"predicted": "deployment"
},
{
"expected": "deployment",
"id": "train-3",
"predicted": "deployment"
},
{
"expected": "rollback",
"id": "train-4",
"predicted": "rollback"
},
{
"expected": "rollback",
"id": "train-5",
"predicted": "rollback"
},
{
"expected": "rollback",
"id": "train-6",
"predicted": "rollback"
},
{
"expected": "rollback",
"id": "train-7",
"predicted": "rollback"
},
{
"expected": "credentials",
"id": "train-8",
"predicted": "credentials"
},
{
"expected": "credentials",
"id": "train-9",
"predicted": "credentials"
},
{
"expected": "credentials",
"id": "train-10",
"predicted": "credentials"
},
{
"expected": "credentials",
"id": "train-11",
"predicted": "credentials"
},
{
"expected": "credentials",
"id": "train-12",
"predicted": "credentials"
}
],
"total": 12
},
"trainable_parameters": 13
},
"lora_rank_2": {
"base_unchanged": true,
"heldout": {
"correct": 9,
"mean_cross_entropy": 0.027204,
"predictions": [
{
"expected": "deployment",
"id": "heldout-1",
"predicted": "deployment"
},
{
"expected": "deployment",
"id": "heldout-2",
"predicted": "deployment"
},
{
"expected": "deployment",
"id": "heldout-3",
"predicted": "deployment"
},
{
"expected": "rollback",
"id": "heldout-4",
"predicted": "rollback"
},
{
"expected": "rollback",
"id": "heldout-5",
"predicted": "rollback"
},
{
"expected": "rollback",
"id": "heldout-6",
"predicted": "rollback"
},
{
"expected": "credentials",
"id": "heldout-7",
"predicted": "credentials"
},
{
"expected": "credentials",
"id": "heldout-8",
"predicted": "credentials"
},
{
"expected": "credentials",
"id": "heldout-9",
"predicted": "credentials"
}
],
"total": 9
},
"train": {
"correct": 12,
"mean_cross_entropy": 0.001077,
"predictions": [
{
"expected": "deployment",
"id": "train-1",
"predicted": "deployment"
},
{
"expected": "deployment",
"id": "train-2",
"predicted": "deployment"
},
{
"expected": "deployment",
"id": "train-3",
"predicted": "deployment"
},
{
"expected": "rollback",
"id": "train-4",
"predicted": "rollback"
},
{
"expected": "rollback",
"id": "train-5",
"predicted": "rollback"
},
{
"expected": "rollback",
"id": "train-6",
"predicted": "rollback"
},
{
"expected": "rollback",
"id": "train-7",
"predicted": "rollback"
},
{
"expected": "credentials",
"id": "train-8",
"predicted": "credentials"
},
{
"expected": "credentials",
"id": "train-9",
"predicted": "credentials"
},
{
"expected": "credentials",
"id": "train-10",
"predicted": "credentials"
},
{
"expected": "credentials",
"id": "train-11",
"predicted": "credentials"
},
{
"expected": "credentials",
"id": "train-12",
"predicted": "credentials"
}
],
"total": 12
},
"trainable_parameters": 26
}
},
"seed": 31,
"task": "small bag-of-words intent classifier; not a pretrained LLM",
"train_sha256": "f7da4b0b5143f626465d1c7911908871c9f41bf20fa20cdff078bffa413aeb2a"
}Distinguish LoRA from QLoRA
LoRA describes the low-rank update. QLoRA combines adaptation with a quantized frozen base model and additional memory-saving techniques. It does not mean that every number in the training process is four-bit.
The classifier here keeps ordinary Python numeric values. Its parameter counts demonstrate which weights receive updates; they do not measure GPU memory. Optimizer state, gradients, activations, tensor formats and runtime overhead also contribute to a real training job’s memory use.
A subsequent QLoRA lesson should run an actual pinned implementation, record model and dataset revisions, measure peak memory, and compare held-out quality with an appropriate baseline. Relabeling rounded Python weights as a QLoRA benchmark would skip that work.
Test the update and choose the next experiment
An invalid adapter rank is rejected before creating the model. Duplicate text or model inputs across splits are rejected before training. Unknown labels and empty splits also stop the experiment instead of producing a misleading report.
The tests check that the adapter begins as the base model, training leaves the frozen base unchanged, and training reduces loss on a controlled example. The repeatability check compares complete reports from repeated experiments with the same seed and fixed data.
Try rank one and rank two on an expanded, separately labeled set. Keep evaluation examples fixed. Record which errors remain, whether extra capacity helps, and whether a simple input or rule change would solve the same task more cheaply.
Design exercise: if a model answers with outdated deployment facts, would you start with LoRA, retrieval, or a source-data correction? State what must change in the system, then propose a test that would distinguish those options.
Read the explanation
Their product supplies the update. The tests check that training leaves the base unchanged.
Failure cases
The split check protects the comparison before an update runs. Different strings can still become the same feature vector, so text deduplication alone is insufficient. Invalid ranks and malformed labels also refuse; they must not silently change the experiment to make a candidate appear better.
Check the input and failure boundaries
| Trigger | Required behavior |
|---|---|
| An invalid adapter rank | Reject a nonpositive or unsupported rank before training. |
| Duplicate text or model inputs across splits | Refuse overlap in either normalized text or computed feature vectors. |
| Unknown labels and empty splits | Refuse inputs that cannot support the declared classification comparison. |
Deployment
The output is a local classifier experiment, not a deployable language-model adapter. Retain the feature vocabulary, base weights, update configuration and evaluation split together when repeating it. A later language-model adapter must additionally bind its base-model and tokenizer revisions and pass that task’s independent evaluations.
Scaling
Trainable adapter values grow with rank and the input/output dimensions of the adapted matrix. Reducing that count does not by itself measure inference latency, total training memory or task quality. Quantized storage, optimizer state and language-model evaluation require separate experiments beyond this fixture.
Key takeaways
- LoRA changes a frozen model through an additive low-rank update.
- Equal initial outputs depend on how the two factors are initialized.
- The small classifier comparison teaches the update mechanism, not language-model adaptation quality.
Best practices
- Fix held-out cases before inspecting candidate results.
- Check both text and model-input overlap across splits.
- Report rank, trainable parameter count and task outcomes together.
References
- LoRA paper: freezing base weights and learning a low-rank update.
- QLoRA paper and authors’ implementation: quantized-base adaptation, NF4, double quantization and paged optimizers.
Sources reviewed October 23, 2026. The recorded results describe the supplied local classifier, not language-model quality or GPU memory savings.