aiml

Lesson 17 of 21Module 5

Compare full updates and low-rank adaptation

Train full and low-rank updates of a small text classifier, compare held-out predictions and count the parameters each method changes.

Prerequisites

Python lists and loops, matrix multiplication, softmax cross-entropy and gradients. Build one training step by hand in the from-scratch course before comparing adaptation methods.

By the end, you can

  • Train full and low-rank updates against the same labeled examples.
  • Count trainable parameters from matrix shapes.
  • Explain what a small classifier experiment says about LoRA and what QLoRA adds.
Runtime
Python 3.12+
Dependencies
Standard library only

Lab code, tests and fixtures (ZIP, 7 KB)Quiz check your understanding

Progress is saved in this browser.

Compare full weight updates with a low-rank adapter on a small intent classifier. Inspect which parameters change, keep held-out inputs separate and judge the update on the same fixed evaluation cases.

Problem

When you need this. You need to inspect which parameters change under a full update and a low-rank update.

Suppose a support-routing model recognizes “rollback” but sends “restore the previous release” to the deployment queue. You could add a rule, change its inputs, or train an update. Before choosing a large training job, make the proposed change observable on a small model.

This lab compares full weight updates with low-rank adaptation, or LoRA, on a small intent classifier. The classifier maps text features to deployment, rollback or credentials. You can inspect every weight and run every experiment on CPU. The from-scratch training step supplies the gradient calculation; this lesson compares the adaptation choices built on it. The classifier shows the update mechanism only; it does not fine-tune a language model.

Concepts

Full adaptation changes the base weight matrix. LoRA freezes that matrix and learns two factors whose product supplies an additive update: W = W_base + B × A. The adapter rank, together with the matrix shape, sets the number of trainable values and limits the rank of the update. It does not change the number of output classes.

Quantization stores weights as lower-precision values. QLoRA combines a quantized frozen base with low-rank adaptation. This CPU classifier implements only the low-rank part.

Architecture

Text featuresxFrozen weightsW baseTrainable adapterB × AAdd both outputsW base x + B A xTarget labelsand training lossupdate only A and BText features xFrozen weights W baseTrainable adapter B × AAdd both outputsW base x + B A xTarget labelsand training lossLoss updates A and B only.
The LoRA path freezes the base matrix and learns A and B from training labels. Full fine-tuning updates the original matrix instead. Held-out examples are used only for evaluation.

Text becomes a fixed vocabulary feature vector. The baseline scores it with hand-written weights. Full adaptation changes those weights; the adapter path freezes them and adds B × A. Before training, the split check rejects duplicate text and duplicate feature vectors. Every candidate is then scored on the same held-out rows.

Working code

Fix the task and the evaluation split

features() turns the presence of ten vocabulary words into a numeric vector. A three-row weight matrix maps that vector to class scores. Softmax converts the scores into probabilities. The initial weights are hand-written: they recognize a few obvious words but miss several support phrases.

train.json contains labeled adaptation examples. heldout.json contains fixed evaluation examples that training never reads. The split checker rejects both duplicate text and duplicate numeric feature vectors across training and evaluation. The 9 evaluation rows use feature combinations that no training row uses. They are a small synthetic test, not a sample of real support traffic.

The baseline, vocabulary and held-out labels are fixed before the training run. Changing the held-out examples to make a candidate win would invalidate the comparison. Real evaluation also needs deduplication beyond exact strings and cases from the target distribution.

Download and extract the lab. In ai-system-adaptation, run:

Train and compare the update methodsTerminal · bash
python3.12 -m venv .venv
source .venv/bin/activate
python adapt.py
python -m unittest -v test_adapt.py

Follow the matrices that change

Full adaptation updates the original weight matrix. With three output classes and ten input features, this matrix has thirty trainable values.

The LoRA path freezes that matrix and adds an update factored into two smaller matrices: W = W_base + B × A. At rank r, A has r × 10 values and B has 3 × r. Their product has the original matrix’s shape, while restricting the rank of its change.

B starts at zero, so the adapted model initially produces exactly the baseline’s outputs. A starts with small seeded random values. If A also started at zero, the gradients for both A and B would be zero, and training would never move the adapter.

adapt.pySource · pythonDownload
"""Full and LoRA updates of a small softmax text classifier, on CPU without an SDK."""
from pathlib import Path
import copy
import hashlib
import json
import math
import random
VOCAB=['deploy','release','rollback','restore','token','credential','rotate','stage','revert','key']
LABELS=['deployment','rollback','credentials']
D=len(VOCAB)
C=len(LABELS)
def features(text):
    import re
    words=set(re.findall(r'[a-z]+',text.lower()))
    return [float(word in words) for word in VOCAB]
def softmax(values):
    top=max(values);raw=[math.exp(v-top) for v in values];total=math.fsum(raw)
    return [v/total for v in raw]
def dot(a,b): return math.fsum(x*y for x,y in zip(a,b))
def base_weights():
    w=[[0.0]*D for _ in LABELS]
    for word,label in [('deploy','deployment'),('release','deployment'),('rollback','rollback'),('token','credentials')]:
        w[LABELS.index(label)][VOCAB.index(word)]=1.5
    return w
class Model:
    def __init__(self,base,rank=None):
        if rank is not None and not 1<=rank<=min(C,D): raise ValueError('rank must be between 1 and 3')
        self.base=copy.deepcopy(base);self.rank=rank;self.w=copy.deepcopy(base)
        rng=random.Random(31)
        self.a=[[rng.uniform(-.1,.1) for _ in range(D)] for _ in range(rank or 0)]
        self.b=[[0.0]*(rank or 0) for _ in range(C)]
    def effective(self):
        if self.rank is None: return self.w
        return [[self.base[c][d]+math.fsum(self.b[c][r]*self.a[r][d] for r in range(self.rank))
                 for d in range(D)] for c in range(C)]
    def predict(self,x): return softmax([dot(row,x) for row in self.effective()])
    def train(self,rows,epochs=200,lr=.12):
        for _ in range(epochs):
            for row in rows:
                x=features(row['text']);y=LABELS.index(row['label']);p=self.predict(x)
                grad=[prob-(c==y) for c,prob in enumerate(p)]
                if self.rank is None:
                    for c in range(C):
                        for d in range(D): self.w[c][d]-=lr*grad[c]*x[d]
                else:
                    hidden=[dot(a,x) for a in self.a]
                    # Both gradients use the same pre-update A and B.
                    back=[math.fsum(self.b[c][r]*grad[c] for c in range(C)) for r in range(self.rank)]
                    for c in range(C):
                        for r in range(self.rank): self.b[c][r]-=lr*grad[c]*hidden[r]
                    for r in range(self.rank):
                        for d in range(D): self.a[r][d]-=lr*back[r]*x[d]
    @property
    def trainable(self): return C*D if self.rank is None else self.rank*(C+D)
def evaluate(model,rows):
    correct=0;loss=0.0;predictions=[]
    for row in rows:
        p=model.predict(features(row['text']));target=LABELS.index(row['label']);guess=max(range(C),key=lambda i:p[i])
        correct+=guess==target;loss-=math.log(max(p[target],1e-15))
        predictions.append({'id':row['id'],'expected':row['label'],'predicted':LABELS[guess]})
    return {'correct':correct,'total':len(rows),'mean_cross_entropy':round(loss/len(rows),6),'predictions':predictions}
def validate_split(train,test):
    if not train or not test: raise ValueError('both splits required')
    if set(r['text'].strip().lower() for r in train)&set(r['text'].strip().lower() for r in test):
        raise ValueError('duplicate text across splits')
    if {tuple(features(r['text'])) for r in train}&{tuple(features(r['text'])) for r in test}:
        raise ValueError('duplicate model input across splits')
    for row in train+test:
        if row['label'] not in LABELS: raise ValueError('unknown label')
        if not any(features(row['text'])): raise ValueError('text has no known feature')
def experiment():
    root=Path(__file__).parent
    train=json.loads((root/'train.json').read_text());test=json.loads((root/'heldout.json').read_text())
    validate_split(train,test);base=base_weights();baseline=Model(base);results={}
    for name,rank in [('full',None),('lora_rank_1',1),('lora_rank_2',2)]:
        model=Model(base,rank);model.train(train)
        results[name]={'trainable_parameters':model.trainable,'train':evaluate(model,train),'heldout':evaluate(model,test),
                       'base_unchanged':model.base==base}
    return {'task':'small bag-of-words intent classifier; not a pretrained LLM','seed':31,'epochs':200,'learning_rate':.12,
            'base_sha256':hashlib.sha256(json.dumps(base).encode()).hexdigest(),
            'train_sha256':hashlib.sha256((root/'train.json').read_bytes()).hexdigest(),
            'heldout_sha256':hashlib.sha256((root/'heldout.json').read_bytes()).hexdigest(),
            'evaluation_feature_duplicates':sum(tuple(features(row['text'])) in {tuple(features(t['text'])) for t in train} for row in test),
            'baseline':evaluate(baseline,test),'results':results}
if __name__ == '__main__':print(json.dumps(experiment(),indent=2,sort_keys=True))

The update loop computes gradients for A and B from the same pre-update parameters, then changes both. Updating B first and using the new B to calculate A’s gradient would implement a different optimization step.

Compare the observed results

All candidates use the same training rows, epoch count and learning rate. Equal epochs do not mean equal compute or equally tuned optimization.

MethodTrainable valuesHeld-out correctTraining loss
full309 / 90.016239
lora rank 1139 / 90.06405
lora rank 2269 / 90.001077

The initial classifier answers 4 of 9 held-out cases correctly. Inspect the individual predictions after training, rather than reading a single aggregate score. In particular, follow the cases containing “restore,” “revert,” “credential” and “key.”

Cross-entropy measures the probability assigned to the target class, not just whether it ranks first. Two candidates can predict the same labels while assigning different probabilities. Lower loss on these training rows is not a reason to skip held-out evaluation. Both ranks label all 9 held-out cases correctly here, but rank two has the lower held-out cross-entropy in the recorded output below. Compare held-out loss as well as the count, and confirm the choice on a separately labeled set before you rely on it.

Inspect predictions, split hashes and training settings
Recorded local adaptation comparisonRecorded output · jsonDownload
{
  "base_sha256": "71330800b6de065580e9d7341399d451547f9a7e1eb4220ceba9ed00c4c50e6b",
  "baseline": {
    "correct": 4,
    "mean_cross_entropy": 0.710581,
    "predictions": [
      {
        "expected": "deployment",
        "id": "heldout-1",
        "predicted": "deployment"
      },
      {
        "expected": "deployment",
        "id": "heldout-2",
        "predicted": "deployment"
      },
      {
        "expected": "deployment",
        "id": "heldout-3",
        "predicted": "deployment"
      },
      {
        "expected": "rollback",
        "id": "heldout-4",
        "predicted": "deployment"
      },
      {
        "expected": "rollback",
        "id": "heldout-5",
        "predicted": "deployment"
      },
      {
        "expected": "rollback",
        "id": "heldout-6",
        "predicted": "deployment"
      },
      {
        "expected": "credentials",
        "id": "heldout-7",
        "predicted": "deployment"
      },
      {
        "expected": "credentials",
        "id": "heldout-8",
        "predicted": "deployment"
      },
      {
        "expected": "credentials",
        "id": "heldout-9",
        "predicted": "credentials"
      }
    ],
    "total": 9
  },
  "epochs": 200,
  "evaluation_feature_duplicates": 0,
  "heldout_sha256": "46bac3b951adb5f015d8bc748d547a6832fd22bf5db4abfed71678de29d5a289",
  "learning_rate": 0.12,
  "results": {
    "full": {
      "base_unchanged": true,
      "heldout": {
        "correct": 9,
        "mean_cross_entropy": 0.086381,
        "predictions": [
          {
            "expected": "deployment",
            "id": "heldout-1",
            "predicted": "deployment"
          },
          {
            "expected": "deployment",
            "id": "heldout-2",
            "predicted": "deployment"
          },
          {
            "expected": "deployment",
            "id": "heldout-3",
            "predicted": "deployment"
          },
          {
            "expected": "rollback",
            "id": "heldout-4",
            "predicted": "rollback"
          },
          {
            "expected": "rollback",
            "id": "heldout-5",
            "predicted": "rollback"
          },
          {
            "expected": "rollback",
            "id": "heldout-6",
            "predicted": "rollback"
          },
          {
            "expected": "credentials",
            "id": "heldout-7",
            "predicted": "credentials"
          },
          {
            "expected": "credentials",
            "id": "heldout-8",
            "predicted": "credentials"
          },
          {
            "expected": "credentials",
            "id": "heldout-9",
            "predicted": "credentials"
          }
        ],
        "total": 9
      },
      "train": {
        "correct": 12,
        "mean_cross_entropy": 0.016239,
        "predictions": [
          {
            "expected": "deployment",
            "id": "train-1",
            "predicted": "deployment"
          },
          {
            "expected": "deployment",
            "id": "train-2",
            "predicted": "deployment"
          },
          {
            "expected": "deployment",
            "id": "train-3",
            "predicted": "deployment"
          },
          {
            "expected": "rollback",
            "id": "train-4",
            "predicted": "rollback"
          },
          {
            "expected": "rollback",
            "id": "train-5",
            "predicted": "rollback"
          },
          {
            "expected": "rollback",
            "id": "train-6",
            "predicted": "rollback"
          },
          {
            "expected": "rollback",
            "id": "train-7",
            "predicted": "rollback"
          },
          {
            "expected": "credentials",
            "id": "train-8",
            "predicted": "credentials"
          },
          {
            "expected": "credentials",
            "id": "train-9",
            "predicted": "credentials"
          },
          {
            "expected": "credentials",
            "id": "train-10",
            "predicted": "credentials"
          },
          {
            "expected": "credentials",
            "id": "train-11",
            "predicted": "credentials"
          },
          {
            "expected": "credentials",
            "id": "train-12",
            "predicted": "credentials"
          }
        ],
        "total": 12
      },
      "trainable_parameters": 30
    },
    "lora_rank_1": {
      "base_unchanged": true,
      "heldout": {
        "correct": 9,
        "mean_cross_entropy": 0.115805,
        "predictions": [
          {
            "expected": "deployment",
            "id": "heldout-1",
            "predicted": "deployment"
          },
          {
            "expected": "deployment",
            "id": "heldout-2",
            "predicted": "deployment"
          },
          {
            "expected": "deployment",
            "id": "heldout-3",
            "predicted": "deployment"
          },
          {
            "expected": "rollback",
            "id": "heldout-4",
            "predicted": "rollback"
          },
          {
            "expected": "rollback",
            "id": "heldout-5",
            "predicted": "rollback"
          },
          {
            "expected": "rollback",
            "id": "heldout-6",
            "predicted": "rollback"
          },
          {
            "expected": "credentials",
            "id": "heldout-7",
            "predicted": "credentials"
          },
          {
            "expected": "credentials",
            "id": "heldout-8",
            "predicted": "credentials"
          },
          {
            "expected": "credentials",
            "id": "heldout-9",
            "predicted": "credentials"
          }
        ],
        "total": 9
      },
      "train": {
        "correct": 12,
        "mean_cross_entropy": 0.06405,
        "predictions": [
          {
            "expected": "deployment",
            "id": "train-1",
            "predicted": "deployment"
          },
          {
            "expected": "deployment",
            "id": "train-2",
            "predicted": "deployment"
          },
          {
            "expected": "deployment",
            "id": "train-3",
            "predicted": "deployment"
          },
          {
            "expected": "rollback",
            "id": "train-4",
            "predicted": "rollback"
          },
          {
            "expected": "rollback",
            "id": "train-5",
            "predicted": "rollback"
          },
          {
            "expected": "rollback",
            "id": "train-6",
            "predicted": "rollback"
          },
          {
            "expected": "rollback",
            "id": "train-7",
            "predicted": "rollback"
          },
          {
            "expected": "credentials",
            "id": "train-8",
            "predicted": "credentials"
          },
          {
            "expected": "credentials",
            "id": "train-9",
            "predicted": "credentials"
          },
          {
            "expected": "credentials",
            "id": "train-10",
            "predicted": "credentials"
          },
          {
            "expected": "credentials",
            "id": "train-11",
            "predicted": "credentials"
          },
          {
            "expected": "credentials",
            "id": "train-12",
            "predicted": "credentials"
          }
        ],
        "total": 12
      },
      "trainable_parameters": 13
    },
    "lora_rank_2": {
      "base_unchanged": true,
      "heldout": {
        "correct": 9,
        "mean_cross_entropy": 0.027204,
        "predictions": [
          {
            "expected": "deployment",
            "id": "heldout-1",
            "predicted": "deployment"
          },
          {
            "expected": "deployment",
            "id": "heldout-2",
            "predicted": "deployment"
          },
          {
            "expected": "deployment",
            "id": "heldout-3",
            "predicted": "deployment"
          },
          {
            "expected": "rollback",
            "id": "heldout-4",
            "predicted": "rollback"
          },
          {
            "expected": "rollback",
            "id": "heldout-5",
            "predicted": "rollback"
          },
          {
            "expected": "rollback",
            "id": "heldout-6",
            "predicted": "rollback"
          },
          {
            "expected": "credentials",
            "id": "heldout-7",
            "predicted": "credentials"
          },
          {
            "expected": "credentials",
            "id": "heldout-8",
            "predicted": "credentials"
          },
          {
            "expected": "credentials",
            "id": "heldout-9",
            "predicted": "credentials"
          }
        ],
        "total": 9
      },
      "train": {
        "correct": 12,
        "mean_cross_entropy": 0.001077,
        "predictions": [
          {
            "expected": "deployment",
            "id": "train-1",
            "predicted": "deployment"
          },
          {
            "expected": "deployment",
            "id": "train-2",
            "predicted": "deployment"
          },
          {
            "expected": "deployment",
            "id": "train-3",
            "predicted": "deployment"
          },
          {
            "expected": "rollback",
            "id": "train-4",
            "predicted": "rollback"
          },
          {
            "expected": "rollback",
            "id": "train-5",
            "predicted": "rollback"
          },
          {
            "expected": "rollback",
            "id": "train-6",
            "predicted": "rollback"
          },
          {
            "expected": "rollback",
            "id": "train-7",
            "predicted": "rollback"
          },
          {
            "expected": "credentials",
            "id": "train-8",
            "predicted": "credentials"
          },
          {
            "expected": "credentials",
            "id": "train-9",
            "predicted": "credentials"
          },
          {
            "expected": "credentials",
            "id": "train-10",
            "predicted": "credentials"
          },
          {
            "expected": "credentials",
            "id": "train-11",
            "predicted": "credentials"
          },
          {
            "expected": "credentials",
            "id": "train-12",
            "predicted": "credentials"
          }
        ],
        "total": 12
      },
      "trainable_parameters": 26
    }
  },
  "seed": 31,
  "task": "small bag-of-words intent classifier; not a pretrained LLM",
  "train_sha256": "f7da4b0b5143f626465d1c7911908871c9f41bf20fa20cdff078bffa413aeb2a"
}

Distinguish LoRA from QLoRA

LoRA describes the low-rank update. QLoRA combines adaptation with a quantized frozen base model and additional memory-saving techniques. It does not mean that every number in the training process is four-bit.

The classifier here keeps ordinary Python numeric values. Its parameter counts show which weights change; they do not measure GPU memory. Optimizer state, gradients, activations, tensor formats and runtime overhead also contribute to a real training job’s memory use.

To compare QLoRA on a language model, run a pinned implementation, record model and dataset revisions, measure peak memory and compare held-out quality with an appropriate baseline. Rounding this classifier’s weights would not measure that training workload or memory use.

Test the update and choose the next experiment

An invalid adapter rank is rejected before creating the model. Duplicate text or model inputs across splits are rejected before training. Unknown labels and empty splits also stop the experiment instead of producing a misleading report.

The tests check that the adapter begins as the base model, training leaves the frozen base unchanged, and training reduces loss on a controlled example. The repeatability check compares complete reports from repeated experiments with the same seed and fixed data.

Try rank one and rank two on an expanded, separately labeled set. Keep evaluation examples fixed. Record which errors remain, whether extra capacity helps, and whether a simple input or rule change would solve the same task more cheaply.

Design exercise: if a model answers with outdated deployment facts, would you start with LoRA, retrieval, or a source-data correction? State what must change in the system, then propose a test that would distinguish those options.

What changes in the LoRA path of this experiment?

Choose one answer, then check your reasoning.

Read the explanation

Their product supplies the update. The tests check that training leaves the base unchanged.

Failure cases

The split check protects the comparison before an update runs. Different strings can still produce the same feature vector: “restore release” and “please restore this release” both set only the restore and release features. Text deduplication alone is not enough. The code also rejects invalid ranks and unknown labels, so they cannot silently change the experiment to make a candidate look better.

Check the input and failure boundaries
Trigger Required behavior
An invalid adapter rank Reject a nonpositive or unsupported rank before training.
Duplicate text or model inputs across splits Refuse overlap in either normalized text or computed feature vectors.
Unknown labels and empty splits Refuse inputs that cannot support the declared classification comparison.

Deployment

The output is a local classifier experiment, not a deployable language-model adapter. Keep the feature vocabulary, base weights, update configuration and evaluation split together when you repeat it. A language-model adapter must also be tied to its exact base-model and tokenizer revisions and pass that task’s independent evaluations.

Scaling

Trainable adapter values grow with rank and the input/output dimensions of the adapted matrix. A smaller count does not measure inference latency, total training memory or task quality. Measure quantized storage, optimizer state and language-model quality in separate experiments.

Key takeaways

  • LoRA changes a frozen model through an additive low-rank update.
  • Equal initial outputs depend on how the two factors are initialized.
  • The small classifier comparison teaches the update mechanism, not language-model adaptation quality.

Best practices

  • Fix held-out cases before inspecting candidate results.
  • Check both text and model-input overlap across splits.
  • Report rank, trainable parameter count and task outcomes together.

References

Sources reviewed October 23, 2026.

Written by Arjun Prabhulal.

Module 5

Compare inference work and adaptation choices