Run inference with a local LLM

You run the docs assistant with and without rollback.md. In one recorded run, the first answer invents a command meaning; the answer with the page describes the documented rollback.

Throughout this course, self-contained labs teach parts of a docs assistant for docuctl, a fictional deployment CLI described by the bundled manual. It answers questions such as “how do I roll back production?” from the team’s docs pages. Here you run a local LLM from Python, ask it about docuctl rollback, and watch it answer two different ways depending on what you send.

When you need this. The first time someone asks your assistant about an internal tool, the model answers fluently, whether or not it knows anything about that tool. Knowing what inference can and cannot do tells you where the facts must come from: the request, not the model.

Recorded example. The saved inference responses used the earlier shipctl manual; run the current docuctl prompts to obtain new responses.

1. Inspect the model you will run

A model is a file of learned numbers, the weights, plus the code that runs them. Training adjusted those weights before this lab; inference uses them without updating them. Inference is running the model on new input to produce output, and it is what your application does on every request.

Download the what-is-inference lab and extract its code, docs, tests and README together.

unzip what-is-inference.zip
cd what-is-inference
python3.12 -m venv .venv
.venv/bin/python -m pip install -r requirements-lock.txt

# Offline tests: no model calls
.venv/bin/python -m pytest -q

# Run after reading the implementation below
ollama pull gemma4:e2b
.venv/bin/python inference.py

The tests replace Ollama; the live script inspects and calls the local Gemma model.

Now we read what the runtime knows about the weight file. describe() calls Ollama’s show endpoint and keeps four fields:

"""Inspect a local model, then run inference with and without the docuctl docs. Run: python inference.py"""
from pathlib import Path

import ollama

MODEL = "gemma4:e2b"
QUESTION = "What does `docuctl rollback --to r41` do? Answer in two sentences."
DOCS = Path(__file__).parent / "docs"


def describe(client: ollama.Client, model: str = MODEL) -> dict:
    """The facts about the weight file that matter to an application."""
    info = client.show(model)
    return {
        "family": info.details.family,
        "parameters": info.details.parameter_size,
        "quantization": info.details.quantization_level,
        "context_length": info.modelinfo[f"{info.details.family}.context_length"],
    }


def ask(client: ollama.Client, question: str, page: str | None = None, model: str = MODEL) -> str:
    """One inference request. The page, if any, travels in the prompt; the weights never change."""
    prompt = question if page is None else f"Answer from this page only.\n\n{page}\n\nQuestion: {question}"
    reply = client.chat(model=model, messages=[{"role": "user", "content": prompt}],
                        think=False, options={"temperature": 0, "seed": 7})
    return reply.message.content.strip()


def main() -> None:
    client = ollama.Client()
    for key, value in describe(client).items():
        print(f"{key:<15} {value}")
    print("\n--- without the docs")
    print(ask(client, QUESTION))
    print("\n--- with docs/rollback.md in the prompt")
    print(ask(client, QUESTION, (DOCS / "rollback.md").read_text()))


if __name__ == "__main__":
    main()

Each field drives a decision later in the course:

• Parameters (4.6B) is the number of weights. It contributes to weight memory; quantization, architecture and serving settings also affect memory and speed.
• Quantization (Q4_K_M) names the lower-precision format used to store weights. Using fewer bits reduces weight storage. Speed and answer quality depend on the quantization method and runtime.
• Context length (131072 tokens) is the model’s reported token capacity. The runtime can use a smaller window; the context lesson shows what happens near the limit you actually configure.

The weights are not on this machine

gemma4:e4b is a real model tag, but it was not installed for the recorded check, so Ollama answers 404 model 'gemma4:e4b' not found. Local inference can only run weights that are on disk. Pin the exact model tag in your configuration, pull it during setup, and check it at startup, with describe() for example, rather than on a user’s first request.

2. Ask without the docs

ask() sends one chat request with temperature 0 and a fixed seed to reduce sampling variation. These settings do not promise identical output across runtime or model changes. think=False turns off Gemma’s reasoning output, so the reply is only the answer. Without a page, the prompt is just the question.

The docuctl manual is fictional and supplied with this course; the model should not be assumed to know its commands. In the original shipctl recording, it answers confidently but incorrectly: rollback --to does not “revert the Ship configuration”. Nothing failed. There was no error and no warning. A language model produces a likely continuation of its input, and a plausible description of a command called rollback is likely whether or not the command exists.

3. Put the page in the prompt

The second call sends the same question with the text of rollback.md in front of it:

# Roll back a release

`docuctl rollback` moves an environment back to an earlier release. It does not rebuild anything; it points traffic at the release you name.

    docuctl rollback --env production --to r41

Without `--to`, docuctl rolls back to the previous release. A rollback is itself recorded as a new release, so `docuctl releases` shows it.

Database migrations are not rolled back. If the release you leave ran a migration, write a new migration that reverses it.

docuctl 1.x used `docuctl release revert <release>`. That command was removed in docuctl 2.0.

Here is one recorded run. With the same weights and question, this run returned an invented answer without the docs and an answer matching rollback.md when the page was included:

family          gemma4
parameters      4.6B
quantization    Q4_K_M
context_length  131072

--- without the docs
The command `shipctl rollback --to r41` is used to revert the Ship configuration to a specific version, denoted by the tag or reference `r41`. This action essentially restores the Ship settings to the state they were in when version `r41` was active.

--- with docs/rollback.md in the prompt
`shipctl rollback --to r41` moves an environment back to the release named r41. It does not rebuild anything; it simply points traffic at that specified release.

Now the answer matches the page: the command moves an environment back to release r41 and does not rebuild. The weights are the same file as in step 2. Only the input changed, and the model used the page because it was part of the text it was continuing.

Here are two failures from the recorded run of failures.py:

"""Two ways inference goes wrong in practice. Run: python failures.py"""
import ollama

from inference import QUESTION, ask

client = ollama.Client()

# 1. The weights know nothing about docuctl, so the model answers anyway, from patterns.
print("1. no docs, three seeds:")
for seed in (1, 2, 3):
    reply = client.chat(model="gemma4:e2b", messages=[{"role": "user", "content": QUESTION}],
                        think=False, options={"temperature": 0.8, "seed": seed})
    print("  ", reply.message.content.strip().split(". ")[0].rstrip(".") + ".")

# 2. Inference needs the weight file on this machine; a tag that was never pulled fails.
print("2. a model that is not pulled:")
try:
    ask(client, QUESTION, model="gemma4:e4b")
except ollama.ResponseError as error:
    print("   ResponseError", error.status_code, error.error)

1. no docs, three seeds:
   The command `shipctl rollback --to r41` is used to revert the current state of a Ship configuration to the version or state designated as "r41".
   The command `shipctl rollback --to r41` is used to revert the current state of a Ship configuration to a previously saved version identified as "r41".
   The command `shipctl rollback --to r41` is used to revert the state of a Ship configuration to a previously saved version labeled "r41." This action effectively undoes recent changes to the Ship configuration, restoring it to the state it was in when version r41 was recorded.
2. a model that is not pulled:
   ResponseError 404 model 'gemma4:e4b' not found

The model answers from patterns

With temperature 0.8 and three seeds, the wording changes but the invented meaning does not: every run describes a “Ship configuration” that does not exist. These three sampled replies repeated the same incorrect description; sampling was not a fix in this run. Include the source text, then check answers against it, as the eval lesson at the end of this course does.

This is the whole strategy of the docs assistant. Instead of hoping the model knows our tools, we fetch the right page and put it in the request. Later lessons find that page with embeddings, fetch it with a tool call and serve it over MCP, but the mechanism stays the one you just ran.

Keep request context separate from model weights

Training and inference run the same network; only training changes it. During training, the provider runs the model over a large body of text, measures how wrong each next-token prediction was, and adjusts trainable weights to reduce the training loss. That loop repeats over the whole training set and produces the weight file you pulled.

During inference the weights are read-only. The request’s text becomes tokens, the network runs once for every token it generates, and the output comes back. Nothing about the conversation is written into the model. That has three consequences to design around:

• Training knowledge is incomplete. Do not assume the model has current release notes, private data or the facts required for this request.
• This inference call does not update weights. A correction can affect later answers when your application includes it in the next request.
• Updating weights requires training; supplying facts does not. For this docs assistant, include current pages in the request and check cited text against them. Fine-tuning can change behavior but does not itself check citations.

Try it: ask about another docs page

Change QUESTION to “How long is a CI deploy token valid?” and run inference.py twice: once as it is, and once with docs/tokens.md as the page. Without the page, note whether the model says it does not know or invents a number. With the page, the answer should say 90 days. Then run the offline tests, which replace Ollama with a fake client, with .venv/bin/python -m pytest -q.

Count tokens with the model’s tokenizer shows what the model actually receives when you send it text, and why the page you added costs more tokens than it has words.

References

Checked October 21, 2026.

• Ollama API reference
• Ollama Python library
• Gemma models overview