A language model builds a response one token at a time. At each step it scores possible next tokens from the text so far, chooses one and appends it. This lesson makes that loop visible and shows why a high-probability continuation can still be the wrong command.
A non-streaming chat call returns a whole reply, which hides how it was made. Underneath, the model produces one token, the application appends it, and the model runs again on the longer text. Here you drive that loop yourself, one call per token, and record what the model considered at every step.
When you need this. When the docs assistant prints a command with a flag that does not exist, you need to know whether to blame the prompt, the settings or the model. Watching the loop pick tokens shows that every choice depends on the text before it, which helps distinguish a missing fact from an output-limit problem.
Recorded example. The candidate probabilities below belong to the original shipctl prompt, not to a new docuctl generation.
1. Score one step
Most chat models are autoregressive: each new token is predicted from all the tokens before it, including the ones the model just produced. One step is one forward pass: the model reads the context and gives a score to every token in its vocabulary. Ollama can return the top few of those as log probabilities; math.exp turns them back into probabilities between 0 and 1.
Download the next-token-prediction lab and extract its code, docs, tests and README together.
unzip next-token-prediction.zip
cd next-token-prediction
python3.12 -m venv .venv
.venv/bin/python -m pip install -r requirements-lock.txt
# Offline tests: no model calls
.venv/bin/python -m pytest -q
# Run after reading the implementation below
ollama pull gemma4:e2b
.venv/bin/python generation_loop.pyThe offline suite checks the generation loop with fixed token scores. The live command asks Ollama for new scores.
"""Run the generation loop one token at a time and record the model's top candidates.
Run: python generation_loop.py"""
import json
import math
from pathlib import Path
import ollama
MODEL = "gemma4:e2b"
HERE = Path(__file__).parent
PROMPT = "To roll back production to release r41, run:\n\n docuctl"
def step(client: ollama.Client, text: str, top: int = 3) -> dict:
"""One forward pass: score the next token, pick the most likely one (temperature 0)."""
reply = client.generate(model=MODEL, prompt=text, raw=True, logprobs=True, top_logprobs=top,
options={"temperature": 0, "num_predict": 1})
if not reply.logprobs: # the model chose its end-of-sequence token, so nothing new was generated
return {"chosen": "", "candidates": [], "done_reason": reply.done_reason}
scored = reply.logprobs[0]
return {"chosen": scored.token,
"candidates": [(c.token, round(math.exp(c.logprob), 3)) for c in scored.top_logprobs]}
def generate(client: ollama.Client, context: str, steps: int = 8, stop: str = "\n") -> list[dict]:
"""Append each chosen token to the context and score again, until the stop text or the step limit."""
trace = []
for _ in range(steps):
result = step(client, context)
trace.append(result)
if not result["chosen"] or stop in result["chosen"]:
break
context += result["chosen"]
return trace
def show(label: str, trace: list[dict]) -> None:
print(f"--- {label}")
for i, row in enumerate(trace, 1):
others = " ".join(f"{token!r} {p:.3f}" for token, p in row["candidates"]) or "(end of sequence)"
print(f"{i}. chose {row['chosen']!r:<12} candidates: {others}")
print("text:", "docuctl" + "".join(row["chosen"] for row in trace).rstrip())
def main() -> None:
client = ollama.Client()
page = (HERE / "docs/rollback.md").read_text()
runs = {"without docs": generate(client, PROMPT),
"with rollback.md": generate(client, f"{page}\n\n{PROMPT}")}
for label, trace in runs.items():
show(label, trace)
output = HERE / "runs/local-docuctl"
output.mkdir(parents=True, exist_ok=True)
(output / "trace.json").write_text(json.dumps(runs, indent=1) + "\n")
if __name__ == "__main__":
main()step() sends the context as a raw prompt, so no chat template is added, and asks for exactly one token (num_predict 1) at temperature 0, selecting the highest-scoring candidate in this run. top_logprobs 3 returns the three best candidates with their scores.
2. Run the loop
generate() is the loop that every chat API runs for you: score, pick, append the chosen token’s text to the context, repeat. It stops when the model picks its end-of-sequence token, when a newline ends the command, or after steps iterations. main() runs it twice on the same prompt, once alone and once with rollback.md in front of it. A fresh run writes runs/local-docuctl/trace.json and leaves the original trace unchanged:
--- without docs
1. chose ' rollback' candidates: ' rollback' 0.977 ' roll' 0.011 ' release' 0.004
2. chose ' r' candidates: ' r' 0.979 ' --' 0.009 ' release' 0.006
3. chose '4' candidates: '4' 1.000 ' r' 0.000 '3' 0.000
4. chose '1' candidates: '1' 1.000 '0' 0.000 '2' 0.000
5. chose '' candidates: (end of sequence)
text: shipctl rollback r41
--- with rollback.md
1. chose ' rollback' candidates: ' rollback' 1.000 ' rollout' 0.000 ' roll' 0.000
2. chose ' --' candidates: ' --' 1.000 ' -' 0.000 ' production' 0.000
3. chose 'env' candidates: 'env' 1.000 'environment' 0.000 ' env' 0.000
4. chose ' production' candidates: ' production' 1.000 ' prod' 0.000 '=' 0.000
5. chose ' --' candidates: ' --' 1.000 ' \\' 0.000 ' -' 0.000
6. chose 'to' candidates: 'to' 1.000 'from' 0.000 '--' 0.000
7. chose ' r' candidates: ' r' 1.000 ' release' 0.000 'r' 0.000
8. chose '4' candidates: '4' 1.000 '1' 0.000 '3' 0.000
text: shipctl rollback --env production --to r4The step limit cuts the answer
The run with the docs was correct until step 8, then stopped at --to r4. A limit on generated tokens stops the loop wherever it is, mid-word or mid-number, and the text still looks like an answer. Chat APIs report this case: Ollama’s done_reason is length instead of stop. Control sampling, output limits and streaming reads that field before accepting a reply.
3. Read the two traces
Without the docs, the model picks ␠rollback, then ␠r with probability 0.979, then 4 and 1, and ends. The result, shipctl rollback r41, is fluent, nearly certain and wrong: the real command needs --env and --to. The probability measured how well ␠r continued this text, not whether the command works.
With rollback.md in the context, every step changes. The same position now picks ␠--, then env, ␠production, ␠--, to, with probabilities that round to 1.000, because the page contains this exact command. Then the loop hits its 8-step limit in the middle of the release name and returns r4.
A confident token can still be wrong
Without the docs, the second token had probability 0.979. Nothing in the loop checks facts: the probability ranks continuations of the context, and a context without the docs makes a guess the best continuation. Do not show token probabilities to users as confidence. Put the source in the context, and check the output against it, as the eval lesson does.
Append each chosen token before scoring again
At each step the model turns the whole context into one score per vocabulary token, called a logit. A softmax turns the logits into probabilities that sum to 1, and a selection rule picks one. Temperature 0 picks the top token, which is called greedy decoding; Control text generation covers sampling, which can pick tokens at random from the resulting distribution.
Our loop re-sends the growing text every step, which is simple but repeats work. Real runtimes keep the computed state for tokens they have already processed, the KV cache, and only compute the new token’s step. Our loop also appends text, and the runtime tokenizes that text again, which can split differently from the token the model chose. That is why production code appends token IDs, and why runtimes run this loop for you.
Try it: watch the docs change a token choice
Change PROMPT to end with docuctl tokens and run the loop with docs/tokens.md in the context instead of rollback.md. Compare the candidates at the first step with and without the page. Then raise steps to 12 and inspect whether the rollback command completes. The offline tests replay the recorded trace: .venv/bin/python -m pytest -q.
Read the explanation
Each step scores every vocabulary token from the context and appends one. High probability means a likely continuation; correctness has to come from the context you supply and from checks after generation.
Send chat messages within the context window moves from a raw prompt to chat messages, and shows what happens when they no longer fit.
References
Checked October 21, 2026.