Control sampling, output limits and streaming

You vary sampling, cap generation and inspect a streamed reply. The recorded 40-token call ends with done_reason=length; the longer stream ends with stop and records its first-chunk time.

Trace next-token generation picked the top token at every step. Real requests set a few options that change that loop: how random each pick is, how many tokens the model may generate, and whether the reply arrives all at once or as it is produced. Here you set each one for the docs assistant and record what changes.

When you need this. The docs assistant writes commit messages in one job and explanations in a chat window. The first needs repeatable output for tests, the second needs text on screen within a second, and both need to distinguish an incomplete reply from a finished one.

1. Repeat or vary the output

Download the generation-controls lab and extract its code, docs, tests and README together.

unzip generation-controls.zip
cd generation-controls
python3.12 -m venv .venv
.venv/bin/python -m pip install -r requirements-lock.txt

# Offline tests: no model calls
.venv/bin/python -m pytest -q

# Run after reading the implementation below
ollama pull gemma4:e2b
.venv/bin/python controls.py

The offline checks cover request options and stream handling. New sampling and streaming observations need the local runtime. complete() passes any options straight to Ollama and returns the reply with its done_reason:

"""Sampling, output limits and streaming against a local model. Run: python controls.py"""
import time

import httpx
import ollama

MODEL = "gemma4:e2b"
TASK = "Write a one-line commit message for a change that adds a --dry-run flag to `docuctl rollback`."
EXPLAIN = "Explain in about 150 words what `docuctl rollback --env production --to r41` does and when to use it."


def complete(client: ollama.Client, prompt: str, **options) -> tuple[str, str]:
    reply = client.chat(model=MODEL, messages=[{"role": "user", "content": prompt}], think=False, options=options)
    return reply.message.content.strip(), reply.done_reason


def stream(client: ollama.Client, prompt: str) -> dict:
    """Count content chunks and time the first one against the whole reply."""
    start, first, chunks = time.perf_counter(), None, 0
    for chunk in client.chat(model=MODEL, messages=[{"role": "user", "content": prompt}], think=False,
                             stream=True, options={"temperature": 0, "num_predict": 400}):
        if chunk.message.content:
            first = time.perf_counter() - start if first is None else first
            chunks += 1
        if chunk.done:
            return {"chunks": chunks, "first_chunk_s": None if first is None else round(first, 2),
                    "total_s": round(time.perf_counter() - start, 2), "done_reason": chunk.done_reason}

    raise RuntimeError("stream ended without a final status; reply is incomplete")


def main() -> None:
    client = ollama.Client()
    print("temperature 0, three runs:")
    for _ in range(3):
        print("  ", complete(client, TASK, temperature=0)[0])
    print("temperature 1.0, seeds 1-3:")
    for seed in (1, 2, 3):
        print("  ", complete(client, TASK, temperature=1.0, seed=seed)[0])
    print("temperature 1.0, seed 1 again:\n  ", complete(client, TASK, temperature=1.0, seed=1)[0])

    text, reason = complete(client, EXPLAIN, temperature=0, num_predict=40)
    print(f"\nnum_predict=40 -> done_reason={reason!r}\n   ...{text[-60:]!r}")

    print("\nstreaming:", stream(client, EXPLAIN))

    try:  # a client timeout shorter than the generation
        complete(ollama.Client(timeout=0.5), EXPLAIN, temperature=0, num_predict=200)
    except httpx.ReadTimeout as error:
        print("timeout=0.5s ->", type(error).__name__, error)


if __name__ == "__main__":
    main()

Temperature scales the scores before a token is drawn. At 0 the runtime uses greedy selection. That reduces sampling variation; runtime changes and numerical differences can still change results. Higher values flatten the probabilities and let less likely tokens through. Seed controls the sampling random number generator. It helps reproduce a run under the same setup.

The saved sampling and timing results used the earlier shipctl prompts. Changing those prompts to docuctl requires a new run; it does not preserve the recorded text or timings.

temperature 0, three runs:
   Feat: Add --dry-run flag to `shipctl rollback`
   Feat: Add --dry-run flag to `shipctl rollback`
   Feat: Add --dry-run flag to `shipctl rollback`
temperature 1.0, seeds 1-3:
   Add `--dry-run` flag to `shipctl rollback`.
   Feat: Add --dry-run flag to shipctl rollback.
   Feat: Add --dry-run flag to `shipctl rollback`
temperature 1.0, seed 1 again:
   Add `--dry-run` flag to `shipctl rollback`.

num_predict=40 -> done_reason='length'
   ..."anaging deployments or configurations.\n\nHere's what it does:"

streaming: {'chunks': 232, 'first_chunk_s': 0.53, 'total_s': 7.68, 'done_reason': 'stop'}
timeout=0.5s -> ReadTimeout timed out

At temperature 0 the three commit messages are identical. At 1.0 they differ in small ways that matter to code: one drops the Feat: prefix, one drops the backticks. In this recording, seed 1 reproduced its first run. Use temperature 0 and a seed for jobs that tests compare; allow sampling where variety helps and nothing parses the text.

2. Cap the output

num_predict limits how many tokens the model may generate. The 150-word explanation needs more than 40, so generation stops at 40, in the middle of a sentence, and done_reason is length. When the model ends the reply itself, done_reason is stop.

The cap is a cost and latency control: you pay for and wait on every generated token. Set it from the longest reply you accept, and treat length as an unfinished reply.

A cut-off reply looks finished

The 40-token explanation ends with “Here’s what it does:” and nothing after it. Text alone does not tell you whether the model finished. Read done_reason on every reply: on length, raise the limit and retry, or mark the reply incomplete. When a stream ends, check done_reason on its final chunk before you store what you showed.

3. Stream the reply

With stream=True the client returns an iterator of chunks, each holding the next piece of text, and the last chunk carries done and done_reason. The recorded explanation arrived as 232 chunks: the first after 0.53 s, the last after 7.68 s. A chat window can show text after the first chunk instead of waiting for the whole reply.

Streaming changes delivery rather than specifying a different generation strategy. It changes when your code sees text, which also means your code sees text before it knows how the reply ends.

The client times out before the model finishes

ollama.Client(timeout=0.5) gave up on a 200-token explanation and raised httpx.ReadTimeout, so the application got no reply at all. Set the client timeout from measured generation time for your output cap, and for long replies stream instead, so that a slow reply still shows progress.

An unfinished stream has no final status

The offline stream tests also pass an empty completed stream and a stream that ends without a final status. stream() has no first-chunk time for the empty stream and raises RuntimeError when the final status is missing. These unit checks exercise stream(), the wrapper around Ollama’s chunk iterator; they do not run a model. Keep that status beside the accumulated text so an interrupted iterator is not reported as a completed answer.

Separate sampling from stream completion

Every option changes the generation loop from Trace next-token generation, not the model. Temperature and seed change the pick at each step. num_predict is the loop’s step limit. Streaming sends each new piece of text over the open HTTP connection as the loop produces it; Ollama sends one JSON object per line. A non-streaming call waits for the loop to finish and sends one object.

Time to first chunk can include model loading, queueing, prompt processing and initial generation. Total time also includes the remaining output. Measure both with the requests you actually send.

Try it: find the output cap for a full explanation

Run the explanation with num_predict 100, 200 and 400, and record done_reason and the word count of each reply. Pick the smallest cap that ends with stop. Then stream the same request and print the time of the first chunk for a short and a long prompt. The offline tests use a fake client and the recorded run: .venv/bin/python -m pytest -q.

Call a local model and a hosted API sends the same request to Gemma on your machine and to a hosted Gemini model, and compares what changes.

References

Checked October 21, 2026.

• Ollama: streaming
• Ollama API: chat, options and done_reason
• Ollama: Modelfile parameters (temperature, seed, num_predict)
• Hugging Face: generation strategies