Send chat messages within the context window

You build and resend a two-turn conversation, then compare two context limits. In the recorded small-window call, the API returns a reply despite processing far fewer prompt tokens and failing to answer the build-log question.

A chat model takes a list of messages, each with a role: system for the application’s instructions, user for what the person typed, and assistant for the model’s earlier replies. Here you give the docs assistant its four docuctl pages as a system message, hold a two-turn conversation, and then push a request past the context window to see what the runtime does.

When you need this. A user pastes a long build log into the docs assistant and asks a follow-up question, and the reply ignores both the log and the service name they gave earlier. Both bugs come from what the request carried, and the API reports success either way.

Recorded example. The context-window measurements were recorded with the earlier shipctl pages; changing them to docuctl changes the request being measured.

1. Build the messages

Download the messages-and-context lab and extract its code, docs, tests and README together.

unzip messages-and-context.zip
cd messages-and-context
python3.12 -m venv .venv
.venv/bin/python -m pip install -r requirements-lock.txt

# Offline tests: no model calls
.venv/bin/python -m pytest -q

# Run after reading the implementation below
ollama pull gemma4:e2b
.venv/bin/python chat_context.py

The tests inspect request construction with a replaced chat client; the live script sends its conversations to Ollama. SYSTEM holds every docs page, so the model can answer from them; chat() sends one request and returns the reply with prompt_eval_count, the number of tokens the runtime processed:

"""Send chat messages to a local model and watch what fits in the context window. Run: python chat_context.py"""
from pathlib import Path

import ollama

MODEL = "gemma4:e2b"
DOCS = "\n\n".join(page.read_text() for page in sorted((Path(__file__).parent / "docs").glob("*.md")))
SYSTEM = {"role": "system", "content": f"You answer questions about docuctl. Use only these docs:\n\n{DOCS}"}
BUILD_LOG = "\n".join(f"compiled module m{i} in 0.{i % 10}s" for i in range(400))


def chat(client: ollama.Client, messages: list[dict], num_ctx: int = 16384) -> tuple[str, int]:
    """Submit this message history; the runtime may truncate an oversized request."""
    reply = client.chat(model=MODEL, messages=messages, think=False,
                        options={"temperature": 0, "num_ctx": num_ctx, "num_predict": 80})
    return reply.message.content.strip(), reply.prompt_eval_count


def main() -> None:
    client = ollama.Client()
    history = [SYSTEM, {"role": "user", "content": "Our service is named search-api. Which environment variable holds its CI deploy token?"}]
    answer, used = chat(client, history)
    print(f"turn 1 ({used} prompt tokens): {answer}")

    history += [{"role": "assistant", "content": answer},
                {"role": "user", "content": "Write the command that creates that token for our service, scoped to staging. Command only."}]
    answer, used = chat(client, history)
    print(f"turn 2 ({used} prompt tokens): {answer}")

    answer, used = chat(client, [SYSTEM, history[-1]])
    print(f"turn 2 without the history ({used} prompt tokens): {answer}")

    long = [SYSTEM, {"role": "user", "content": f"My build log:\n{BUILD_LOG}"},
            {"role": "assistant", "content": "Noted."},
            {"role": "user", "content": "Which module did my build log compile last? Reply with the module name only."}]
    for num_ctx in (16384, 4096):
        answer, used = chat(client, long, num_ctx)
        print(f"build log, num_ctx={num_ctx} ({used} prompt tokens): {answer}")


if __name__ == "__main__":
    main()

num_ctx sets the context window for the request: the most tokens the runtime keeps for this input plus the reply. We set it on every call instead of relying on the server default, which depends on the machine.

2. Hold a conversation

Here is the recorded run:

turn 1 (453 prompt tokens): The documentation does not specify an environment variable for the CI deploy token. It states that CI jobs authenticate with a deploy token, and you should store it in your CI secret store as `SHIPCTL_TOKEN`.
turn 2 (523 prompt tokens): `shipctl tokens create --name ci-search-api --scope deploy:staging`
turn 2 without the history (454 prompt tokens): shipctl tokens create --name ci-docs-api --scope deploy:staging
build log, num_ctx=16384 (5559 prompt tokens): m399
build log, num_ctx=4096 (460 prompt tokens): I do not have access to your build logs, so I cannot tell you which module was compiled last.

In turn 1 the user names the service, search-api. Turn 2 asks for “the command that creates that token for our service”. The client appends the assistant’s first reply and the new question to history and sends all of it, so the model writes ci-search-api.

The chat API is stateless: the model has no memory between requests, and the server keeps no conversation for you. The only reason turn 2 knew the service name is that the client sent turn 1 again. When the same question goes out without the history, the model has no service name, so it copies the example name from the docs page, ci-docs-api, and nothing reports a problem.

The follow-up loses the earlier turns

Turn 2 without the history returned ci-docs-api instead of ci-search-api. The model filled the gap with the closest text it had, the example in the docs. Keep the conversation in your application, resend it on every request, and when it grows too long, shorten it on purpose: summarize or drop the oldest turns yourself, so that you know what the model saw.

3. Overflow the context window

The last request adds a 400-line build log as a user message and asks which module compiled last. With num_ctx 16384 the runtime processed 5559 prompt tokens and the answer is m399, the last line of the log.

With num_ctx 4096 the same messages do not fit. The request still succeeded, but the runtime processed only 460 prompt tokens, about the size of the system message plus the last question. The lower prompt count and the reply saying it has no build log are consistent with truncation. This recording does not expose exactly which tokens the runtime retained.

Messages are dropped without an error

At num_ctx 4096, prompt_eval_count fell from 5559 to 460, and the reply said it had no build log. Count the formatted request with the model’s tokenizer before sending, set num_ctx above that count plus the output budget, and after each call compare prompt_eval_count with your count. A large gap means the model did not see what you sent.

Keep conversation history in the application

The runtime renders the message list through the model’s chat template into one sequence of tokens, as the tokenizer lesson showed. That sequence, plus every token of the reply, must fit in the window, because the model attends over all of them at every generation step.

A larger window costs memory: the runtime keeps the KV cache, the per-token state from the previous lesson, for every position in the window. Choose num_ctx for the requests you expect instead of choosing the largest value for every request.

When a request is too long, a runtime can refuse it or shorten it. In this run Ollama processed fewer prompt tokens and returned no error; its answer said it had no build log. The recording does not identify the exact retained tokens. Whatever your runtime does, the safe design is the same: count before you send, and check what was processed.

Try it: find the smallest window that keeps the build log

Run the build-log request with num_ctx 6144 and 8192, and record prompt_eval_count and the answer for each. Then add a check to chat() that raises an error when prompt_eval_count is far below a count you pass in. The offline tests use a fake client and the recorded run: .venv/bin/python -m pytest -q.

Control sampling, output limits and streaming covers the other side of the request: how much the model may generate, how random it is, and how the reply arrives.

References

Checked October 21, 2026.

• Ollama API: chat
• Ollama: context length
• Hugging Face: chat templates