Count tokens with the model's tokenizer

Turn document text into token IDs, then count the complete model request. A word count cannot tell you whether documentation, conversation history and the answer will fit into the model’s context window.

Before a text model runs, its tokenizer converts the request into integer token IDs. Context limits, output limits and many API usage charges are measured in tokens. Here you tokenize pieces of the docuctl docs with Gemma’s own tokenizer, then check your counts against what Ollama reports it processed.

When you need this. The docs assistant packs several pages, a conversation and a question into one request. To know whether that request fits, or what it costs, you need the count the model will see, and the commands, IDs and config blocks in developer docs are exactly where guesses go wrong.

Recorded example. Token IDs and runtime counts in the archived comparison describe the original shipctl text; the current docuctl counts are shown separately in step 2.

1. Tokenize the docs text

A tokenizer splits text into pieces from a fixed vocabulary and maps each piece to an integer ID. Common words often fit in one piece; unfamiliar strings may need several smaller pieces. The tokenizer is part of the model: its vocabulary was fixed when the model was trained, so you must count with the same one.

Download the tokenizer lab, then obtain its model-specific tokenizer file with the pinned command below.

unzip tokens-and-tokenizers.zip
cd tokens-and-tokenizers
python3.12 -m venv .venv
.venv/bin/python -m pip install -r requirements-lock.txt
mkdir -p gemma-tokenizer
curl --fail --location \
  https://huggingface.co/google/gemma-4-E2B-it/resolve/3e22461f65e89153144f8adb70e3b8c2cc9845a7/tokenizer.json \
  --output gemma-tokenizer/tokenizer.json
export TOKENIZER_PATH="$PWD/gemma-tokenizer"

# Offline tests: no model calls
.venv/bin/python -m pytest -q

# Run after reading the implementation below
.venv/bin/python count_tokens.py

Only the tokenizer file is downloaded, about 32 MB; model weights are not needed for counting. The pinned file was checked against the existing local tokenizer and reproduces the saved counts. Compare its SHA-256 with cc8d3a0ce36466ccc1278bf987df5f71db1719b9ca6b4118264f45cb627bfe0f. The offline tests use this file without a model request; compare_runtime.py additionally needs Ollama with both models named in its source.

"""Count tokens in docuctl docs text with Gemma's own tokenizer. Run: TOKENIZER_PATH=... python count_tokens.py"""
import os
import sys
from pathlib import Path

from tokenizers import Tokenizer

HERE = Path(__file__).parent
SAMPLES = {
    "prose": "docuctl rollback moves an environment back to an earlier release.",
    "command": "docuctl rollback --env production --to r41",
    "config": '[service]\nname = "docs-api"\nbuild = "make build"',
    "id": "ci-docs-api-2026-10-21T09:14:03Z",
    "page": (HERE / "docs/rollback.md").read_text(),
}


def load(folder: str | None = None) -> Tokenizer:
    path = Path(folder or os.environ.get("TOKENIZER_PATH", "gemma-tokenizer")) / "tokenizer.json"
    if not path.is_file():
        sys.exit(f"no tokenizer.json in {path.parent}; set TOKENIZER_PATH to the Gemma tokenizer folder")
    return Tokenizer.from_file(str(path))


def count(tokenizer: Tokenizer, text: str) -> int:
    """Tokens in the raw text, without the BOS token a runtime adds in front."""
    return len(tokenizer.encode(text, add_special_tokens=False).ids)


def pieces(tokenizer: Tokenizer, text: str) -> list[tuple[str, int]]:
    encoding = tokenizer.encode(text, add_special_tokens=False)
    return list(zip(encoding.tokens, encoding.ids))


def main() -> None:
    tokenizer = load()
    print(f"vocabulary size {tokenizer.get_vocab_size()}\n")
    print(f"{'sample':<8} {'chars':>5} {'words':>5} {'tokens':>6}")
    for name, text in SAMPLES.items():
        print(f"{name:<8} {len(text):>5} {len(text.split()):>5} {count(tokenizer, text):>6}")
    print("\npieces of the command:")
    for piece, token_id in pieces(tokenizer, SAMPLES["command"]):
        print(f"  {piece!r:<14} {token_id}")


if __name__ == "__main__":
    main()

add_special_tokens=False counts only your text. The runtime adds special tokens of its own, and step 3 measures them.

2. Read the counts and the pieces

vocabulary size 262144

sample   chars words tokens
prose       65    10     13
command     42     6     12
config      48     8     18
id          32     1     26
page       572    92    143

pieces of the command:
  'doc'          8929
  'uct'          17092
  'l'            236752
  '▁rollback'    166836
  '▁--'          2617
  'env'          7291
  '▁production'  3719
  '▁--'          2617
  'to'           1071
  '▁r'           637
  '4'            236812
  '1'            236770

This run loads only the pinned tokenizer, not model weights. The raw local counts and pieces describe the current docuctl text. The earlier runtime comparison below uses its original text and cannot establish counts for a changed request.

vocabulary size 262144

sample   chars words tokens
prose       65    10     12
command     42     6     11
config      48     8     18
id          32     1     26
page       572    92    136

pieces of the command:
  'ship'         5976
  'ctl'          58514
  '▁rollback'    166836
  '▁--'          2617
  'env'          7291
  '▁production'  3719
  '▁--'          2617
  'to'           1071
  '▁r'           637
  '4'            236812
  '1'            236770

The prose sentence is close to one token per word. The rest are not. The timestamped identifier ci-docs-api-2026-10-21T09:14:03Z is one word and 26 tokens, because the tokenizer has no piece for it and spells it out in fragments. The config block costs more than twice its word count. The archived command shows why: the earlier name shipctl is not in the vocabulary, so it becomes ship + ctl; r41 becomes ▁r, 4, 1; and the ▁ marks a piece that starts with a space.

Developer docs are full of these strings. A rule of thumb tuned on prose, such as words times 1.3, undercounts exactly the pages a docs assistant sends.

Another model’s tokenizer gives another count

Qwen’s column differs from Gemma’s on most samples, including the config block and the ID, even though both are raw prompts of the same text. A count from one model’s tokenizer is not a budget for another model. When an application can switch models, keep one tokenizer per model, and check the count again whenever you change models.

3. Compare with what the runtime processed

Your count is only useful if it matches the model’s. compare_runtime.py sends each sample to Ollama with num_predict 1 and reads prompt_eval_count, the number of tokens the runtime fed to the model. It sends each sample three ways: as a raw prompt to Gemma, as a chat message to Gemma, and as a raw prompt to a different model, Qwen 3.5. Fresh runtime counts are saved to runs/local-docuctl/counts.json; the root counts.json remains the original comparison:

"""Compare local counts with what Ollama actually processed. Run: TOKENIZER_PATH=... python compare_runtime.py"""
import json
from pathlib import Path

import ollama

from count_tokens import SAMPLES, count, load


def processed(client: ollama.Client, model: str, text: str, chat: bool) -> int:
    """prompt_eval_count: the tokens the runtime fed to the model for this request."""
    options = {"num_predict": 1, "temperature": 0}
    if chat:
        reply = client.chat(model=model, messages=[{"role": "user", "content": text}], think=False, options=options)
    else:
        reply = client.generate(model=model, prompt=text, raw=True, options=options)
    return reply.prompt_eval_count


def main() -> None:
    tokenizer, client, rows = load(), ollama.Client(), {}
    print(f"{'sample':<8} {'local':>5} {'gemma raw':>9} {'gemma chat':>10} {'qwen raw':>8}")
    for name, text in SAMPLES.items():
        row = {"local": count(tokenizer, text),
               "gemma_raw": processed(client, "gemma4:e2b", text, chat=False),
               "gemma_chat": processed(client, "gemma4:e2b", text, chat=True),
               "qwen_raw": processed(client, "qwen3.5:0.8b", text, chat=False)}
        rows[name] = row
        print(f"{name:<8} {row['local']:>5} {row['gemma_raw']:>9} {row['gemma_chat']:>10} {row['qwen_raw']:>8}")
    output = Path(__file__).parent / "runs/local-docuctl"
    output.mkdir(parents=True, exist_ok=True)
    (output / "counts.json").write_text(json.dumps(rows, indent=2) + "\n")


if __name__ == "__main__":
    main()

sample   local gemma raw gemma chat qwen raw
prose       12        13         21       12
command     11        12         20       11
config      18        19         27       17
id          26        27         35       25
page       136       137        144      137

For these recorded samples, the raw Gemma count was the local count plus one: the beginning-of-sequence (BOS) token the runtime puts in front. This offset explains these samples. Recheck it after changing the model, tokenizer or runtime instead of assuming it applies to every request.

The chat template adds tokens you did not write

As a chat message, the page cost 144 tokens, not 136. Every other sample shows the same overhead of a few tokens per message. It looks small, but it repeats for every message in a conversation and for every page you add as its own message. Budget from the formatted request: render it with the model’s chat template, or read prompt_eval_count from a real call.

Count the formatted request, not its words

Gemma’s tokenizer is a subword tokenizer with a vocabulary of a quarter of a million pieces. Building it, before training, chose pieces that cover frequent text compactly. At runtime, encoding finds the pieces that spell your text from that vocabulary, and decoding joins the pieces back. The lab’s tests check that decoding the config block’s IDs returns the exact original text.

A chat request is not your text alone. The runtime renders the messages through the model’s chat template, which wraps each message in role markers such as the start of a user turn and the start of the model’s turn. Those markers are tokens too. Count the formatted request, not the strings you wrote.

Try it: budget a request for all four docs pages

Add a sample that joins all four pages in docs/ and count it. Then estimate the same text as words times 1.3 and compare. Finally, decide how many output tokens to reserve for a two-sentence answer, and check that pages, question and reply fit in a 4,096-token window. Run the offline tests with .venv/bin/python -m pytest -q.

Trace next-token generation runs the model one token at a time and shows the candidates it scores at each step.

References

Checked October 21, 2026.

• Hugging Face Tokenizers documentation
• Hugging Face: chat templates
• Ollama API reference: prompt_eval_count