Build a byte-pair encoding tokenizer

Problem

When you need this. The four-word model cannot read a new word. Build a tokenizer that splits text into reusable pieces and gives each piece the integer ID the model uses.

The model in the previous build had a vocabulary of four whole words. A language model must turn any text into IDs from a fixed vocabulary, and turn its predicted IDs back into text. In this build you write the tokenizer that the rest of this series uses.

Two simple choices fail in opposite ways. One ID per character gives a tiny vocabulary but long sequences, and every prediction covers only one letter. One ID per word gives short sequences, but the vocabulary grows with every new word, and a word missing from it cannot be encoded at all. Byte-pair encoding (BPE) sits between them. It starts from characters and learns longer pieces from the text, so frequent words become one token and rare words still split into characters.

The training text is a synthetic corpus written by make_corpus.py: 400 four-sentence stories, 45,223 characters in all. Each story gives a child an object of some color, moves it to a place and hands it to another child, for example: “mia has a red hat. mia goes to the garden. at the garden, mia gives the hat to ava. now ava has the red hat.” The stories are original text written for this series; no third-party text is included. The last sentence repeats the color and object from the first, so a later build’s model needs earlier text to predict it. A fixed seed regenerates the identical file, and nothing is downloaded.

The contract is small. Tokenizer.train(text, vocab_size) learns a tokenizer. encode(text) returns a list of IDs, decode(ids) returns text, and save() and load() store it as a versioned JSON file. Success means: decoding the encoded corpus gives back the exact corpus, every learned token encodes to itself, a fresh training run reproduces the saved file, and each bad input in the failure cases fails with a clear error.

Concepts

A token is a piece of text the model treats as one unit. The vocabulary lists the available tokens, and a token ID is the integer used to look one up. The first lesson used four manually chosen words; real text needs a repeatable way to split new input. This lesson learns reusable pieces from the corpus rather than assigning one token to every whole word. For a broader introduction, see What are tokens and tokenizers?.

Five terms describe the implementation:

Alphabet: The characters found in the training text. Each one becomes a token. This corpus has 27: the newline, the space, the comma, the period and 23 lowercase letters.
Chunk: A piece of text that merges may not cross: a word with its leading space, a run of punctuation or a run of whitespace. Splitting into chunks before BPE is called pre-tokenization. GPT-2 uses a similar rule: its merges do not cross between letters and punctuation, except for spaces.
Merge: A rule that joins two adjacent tokens into one new token, such as ” t” + “he” → ” the”. Each merge adds one entry to the vocabulary.
Merge rank: The position of a merge in the order training learned it. Encoding applies lower-ranked merges first.
<unk>: ID 0, used for a character that was not in the training alphabet. It decodes to the replacement character �, so the original character is lost.

Training counts every pair of adjacent tokens across the corpus, merges the most frequent pair everywhere it occurs, and repeats. When two pairs have the same count, this implementation takes the one that comes first in Python’s string order, which compares Unicode code points, so every run learns the same merges. Training stops when the vocabulary reaches the requested size, or earlier when no pair occurs at least twice. Merging a pair seen only once would still shorten the text by one token, but it would spend a vocabulary entry on text that appeared a single time; this implementation does not.

Production tokenizers for large models, such as GPT-2’s, use the 256 possible byte values as their base instead of the characters in the training text. Any text is then a sequence of known bytes, and no input needs an unknown token. This build starts from characters to keep the vocabulary at 96 entries, which keeps the later builds’ output layer small enough to train in plain Python. The cost is the <unk> token and the information it loses.

Architecture

bpe.py · train once, save the merges, replay them on new text

Tokenizer training counts adjacent pairs inside chunks and repeatedly merges the most frequent pair. It saves the alphabet and ordered merges. Encoding loads that artifact, applies the merges to new text and returns token IDs. Decoding turns the IDs back into text. Unsupported characters become the unknown token; invalid files or IDs are refused.

The figure has two parts that run at different times. In Training, the corpus chunks go to Count pairs. Merge top pair joins the most frequent pair and appends it to the merge list, and the return path labeled repeat sends the updated chunks back to be counted again. When the vocabulary is full or no pair occurs twice, the path labeled “no: stop” ends training and saves the Merge list, the one artifact training produces. In Encoding, which runs every time text enters the model, the dashed hand-off brings that saved list to Replay merges, which applies it to the characters of new text and outputs token IDs.

Everything runs in one Python process. bpe.py holds the tokenizer. make_corpus.py writes corpus.txt. trace_bpe.py trains on the corpus, saves tokenizer.json and records bpe-trace.json. The tokenizer file stores only the alphabet and the ordered merges; the vocabulary and IDs are rebuilt from them in the same order on load.

Working code

Runtime: Python 3.10 or newer. Dependencies: Python standard library only.

Download and extract the complete lab using Download the complete lab above. Open build-a-byte-pair-encoding-tokenizer/; keep its Python modules, tests and fixture files together. The bundled README includes setup commands for your platform.

bpe.pySource · pythonDownload

"""A byte-pair encoding (BPE) tokenizer trained on characters, standard library only.

Training starts from the characters seen in the training text and repeatedly
merges the most frequent adjacent pair of tokens into a new token. Encoding
replays those merges in the order they were learned. Merges never cross the
boundary of a chunk: a word with its leading space, a run of punctuation or a
run of whitespace.

ID 0 is reserved for <unk>: a character that never appeared in training.
"""
import json
import re
from collections import Counter

UNK = "<unk>"
FORMAT_VERSION = 1
# A chunk is an optional space plus letters or digits, an optional space plus
# punctuation, or whitespace. BPE merges only inside a chunk.
CHUNK = re.compile(r" ?[A-Za-z0-9]+| ?[^A-Za-z0-9\s]+|\s+")


def chunks(text):
    found = CHUNK.findall(text)
    if "".join(found) != text:  # every character must belong to exactly one chunk
        raise AssertionError("chunk pattern lost characters")
    return found


def most_frequent_pair(words):
    """words: {tuple of tokens: count}. Ties go to the pair first in string (code-point) order."""
    pairs = Counter()
    for word, count in words.items():
        for pair in zip(word, word[1:]):
            pairs[pair] += count
    if not pairs:
        return None, 0
    best = min(pairs, key=lambda p: (-pairs[p], p))
    return best, pairs[best]


def merge_word(word, pair, joined):
    out, i = [], 0
    while i < len(word):
        if i + 1 < len(word) and (word[i], word[i + 1]) == pair:
            out.append(joined)
            i += 2
        else:
            out.append(word[i])
            i += 1
    return tuple(out)


class Tokenizer:
    def __init__(self, alphabet, merges):
        self.alphabet = list(alphabet)
        self.merges = [tuple(m) for m in merges]
        self.vocab = [UNK] + self.alphabet + ["".join(m) for m in self.merges]
        self.ids = {token: i for i, token in enumerate(self.vocab)}
        if len(self.ids) != len(self.vocab):
            raise ValueError("vocabulary has duplicate tokens")
        self.rank = {m: r for r, m in enumerate(self.merges)}

    @classmethod
    def train(cls, text, vocab_size):
        """Learn merges until the vocabulary (including <unk>) has vocab_size entries."""
        if not text:
            raise ValueError("cannot train on empty text")
        alphabet = sorted(set(text))
        base = 1 + len(alphabet)
        if vocab_size < base:
            raise ValueError(f"vocab_size {vocab_size} is smaller than the {base} base tokens")
        words = Counter(tuple(c) for c in chunks(text))
        merges, history = [], []
        while base + len(merges) < vocab_size:
            pair, count = most_frequent_pair(words)
            if pair is None or count < 2:
                break  # no pair repeats; a once-seen pair is not worth a vocabulary entry
            joined = pair[0] + pair[1]
            words = Counter({merge_word(w, pair, joined): c for w, c in words.items()})
            merges.append(pair)
            history.append({"pair": list(pair), "token": joined, "count": count})
        tok = cls(alphabet, merges)
        tok.history = history
        return tok

    def encode_chunk(self, chunk):
        parts = [c if c in self.ids else UNK for c in chunk]
        while True:
            candidates = [(self.rank[p], i) for i, p in enumerate(zip(parts, parts[1:])) if p in self.rank]
            if not candidates:
                return parts
            _, i = min(candidates)
            pair = (parts[i], parts[i + 1])
            parts = list(merge_word(tuple(parts), pair, pair[0] + pair[1]))

    def tokens(self, text):
        return [t for chunk in chunks(text) for t in self.encode_chunk(chunk)]

    def encode(self, text):
        return [self.ids[t] for t in self.tokens(text)]

    def decode(self, ids):
        bad = [i for i in ids if isinstance(i, bool) or not (isinstance(i, int) and 0 <= i < len(self.vocab))]
        if bad:
            raise ValueError(f"token IDs outside the vocabulary of size {len(self.vocab)}: {bad[:5]}")
        return "".join("�" if i == 0 else self.vocab[i] for i in ids)

    def to_json(self):
        return {"format_version": FORMAT_VERSION, "alphabet": self.alphabet,
                "merges": [list(m) for m in self.merges]}

    @classmethod
    def from_json(cls, data):
        if data.get("format_version") != FORMAT_VERSION:
            raise ValueError(f"unsupported tokenizer format {data.get('format_version')!r}; expected {FORMAT_VERSION}")
        return cls(data["alphabet"], data["merges"])

    def save(self, path):
        with open(path, "w") as f:
            json.dump(self.to_json(), f, indent=1)
            f.write("\n")

    @classmethod
    def load(cls, path):
        with open(path) as f:
            return cls.from_json(json.load(f))

train() keeps a count for each distinct chunk instead of a list of every chunk in the corpus. The word “ the” appears 1,600 times but is stored once with its count, so each counting pass touches only the distinct chunks. encode_chunk() repeatedly finds the adjacent pair with the lowest merge rank and joins it, which replays training’s decisions in training’s order. decode() rejects IDs outside the vocabulary instead of failing with an IndexError deeper in the code.

Read make_corpus.py

make_corpus.pySource · pythonDownload

"""Write corpus.txt: short synthetic stories used by every build in this series.

Each story names a child, an object and its color, moves the object to a place
and hands it to a second child. The last sentence repeats the object and color,
so predicting it needs information from earlier in the story. The text is
generated from a fixed seed, so anyone can rebuild the identical file.

Run: python3 make_corpus.py          (rewrites corpus.txt)
     python3 make_corpus.py --check  (exit 1 if corpus.txt differs)
Python 3.10+, standard library only. No download.
"""
import random
import sys
from pathlib import Path

SEED = 2026
STORIES = 400
NAMES = ["mia", "leo", "ava", "sam", "zoe", "ben"]
OBJECTS = ["ball", "kite", "book", "cup", "hat", "box"]
COLORS = ["red", "blue", "green", "yellow"]
PLACES = ["park", "garden", "kitchen", "school"]


def story(rng):
    giver, taker = rng.sample(NAMES, 2)
    thing, color, place = rng.choice(OBJECTS), rng.choice(COLORS), rng.choice(PLACES)
    return (f"{giver} has a {color} {thing}. {giver} goes to the {place}. "
            f"at the {place}, {giver} gives the {thing} to {taker}. "
            f"now {taker} has the {color} {thing}.")


def corpus(seed=SEED, stories=STORIES):
    rng = random.Random(seed)
    return "\n".join(story(rng) for _ in range(stories)) + "\n"


if __name__ == "__main__":
    path = Path(__file__).with_name("corpus.txt")
    text = corpus()
    if "--check" in sys.argv:
        sys.exit(0 if path.read_text() == text else "corpus.txt is out of date")
    path.write_text(text)
    print(f"wrote {len(text)} characters, {STORIES} stories")

Write the corpus, then train and record:

Terminal commandsTerminal · bash

python3 make_corpus.py
python3 trace_bpe.py --write

trace_bpe.py --check exits with an error if your files differ from the recorded tokenizer.json and bpe-trace.json.

Watch the first merges

The base vocabulary has 28 entries: <unk> and the 27 characters. Training adds 68 merges to reach 96. The first 12:

Rank	Pair	New token	Count
1	`" " + "t"`	`" t"`	2,400
2	`"h" + "e"`	`"he"`	1,798
3	`" t" + "he"`	`" the"`	1,600
4	`" " + "g"`	`" g"`	1,186
5	`" " + "b"`	`" b"`	1,101
6	`" " + "a"`	`" a"`	1,072
7	`" " + "h"`	`" h"`	1,010
8	`" h" + "a"`	`" ha"`	1,010
9	`" ha" + "s"`	`" has"`	800
10	`" t" + "o"`	`" to"`	800
11	`"e" + "s"`	`"es"`	800
12	`"e" + "n"`	`"en"`	716

The space before “the” is part of the first merge, " " + "t", because chunks keep a word’s leading space. Merge 3 builds " the" from two tokens learned earlier. Counts never rise from row to row, because each merge takes the most frequent remaining pair; equal counts can follow each other. The last merges build up longer words one merge at a time, such as " kitchen", " par", " park".

Choose the vocabulary size

More merges give longer tokens and shorter sequences. The recorded sweep trains 4 tokenizers on the same corpus:

Requested size	Vocabulary size	Corpus tokens	Characters per token
40	40	30,930	1.46
64	64	21,494	2.10
96	96	14,381	3.15
128	109	12,400	3.65

At 96 entries the corpus becomes 14,381 tokens, 3.15 characters per token. A request for 128 stops at 109: by then every word in this small corpus is a single token and no pair occurs twice. The series uses 96. Growing to 109 entries would cut the corpus by a further 14% of its tokens, but each extra entry adds a column to the model’s output layer, which the later builds compute in plain Python.

Encode new text

mia has a red ball. uses only corpus words, so it becomes 6 tokens: "mia", " has", " a", " red", " ball", ".", with IDs 58, 36, 33, 89, 85, 4. Decoding those IDs returns the same sentence.

Check your understanding

The corpus has no “maya”, “purple” or “umbrella”. Before you open the comparison, predict how the tokenizer splits maya has a purple umbrella., and whether decoding gives back the same text.

Compare your prediction

It becomes 20 tokens: "m", "a", "y", "a", " has", " a", " p", "u", "r", "p", "l", "e", " ", "u", "m", "b", "re", "ll", "a", ".". The known words “ has” and “ a” are single tokens. The new words fall back to single characters, except where a learned merge happens to fit, such as "re" and "ll" inside “umbrella”. Every character is in the alphabet, so decoding returns the exact sentence. New words still encode, but they cost more tokens.

Read the explanation

BPE adds one entry per merge. This trainer also stops when no pair occurs at least twice, so a larger request cannot help a corpus whose repeated words are already single tokens.

Inspect the recorded trace

bpe-trace.jsonRecorded output · jsonDownload

{
  "corpus": {
    "characters": 45223,
    "stories": 400
  },
  "alphabet": [
    "\n",
    " ",
    ",",
    ".",
    "a",
    "b",
    "c",
    "d",
    "e",
    "g",
    "h",
    "i",
    "k",
    "l",
    "m",
    "n",
    "o",
    "p",
    "r",
    "s",
    "t",
    "u",
    "v",
    "w",
    "x",
    "y",
    "z"
  ],
  "base_vocab_size": 28,
  "vocab_size": 96,
  "merge_count": 68,
  "first_merges": [
    {
      "pair": [
        " ",
        "t"
      ],
      "token": " t",
      "count": 2400
    },
    {
      "pair": [
        "h",
        "e"
      ],
      "token": "he",
      "count": 1798
    },
    {
      "pair": [
        " t",
        "he"
      ],
      "token": " the",
      "count": 1600
    },
    {
      "pair": [
        " ",
        "g"
      ],
      "token": " g",
      "count": 1186
    },
    {
      "pair": [
        " ",
        "b"
      ],
      "token": " b",
      "count": 1101
    },
    {
      "pair": [
        " ",
        "a"
      ],
      "token": " a",
      "count": 1072
    },
    {
      "pair": [
        " ",
        "h"
      ],
      "token": " h",
      "count": 1010
    },
    {
      "pair": [
        " h",
        "a"
      ],
      "token": " ha",
      "count": 1010
    },
    {
      "pair": [
        " ha",
        "s"
      ],
      "token": " has",
      "count": 800
    },
    {
      "pair": [
        " t",
        "o"
      ],
      "token": " to",
      "count": 800
    },
    {
      "pair": [
        "e",
        "s"
      ],
      "token": "es",
      "count": 800
    },
    {
      "pair": [
        "e",
        "n"
      ],
      "token": "en",
      "count": 716
    }
  ],
  "last_merges": [
    {
      "pair": [
        " kit",
        "c"
      ],
      "token": " kitc",
      "count": 198
    },
    {
      "pair": [
        " kitc",
        "he"
      ],
      "token": " kitche",
      "count": 198
    },
    {
      "pair": [
        " kitche",
        "n"
      ],
      "token": " kitchen",
      "count": 198
    },
    {
      "pair": [
        " p",
        "ar"
      ],
      "token": " par",
      "count": 198
    },
    {
      "pair": [
        " par",
        "k"
      ],
      "token": " park",
      "count": 198
    }
  ],
  "corpus_tokens": 14381,
  "characters_per_token": 3.145,
  "round_trip": true,
  "sweep": [
    {
      "requested": 40,
      "vocab_size": 40,
      "tokens": 30930,
      "characters_per_token": 1.462
    },
    {
      "requested": 64,
      "vocab_size": 64,
      "tokens": 21494,
      "characters_per_token": 2.104
    },
    {
      "requested": 96,
      "vocab_size": 96,
      "tokens": 14381,
      "characters_per_token": 3.145
    },
    {
      "requested": 128,
      "vocab_size": 109,
      "tokens": 12400,
      "characters_per_token": 3.647
    }
  ],
  "samples": [
    {
      "text": "mia has a red ball.",
      "tokens": [
        "mia",
        " has",
        " a",
        " red",
        " ball",
        "."
      ],
      "ids": [
        58,
        36,
        33,
        89,
        85,
        4
      ],
      "decoded": "mia has a red ball."
    },
    {
      "text": "maya has a purple umbrella.",
      "tokens": [
        "m",
        "a",
        "y",
        "a",
        " has",
        " a",
        " p",
        "u",
        "r",
        "p",
        "l",
        "e",
        " ",
        "u",
        "m",
        "b",
        "re",
        "ll",
        "a",
        "."
      ],
      "ids": [
        15,
        5,
        26,
        5,
        36,
        33,
        90,
        22,
        19,
        18,
        14,
        9,
        2,
        22,
        15,
        6,
        44,
        43,
        5,
        4
      ],
      "decoded": "maya has a purple umbrella."
    },
    {
      "text": "café",
      "tokens": [
        "c",
        "a",
        "<unk>",
        "<unk>"
      ],
      "ids": [
        7,
        5,
        0,
        0
      ],
      "decoded": "ca��"
    }
  ]
}

Failure cases

Each case has a named test in test_bpe.py.

Case	Result with this implementation	What it tells you
Characters outside the training alphabet	`café` encodes to 7, 5, 0, 0 and decodes to `ca��`	The corpus has no “f” or “é”; both become `<unk>` and cannot be recovered
A word the corpus never contained	`maya` falls back to single characters	New words encode, but take more tokens
An ID outside the vocabulary passed to `decode()`	`ValueError` naming the ID	IDs from a different tokenizer or a bug are caught at the boundary
Empty training text	`ValueError`	There is nothing to learn an alphabet from
A vocabulary size below the base vocabulary	`ValueError` naming the base size	The alphabet and `<unk>` always need their own IDs
A tokenizer file with a different format version	`load()` raises `ValueError`	A file written by a changed program is refused instead of misread

The complete lab includes these tests and their fixtures. Run the 16 standard-library tests:

Terminal commandsTerminal · bash

python3 -m unittest -v test_bpe.py

Read the tests

test_bpe.pySource · pythonDownload

"""Tests for bpe.py. Run: python3 -m unittest -v test_bpe.py"""
import json
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path

from bpe import Tokenizer, chunks
from make_corpus import corpus

HERE = Path(__file__).parent
TEXT = (HERE / "corpus.txt").read_text()


class Recorded(unittest.TestCase):
    def test_corpus_matches_generator(self):
        self.assertEqual(TEXT, corpus())

    def test_recorded_trace_and_tokenizer(self):
        result = subprocess.run([sys.executable, "trace_bpe.py", "--check"], cwd=HERE)
        self.assertEqual(result.returncode, 0)

    def test_saved_tokenizer_equals_fresh_training(self):
        saved = Tokenizer.load(HERE / "tokenizer.json")
        fresh = Tokenizer.train(TEXT, 96)
        self.assertEqual(saved.vocab, fresh.vocab)


class Mechanism(unittest.TestCase):
    def setUp(self):
        self.tok = Tokenizer.train(TEXT, 96)

    def test_round_trip_on_corpus(self):
        self.assertEqual(self.tok.decode(self.tok.encode(TEXT)), TEXT)

    def test_every_learned_token_encodes_to_itself(self):
        # Replaying merges in rank order must rebuild each merged token as one piece.
        for token in self.tok.vocab[1 + len(self.tok.alphabet):]:
            with self.subTest(token=token):
                self.assertEqual(self.tok.tokens(token), [token])

    def test_most_frequent_pair_is_merged_first(self):
        self.assertEqual(self.tok.history[0]["pair"], [" ", "t"])
        counts = [h["count"] for h in self.tok.history]
        self.assertEqual(counts, sorted(counts, reverse=True))

    def test_merges_stay_inside_chunks(self):
        for token in self.tok.vocab[1:]:
            self.assertEqual(chunks(token), [token], token)

    def test_unseen_word_falls_back_to_characters(self):
        self.assertEqual(self.tok.tokens("maya"), ["m", "a", "y", "a"])

    def test_training_stops_when_no_pair_repeats(self):
        self.assertEqual(len(Tokenizer.train(TEXT, 128).vocab), 109)

    def test_encoding_applies_the_lowest_rank_merge_first(self):
        # "bc" was learned before "ab", so "abc" becomes a + bc, not ab + c.
        tok = Tokenizer(["a", "b", "c"], [("b", "c"), ("a", "b")])
        self.assertEqual(tok.tokens("abc"), ["a", "bc"])

    def test_ties_break_alphabetically(self):
        tok = Tokenizer.train("ab cd", 10)
        self.assertEqual(tok.merges, [])  # every pair occurs once, so the trainer adds no entry
        # (" ", "c"), ("a", "b") and ("c", "d") each occur twice; the space sorts first.
        tok = Tokenizer.train("ab cd ab cd", 10)
        self.assertEqual(tok.merges[0], (" ", "c"))


class FailureCases(unittest.TestCase):
    def setUp(self):
        self.tok = Tokenizer.train(TEXT, 96)

    def test_unknown_characters_lose_information(self):
        ids = self.tok.encode("café")
        self.assertEqual(ids[-2:], [0, 0])
        self.assertEqual(self.tok.decode(ids), "ca��")

    def test_decode_rejects_out_of_range_ids(self):
        for ids in ([5, 96], [-1], [True]):
            with self.subTest(ids=ids), self.assertRaisesRegex(ValueError, "outside the vocabulary"):
                self.tok.decode(ids)

    def test_empty_training_text(self):
        with self.assertRaisesRegex(ValueError, "empty text"):
            Tokenizer.train("", 96)

    def test_vocab_size_below_base(self):
        with self.assertRaisesRegex(ValueError, "smaller than the 28 base tokens"):
            Tokenizer.train(TEXT, 20)

    def test_wrong_format_version(self):
        data = self.tok.to_json() | {"format_version": 2}
        with tempfile.TemporaryDirectory() as d:
            path = Path(d) / "tok.json"
            path.write_text(json.dumps(data))
            with self.assertRaisesRegex(ValueError, "unsupported tokenizer format"):
                Tokenizer.load(path)


if __name__ == "__main__":
    unittest.main()

test_every_learned_token_encodes_to_itself guards the encoder: if encode_chunk() replays merges in the wrong order, some learned tokens split into pieces and the test fails. test_most_frequent_pair_is_merged_first guards training: if it picks pairs in any other order, the counts are no longer in falling order.

Deployment

The deployable artifact is tokenizer.json, not the training script. Version it together with any model trained on its IDs, and load it with Tokenizer.load(). Loading rebuilds the vocabulary from the alphabet and merges in a fixed order, so the IDs are the same on every machine. The format version lets a later program refuse a file it does not understand.

Scaling

Training repeats one counting pass over the distinct chunks for every merge. Here there are few distinct chunks, because the corpus reuses a small set of words; real text has far more. encode_chunk() rescans the chunk after every merge it applies, which is fine for short words and slow for long runs of characters. Libraries built for large corpora use faster data structures and compiled code for both steps. This build records no timings and makes no speed claim.

Key takeaways

BPE learns a vocabulary from text: start from characters, merge the most frequent adjacent pair, repeat. The ordered merge list is the tokenizer.
Encoding replays the merges in the order they were learned. Frequent words become single tokens; new words fall back to smaller pieces.
A character outside the base alphabet cannot be represented. Byte-level BPE avoids that by starting from all 256 byte values.

Best practices

Save the tokenizer as a versioned file and keep it with every model trained on it.
Test the round trip on the training text and the encoding of text the tokenizer never saw, including characters outside its alphabet.
Make tie-breaking explicit so that two training runs on the same text produce the same file.

References

Sources checked October 20, 2026.

Sennrich, Haddow and Birch, Neural Machine Translation of Rare Words with Subword Units (ACL 2016): adapts Gage’s 1994 compression algorithm to merge characters into subword units. Training starts from the character vocabulary, merges the most frequent pair repeatedly, and ends with the initial size plus the number of merges. Merges do not cross word boundaries.
Radford et al., Language Models are Unsupervised Multitask Learners (GPT-2, 2019), section 2.2: byte-level BPE with a 256-entry base vocabulary, merges blocked across character categories except for spaces.
Hugging Face Transformers: tokenization algorithms: pre-tokenization into words, the unknown token for characters outside the base vocabulary, and byte-level BPE as the way to avoid it.
OpenAI tiktoken: a production byte-level BPE library; its README describes BPE as reversible and lossless on arbitrary text.