How Tokenizers Work: BPE Explained Simply (2026)

Published: August 8, 2026 — Before a model can think, text must become numbers — and the tokenizer is the translator. BPE (byte-pair encoding) is the algorithm behind GPT, Llama, Qwen, and most of the models on this site, and it quietly decides your cost, your context budget, and part of your quality. This guide explains how it works with a worked example, why it matters, and a from-scratch implementation.

⚡ Quick Takeaways

What a Tokenizer Does

Models process numbers, not characters. The tokenizer's job: split text into tokens and map each to an integer ID. A token can be a word (the), a subword (ization), or a single byte. The vocabulary — the fixed set of tokens the model knows — is the dictionary this mapping uses, and its size is literally the first dimension of the model's embedding table.

"the cat sat" → [1162, 6756, 7515]   # three IDs, three tokens

How BPE Works: A Worked Example

BPE starts at single characters (bytes) and iteratively merges the most frequent adjacent pair into a new token:

  1. Start: every character is a token. Vocabulary = unique characters.
  2. Count: find the most frequent adjacent pair in the corpus. In English, "t h" and "h e" are early winners.
  3. Merge: create a new token for that pair; replace all occurrences.
  4. Repeat until the vocabulary reaches its target size (GPT models: 50K–150K tokens).
# Conceptual trace on "low lower lowest"
# Step 1 (chars):   l o w _ l o w e r _ l o w e s t
# Merge "lo" → LO:  LO w _ LO w e r _ LO w e s t
# Merge "LOw" → LOW: LOW _ LOW e r _ LOW e s t
# ... eventually:   "low" is ONE token, "lowest" = "low"+"est"

# The greedy rule: merge the pair that appears most often.
# Common words get their own token; rare words stay assembled.

That's the entire algorithm — frequency-driven compression, learned once at tokenizer-training time, then frozen. The tokenizer itself is trained on a large corpus before the model is; the model then trains on the fixed token stream.

A From-Scratch Implementation

def train_bpe(text, vocab_size):
    vocab = {bytes([b]) for b in text.encode("utf-8")}  # bytes
    tokens = [bytes([b]) for b in text.encode("utf-8")]

    while len(vocab) < vocab_size:
        # Count adjacent pairs
        pairs = {}
        for a, b in zip(tokens, tokens[1:]):
            pairs[(a, b)] = pairs.get((a, b), 0) + 1
        if not pairs: break
        # Merge the most frequent
        best = max(pairs, key=pairs.get)
        merged = best[0] + best[1]
        vocab.add(merged)
        # Replace all occurrences of the pair
        out, i = [], 0
        while i < len(tokens):
            if (i < len(tokens) - 1
                    and tokens[i] == best[0]
                    and tokens[i+1] == best[1]):
                out.append(merged); i += 2
            else:
                out.append(tokens[i]); i += 1
        tokens = out
    return vocab

# Usage (on real corpora you'd use the `tokenizers` library —
# this shows the algorithm, not the production engineering)

For production tokenizers, use Hugging Face's tokenizers library (Rust-backed, fast, used by most open models) — the concept is identical.

Why Tokenization Matters So Much

Impact How the tokenizer decides it
Cost Billing is per-token. A text that tokenizes badly costs more — and cloud AI bills exactly this way
Context budget The context window is measured in tokens; a wasteful tokenizer shrinks your effective context
Quality Words split awkwardly across tokens are harder to generate correctly — the source of many 'wrong language/typo' failures
Language equity Scripts under-represented in training corpora tokenize inefficiently — the Dari/Pashto reality

This is why the translation pipeline and RAG guides both mention token budgets: a chunk in Arabic script can consume 1.5–2x the tokens of the same English content, silently shrinking your context and raising your cost.

Frequently Asked Questions (FAQ)

What is a tokenizer?

A tokenizer converts text into the integer IDs a model processes — the front door of every LLM. It splits text into tokens (words, subwords, or bytes) and maps each to an ID. Everything a model "reads" or "writes" passes through it, which is why tokenizer choices shape quality and cost.

What is BPE (byte-pair encoding)?

BPE is the tokenization algorithm behind GPT, Llama, Qwen, and most modern LLMs. It starts with single characters (bytes) and iteratively merges the most frequent adjacent pair into a new token — so common words become single tokens ("the") while rare words stay split into pieces ("tokenization" → "token" + "ization").

Why does tokenization matter?

Because models are billed and capped by tokens: the tokenizer decides how many tokens your text consumes, which drives cost, context limits, and speed. It also affects quality — a word split awkwardly across tokens is harder for the model to handle, which is why some languages (and some scripts) behave worse than others.

How many tokens does a word usually take?

Roughly one token per 3–4 characters of English text — about 750 words per 1000 tokens. But it varies wildly: common words take 1 token, while technical terms, names, and words in other scripts can take 3–6 tokens each. This is why Arabic-script text often costs more tokens than the same meaning in English.

Can I change a model's tokenizer?

Not without retraining — the tokenizer and model are trained together; the embedding table's size is the vocabulary size. Tokenizers are swapped only when a model is trained from scratch or extended (some teams extend vocabularies for new languages, which requires continued pretraining).

How do I count tokens in my own app?

Use the model's own tokenizer — tiktoken for OpenAI models, tokenizers library for Hugging Face models, and each local runtime (Ollama, llama.cpp) exposes token counting. Never estimate by characters; the variance is too large. Counting matters for context budgeting and cost forecasting.

Sources & Further Reading