Tokenization
How does tokenization work?
Before the transformer does anything at all, your text has to become numbers. That conversion step is tokenization, and it quietly sits at the root of most LLM behaviours that feel weird or inexplicable. Why can’t GPT count letters in “strawberry”? Why does it struggle with Arabic or Chinese but breeze through English? Why does a trailing space sometimes produce a completely different completion?
Tokenization. Andrej Karpathy once called it the “root of suffering” in LLMs. That’s a strong thing to say about what looks like a preprocessing step. By the end of this post, you’ll understand exactly why he said it.
What is Token, Actually?
A token is not a word. It’s not a character. It’s not a syllable. It’s a chunk of text that was statistically useful to treat as a unit, learned from training data. The same word can tokenize differently depending on context, capita-lization, and whether there’s a space before it. ‘Hello’ and ‘Hello ’(with a leading space) are often different tokens. ‘Running’ might be one token; ‘unbelievable’ might be three.
This is the core tradeoff that modern tokenizers navigate. Too coarse (whole words) and you hit vocabulary explosion — the English language has hundreds of thousands of words, and that number blows up the moment you add names, code, multilingual text, and typos. Too fine (individual characters) and your sequences become so long that the attention mechanism, which is quadratic in sequence length, becomes catastrophically expensive.
Subword tokenization sits in the middle, and BPE is currently the most widely used approach to get there.
From Text to Numbers
Before getting into BPE, it helps to understand what actually happens when you type text into an LLM. The model never sees your text. It only ever sees integers. Every word, every punctuation mark, every emoji goes through a conversion pipeline before the model touches it.
Here is what each stage does.
Step 1 is Unicode and UTF-8. Every character in every human language has an assigned number in the Unicode standard. There are over 1.1 million of these numbers, called code points. UTF-8 is the encoding system that converts those code points into actual bytes, the raw data a computer can store and transmit.
The byte cost is not equal for all characters. A plain ASCII letter like ‘a’ takes 1 byte. An accented character like é takes 2 bytes [195, 169]. Arabic letters each take 2 bytes. An emoji like 🔥 takes 4 bytes. This is the first and most fundamental reason why non-English text is more expensive to process: it starts with more raw bytes before any tokenization even begins.
Step 2 is BPE merging. Those bytes get merged, repeatedly, according to rules the tokenizer learned during training. Common pairs get combined into single tokens. Common triplets become single tokens. This continues until you have a much shorter sequence of token IDs. A 20-byte word might become 2 or 3 tokens. This is where most of the interesting behaviour lives, and where most of the surprises come from.
Step 3 is the embedding lookup. Each token ID is just an index into a large matrix. The model looks up the row that corresponds to each ID and retrieves a vector — a list of floating point numbers, typically hundreds or thousands of values long. That vector is what the model actually computes on. Strip everything else away, and that is all an LLM ever processes: sequences of vectors, one per token.
Why Not Just Use Words or Characters?
Worth spending one minute here because people ask this. Word-level tokenization seems natural. Split by spaces, done. The problem is vocabulary size. A moderate English corpus gives you 170,000+ unique words. Add code, math symbols, multilingual text, usernames, and the long tail of internet text, and you’re looking at millions. Every word needs a row in the embedding matrix. And when you encounter a word that wasn’t in training, a new brand name, a misspelling, a niche technical term — you have no choice but to map it to <UNK>, a single "unknown" token, throwing away all information.
Character-level tokenization solves the vocabulary problem completely — you only need ~256 characters to cover everything. But a paragraph of text is now a sequence of thousands of characters, and transformer attention is O(n²) in sequence length. You either truncate heavily or pay enormous compute costs.
Subword tokenization merges frequent character sequences into single tokens, so common words become one token while rare words decompose into recognizable pieces. unbelievable becomes [un][believ][able].The model still has access to the semantic subunits. This is the right tradeoff, and BPE is the most practical way to find those subunits.
How BPE Works
Byte Pair Encoding was originally a data compression algorithm from 1994. Someone noticed it could be repurposed for tokenization, and it worked.
The core idea is simple: find the most frequent pair of adjacent tokens, merge them into one new token, repeat until you reach your target vocabulary size. Here’s the algorithm step by step on a tiny example. Say your entire training corpus is the text "hello hello world", encoded to bytes: [104, 101, 108, 108, 111, 32, 104, 101, 108, 108, 111, 32, 119, 111, 114, 108, 100]
Round 1: The pair (108, 108),the byte sequence for "ll" appears twice. Merge it into a new token 256. Sequence length drops from 17 to 15.
Round 2: Now (104, 101) — "he" — appears twice. Merge it into token 257. Length drops to 13.
You keep doing this. After enough rounds, common whole words become single tokens. Frequent subwords like “ing”, “un-”, “-tion” become single tokens. The vocabulary fills up from the bottom, starting at 256 base byte tokens and adding one merged token per round.
Why start from bytes and not characters? Because bytes are universal. Every possible string of text, in every language, every emoji, every special symbol, can be expressed as a sequence of 0–255. You start with a guaranteed full-coverage vocabulary of 256 tokens. Character-level BPE would require handling the full Unicode space, which is unwieldy. Byte-level BPE means you can’t have an OOV token — ever.
Tokenizer during Inference
After training, the tokenizer is essentially just a lookup table of merge rules, stored in the exact order in which they were learned during training. Earlier merge rules are more important because they were discovered first, meaning those pairs appeared more frequently in the training data. To tokenize new text during inference:
- Convert text to UTF-8 bytes
- Look at all adjacent pairs in the current sequence. The tokenizer scans the sequence and checks every neighbouring pair.
- There will be many pairs, but find the pair whose merge rule was learned earliest.
- Apply the merge and replace the selected pair with its merged token ID.
- Recompute the pairs again, as after one merge, the sequence has changed.
- Repeat until no more merges apply
Merges must be applied in the exact same order they were learned. You don’t just apply all possible merges at once; you apply the most frequent one first, then re-evaluate. This is what makes BPE deterministic and reproducible.
Pre-Tokenization
Real-world tokenizers don’t run BPE on the entire document as one stream. They first split text using regex patterns, then run BPE on each chunk independently.
Get Ali’s stories in your inbox
Join Medium for free to get updates from this writer.
Why? Because you don’t want BPE merging across word boundaries. Without pre-tokenization, the space before a word and the word itself could merge into a joint token (and sometimes they do, intentionally). For example, in “dog is”, BPE might learn “g i”, which is weird. The GPT-2 pattern splits on contractions, letter sequences, digit sequences, punctuation, and whitespace — keeping linguistic units separate before BPE runs on each one.
This is also why' running' (with a leading space) and running (without) are different tokens in most GPT-family models. The space gets absorbed into the token during pre-tokenization.
WordPiece and Unigram
Two other algorithms are worth knowing, mostly because major models use them and because understanding their differences sharpens your intuition for BPE’s design choices.
WordPiece (BERT, Google’s translation systems) is BPE but with a different selection criterion. Instead of picking the most frequent pair, it picks the pair that most increases the likelihood of the training data, specifically pairs where co-occurrence is much higher than chance:
This produces slightly different vocabularies because it prefers pairs that are meaningfully associated, not just common. This is called statistical assosiation. BERT’s vocabulary also marks continuation subwords with ‘##’ so "running" becomes ["run", "##ning"], which helps the model explicitly track word boundaries.
Unigram (T5, XLNet, SentencePiece) works in reverse. Start with a huge over-generated vocabulary(seed vocabulary), then assign probabilities to tokens. These probabilities define how likely a token is used when generating a word. We want the tokenzier to assign high probability to reall training text. Then prune the entries that contribute least to explaining the training data. It’s probabilistic — during training, the same word can be tokenized multiple ways, which acts as implicit data augmentation and makes models more robust to typos. Now when you tokenize a sentence, you don’t greedily split. Instead you compute multiple segmentations and pick the most probable one.
BPE won the practical race for a few reasons: it’s fully deterministic, fast to implement, and scales predictably. The slight suboptimality in merge selection compared to WordPiece turns out not to matter much when you’re training at scale. GPT-4’s cl100k_base and the newer o200k_base vocabularies are both BPE-based, as are Llama, Mistral, Falcon, and most modern open-source models.
Special Tokens
Beyond regular text tokens, tokenizers reserve certain IDs for structural purposes. These are not learned from data, they’re manually added and serve specific roles in how the model understands its input.
Key special tokens:
<|endoftext|>tells the model a document ends here. If a user can inject this into a prompt, they can prematurely terminate the model's output.<|pad_token|>is used to fill batches to equal length so the GPU can process them in parallel; the attention mask ensures the model ignores it.<|im_start|>/<|im_end|>mark the start and end of turns in a chat format (ChatML format used by many models).- FIM tokens (
<|fim_prefix|>,<|fim_middle|>,<|fim_suffix|>) Fill-in-the-Middle tokens for code completion, letting the model generate code that fits between two existing snippets.
Mishandling these tokens is a common source of prompt injection vulnerabilities. If user-supplied input can embed special tokens, it can corrupt the model’s understanding of conversation structure.
Why LLMs Are Moving Away from Tokenizers
For all of BPE’s practical success, the field has been slowly asking: what if we just didn’t tokenize?
The problems are real. Tokenization creates arbitrary boundaries that the model can’t reason across. It bakes in structural biases toward English. Glitch tokens pollute the embedding space. The whole thing is a separate trained artefact with its own failure modes.
Two approaches are gaining traction.
Byte Latent Transformers (BLT)
BLT, developed by Meta, eliminates the fixed vocabulary. Instead of a static merge table, it groups bytes into dynamic patches based on how surprising each byte is.
The mechanism: a small lightweight model reads the byte stream and estimates entropy, like how predictable is the next byte? When entropy is low (the next byte is highly predictable), bytes get grouped into a larger patch. When entropy is high (something complex or rare is happening), the patch boundary falls there, and the main model spends more compute on that region.
This adaptive compute allocation is the real win. BLT matched Llama 3’s performance benchmarks while using 50% fewer FLOPs at inference. The model is literally spending less time on predictable text and more time on complex text.
Mamba and State Space Models
The other blocker for byte-level models is raw sequence length. A 1,000-word document is maybe 1,500 tokens with BPE, but it’s 5,000–6,000 raw bytes. Standard transformer attention is O(n²) in sequence length — 4x longer sequences means 16x more attention compute.
Mamba addresses this using selective state space models (SSMs). Rather than attending to all previous positions, Mamba maintains a compact hidden state that summarizes context, updated at each step. This gives linear scaling in sequence length — the same compute regardless of how long the sequence gets.
This makes byte-level processing practically feasible at scale. Mamba-based architectures can process hundreds of thousands of bytes without the memory explosion that would kill a transformer.
The combination — dynamic patching + linear attention — is likely the direction tokenization-free models will follow. We’re not there yet at scale, but the trajectory is clear.
Closing thoughts:
Tokenization exists because training large transformers on raw bytes was too slow with early hardware and early architectures. BPE is an elegant workaround which compress the byte stream into a smaller vocabulary of frequent subwords, so the attention mechanism has less to attend over.
It works well enough that GPT-4, Llama 3, and Mistral all use it. But it introduces fragility at every layer: inconsistent number representation, language bias, glitch tokens, context window inefficiency, and vulnerabilities around special token injection.
The models that come after might not need it. BLT shows that byte-level models can reach parity with token-based ones. Mamba shows that long byte sequences don’t have to be prohibitively expensive.
Until then if you’re building with LLMs, tokenization is not a background detail. It’s a constraint that shapes what your model can and can’t do, how much it costs to run, and how it fails. Understanding it at this level puts you well ahead of treating the API as a black box.
