Tokenization is the process of splitting text into small units called tokens and converting each one into a numeric ID that a language model can process. It is the first step whenever an LLM reads text, and the last step in reverse when it writes. Most modern LLMs use subword tokenization, which breaks rare words into smaller pieces while keeping common words whole.
Key takeaways
- Tokenization turns text into a list of numbers from a fixed vocabulary.
- Subword methods such as byte-pair encoding (BPE), WordPiece and Unigram dominate.
- Vocabulary sizes typically range from tens of thousands to a few hundred thousand tokens.
- The tokenizer is trained before the model and is fixed afterwards.
- Tokenizer design affects cost, multilingual performance and some odd model behaviours.
Tokenization explained in simple terms
A model cannot read letters or words directly; it only works with numbers. Tokenization is like a dictionary lookup that turns text into page numbers. The dictionary contains whole common words and useful fragments. Any word, even a brand-new one, can be spelled out using fragments, so the model never meets a word it cannot represent.
Why subword tokenization?
| Approach | Problem |
|---|---|
| Word-level | Huge vocabulary; cannot handle new or misspelled words |
| Character-level | Sequences become very long, making models slower and less effective |
| Subword-level | Balances both: common words stay whole, rare words split into known pieces |
How byte-pair encoding (BPE) works
BPE started as a data compression method and was adapted for language models in 2016 by Rico Sennrich and colleagues. To build a BPE vocabulary:
- Start with individual characters (or bytes) as the vocabulary.
- Count every adjacent pair of symbols in a large training text.
- Merge the most frequent pair into a new single symbol and add it to the vocabulary.
- Repeat thousands of times until the vocabulary reaches the target size.
Worked example: in a text where “low”, “lower” and “lowest” are common, the pair “l” + “o” is merged into “lo”, then “lo” + “w” into “low”. Later, “est” might become a token. Now “lowest” can be written as “low” + “est”, and a new word like “slowest” as “s” + “low” + “est”. Byte-level BPE, used by GPT-2 and many later models, starts from raw bytes, so it can represent any text, emoji or script.
Common tokenization methods
| Method | How it works | Used in |
|---|---|---|
| Byte-pair encoding (BPE) | Merges frequent symbol pairs | GPT family and many modern LLMs |
| WordPiece | Merges pairs based on likelihood | BERT |
| Unigram | Starts large and prunes the least useful pieces | Many multilingual models |
| SentencePiece | Library that treats text as raw characters including spaces; supports BPE and Unigram | T5, many Llama-era models |
Vocabulary size trade-offs
A larger vocabulary means more words and scripts are represented as single tokens, so text uses fewer tokens and costs less. But it also makes the model’s embedding and output layers larger. Newer models have generally moved to larger vocabularies to improve multilingual and code efficiency.
Tokenization and languages
Tokenizers trained mostly on English split other languages into more pieces. The same sentence in Hindi, Marathi or Tamil can take noticeably more tokens than in English, which increases cost, uses more of the context window and can reduce quality. Multilingual models address this with more balanced training data and larger vocabularies.
Quirks caused by tokenization
- Spelling and counting letters: the model sees “straw” + “berry”, not individual letters, so letter-counting questions can fail.
- Arithmetic: numbers split in inconsistent chunks can make maths harder.
- Leading spaces: “hello” and ” hello” may be different tokens.
- Rare names and code: unusual strings split into many tokens.
Tokenization in practice
- Use the correct tokenizer for each model; they are not interchangeable.
- Count tokens before sending large documents.
- When chunking documents for retrieval, split by token count, not character count.
- Libraries such as Hugging Face Tokenizers and OpenAI’s tiktoken make this easy.