An LLM’s vocabulary is the fixed set of tokens the model can read and produce. It is created when the tokenizer is trained, before the model itself, and cannot change afterwards without retraining. Every piece of text entering the model is broken into these tokens, and every token the model generates is chosen from this list.
Key takeaways
- Vocabulary sizes typically range from about 30,000 to over 250,000 tokens.
- Each vocabulary entry has a row in the model’s embedding table and a slot in the output layer.
- Larger vocabularies make text shorter in tokens but increase model size.
- Special tokens mark boundaries such as start, end and chat roles.
- Rare words are still representable because they split into subword pieces.
An LLM vocabulary explained in simple terms
Think of a printing press with a fixed tray of blocks. Some blocks are whole common words, others are fragments and letters. Any text can be set using these blocks, but the tray never changes once built. An LLM’s vocabulary is that tray.
How vocabulary size affects a model
| Effect | Smaller vocabulary | Larger vocabulary |
|---|---|---|
| Tokens per sentence | More | Fewer |
| Context window usage | Higher | Lower |
| API cost for the same text | Higher | Lower |
| Embedding and output layer size | Smaller | Larger |
| Coverage of many languages | Weaker | Stronger |
Worked example: with an embedding dimension of 4,096, a vocabulary of 32,000 needs 32,000 × 4,096 ≈ 131 million parameters in the embedding table, and a similar number in the output layer if weights are not shared. Growing the vocabulary to 128,000 raises that to about 524 million each. In exchange, the same text may need noticeably fewer tokens, especially in non-English languages and code.
The trend towards larger vocabularies
Early models such as BERT used around 30,000 tokens. Later models moved to 32,000, then 100,000 and beyond, as developers sought better multilingual and code efficiency. Larger vocabularies particularly help languages written in non-Latin scripts, including Hindi, Marathi and Tamil, which are otherwise split into many small pieces.
Special tokens
| Token type | Purpose |
|---|---|
| Beginning of sequence | Marks where text starts |
| End of sequence | Signals generation should stop |
| Padding | Fills short sequences in a batch |
| Unknown | Placeholder in older tokenizers |
| Chat role markers | Separate system, user and assistant turns |
| Tool or function markers | Delimit structured tool calls |
| Fill-in-the-middle markers | Used in code models |
Chat templates depend on these tokens, which is why using the wrong template can noticeably degrade a model’s behaviour.
Vocabulary and out-of-vocabulary words
Subword tokenization means there are no truly unknown words. A new brand name or a typo is split into smaller pieces the model knows. The trade-off is that unusual strings consume more tokens and may be handled less reliably.
Practical implications
- Token counts differ between models for the same text, so budget per model.
- Vocabularies are model-specific; tokenizers are not interchangeable.
- Extending a vocabulary for a new language requires adding embedding rows and further training.
- Output layers over very large vocabularies add compute at every generation step.
Limitations
- Fixed at training time.
- Biased towards the languages and domains in the tokenizer’s training data.
- Very large vocabularies increase memory and slow the output layer.
- Poor vocabulary fit raises cost and can reduce quality for some languages.