Perplexity is a measure of how well a language model predicts text. It is the exponential of the average cross-entropy loss per token, and can be read as the average number of equally likely options the model is weighing at each step. A perplexity of 10 means the model is about as uncertain as if it were choosing uniformly among 10 tokens. Lower is better.
Key takeaways
- Perplexity = exp(average cross-entropy loss per token).
- It measures prediction quality, not helpfulness or truthfulness.
- Values are only comparable when the tokenizer and test data are the same.
- It is widely used during pretraining to track progress.
- Modern assistants are judged mainly on task benchmarks and human preference instead.
Perplexity explained in simple terms
Imagine guessing the next word in a sentence. If you are confident, you have effectively one option in mind: low perplexity. If you have no idea and a hundred words seem equally possible, your perplexity is around 100. Perplexity puts a number on how “confused” the model is, averaged over a whole text.
The formula and a worked example
If the model assigns probability p to each true token, the cross-entropy loss per token is the average of −ln(p), and perplexity is e raised to that average.
Simple case: a model assigns probability 0.5 to every correct token.
- Loss per token: −ln(0.5) = 0.693
- Perplexity: e^0.693 = 2.0
The model is as uncertain as flipping a fair coin between two options.
Second case: average probability 0.25 → loss 1.386 → perplexity 4.0. Third case: average probability 0.9 → loss 0.105 → perplexity 1.11, a very confident model.
What counts as good perplexity?
There is no universal answer. Perplexity depends on:
- The tokenizer: a model with a larger vocabulary predicts fewer, longer tokens, which changes the number.
- The test data: predicting technical papers is harder than predicting common web text.
- Language: values differ between languages.
For this reason, perplexity is meaningful when comparing models on the same data with the same tokenizer, and misleading otherwise.
Where perplexity is used
| Use | How |
|---|---|
| Pretraining monitoring | Track perplexity falling as training progresses |
| Model comparison | Compare architectures on identical data and tokenizer |
| Data quality checks | High perplexity on a dataset can indicate noise or unusual text |
| Domain fit | Measure how well a model predicts industry-specific text before fine-tuning |
| Quantization checks | Verify that compression has not badly degraded the model |
Why perplexity is not enough
A model can predict text well and still be unhelpful, unsafe or factually wrong. Perplexity says nothing about instruction following, reasoning, tone or safety. That is why modern evaluation combines perplexity with task benchmarks, human preference comparisons and safety testing. It also cannot measure a chat model’s usefulness, since good answers are not always the most statistically likely text.
Limitations
- Not comparable across tokenizers or datasets.
- Ignores helpfulness, factual accuracy and safety.
- Can be gamed by training on data similar to the test set.
- Small perplexity differences may not matter in practice.