Cross-entropy loss measures how far a model’s predicted probabilities are from the correct answer. For each example, it is the negative logarithm of the probability the model gave to the true class or token. Confident correct predictions produce a small loss; confident wrong predictions produce a very large one. It is the standard loss function for classification and for training LLMs.
Key takeaways
- Loss for one example = −log(probability assigned to the correct answer).
- It punishes confident mistakes heavily.
- Average cross-entropy over tokens is what LLM training minimises.
- Perplexity is simply the exponential of average cross-entropy.
- It is computed from logits for numerical stability.
Cross-entropy explained in simple terms
Imagine a weather forecaster. If they say 90% chance of rain and it rains, they scored well: small penalty. If they said 10% and it rains, the penalty is much larger. If they had said 1%, larger still. Cross-entropy is that penalty system, and training pushes the model to avoid confident mistakes.
The formula and a worked example
For a single prediction, loss = −ln(p), where p is the probability assigned to the correct answer.
| Probability given to the truth | Loss |
|---|---|
| 0.99 | 0.01 |
| 0.90 | 0.105 |
| 0.50 | 0.693 |
| 0.25 | 1.386 |
| 0.10 | 2.303 |
| 0.01 | 4.605 |
The curve rises steeply as probability approaches zero, which is why models learn quickly from confident errors.
In an LLM: if the true next token is “Mumbai” and the model gave it 0.30, the loss at that position is −ln(0.30) ≈ 1.20. Training averages this across every token in the batch.
Why cross-entropy rather than accuracy?
Accuracy only counts right or wrong, so it gives no gradient to learn from: nudging a weight slightly rarely flips a prediction. Cross-entropy responds smoothly to probability changes, so even small improvements produce a useful learning signal. It is also differentiable, which gradient descent requires.
Variants
| Variant | Use |
|---|---|
| Binary cross-entropy | Two classes, or multi-label problems |
| Categorical cross-entropy | One correct class among many |
| Label smoothing | Softens targets slightly to reduce overconfidence |
| Weighted cross-entropy | Gives rare classes more importance |
| Focal loss | Focuses on hard examples in imbalanced data |
Cross-entropy and perplexity
Perplexity = exp(average cross-entropy loss). A loss of 2.0 corresponds to a perplexity of about 7.4; a loss of 1.0 corresponds to about 2.7. Training curves for LLMs are usually shown as loss, while evaluation reports often use perplexity, but they carry the same information.
Practical notes
- Loss is computed from logits using stable implementations rather than computing softmax first.
- In LLM fine-tuning, loss is often applied only to response tokens, not the prompt.
- Padding tokens are masked out so they do not affect the loss.
Limitations
- Assumes exactly one correct answer, which is a poor fit when several continuations are equally valid.
- Encourages matching training text, not truth or helpfulness.
- Very confident wrong predictions can produce huge gradients, which is one reason for gradient clipping.
- A low loss does not guarantee a useful assistant.