A loss function is a mathematical formula that measures how far a machine learning model’s predictions are from the correct answers. It turns “how wrong is the model?” into a single number. Training works by adjusting the model’s parameters, usually with gradient descent, to make that number as small as possible.
Key takeaways
- The loss function defines what “good” means for a model during training.
- Mean squared error is standard for regression; cross-entropy is standard for classification.
- Loss functions are for training; evaluation metrics are for reporting to people. They can differ.
- Choosing the wrong loss makes the model optimise the wrong goal.
A loss function explained in simple terms
A darts player needs a score to know how well they are doing. A loss function is the model’s scorecard, except lower is better. Each prediction is compared with the true answer, the size of the miss is calculated, and the misses are combined into one number. The model then adjusts itself to reduce that number next time.
Loss vs cost vs objective
- Loss function: error for a single example.
- Cost function: average loss across the whole dataset or batch, sometimes plus regularisation.
- Objective function: the general term for whatever is being optimised, minimised or maximised.
In practice, many people use “loss” and “cost” interchangeably.
Common loss functions
| Loss function | Task | What it does |
|---|---|---|
| Mean squared error (MSE) | Regression | Averages squared errors; heavily punishes large mistakes |
| Mean absolute error (MAE) | Regression | Averages absolute errors; less sensitive to outliers |
| Huber loss | Regression | Behaves like MSE for small errors and MAE for large ones |
| Binary cross-entropy (log loss) | Binary classification | Punishes confident wrong probabilities strongly |
| Categorical cross-entropy | Multi-class classification | Compares predicted class probabilities with the true class |
| Hinge loss | Classification (SVMs) | Encourages a margin between classes |
| Focal loss | Imbalanced classification | Focuses on hard, misclassified examples |
| Contrastive and triplet loss | Similarity learning | Pulls similar items together and pushes different ones apart |
Large language models are trained with cross-entropy on next-token prediction: the loss measures how much probability the model gave to the actual next word.
How a loss function guides training
- The model makes predictions on a batch of training data.
- The loss function scores the error.
- The gradient of the loss shows how to change each parameter to reduce it.
- The optimiser updates the parameters.
- Training and validation loss are tracked to spot overfitting.
Loss function vs evaluation metric
The loss must usually be smooth and differentiable so gradient descent can use it. Business metrics often are not. For example, a fraud model may be trained with cross-entropy but judged on recall at a fixed false-alarm rate. The two should be aligned, but they are not always the same thing. If loss falls but the business metric does not improve, the loss may not match the real goal.
How to choose a loss function
- Match the task: numbers, categories, rankings or similarity.
- Consider outliers: MSE is sensitive; MAE and Huber are more robust.
- Consider imbalance: weighted cross-entropy or focal loss help rare classes.
- Consider error costs: if under-predicting stock is worse than over-predicting, use an asymmetric loss such as quantile loss.
- Add regularisation terms to discourage overly complex models.
Limitations
A model optimises exactly what the loss measures, nothing more. A poorly chosen loss can produce a model that scores well mathematically but fails the real goal. Loss values are also hard to interpret on their own; compare them over time and against baselines rather than in isolation.