L1 regularization adds a penalty equal to the sum of the absolute values of a model’s weights. Its defining property is sparsity: it drives some weights to exactly zero, effectively removing those features from the model. In linear regression this is known as lasso, introduced by Robert Tibshirani in 1996.
Key takeaways
- Penalty = λ × sum of |weights|.
- Some weights become exactly zero, so L1 performs feature selection.
- It suits problems with many features where only a few genuinely matter.
- With correlated features it tends to keep one and discard the others.
- Elastic net combines L1 and L2 to get sparsity with more stability.
L1 explained in simple terms
Imagine packing for a trip with a strict rule: pay a fixed fee for every item you bring, regardless of weight. You will leave behind anything not clearly worth its fee, ending with a short list of genuinely useful things. L1 charges that flat fee per feature, so marginal features get dropped entirely.
How it works
Training normally minimises prediction error alone. With L1, it minimises:
total loss = prediction error + λ × Σ|wᵢ|
Because the penalty’s gradient is constant regardless of how small a weight becomes, there is always the same downward pressure. A weight whose predictive contribution is smaller than that pressure is pushed to exactly zero and stays there. L2, by contrast, shrinks weights proportionally, so they approach zero without reaching it.
A worked example
A model predicting house prices starts with ten features. As λ increases:
| λ | Non-zero features | Validation error |
|---|---|---|
| 0 | 10 | 0.182 |
| 0.01 | 9 | 0.176 |
| 0.1 | 6 | 0.171 |
| 1.0 | 3 | 0.189 |
| 10 | 0 | 0.412 |
At λ = 0.1 the model keeps six features and performs best: four weak features were adding noise. At λ = 1.0 it has discarded useful signal. At λ = 10 everything is zero and the model predicts a constant. The chosen λ comes from cross-validation, not intuition.
When to use L1
- You have many features and suspect most are irrelevant.
- You want an interpretable model with a short feature list.
- Feature collection or computation is expensive, so dropping features saves money.
- You need a compact model for deployment.
- You are exploring which features matter before building a final model.
When L2 is better
- Most features contribute a little and you want to keep them all.
- Features are highly correlated and you do not want arbitrary selection.
- You are training neural networks, where weight decay (L2) is standard.
- Stability across retraining matters more than sparsity.
L1 vs L2 vs elastic net
| Property | L1 (lasso) | L2 (ridge) | Elastic net |
|---|---|---|---|
| Penalty | Sum of absolute weights | Sum of squared weights | Mix of both |
| Produces zeros | Yes | No | Yes |
| Feature selection | Built in | None | Built in |
| Correlated features | Picks one arbitrarily | Spreads weight across them | Selects groups together |
| Stability across resamples | Lower | Higher | Higher than L1 |
| Typical use | Sparse, interpretable models | General shrinkage, neural networks | Many correlated features |
Practical notes
- Scale features first. L1 penalises coefficient size, so unscaled features are penalised unequally.
- Tune λ on validation data, trying values on a log scale.
- Report the selected features, since that list is a useful output in itself.
- Check stability: re-run on bootstrap samples; if the selected set changes a lot, prefer elastic net.
- In neural networks, L1 is used far less than L2; sparsity there is more often achieved through pruning.
Limitations
- Selection can be unstable with correlated features.
- With more features than observations, it selects at most as many features as there are rows.
- The zeros are a modelling convenience, not proof a feature is irrelevant.
- Less useful for deep networks than for linear models.