L2 regularization adds a penalty equal to the sum of the squared weights, shrinking them smoothly towards zero without setting any to exactly zero. In linear models it is called ridge regression; in neural networks it is usually called weight decay. It is the most widely used form of regularization in deep learning.
Key takeaways
- Penalty = λ × sum of weights squared.
- It shrinks all weights proportionally; none become exactly zero.
- It handles correlated features gracefully by spreading weight across them.
- In neural networks it appears as weight decay, standard in AdamW.
- It reduces variance at the cost of a small increase in bias.
L2 explained in simple terms
Packing for a trip where you pay by total weight squared: a few heavy items cost far more than many light ones. You therefore spread the load, bringing lighter versions of everything rather than dropping items entirely. L2 pushes models towards many small weights instead of a few large ones, which tends to generalise better.
How it works
Training minimises:
total loss = prediction error + λ × Σwᵢ²
Because the penalty’s gradient is proportional to the weight itself, large weights are pushed down hard and small weights barely at all. Weights therefore shrink towards zero asymptotically but never arrive, which is why L2 does not perform feature selection.
A worked example
Two candidate solutions fit the training data almost equally well:
| Model | Weights | Σw² | Penalty at λ = 0.01 |
|---|---|---|---|
| A | 2, −1, 3 | 4 + 1 + 9 = 14 | 0.14 |
| B | 25, −30, 18 | 625 + 900 + 324 = 1,849 | 18.49 |
Model B’s enormous opposing weights are a classic sign of fitting noise, often triggered by correlated features. L2 makes B’s total loss far worse, so training prefers A. The result is a smoother, more stable model.
Why large weights are a problem
Large weights make a model’s output highly sensitive to small input changes. A tiny measurement difference can swing the prediction. Keeping weights small produces smoother decision boundaries, which usually generalise better to new data and are more robust to noise.
L2 in neural networks: weight decay
In deep learning, L2 appears as weight decay, typically with values between 0.01 and 0.1 for large models and smaller for small ones. An important detail: with adaptive optimisers such as Adam, adding L2 to the loss does not behave as intended because the adaptive scaling distorts it. AdamW, proposed in 2017, decouples weight decay from the gradient update and fixes this. AdamW is now the standard optimiser for training large models.
L1 vs L2
| Property | L1 | L2 |
|---|---|---|
| Penalty | Sum of absolute weights | Sum of squared weights |
| Effect on weights | Some become exactly zero | All shrink, none vanish |
| Feature selection | Yes | No |
| Correlated features | Picks one | Shares weight between them |
| Solution stability | Lower | Higher |
| Neural network use | Rare | Standard as weight decay |
| Closed-form solution | No | Yes, for linear regression |
Choosing λ
| λ | Effect |
|---|---|
| 0 | No regularisation; risk of overfitting |
| Small (0.0001–0.01) | Light smoothing; common for neural networks |
| Moderate (0.01–1) | Noticeable shrinkage |
| Large (10+) | Underfitting; weights crushed towards zero |
Tune it on validation data using a logarithmic range, and scale features first so the penalty applies evenly.
When L2 is the right choice
- Features are correlated, which is common in real tabular data.
- All features plausibly contribute something.
- You are training neural networks, where it is the default.
- You want stable results across retraining.
- Multicollinearity makes unregularised linear regression unstable.
Limitations
- No feature selection, so the model keeps every input.
- Requires scaled features.
- Too much λ causes underfitting.
- Does not remove the need for other regularisation such as dropout or early stopping.