Model pruning removes weights, neurons or whole components from a trained neural network to make it smaller and faster. The idea rests on the observation that large models are over-parameterised: many weights contribute little to the output. Pruning identifies and removes them, ideally with minimal accuracy loss.
Key takeaways
- Unstructured pruning zeroes individual weights; structured pruning removes whole units such as attention heads, channels or layers.
- Unstructured pruning preserves accuracy better but needs hardware support to deliver speedups.
- Structured pruning gives real speedups on any hardware but usually costs more accuracy.
- Modern one-shot methods such as SparseGPT and Wanda prune LLMs without retraining.
- 2:4 semi-structured sparsity is supported on recent NVIDIA GPUs for practical acceleration.
Model pruning explained in simple terms
A bonsai grower removes branches that add little to the tree’s shape. Done carefully, the tree looks the same but is lighter and easier to maintain. Done carelessly, it loses its form. Pruning a model is the same balance: cut what contributes least, keep what matters.
Types of pruning
| Type | What is removed | Speedup on standard hardware | Accuracy impact |
|---|---|---|---|
| Unstructured | Individual weights set to zero | Little without sparse kernels | Smallest |
| Semi-structured (2:4) | Two of every four weights in a block | Yes, on supported GPUs | Small |
| Structured | Heads, channels, feed-forward units | Yes, everywhere | Larger |
| Layer pruning | Whole transformer blocks | Large | Largest, but sometimes surprisingly tolerable |
How weights are chosen for removal
- Magnitude: remove the smallest weights, the classic baseline.
- Activation-aware: consider both weight size and the size of the inputs it multiplies, as in Wanda (2023).
- Second-order: estimate each weight’s effect on the loss, as in SparseGPT (2023), which can prune large models to about 50% sparsity in one pass.
- Importance scores for structured units such as heads or layers.
A worked example
A 7-billion-parameter model is pruned to 50% unstructured sparsity. Half the weights become zero, so the model can in principle be stored in roughly half the space using sparse formats. On a GPU without sparse kernel support, however, it may run no faster, because the zeros are still processed. Switching to 2:4 semi-structured sparsity, where exactly two of every four weights are zero, lets supported hardware skip work and deliver real acceleration.
Pruning vs quantization vs distillation
| Technique | What changes | Typical benefit |
|---|---|---|
| Pruning | Removes parameters | Smaller, potentially faster |
| Quantization | Lowers precision | Reliable memory and speed gains |
| Distillation | Trains a smaller model | Big reductions, needs training |
In practice, quantization is the first choice for LLMs because its benefits are more dependable. Pruning is often combined with it rather than used alone.
Recovery after pruning
Light pruning may need no repair. Heavier pruning usually requires a short recovery phase: continued training or LoRA fine-tuning on a modest dataset to regain lost accuracy.
Limitations
- Unstructured sparsity rarely speeds up standard hardware.
- Structured pruning can noticeably reduce capability.
- Effects are uneven across tasks, so broad evaluation is needed.
- For LLMs, quantization usually gives better returns for the effort.