Sparsity in LLMs means that a large share of a model’s values are zero, or that only part of the model is used for any given input. It appears in three main forms: weight sparsity from pruning, activation sparsity that arises naturally during inference, and conditional sparsity, where a router activates only some parameters, as in mixture-of-experts models. Sparsity is attractive because computation on zeros is wasted work.
Key takeaways
- Dense models use every parameter for every token; sparse ones do not.
- Weight sparsity needs hardware or kernel support to deliver real speedups.
- Activation sparsity is common with ReLU-style activation functions.
- Conditional sparsity in MoE models is the most commercially successful form.
- 2:4 structured sparsity is supported on recent NVIDIA GPUs.
Sparsity explained in simple terms
A large hotel has 500 rooms, but on any given night only 60 are occupied. Heating every room would be wasteful. Sparsity is the recognition that most of a model, for most inputs, is effectively empty rooms, and the engineering challenge is skipping them efficiently.
Types of sparsity
| Type | What is sparse | Source | Practical benefit |
|---|---|---|---|
| Weight sparsity | Many weights are zero | Pruning | Smaller storage; speed needs support |
| Activation sparsity | Many activations are zero | ReLU-style functions | Skippable computation |
| Conditional (MoE) | Only some experts run per token | Routing | Large, proven gains |
| Attention sparsity | Each token attends to a subset | Sparse attention patterns | Cheaper long context |
| KV cache sparsity | Only important cached tokens kept | Cache eviction policies | Lower memory in long conversations |
Why zeros do not automatically mean speed
A GPU multiplying a matrix processes values in fixed blocks. Scattering zeros randomly through a matrix does not let it skip work, because the hardware still loads and multiplies the block. Only structured patterns, where the zeros follow a predictable arrangement, can be skipped. This is why 2:4 sparsity, with exactly two zeros in every group of four, exists: the hardware knows what to skip.
A worked example
A model is pruned to 50% unstructured sparsity.
- Storage: with a sparse format, roughly half the memory, though index overhead reduces the saving.
- Speed on standard GPU kernels: little or no improvement.
- Same 50% as 2:4 structured sparsity: supported GPUs can deliver meaningful speedups on matrix operations.
The lesson is that the pattern matters more than the percentage.
Conditional sparsity: the practical winner
MoE models replace the feed-forward network with many experts and route each token to only a few. A model may hold hundreds of billions of parameters in total while using only tens of billions per token. This gives the capacity of a very large model at the compute cost of a much smaller one, and is used in several leading models.
Activation sparsity
Models using ReLU produce many exact zeros in activations, and research has explored exploiting this to skip work, including loading only the needed weights from memory. Models using smoother activations such as GELU or SwiGLU produce fewer exact zeros, which makes this harder, so some research reintroduces ReLU-style functions for efficiency.
Limitations
- Unstructured sparsity rarely accelerates standard hardware.
- Structured patterns constrain which weights can be removed, costing accuracy.
- MoE models need more total memory even though compute per token is lower.
- Sparse kernels and tooling are less mature than dense equivalents.