4-bit quantization stores a model’s weights using four bits each instead of 16, reducing memory use to roughly a quarter with a modest quality cost. It is the level that makes large open-weight models practical on consumer GPUs and laptops, and it underpins QLoRA fine-tuning.
Key takeaways
- Four bits allow only 16 distinct values per weight, plus a scale per group.
- A 7-billion-parameter model needs about 3.5 GB for weights at 4-bit, versus 14 GB at 16-bit.
- Common formats include NF4, GPTQ 4-bit, AWQ 4-bit and GGUF K-quants.
- Quality loss is usually small for general chat, larger for maths and code.
- Group size and calibration strongly affect results.
4-bit quantization explained in simple terms
Imagine recording every measurement in a workshop to the nearest millimetre rather than the nearest micrometre. Almost every job still comes out fine, the notebook is far smaller, and only the most precise work suffers. Four-bit quantization makes that trade for model weights.
The memory maths
| Model size | 16-bit weights | 8-bit | 4-bit |
|---|---|---|---|
| 7B | about 14 GB | 7 GB | about 3.5 GB |
| 13B | about 26 GB | 13 GB | about 6.5 GB |
| 70B | about 140 GB | 70 GB | about 35 GB |
Real usage is higher because of the attention cache and activations, but the pattern is what matters: 4-bit turns a multi-GPU model into a single-GPU one.
How 16 levels can be enough
Weights are quantized in groups, often 32, 64 or 128 values, each with its own scale factor. Within a group, weights are mapped to 16 levels spread across the group’s range. Because weights within a small group tend to be similar in magnitude, the error per weight stays small. Smaller groups mean better accuracy and slightly more overhead for storing scales.
Worked example: a group of 64 weights ranging from −0.12 to +0.12 is mapped onto 16 levels spaced about 0.016 apart. A weight of 0.071 becomes the nearest level, 0.072, an error of 0.001. Averaged over billions of weights, such errors have limited effect on output.
Common 4-bit formats
| Format | Notes |
|---|---|
| NF4 | 4-bit NormalFloat, designed for normally distributed weights; used in QLoRA |
| GPTQ | Post-training method using calibration data to minimise error |
| AWQ | Activation-aware; protects the most important weights |
| GGUF K-quants | Mixed-precision schemes such as Q4_K_M, widely used in llama.cpp |
Quality impact
For general conversation and writing, 4-bit models are often hard to distinguish from 16-bit. Degradation is more noticeable in:
- Multi-step maths and arithmetic.
- Code generation and debugging.
- Long-context reasoning.
- Less-resourced languages.
A useful rule from practical experience: a larger model quantized to 4-bit usually beats a smaller model at 16-bit for the same memory budget, though this should be tested on your own tasks.
When to use 4-bit
- Running models locally on consumer hardware.
- Serving large models on limited GPUs.
- Fine-tuning with QLoRA.
- Cost-sensitive, high-volume deployments where small quality losses are acceptable.
Limitations
- Clear quality loss on precision-sensitive tasks.
- Speed gains depend on hardware support for low-precision operations.
- Formats are not interchangeable across tools.
- Quantized checkpoints must match the runtime that produced them.