Model quantization is a technique that stores and computes a model’s numbers, mainly its weights, with fewer bits, for example 8-bit or 4-bit instead of 16-bit. This shrinks the model’s memory footprint and often speeds up inference, making it possible to run large models on smaller GPUs, laptops and phones. The trade-off is usually a small loss in accuracy.
Key takeaways
- Fewer bits per number means smaller, cheaper, often faster models.
- A 7-billion-parameter model needs about 14 GB at 16-bit, 7 GB at 8-bit and about 3.5 GB at 4-bit, for weights alone.
- Post-training quantization is fast; quantization-aware training preserves more quality.
- Popular methods and formats include GPTQ, AWQ, bitsandbytes and GGUF.
- Very low bit levels, such as 2–3 bits, can noticeably hurt quality.
Model quantization explained in simple terms
Imagine storing prices as ₹1,249.3781 when ₹1,249 is good enough. Rounding saves space with little practical loss. Quantization rounds a model’s billions of numbers to simpler values that fit in fewer bits. Do it carefully and the model behaves almost the same. Round too aggressively and it starts making more mistakes.
The memory maths
| Precision | Bytes per parameter | 7B model (weights only) | 70B model (weights only) |
|---|---|---|---|
| FP32 (32-bit) | 4 | about 28 GB | about 280 GB |
| FP16 / BF16 (16-bit) | 2 | about 14 GB | about 140 GB |
| INT8 (8-bit) | 1 | about 7 GB | about 70 GB |
| 4-bit | 0.5 | about 3.5 GB | about 35 GB |
Real memory use is higher, because running a model also needs space for the key-value cache and activations. Still, 4-bit quantization can let a 70-billion-parameter model fit on hardware that could never hold it at 16-bit.
How quantization works
A group of weights is mapped from a wide range of precise values onto a small set of levels. For 4-bit, there are only 16 possible levels. Each group stores a scale (and sometimes a zero point) to convert levels back to approximate real values during computation.
Worked example: suppose a group of weights ranges from −1.0 to +1.0 and we use 8-bit integers (−127 to 127). The scale is 1.0 ÷ 127 ≈ 0.00787. A weight of 0.4321 becomes round(0.4321 ÷ 0.00787) = 55, stored as the integer 55. When used, it is converted back: 55 × 0.00787 ≈ 0.4329. The error, less than 0.001, is tiny, and across billions of weights such small errors usually have little effect.
Types of quantization
| Type | When applied | Notes |
|---|---|---|
| Post-training quantization (PTQ) | After training | Fast and cheap; most common for LLMs |
| Quantization-aware training (QAT) | During training | Model learns to cope with low precision; better quality, more effort |
| Weight-only | Weights quantized, activations kept higher precision | Common for LLM inference |
| Weight and activation | Both quantized | Bigger speed gains; harder to keep quality |
Popular methods and formats
- GPTQ (2022): post-training method using calibration data to minimise error.
- AWQ (2023): protects the most important weights based on activations.
- bitsandbytes: a library for 8-bit and 4-bit loading, used in QLoRA fine-tuning.
- GGUF: a file format used by llama.cpp and tools like Ollama to run quantized models locally on CPUs and consumer GPUs.
Quantization vs distillation vs pruning
- Quantization: same model, fewer bits per number.
- Distillation: train a smaller model to imitate a larger one.
- Pruning: remove weights or parts of the network.
These are often combined.
Limitations
- Quality loss increases at very low bit widths.
- Some tasks, such as maths, code and less common languages, can be more sensitive.
- Speed gains depend on hardware support for low-precision maths.
- Different formats and tools are not always compatible.