8-bit quantization stores a model’s weights using eight bits each, halving memory compared with 16-bit precision while typically preserving quality almost exactly. It is the conservative choice when you want meaningful savings without the quality risk of 4-bit. Eight bits allow 256 distinct values per weight, which is enough for most of what a model needs.
Key takeaways
- 256 levels per weight, versus 16 at 4-bit.
- A 7-billion-parameter model needs about 7 GB for weights at 8-bit.
- Usually close to lossless for general tasks.
- Outlier values are the main technical challenge, solved by mixed-precision handling.
- A good default when quality matters more than squeezing the last gigabyte.
8-bit quantization explained in simple terms
Rounding prices to the nearest rupee instead of tracking paise. Totals across thousands of transactions barely change, the ledger is half the size, and nobody notices. Eight-bit quantization is that level of rounding for model weights.
The memory picture
| Model size | 16-bit | 8-bit | Saving |
|---|---|---|---|
| 7B | about 14 GB | about 7 GB | 7 GB |
| 13B | about 26 GB | about 13 GB | 13 GB |
| 70B | about 140 GB | about 70 GB | 70 GB |
A 70-billion-parameter model at 8-bit fits on a single high-memory data-centre GPU, which is often the deciding factor for deployment.
The outlier problem
Research by Tim Dettmers and colleagues in 2022 (LLM.int8()) found that large transformer models contain rare outlier features: a small number of dimensions with unusually large values. Quantizing these naively to 8-bit damages accuracy badly. Their solution was mixed precision: keep the small number of outlier dimensions in 16-bit and quantize the rest to 8-bit. This made 8-bit inference near-lossless for large models and is widely implemented in libraries such as bitsandbytes.
A worked example
A group of weights ranges from −0.5 to +0.5, quantized to signed 8-bit with 255 usable levels.
- Step size: 1.0 ÷ 255 ≈ 0.0039.
- A weight of 0.3127 maps to the nearest level, about 0.3138.
- Error: roughly 0.001, about four times smaller than typical 4-bit error in the same range.
8-bit vs 4-bit: choosing
| Consideration | 8-bit | 4-bit |
|---|---|---|
| Memory saving vs 16-bit | Half | About a quarter |
| Quality loss | Very small | Small to moderate, task-dependent |
| Maths and code | Reliable | More affected |
| Local laptop use | Sometimes too large | Usually the practical choice |
| Production serving where quality matters | Preferred | Use with testing |
Beyond weights
8-bit is also used for activations in some setups, and for KV cache compression, which reduces memory growth with long context windows. FP8 formats, supported on newer GPUs, are increasingly used for both training and inference at similar precision levels.
Limitations
- Half the savings of 4-bit, so very large models may still need multiple GPUs.
- Speed gains depend on hardware support for 8-bit operations.
- Outlier handling adds implementation complexity.
- Still a lossy process, so accuracy should be validated on your tasks.