GPTQ is a post-training quantization method that compresses a model’s weights to 3 or 4 bits in a single pass, using a small calibration dataset to minimise the error introduced. Published by Elias Frantar and colleagues in 2022, it made accurate 4-bit quantization of very large models practical without any retraining.
Key takeaways
- One-shot: no retraining or fine-tuning required.
- Uses a few hundred calibration samples to guide quantization.
- Works layer by layer, adjusting remaining weights to compensate for errors already introduced.
- Typically 3-bit or 4-bit, with group sizes such as 128.
- AWQ is the main alternative, with a different strategy.
GPTQ explained in simple terms
Imagine rounding every number in a spreadsheet, but after each rounding you slightly adjust the remaining numbers so the totals still come out right. Naive rounding accumulates error; this approach keeps correcting as it goes. GPTQ does that across a model’s weight matrices.
How it works
- Run a small calibration set, often a few hundred text samples, through the model to record how each layer behaves.
- Process one layer at a time, and within it, quantize weights in order.
- After quantizing each weight, update the not-yet-quantized weights in that layer to compensate for the error introduced.
- Use second-order information, derived from the Hessian, to decide how to distribute that compensation.
- Move to the next layer with the quantized outputs of the previous one.
The method descends from earlier optimal brain compression research, applied efficiently enough to handle billions of parameters in hours rather than weeks.
Key settings
| Setting | Meaning | Typical value |
|---|---|---|
| Bits | Precision per weight | 4, sometimes 3 or 8 |
| Group size | Weights sharing a scale factor | 128, sometimes 64 or 32 |
| Act-order (desc_act) | Quantize the most important columns first | Often improves accuracy slightly |
| Calibration data | Sample text used to guide quantization | 128–512 samples from relevant text |
Smaller group sizes improve accuracy and slightly increase file size. Calibration data that matches your domain, for example code or Hindi text, usually produces better results for that domain.
GPTQ vs AWQ vs GGUF
| Aspect | GPTQ | AWQ | GGUF |
|---|---|---|---|
| Approach | Error-compensating layer-wise quantization | Protects salient weights using activation statistics | File format with several K-quant schemes |
| Calibration needed | Yes | Yes | Depends on scheme |
| Typical target | GPU inference | GPU and edge inference | CPU and mixed CPU/GPU |
| Ecosystem | vLLM, Transformers, text-generation tools | vLLM, Transformers, edge runtimes | llama.cpp, Ollama, LM Studio |
Practical notes
- Quantizing a 7-billion-parameter model typically takes tens of minutes to a few hours on a single GPU.
- Always evaluate the quantized model on your own tasks; benchmark perplexity alone can hide task-specific losses.
- Many popular open models are available pre-quantized with GPTQ, so you may not need to run it yourself.
- Serving engines must support the format, so check compatibility before committing.
Limitations
- Results depend on calibration data quality and relevance.
- Accuracy degrades noticeably at 3 bits and below.
- Quantization is not free: the process itself needs a GPU and time.
- Format compatibility varies across inference engines.