QLoRA (quantized low-rank adaptation) is a fine-tuning method that loads a frozen base model in 4-bit precision and trains small LoRA adapters on top of it. Because the large base model is quantized and never updated, memory use drops dramatically. QLoRA was introduced by Tim Dettmers and colleagues in 2023 and made fine-tuning very large open-weight models possible on a single GPU.
Key takeaways
- QLoRA = 4-bit frozen base model + trainable LoRA adapters.
- The paper showed a 65-billion-parameter model fine-tuned on one 48 GB GPU.
- Three key techniques: NF4 quantization, double quantization and paged optimizers.
- Quality was reported to be close to 16-bit full fine-tuning on the tasks tested.
- The saved adapter is small and can be merged or swapped.
QLoRA explained in simple terms
Normal fine-tuning is like renovating a building while keeping full architectural drawings of every room in memory. QLoRA compresses the drawings into a compact form you can still read, and then only designs a few small extensions. You need a far smaller desk to work on, yet the finished building does the job.
The memory problem it solves
| Approach | Approximate GPU memory for a 7B model |
|---|---|
| Full fine-tuning in 16-bit | Roughly 70–120 GB including optimiser state |
| LoRA on a 16-bit base | Around 16–20 GB |
| QLoRA on a 4-bit base | Around 6–8 GB |
Exact figures depend on sequence length, batch size and settings, but the pattern holds: QLoRA brings large-model fine-tuning within reach of a single consumer or mid-range GPU.
The three innovations
- NF4 (4-bit NormalFloat): a 4-bit data type designed for weights that follow a roughly normal distribution, preserving accuracy better than plain 4-bit integers.
- Double quantization: the quantization constants themselves are quantized, saving additional memory per parameter.
- Paged optimizers: optimiser state can be moved between GPU and CPU memory when spikes occur, preventing out-of-memory crashes during long sequences.
How QLoRA training works
- Load the base model in 4-bit NF4 and freeze it.
- Attach LoRA adapters to attention and often feed-forward layers.
- During the forward pass, weights are dequantized on the fly for computation.
- Gradients flow to the adapters only; the base model never changes.
- Save just the adapter, typically tens to a few hundred megabytes.
A worked example
A small team wants a support assistant based on a 13-billion-parameter open model. Full fine-tuning would need multiple high-end GPUs. With QLoRA on a single 24 GB GPU, they:
- Load the model in 4-bit NF4.
- Set LoRA rank 16 with a learning rate of 2e-4.
- Use gradient accumulation to reach an effective batch size of 32.
- Train for 2 epochs on 8,000 examples.
The result is a small adapter file, deployable on top of the same quantized base model.
Trade-offs to know
- Training is slower per step than 16-bit LoRA because of dequantization work.
- Very low precision can slightly affect sensitive tasks such as maths and code.
- Serving usually runs the quantized base model, so inference quality reflects quantization too.
- Adapters are tied to the specific base model and quantization setup.
Limitations
- Still requires a GPU with enough memory for the quantized model.
- Slower throughput than full-precision training.
- Does not overcome the base model’s underlying limits.
- Quality gains depend on data, as with any fine-tuning.