GGUF is a file format for storing quantized language models, used by llama.cpp and the tools built on it, including Ollama and LM Studio. It packages the quantized weights, the tokenizer, the chat template and metadata into a single file designed for efficient local inference on CPUs, GPUs or a mix of both. It replaced the earlier GGML format in 2023.
Key takeaways
- One self-contained file: weights, tokenizer, prompt template and configuration.
- Designed for CPU inference with optional GPU offloading of layers.
- Offers many quantization levels, named with K-quant labels such as Q4_K_M.
- The standard format for running models locally on laptops and desktops.
- Not typically used for GPU-only server deployments, where other formats are preferred.
GGUF explained in simple terms
Other formats are like a folder of parts that a program must assemble: weights here, tokenizer there, configuration somewhere else. GGUF is a single sealed box with everything inside and a label on the front. Point a local tool at the file and it runs.
What is inside a GGUF file
| Component | Purpose |
|---|---|
| Quantized tensors | The model’s compressed weights |
| Tokenizer | Vocabulary and merge rules |
| Chat template | How to format system, user and assistant turns |
| Architecture metadata | Layer counts, dimensions, attention settings |
| Quantization details | Which scheme was used for each tensor |
Because metadata travels with the model, tools can load a new model without separate configuration files.
Understanding K-quant names
A name such as Q4_K_M breaks down as:
- Q4: roughly 4 bits per weight.
- K: K-quant, a scheme that uses different precision for different parts of the model.
- M: size variant, where S is small, M is medium and L is large.
| Common type | Approximate size for a 7B model | Typical use |
|---|---|---|
| Q8_0 | about 7.2 GB | Near-lossless, when memory allows |
| Q6_K | about 5.5 GB | High quality |
| Q5_K_M | about 4.8 GB | Good balance |
| Q4_K_M | about 4.1 GB | The most common default |
| Q3_K_M | about 3.3 GB | Tight memory, noticeable quality loss |
| Q2_K | about 2.8 GB | Last resort; quality drops sharply |
Q4_K_M is the usual recommendation: a strong balance of size and quality for most local use.
Why GGUF suits local inference
- Runs on CPU, using memory mapping so files load quickly.
- Offloads as many layers to a GPU as memory allows, so a laptop GPU with 8 GB can still accelerate a larger model.
- Works across Windows, macOS and Linux, including Apple Silicon.
- Files can be downloaded and shared as a single artefact.
A worked example
A developer with a 16 GB laptop wants to run a 13-billion-parameter model. At 16-bit it would need about 26 GB, which is impossible. As Q4_K_M it is roughly 7.5 GB, leaving room for the context cache and the operating system. They offload some layers to the integrated GPU and get usable speed for local drafting and code help.
Limitations
- Aimed at local and mixed CPU/GPU inference, not high-throughput server serving.
- Quality varies significantly across quant types, so choose carefully.
- Converting to GGUF requires a conversion step from the original weights.
- Support for new architectures can lag behind their release.