An AI model checkpoint is a saved snapshot of a model at a specific point in time, containing its learned weights and often the extra information needed to continue training. Training large models takes days or months, so teams save checkpoints regularly. They are used to recover from failures, compare progress, pick the best version, share models publicly and deploy them for inference.
Key takeaways
- A checkpoint captures model weights, and a full training checkpoint also includes optimiser and training state.
- Checkpoints protect long training runs from hardware failures.
- The “best” checkpoint is often chosen by validation performance, not simply the last one.
- Common formats include safetensors, PyTorch files and GGUF.
- Loading checkpoints from untrusted sources can be a security risk with some formats.
A checkpoint explained in simple terms
Think of saving your progress in a long video game. If the power goes out, you restart from your last save instead of the beginning. You might also keep several saves and go back to one where things were going better. In AI training, checkpoints are those save points.
What is inside a checkpoint
| Component | Purpose | Needed for |
|---|---|---|
| Model weights | The learned parameters | Inference and further training |
| Optimiser state | For example, Adam’s running averages for each parameter | Resuming training smoothly |
| Training progress | Step number, epoch, learning rate schedule | Resuming at the right point |
| Random states | Seeds for reproducibility | Exact resumption |
| Configuration | Architecture and settings | Rebuilding the model correctly |
| Tokenizer files | Vocabulary and rules | Using language models correctly |
A weights-only checkpoint is much smaller than a full training checkpoint, because optimisers such as Adam store extra values for every parameter.
A worked example: size of checkpoints
A 7-billion-parameter model stored in 16-bit precision has weights of about 14 GB. A full training checkpoint that also stores higher-precision master weights and Adam’s two running averages per parameter can be several times larger, often around 80–110 GB. That is why teams usually share weights-only checkpoints publicly and keep full training checkpoints internally.
How checkpoints are used
- Fault tolerance: resume after crashes, which are common in runs across thousands of GPUs.
- Model selection: evaluate several checkpoints and pick the one with the best validation score, which helps avoid overfitting.
- Research: study how abilities emerge during training.
- Starting points: a pretrained checkpoint is the base for fine-tuning.
- Versioning and rollback: return to an earlier checkpoint if a new one performs worse.
- Distribution: open-weight models are shared as checkpoint files on platforms such as Hugging Face.
Common checkpoint formats
| Format | Notes |
|---|---|
| .pt / .pth / .bin (PyTorch) | Traditional format; older ones use Python pickle, which can execute code when loaded |
| .safetensors | Developed by Hugging Face; stores tensors safely without executable code and loads fast |
| .gguf | Used by llama.cpp and Ollama for quantized local models |
| TensorFlow checkpoints | Used in TensorFlow and Keras workflows |
| ONNX | Portable format for deployment across runtimes |
Security note
Pickle-based checkpoint files can run arbitrary code when loaded, so a malicious checkpoint could compromise a machine. Prefer safetensors, download only from trusted sources, and use tools that scan model files.
Checkpoint vs model vs adapter
- Checkpoint: a saved state at a point in training.
- Model: the architecture plus a chosen checkpoint’s weights, ready to use.
- Adapter: a small add-on, such as a LoRA file, applied to a base checkpoint.
Limitations
- Large checkpoints require significant storage and transfer time.
- Checkpoints only work with compatible architecture and code versions.
- Frequent saving slows training; infrequent saving risks lost work.
- Unsafe formats pose security risks.