Continued pretraining is the practice of taking an existing pretrained model and training it further on large amounts of new unlabelled text, usually from a specific domain or language, before any task fine-tuning. It uses the same next-token prediction objective as the original pretraining, so no labels are needed. When the new data is domain-specific, it is often called domain-adaptive pretraining (DAPT).
Key takeaways
- It deepens knowledge; fine-tuning shapes behaviour.
- It needs large volumes of raw text, typically hundreds of millions to billions of tokens.
- It is far cheaper than pretraining from scratch but far costlier than fine-tuning.
- Mixing in general data helps prevent catastrophic forgetting.
- Influential research includes “Don’t Stop Pretraining” (Gururangan and colleagues, 2020).
Continued pretraining explained in simple terms
A general model is like a well-read graduate. Fine-tuning is teaching them your company’s procedures. Continued pretraining is sending them to read every textbook, case file and journal in your industry first. They absorb the vocabulary, conventions and background knowledge, so later training goes further.
When to use it
| Situation | Is continued pretraining a good fit? |
|---|---|
| The domain has specialised vocabulary the model handles poorly (law, medicine, chemistry) | Yes |
| You need better support for a language with limited coverage | Yes |
| You have large amounts of unlabelled domain text | Yes |
| You want a specific tone or output format | No, use fine-tuning |
| You need current facts from documents | No, use retrieval |
| You have only a few thousand examples | No, use fine-tuning or LoRA |
A worked example
An Indian legal-tech company finds that a general model misreads statute references and court formats. They gather 3 billion tokens of Indian judgements, statutes and legal commentary, mixed with about 30% general text to preserve broad ability. They continue pretraining an open-weight model for a few thousand GPU hours, then apply supervised fine-tuning on 5,000 legal question-answer pairs. The final model understands legal language far better than fine-tuning alone would achieve.
The typical pipeline
- Collect and clean a large domain corpus (data curation).
- Mix in general-purpose data, often a significant share, to limit forgetting.
- Continue training with a lower learning rate than original pretraining, with warmup.
- Monitor perplexity on domain and general validation sets.
- Follow with instruction fine-tuning and preference tuning, since continued pretraining alone produces a base model.
Practical guidance
- Consider extending the vocabulary if the domain or script tokenises poorly, then train the new embeddings.
- Deduplicate aggressively; repeated documents encourage memorisation.
- Remove benchmark data to avoid contamination.
- Save checkpoints often, and evaluate several, not just the last.
Alternatives to consider first
Retrieval-augmented generation handles changing facts better and costs far less. Fine-tuning handles format and style. Continued pretraining is worth the expense mainly when the model’s underlying understanding of the domain language is the bottleneck.
Limitations
- Expensive in compute and data preparation.
- Risks forgetting general abilities.
- Produces a base model that still needs instruction tuning.
- Knowledge still becomes outdated, so it does not replace retrieval.