Pretraining is the first and largest stage of training a large AI model, in which it learns general knowledge and patterns from vast amounts of unlabelled data. For LLMs, pretraining usually means predicting the next token across trillions of tokens of text and code. The result is a base model: broadly knowledgeable, but not yet good at following instructions or holding a helpful conversation.
Key takeaways
- Pretraining uses self-supervised learning: the data provides its own labels.
- It is by far the most expensive training stage, often needing thousands of specialised chips for weeks or months.
- Data quality and mix strongly affect the final model.
- A pretrained base model is later adapted by fine-tuning, instruction tuning and preference tuning.
- Pretraining gives a model its knowledge cut-off.
Pretraining explained in simple terms
Think of a doctor’s education. Years of broad schooling and medical college come first: general knowledge, language, science. Only later comes specialisation, such as cardiology, and bedside manner. Pretraining is the broad education. It is long and costly, and it shapes everything that follows. Later training stages add specialisation and manners.
How pretraining works
- Collect data: web pages, books, academic papers, code and other licensed or public text.
- Filter and clean: remove duplicates, spam, low-quality pages and some harmful content; balance languages and domains.
- Tokenize everything with the model’s tokenizer.
- Train: show the model sequences of tokens, ask it to predict each next token, measure error with cross-entropy loss, and update weights with backpropagation and gradient descent.
- Repeat across the whole dataset, saving checkpoints regularly.
- Evaluate the base model on benchmarks.
A worked example: one training step
The training text contains “The Reserve Bank of India sets the repo rate.” The model sees “The Reserve Bank of India sets the repo” and must predict the next token. Early in training it might give “rate” a probability of 0.001. The loss is high, so the weights are adjusted. After seeing many similar sentences, it might give “rate” 0.9. Across trillions of such predictions, the model absorbs grammar, facts, reasoning patterns and writing styles.
Scaling: data, parameters and compute
Researchers found that model performance improves predictably as parameters, data and compute increase. A 2022 DeepMind study, known as the “Chinchilla” paper, suggested many models had been undertrained and that, for a fixed compute budget, roughly 20 training tokens per parameter was a good balance. Since then, many developers have trained smaller models on far more tokens than that, because smaller models are cheaper to run after training.
Pretraining vs fine-tuning
| Aspect | Pretraining | Fine-tuning |
|---|---|---|
| Goal | Broad general knowledge | Specific behaviour or task |
| Data | Trillions of unlabelled tokens | Thousands to millions of curated examples |
| Cost | Very high | Much lower |
| Frequency | Rarely | Often |
| Output | Base model | Assistant or specialised model |
Continued pretraining
Organisations sometimes take an existing base model and continue pretraining on domain text, such as legal documents, medical papers or a specific language, before fine-tuning. This adds domain knowledge more deeply than prompting alone.
Challenges
- Cost and energy: large runs consume huge compute budgets.
- Data rights: lawsuits and debates over using copyrighted material.
- Data quality: low-quality or duplicated data wastes compute and harms results.
- Contamination: benchmark questions leaking into training data inflate scores.
- Knowledge cut-off: the model knows nothing after its data was collected.
- Bias: patterns in the data become patterns in the model.