Self-supervised learning is a machine learning approach in which a model learns from unlabelled data by creating its own training labels from the data itself. For example, the model hides part of a sentence and learns to predict the missing word. No human labelling is needed, so it can learn from enormous amounts of raw text, images or audio. It is the main training method behind large language models.
Key takeaways
- The data provides its own supervision, so no manual labels are required.
- Models solve “pretext tasks”, such as predicting missing or next words, to learn general patterns.
- It is usually a pre-training step, followed by fine-tuning for specific tasks.
- It enabled foundation models such as BERT and GPT-style language models.
Self-supervised learning explained in simple terms
Imagine learning a language by reading books where some words are covered. You guess the hidden word from context, then lift the cover to check. After millions of guesses, you understand grammar, meaning and even facts about the world. Nobody taught you directly. The text itself provided the questions and answers. That is self-supervised learning.
How does self-supervised learning work?
- Gather large unlabelled data: web text, images, video or audio.
- Create a pretext task that hides or changes part of the data.
- Train the model to predict or reconstruct the hidden part, reducing a loss function.
- Keep the learned representations: the internal patterns the model built.
- Fine-tune on a smaller labelled dataset for the real task, or use the model directly.
Common self-supervised methods
| Method | Pretext task | Known example |
|---|---|---|
| Next-token prediction | Predict the next word in a sequence | GPT-style language models |
| Masked language modelling | Predict hidden words in a sentence | BERT (2018) |
| Masked image modelling | Reconstruct hidden patches of an image | Masked autoencoders |
| Contrastive learning | Recognise two altered views of the same item as similar | SimCLR, CLIP |
| Audio prediction | Predict masked parts of speech audio | Speech representation models |
Why it changed AI
Labelled data was the main bottleneck in supervised learning. Self-supervised learning removed that limit. Models could now learn from almost any raw data at massive scale. The result was foundation models: large, general-purpose models that can be adapted to many tasks with little extra labelled data. AI researcher Yann LeCun has long argued that self-supervised learning is key to more capable AI.
Self-supervised vs supervised vs unsupervised
Self-supervised learning technically uses no human labels, like unsupervised learning. But it trains exactly like supervised learning, with an input and a target to predict. The difference is that the target comes from the data. Some people treat it as a type of unsupervised learning; others treat it as its own category. It also differs from semi-supervised learning, which uses some human labels.
Pre-training and fine-tuning
A typical pattern has two stages. First, pre-training with self-supervised learning on huge datasets, which is expensive and done rarely. Second, fine-tuning on smaller labelled data for specific tasks, which is cheaper and done often. Chat assistants add further steps, such as instruction tuning and reinforcement learning from human feedback.
Limitations
Self-supervised pre-training needs massive data and computing power. Models absorb whatever is in the data, including errors, bias and outdated facts. Good pretext-task performance does not guarantee good performance on the real task. And because training data is so large, it is hard to audit exactly what a model learned.