A language model is a model that estimates how likely a sequence of words or tokens is, most often by predicting what comes next. Given “The monsoon arrived in”, a good language model gives a high probability to “June” and a low probability to “refrigerator”. This simple ability underpins autocomplete, speech recognition, translation and today’s large language models.
Key takeaways
- A language model assigns probabilities to text.
- Most generate text by predicting the next token repeatedly.
- They evolved from counting word sequences (n-grams) to neural networks and transformers.
- An LLM is a language model that is very large and broadly capable.
- Quality is often measured with perplexity: how surprised the model is by real text.
A language model explained in simple terms
Your phone’s keyboard suggests the next word as you type. After “Happy”, it suggests “birthday”. That suggestion comes from a language model: it has learned which words tend to follow others. A large language model does the same thing, but it considers much longer context and has learned far richer patterns, so it can continue a legal contract, a Python script or a poem.
How language models evolved
| Era | Approach | Strength | Weakness |
|---|---|---|---|
| 1940s–2000s | N-gram models count how often word sequences occur | Simple and fast | Only a few words of context; cannot generalise to unseen phrases |
| 2003 onwards | Neural language models with word embeddings | Generalise between similar words | Limited context |
| 2010s | Recurrent models (RNN, LSTM) | Longer context | Slow training; forgetting over long text |
| 2017 onwards | Transformers | Long context, parallel training, scales well | High compute cost |
| 2020s | Large language models | Broad, general abilities | Hallucinations, cost |
Claude Shannon used simple statistical models of English in his 1948 work on information theory. Yoshua Bengio and colleagues introduced an influential neural language model in 2003.
A worked example: a tiny n-gram model
Suppose a small training text contains “I like tea” twice and “I like coffee” once. A bigram model counts which word follows “like”:
- “tea” follows 2 of 3 times → probability 0.67
- “coffee” follows 1 of 3 times → probability 0.33
Asked to continue “I like”, it will most often say “tea”. This is a real language model, just a very small one. It cannot handle “I really enjoy” because it has never seen that phrase, which is exactly the problem neural models solved by learning that “enjoy” and “like” are similar.
Types of language models
| Type | Direction | Example use |
|---|---|---|
| Autoregressive (causal) | Predicts the next token left to right | Text generation, chat (GPT-style) |
| Masked | Predicts hidden words using both sides of context | Understanding, classification (BERT) |
| Sequence-to-sequence | Encodes input, generates new output | Translation, summarisation |
Measuring language models
Perplexity measures how well a model predicts real text; lower is better. For modern assistants, teams also measure task accuracy, factuality, safety and human preference, because low perplexity alone does not guarantee helpful answers.
Language model vs LLM vs small language model
- Language model: any model of text probabilities, big or small.
- LLM: a large neural language model, usually billions of parameters.
- Small language model: a compact neural language model designed for efficiency.
Limitations
- Language models predict plausible text, not verified truth.
- They reflect patterns and biases in training data.
- They have no direct knowledge of events after training.
- Probability of text is not the same as understanding in the human sense.