Next-token prediction is the training objective behind most modern LLMs: given a sequence of tokens, predict the token that comes next. The model outputs a probability for every token in its vocabulary, and training adjusts its weights so that the true next token gets a higher probability. Repeated across trillions of tokens, this simple task produces grammar, facts, reasoning patterns and style.
Key takeaways
- The objective is simple, but at scale it produces broad capability.
- Every token in the training text is a training example.
- Loss is measured with cross-entropy, which relates directly to perplexity.
- The same mechanism drives generation: predict, append, repeat.
- Critics argue prediction alone is not understanding; the debate continues.
Next-token prediction explained in simple terms
Play a game where someone reads a sentence and stops mid-way, and you must guess the next word. To win consistently, you need grammar, vocabulary, world knowledge, and a sense of tone and logic. A model plays this game constantly during pretraining. Improving at it forces it to learn the structure of language and much of what language describes.
How one training step works
- Take a sequence, for example “The capital of Maharashtra is”.
- The model processes it and produces logits, one score per vocabulary token.
- Softmax converts logits into a probability distribution.
- Cross-entropy loss is the negative log probability assigned to the true token, “Mumbai”.
- Backpropagation and gradient descent adjust weights so that probability rises next time.
A worked example
Suppose the model gives “Mumbai” a probability of 0.30. The loss is −log(0.30) ≈ 1.20. After training on many similar sentences, the probability rises to 0.85, giving a loss of −log(0.85) ≈ 0.16. Lower loss means better prediction. Average loss across a dataset, exponentiated, gives perplexity.
Note that during training the model predicts every position at once. In a 2,048-token sequence, one forward pass produces 2,048 predictions and 2,048 loss values.
From training to generation
The same operation powers text generation, one token at a time:
- Predict a distribution for the next token.
- Select a token using temperature, top-p or greedy decoding.
- Append it and repeat.
This is autoregressive generation.
Why such a simple objective works so well
To predict the next token accurately across diverse text, a model must implicitly learn syntax, facts, arithmetic patterns, code structure, dialogue conventions and more. Skills that were not explicitly trained, such as translating between languages or following instructions from examples, emerge as side effects of getting better at prediction at scale.
Criticism and debate
Some researchers argue that predicting text is not the same as understanding it, and that models can produce fluent output without grounded meaning. Others point to strong performance on reasoning tasks as evidence of deeper internal structure. Both sides agree on the practical conclusion: outputs should be verified, since fluency is not accuracy.
Limitations
- Optimises plausibility, not truth, which contributes to hallucination.
- Knowledge is frozen at the training cut-off.
- Errors early in generation shape everything after.
- Token-level prediction causes quirks with letters, numbers and rare words.