Greedy decoding is the simplest generation strategy: at each step, the model picks the token with the highest probability. There is no randomness and no lookahead. It is fast, reproducible and often the right choice for factual or structured tasks, but it can produce repetitive or bland text in open-ended writing.
Key takeaways
- Greedy decoding takes the top token every step, equivalent to temperature 0.
- It is deterministic in principle, though small hardware variations can still occur.
- It can get stuck in repetition loops.
- It does not guarantee the most likely overall sentence, only the most likely next token.
- Beam search and sampling methods address different weaknesses.
Greedy decoding explained in simple terms
Imagine walking through a maze and always taking the path that looks best right now, without considering where it leads. Often you reach the exit. Sometimes you walk confidently into a dead end that a little foresight would have avoided. Greedy decoding makes each choice locally optimal, not globally.
How it works
- The model produces a probability distribution over the vocabulary.
- Select the token with the highest probability.
- Append it and repeat until an end token or length limit.
Worked example: after “The capital of Karnataka is”, probabilities might be: ” Bengaluru” 0.88, ” Bangalore” 0.09, ” a” 0.01. Greedy decoding always chooses ” Bengaluru”. Run it a hundred times and you get the same answer every time, which is exactly what you want for factual questions.
Why locally best is not globally best
Suppose the model gives “The” 0.5 and “A” 0.45 as the first word. Greedy takes “The”. But the best continuation after “A” might have far higher combined probability than anything after “The”. Greedy decoding never discovers this, because it never looks ahead. Beam search exists to address this.
When to use greedy decoding
| Situation | Suitable? |
|---|---|
| Data extraction and structured output | Yes |
| Classification with an LLM | Yes |
| Factual question answering | Usually |
| Code generation | Often |
| Evaluations needing reproducibility | Yes |
| Creative writing and brainstorming | No, output becomes bland |
| Generating multiple varied options | No |
The repetition problem
Greedy decoding is prone to loops: “I think that I think that I think…”. Once a phrase becomes locally most likely, it reinforces itself. Remedies include a repetition penalty, a frequency penalty, or switching to sampling with top-p.
Greedy vs other strategies
| Strategy | Chooses | Best for |
|---|---|---|
| Greedy | Top token | Deterministic, factual output |
| Beam search | Best overall sequence among k candidates | Translation, summarisation |
| Top-k sampling | Random from the k most likely | Varied text |
| Top-p sampling | Random from the smallest set reaching probability p | Balanced, natural text |
Limitations
- Can produce repetitive or generic text.
- No variety across runs, so it cannot generate alternatives.
- Ignores better sequences that begin with a slightly less likely token.
- Not fully deterministic in practice across different hardware or batch settings.