Top-k sampling is a decoding method that keeps only the k most likely next tokens, renormalises their probabilities, and samples one from that shortlist. It adds variety compared with greedy decoding while preventing the model from occasionally picking an absurd, very low-probability token. It was popularised for story generation by Angela Fan and colleagues in 2018.
Key takeaways
- k is a fixed number of candidates, commonly 40 or 50.
- Everything outside the top k is given zero chance.
- It is often combined with temperature.
- Its weakness is that a fixed k ignores how confident the model is.
- Top-p sampling adapts the shortlist size instead.
Top-k sampling explained in simple terms
Imagine choosing a restaurant. Instead of always picking your top-rated option, or picking randomly from every restaurant in the city, you shortlist the five best and choose among those. You get variety without ending up somewhere terrible.
How it works
- The model produces a probability distribution over the vocabulary.
- Keep only the k highest-probability tokens.
- Renormalise so those k probabilities sum to 1.
- Sample one token from the shortlist.
- Append and repeat.
Worked example with k = 3. Original probabilities: ” excellent” 0.40, ” good” 0.25, ” useful” 0.15, ” fine” 0.08, and 4,000 other tokens sharing 0.12.
- Keep the top 3: 0.40, 0.25, 0.15, which sum to 0.80.
- Renormalise: 0.50, 0.31, 0.19.
- Sample from these three.
” fine” and every rarer token now have zero probability at this step.
Choosing k
| k value | Effect |
|---|---|
| 1 | Identical to greedy decoding |
| 5–10 | Conservative, fairly predictable |
| 40–50 | Common default for varied text |
| 100+ | More adventurous, higher risk of odd choices |
The fixed-size weakness
Language is not uniformly uncertain. After “The capital of France is”, one token deserves almost all the probability; a shortlist of 40 needlessly includes 39 poor options. After “She opened the door and saw a”, dozens of nouns are reasonable; k = 40 may cut off good candidates. Top-p fixes this by choosing the shortlist by cumulative probability rather than a fixed count.
Combining settings
Most APIs allow temperature with top-k or top-p. A typical order is: apply temperature to the logits, then filter by top-k or top-p, then sample. Providers often recommend tuning one filter rather than both, to keep behaviour predictable.
Limitations
- A fixed k ignores context-dependent uncertainty.
- Can exclude good options when many are plausible.
- Can include poor options when the model is confident.
- Less commonly exposed in modern chat APIs than top-p.