Top-p sampling, also called nucleus sampling, keeps the smallest group of tokens whose probabilities add up to at least p, then samples from that group. Unlike top-k sampling, which uses a fixed shortlist size, top-p adapts: when the model is confident the shortlist is tiny, and when many continuations are plausible the shortlist grows. It was introduced by Ari Holtzman and colleagues in 2019 in work on neural text degeneration.
Key takeaways
- p is a probability threshold, commonly 0.9 or 0.95.
- The shortlist, or “nucleus”, changes size at every step.
- It avoids both bland repetition and nonsensical rare tokens.
- p = 1.0 means no filtering; very low p approaches greedy decoding.
- Most providers suggest tuning either top-p or temperature, not both.
Top-p sampling explained in simple terms
Instead of always shortlisting five restaurants, you shortlist however many it takes to cover 90% of what you would actually consider. On a night when one place is clearly right, the shortlist has one entry. When many appeal equally, it has fifteen. The rule adapts to the situation.
How it works
- Sort tokens by probability, highest first.
- Add them up until the running total reaches p.
- Discard everything else.
- Renormalise the kept probabilities.
- Sample one token from that nucleus.
A worked example with p = 0.9
Confident case, after “The capital of Japan is”:
| Token | Probability | Running total |
|---|---|---|
| ” Tokyo” | 0.95 | 0.95 ✓ reached |
The nucleus contains one token. Output is effectively deterministic.
Uncertain case, after “She picked up the”:
| Token | Probability | Running total |
|---|---|---|
| ” phone” | 0.20 | 0.20 |
| ” book” | 0.15 | 0.35 |
| ” bag” | 0.12 | 0.47 |
| ” pen” | 0.10 | 0.57 |
| … several more … | 0.90 ✓ reached after about 12 tokens |
Here the nucleus holds around a dozen tokens, giving natural variety.
Why it beats a fixed k
Human text is not the most probable text, but it is also not random. Holtzman’s research showed that maximisation methods such as beam search produce repetitive, unnatural output, while unrestricted sampling produces incoherence. Nucleus sampling targets the middle: sample from the meaningful mass of probability and cut the long tail.
Typical settings
| Task | Suggested p |
|---|---|
| Data extraction, structured output | 0.1–0.5, or greedy |
| Factual answers and code | 0.5–0.8 |
| General chat and writing | 0.9–0.95 |
| Creative brainstorming | 0.95–1.0 |
Combining with other settings
Many APIs apply temperature first, then top-p filtering, then sampling, sometimes with top-k and penalties as well. Since temperature and top-p both control randomness, adjusting one at a time makes behaviour easier to reason about.
Limitations
- Does not prevent factual errors or hallucination.
- Very low p can cause repetition, like greedy decoding.
- Interacts with temperature in ways that can be confusing.
- Optimal values differ between models and tasks.