Temperature is a setting that controls how random or predictable a language model’s output is. At each step, an LLM calculates probabilities for every possible next token. Temperature reshapes those probabilities: low values make the model stick to the most likely tokens, and high values give less likely tokens more chance. It is one of the most commonly adjusted settings when using LLM APIs.
Key takeaways
- Low temperature (around 0–0.3): focused, consistent, predictable output.
- Medium (around 0.5–0.8): balanced output for general use.
- High (around 1 or above): more varied, creative and sometimes less coherent output.
- Temperature 0 is close to always picking the top token, but outputs may still vary slightly.
- It often works together with top-p and top-k sampling.
Temperature explained in simple terms
Imagine a chef choosing spices. At low temperature, they always use the house favourite. At medium, they occasionally try the second or third favourite. At high temperature, they reach for unusual spices more often: sometimes brilliant, sometimes strange. Temperature doesn’t change what the chef knows, only how adventurous they are when choosing.
How temperature works
The model produces raw scores (logits) for each possible token. Before converting them into probabilities with softmax, each score is divided by the temperature T.
- T below 1 makes large scores larger relative to small ones, sharpening the distribution.
- T = 1 leaves the model’s original distribution unchanged.
- T above 1 flattens the distribution, so unlikely tokens gain probability.
A worked example
Suppose the model’s scores for the next word after “The weather today is” are: sunny 2.0, cloudy 1.0, purple 0.1.
| Temperature | sunny | cloudy | purple |
|---|---|---|---|
| 0.5 | 86% | 12% | 2% |
| 1.0 | 66% | 24% | 10% |
| 2.0 | 50% | 30% | 19% |
At 0.5, the model almost always says “sunny”. At 2.0, “purple” appears about one time in five, which might be creative in a poem but nonsense in a weather report.
Recommended settings by task
| Task | Suggested temperature |
|---|---|
| Data extraction, classification, structured output | 0–0.2 |
| Factual Q&A, summaries, code | 0–0.4 |
| General chat and writing | 0.5–0.8 |
| Brainstorming, fiction, taglines | 0.8–1.2 |
These are starting points; test with your model and task. Some providers use different scales, and some reasoning models fix or limit temperature.
Temperature, top-p and top-k
- Top-k: only consider the k most likely tokens, for example the top 40.
- Top-p (nucleus sampling): only consider the smallest set of tokens whose probabilities add up to p, for example 0.9.
- Temperature: reshapes probabilities within whatever set remains.
Providers often recommend adjusting temperature or top-p, not both at once, to keep behaviour predictable.
Common misconceptions
- “Temperature 0 means fully deterministic.” Not always; hardware and batching effects can cause small variations.
- “Low temperature prevents hallucinations.” It reduces randomness but does not stop confident errors. Grounding and verification are still needed; see AI hallucination.
- “Higher temperature means smarter answers.” It means more varied answers, not better reasoning.
Limitations
- Too low can produce repetitive or bland text.
- Too high can produce incoherent or off-topic text.
- Best values differ between models and tasks.