Autoregressive generation is the process of producing output one token at a time, feeding each generated token back into the model as input for the next step. “Auto-regressive” means the model regresses on its own previous outputs. Nearly all text generation by LLMs, and some image and audio generation, works this way.
Key takeaways
- Each token depends on the prompt plus every token already generated.
- Generation continues until an end token, a length limit or a stop sequence.
- It is inherently sequential, which limits speed.
- The KV cache avoids reprocessing the whole sequence each step.
- Speculative decoding and parallel methods aim to reduce the sequential bottleneck.
Autoregressive generation explained in simple terms
Think of building a sentence with fridge magnets. You place one word, step back, read the whole line, then choose the next word that fits. You cannot place the fifth word before the third, because what fits depends on what came before. A model does the same at high speed.
The generation loop
- Process the prompt and produce a probability distribution for the next token.
- Select a token using greedy decoding, sampling with temperature, or top-p.
- Append the token to the sequence.
- Repeat from step 1 with the extended sequence.
- Stop at an end-of-sequence token, a maximum token limit or a stop string.
A worked example
Prompt: “List two benefits of UPI:”
| Step | Sequence so far | Next token chosen |
|---|---|---|
| 1 | prompt | “1” |
| 2 | prompt + “1” | “.” |
| 3 | prompt + “1.” | ” Instant” |
| 4 | prompt + “1. Instant” | ” transfers” |
Each step is a full forward pass through the model, though the KV cache means only the newest token is processed in detail. A 300-token answer therefore takes 300 sequential steps.
Why it is slow, and how that is addressed
Because steps cannot run in parallel, output speed is capped by how fast one token can be produced, usually limited by memory bandwidth rather than raw compute. Common remedies:
| Technique | Effect |
|---|---|
| KV cache | Avoids reprocessing earlier tokens |
| Continuous batching | Serves many users at once to use the GPU fully |
| Speculative decoding | A small draft model proposes several tokens; the large model verifies them together |
| Quantization | Less memory traffic per token |
| Shorter outputs | Fewer steps, directly less time |
Autoregressive vs non-autoregressive
Some research explores generating all tokens at once, or in parallel blocks, including diffusion-style text models. These can be faster but have generally lagged in quality for open-ended text. In images, diffusion models are non-autoregressive and dominate, while text remains mostly autoregressive.
Exposure bias
During training, the model always sees correct previous tokens. During generation, it sees its own, possibly flawed, output. This mismatch is called exposure bias and helps explain why long generations can drift off track. Techniques such as preference tuning and better decoding settings reduce, but do not remove, the problem.
Limitations
- Sequential steps limit speed and raise latency.
- Errors compound over long outputs.
- Cost grows with output length.
- The model cannot revise text it has already produced unless prompted to rewrite.