Speculative decoding speeds up generation by having a small, fast draft model propose several tokens ahead, which the large target model then verifies in a single forward pass. Tokens the large model agrees with are accepted immediately; the first disagreement is corrected and drafting resumes. Because verification of several tokens costs about the same as generating one, the result is faster output with the same distribution as the large model alone. It was described in 2023 by research teams at Google and DeepMind.
Key takeaways
- A small draft model guesses; the large model checks.
- Verification is parallel, so accepted tokens are effectively free.
- Output quality is preserved: the accepted tokens match what the target model would have produced.
- Typical speedups are around two to three times, depending on agreement rates.
- Variants such as Medusa, EAGLE and prompt lookup avoid needing a separate draft model.
Speculative decoding explained in simple terms
A junior colleague drafts the next few sentences of a letter. The senior reviewer reads them in one go and says “the first three are exactly what I would have written, the fourth is wrong”. Three sentences were produced in the time it would have taken the reviewer to write one, and the final text is still entirely the reviewer’s.
How it works
- The draft model generates k candidate tokens, for example 4.
- The target model processes the prompt plus all k candidates in one forward pass, producing its own probability for each position.
- Candidates are accepted while they match what the target model would have sampled, using a rejection-sampling rule that preserves the target distribution.
- At the first rejection, the target model’s own token is used instead.
- Drafting resumes from there.
A worked example
The draft model proposes: ” the”, ” report”, ” is”, ” attached”.
- The target model verifies all four in one pass.
- It agrees with ” the”, ” report” and ” is”, but would have chosen ” ready” rather than ” attached”.
- Three drafted tokens are accepted, plus one corrected token: 4 tokens produced in one target-model pass instead of four passes.
If acceptance rates are high, throughput rises sharply. If the draft model is poorly matched, most guesses are rejected and the overhead can outweigh the benefit.
Why it works
Decode speed is usually limited by reading the large model’s weights from memory, not by arithmetic. Checking four tokens requires roughly the same memory read as checking one, so verification is nearly free. The draft model is small enough that its own passes cost little.
Choosing a draft model
| Option | Notes |
|---|---|
| A much smaller model from the same family | Best alignment, common choice |
| A quantized version of the target | Reasonable agreement |
| Self-drafting heads (Medusa, EAGLE) | No separate model to serve |
| Prompt or n-gram lookup | Drafts by copying from the prompt; strong for summarisation and code editing |
A good draft model is small, fast and agrees often. Alignment matters more than the draft model’s standalone quality.
Where it helps most
Predictable text benefits most: code, structured formats, summarisation of provided text, and repetitive business writing. Highly creative or unpredictable output sees lower acceptance rates and smaller gains.
Limitations
- Requires extra memory to host the draft model.
- Gains depend on the acceptance rate.
- Complicates serving stacks and batching logic.
- Provides little benefit when batch sizes are already very large, since the GPU is then compute-bound.