AI Explained
Speculative decoding: faster LLMs, same output
A small model can make a big one faster with exactly the same output. How speculative decoding drafts tokens and verifies them in one pass, and when it helps.
On this page
A small model can make a big one faster, with exactly the same output. That's speculative decoding.
Why generation is slow
Generating text is slow because each token is one step, and every step reads all of the model's weights from memory. Chips can do hundreds of trillions of operations per second, but memory moves only trillions of bytes per second. The math sits waiting for data.
That has a useful side effect: checking several tokens at once costs about the same as producing one.
How it works
- A small, fast draft model guesses the next few tokens.
- The big model checks all the guesses in one parallel pass, and keeps the ones it agrees with.
- At the first wrong guess, it throws away the rest and writes the correct token itself.
- Repeat.
A special sampling rule (a modified rejection sampling scheme) keeps the big model's probability distribution, so the output matches running the big model alone.
Why it pays off: many tokens are easy, like a closing bracket or the end of a common phrase. The draft gets those right, so several tokens land per pass.
The results
- The original paper (Leviathan et al.) showed 2 to 3 times faster generation on T5-XXL.
- DeepMind reported 2 to 2.5 times on Chinchilla 70B.
- It's in production: Google says it helps AI Overviews in Search answer faster.
Drafts without a draft model
The guesses don't have to come from a separate model:
- N-gram lookup copies spans from the prompt. Great when the answer repeats your input, like editing code.
- EAGLE and multi-token prediction train small heads on the big model to guess ahead.
APIs offer a cousin of this: OpenAI's Predicted Outputs. Pass the code you expect back, and an edit to it comes back faster.
When it helps
It shines at low traffic, when latency matters most. Under heavy load the gains shrink, because the GPU is already busy with other requests.
Tip
If you self-host, test it on your own prompts and watch the acceptance rate. More accepted guesses means more speed.
Guess fast, check once.