Speculative Decoding
A technique that speeds up text generation by having a small, fast model draft several tokens that the large model then checks in one pass, without changing the output.
What is speculative decoding?
Speculative decoding is a way to make a large language model generate text faster. A small, cheap model guesses the next several tokens, and the large model checks all of the guesses at once. Correct guesses are kept, so several tokens are produced in the time it normally takes to produce one.
How does it work?
Normally a large model generates one token at a time, and each step needs a full pass through the model. Here the draft model proposes a short run of tokens, and the large model verifies them in a single pass. Tokens are accepted up to the first one the large model would not have chosen, and generation continues from there. The original paper, by Leviathan, Kalman and Matias, showed this keeps the output distribution unchanged while speeding up generation by about 2 to 3 times on the model it tested.
What is a simple example?
When the next words are easy to predict, such as the end of a common phrase, the small model gets them right and several words are accepted at once. When the text is harder, fewer guesses are accepted and the speed-up is smaller.
Why does it matter?
It lowers latency and inference cost without making answers worse, which is why it is widely used when serving large models.