Intermediate MLOps & Production
10 MIN READ
Speculative Decoding: Guess Cheaply, Check in One Pass
Why a large language model can check several guessed tokens for about the cost of generating one, how the acceptance rule keeps the output exactly the same, the main ways to make guesses (draft models, n-grams, Medusa, EAGLE, multi-token prediction), and when it stops helping, including for vision-language and mixture-of-experts models.