ML//inference//speculative decoding

Speculative decoding is an inference technique that speeds up generation by letting a small draft model predict several tokens ahead, which the large model then verifies in one forward pass. When the draft is right (common for easy tokens like *the* or *is*), a single step yields multiple tokens.

Speculative decoding is an inference technique that speeds up generation by letting a small draft model predict several tokens ahead, which the large model then verifies in one forward pass. When the draft is right (common for easy tokens like the or is), a single step yields multiple tokens.

The speedup is free, with zero quality loss: the large model's distribution is preserved exactly.