ML//inference//speculative decoding//multi-token prediction

Multi-token prediction is a training setup in which a language model learns to predict not only the next token but also one or more tokens further ahead, through extra output heads or extra layers, and it is used both to train stronger models and, at inference, as a built-in draft for speculative decoding that needs no second model. Gloeckle and colleagues at Meta studied it in 2024, and DeepSeek-V3 was trained with one extra prediction module.

Multi-token prediction is a training setup in which a language model learns to predict not only the next token but also one or more tokens further ahead, through extra output heads or extra layers, and it is used both to train stronger models and, at inference, as a built-in draft for speculative decoding that needs no second model. Gloeckle and colleagues at Meta studied it in 2024, and DeepSeek-V3 was trained with one extra prediction module.

At inference the extra layer proposes the token after the next one while the main model writes the next; the main model checks the proposal in its following pass, as with any draft, and keeps it only when it agrees. The output is unchanged, and the gain depends on how often the guess is accepted.

The extra layer shows up in model files: the GGUF of Qwen3.6 35B-A3B declares nextn_predict_layers = 1 and 41 blocks for 40 layers. Whether an engine uses it as a draft depends on the engine and its version.

A built-in draft avoids the two costs of a separate draft model: finding a small model with the same tokenizer, and the memory it takes.