ML//inference//decoding step

A decoding step is one forward pass, which generates one new token; the full inference, the entire generation, is a sequence of decoding steps.

A decoding step is one forward pass, which generates one new token; the full inference, the entire generation, is a sequence of decoding steps.

At each step the new token attends to all previous tokens, but the previous tokens do not attend to the new one (causal masking).

The KV cache makes this efficient: KKK and VVV are computed only for the new token and reused from the cache for all previous tokens. Without it, a sequence of length NNN recomputes attention over all NNN tokens at every step; with it, only the new token is computed, a 10 to 50× speedup for sequences of 4K tokens and more.

The cache grows linearly with context length, which is part of why long context is expensive in memory rather than in compute.