ML//autoregressive

2026-02-25

Autoregressive generation is the paradigm behind GPT and all decoder-only models: the model predicts the next token given all previous tokens, one at a time, left to right. The word comes from time-series analysis, where an autoregressive model predicts a signal as a linear sum of its own past values.

1Run a forward passlogits from the LM head2Sample the next token3Append it to the sequence

Each iteration of that loop is one decoding step.

Left-to-right prediction is why causal masking exists. If the model could see future tokens during training, it would cheat. Masking forces it to learn genuine left-to-right prediction, matching how it must generate at inference time. The training objective that results is causal language modeling: it describes what the model learns, autoregression how it generates.

The KV cache optimization is a direct consequence: since each token's representation depends only on tokens before it, previous computations are closed and can be cached. Non-autoregressive models cannot do this.

The fundamental bottleneck is that generation is sequential. It cannot be parallelized across tokens because each depends on the previous. Speculative decoding tries to work around this with a draft-then-verify approach.

Training is parallel (all positions computed at once via masking), but inference is serial. This asymmetry is why training a model is fast per token but generation is slow. It has a second cost: the model trains on real sequences but generates from its own outputs, so an error compounds as each token conditions on possibly wrong predecessors (exposure bias).

BERT is the counterexample: it is not autoregressive. It sees all tokens simultaneously (bidirectional), which is why it's better at understanding but can't generate text naturally.