ML//transformer//attention//causal masking

Causal masking is an attention mask for autoregressive models that sets the upper triangle of the attention matrix to \(-\infty\) (which becomes 0 after softmax) so that tokens cannot attend to future positions. Attention then never carries information from later tokens back to earlier ones, which preserves the autoregressive property: the new token can attend to all previous tokens, and previous tokens cannot attend to the new one.

Causal masking is an attention mask for autoregressive models that sets the upper triangle of the attention matrix to −∞-\infty−∞ (which becomes 0 after softmax) so that tokens cannot attend to future positions. Attention then never carries information from later tokens back to earlier ones, which preserves the autoregressive property: the new token can attend to all previous tokens, and previous tokens cannot attend to the new one.

It enables the KV cache: since the representations of previous tokens are closed (they will not change), their K and V can be cached.

Without causal masking (encoder-style), all tokens see all others, which enriches early positions better but rules out generation. It is also another reason not to save enriched embeddings: without masking, everything would have to be recomputed each time a new token arrives.

BERT has no masking, which gives bidirectional understanding; GPT has masking, which gives autoregressive generation. Encoder-decoder cross-attention has no masking (the decoder sees all encoder tokens freely).