ML//transformer//attention//cross-attention

Cross-attention is the attention in which the queries \(Q\) come from one sequence and the keys \(K\) and values \(V\) from another, and it is how the decoder reads the encoder output. It has no causal masking: the decoder sees every encoder token freely, which makes sense in translation, where the whole source sentence should be visible.

Cross-attention is the attention in which the queries QQQ come from one sequence and the keys KKK and values VVV from another, and it is how the decoder reads the encoder output. It has no causal masking: the decoder sees every encoder token freely, which makes sense in translation, where the whole source sentence should be visible.

The first query starts from [BOS], the cold start. Before anything has been generated there is nothing to translate from, so the [BOS] embedding, processed by the decoder layers, is the initial query.

KKK and VVV from the encoder are computed once and cached, because they do not change as the decoder generates token by token.

The same mechanism is how DALL-E and Stable Diffusion condition image generation on text.

Of the three attentions in an encoder-decoder, only the decoder's self-attention uses causal masking; cross-attention from the decoder to the encoder and the encoder's self-attention use no mask.