ML//transformer//attention//self-attention

Self-attention is attention in which Q, K and V all come from the same sequence, so each token attends to all the others, itself included.

Self-attention is attention in which Q, K and V all come from the same sequence, so each token attends to all the others, itself included.

It computes softmax⁡ ⁣(QKT/d)V\operatorname{softmax}!\left(QK^{\mathsf T}/\sqrt{d}\right) Vsoftmax(QKT/d​)V: the dot product measures relevance, the softmax normalizes it, and the values get weighted. Every pair of tokens interacts, which gives the n2n^2n2 cost that defines transformer scaling. Self-attention was the novel contribution of the 2017 paper: attention existed before, but self-attention for sequence processing was new.

In an encoder (BERT) self-attention is bidirectional, with no masking, so every token sees everything. In a decoder (GPT) it is causal, with masking, so each token only sees the past.