ML//transformer//attention//dot product attention

Dot-product attention is the mechanism that measures relevance between tokens: \(QK^{\mathsf T}\) produces an \(N \times N\) matrix of alignment scores between every pair of tokens. The whole computation is

Dot-product attention is the mechanism that measures relevance between tokens: QKTQK^{\mathsf T}QKT produces an N×NN \times NN×N matrix of alignment scores between every pair of tokens. The whole computation is

softmax⁡ ⁣(QKTdk)V,\operatorname{softmax}\!\left(\frac{QK^{\mathsf T}}{\sqrt{d_k}}\right) V,softmax(dk​​QKT​)V,

and that whole process is one attention head.

The raw scores are arbitrarily large and unnormalized, so they are scaled by dk\sqrt{d_k}dk​​ to prevent softmax saturation.

Softmax is applied per column (each query's distribution over all keys), which turns the scores into probabilities that sum to 1.

Multiplying the softmax weights by VVV gives a weighted sum of values, the suggested change to each token's representation.

The attention matrix is what makes transformers O(n2)O(n^2)O(n2) in sequence length: for N=4096N = 4096N=4096, that's 16M values per head.