ML//transformer//positional encoding

Positional encoding is the information added to a Transformer's token embeddings to give the model the order of its tokens: without it the model is permutation invariant and treats *cat bites dog* the same as *dog bites cat*. The reason is in attention itself: it is a weighted average over a set, so shuffling the inputs changes nothing unless the position is written into the vectors (added to the token's embedding, so each vector carries what the token means and where it sits) or into the attention scores.

Positional encoding is the information added to a Transformer's token embeddings to give the model the order of its tokens: without it the model is permutation invariant and treats cat bites dog the same as dog bites cat. The reason is in attention itself: it is a weighted average over a set, so shuffling the inputs changes nothing unless the position is written into the vectors (added to the token's embedding, so each vector carries what the token means and where it sits) or into the attention scores.

The original paper summed sinusoidal waves to the embeddings. Its problem is length: a model trained on 512 tokens that sees 1000 at inference degrades, because it never saw those positions. The later methods are learned embeddings (GPT-2), RoPE (LLaMA, which rotates Q and K) and ALiBi (a linear bias on the attention scores), and each handles extrapolation differently. The choice of method matters most for long sequences beyond the training length.