ML//transformer//positional encoding//RoPE

RoPE (Rotary Position Embedding) is a positional encoding for transformers that encodes position by rotating the Q and K vectors by an angle proportional to their position before the dot product, instead of summing a position vector.

RoPE (Rotary Position Embedding) is a positional encoding for transformers that encodes position by rotating the Q and K vectors by an angle proportional to their position before the dot product, instead of summing a position vector.

Because rotation angles compose, Q⋅KQ \cdot KQ⋅K then depends naturally on the relative distance between tokens rather than on their absolute position. The rotation pattern generalizes, so RoPE extrapolates to longer sequences better than learned absolute embeddings. It is used in LLaMA, Mistral and most modern open LLMs.