rotary position embeddings (RoPE) do

RoPE encodes position: multiply Q,K by rotation matrix R(θ_i) at each position

RoPE encodes position by multiplying Q,K by R(θ_i) at each position

RoPE's advantage is: supports length extrapolation beyond training context length

RoPE (Relative Position Encoding) advantage: supports length extrapolation beyond training context length

ALiBi allows length extrapolation better than learned position embeddings

ALiBi uses relative positional encoding, avoiding fixed-size embeddings, enabling better handling of variable-length sequences

sinusoidal position encoding works: each dimension has a different frequency

Sinusoidal position encoding assigns unique frequencies to each dimension, enabling the model to distinguish positions effectively

weight tying does in language models: shares embedding and output projection matrices

Tying reduces the number of parameters by sharing embedding and output projection matrices

List of algorithms

Cosine similarity measures the angle between vectors, not their magnitude