Position Encoding in Transformers: From Absolute and Relative Methods to Rotary Position Embeddings and Long-Context Scaling

2026-08-09

AI Summary

Transformer models inherently lack positional awareness and rely on positional encodings to capture sequence order. This work presents the first unified framework encompassing mainstream positional encoding methods—including sinusoidal and learnable absolute encodings, Shaw’s relative encoding, Transformer-XL, T5 bias, ALiBi, RoPE, and their long-context extensions such as YaRN and LongRoPE—and systematically analyzes their characteristics in terms of representation injection mechanisms, computational overhead, compatibility with KV caching, and length extrapolation capabilities. The study demonstrates that the computability of positional features does not necessarily imply strong generalization to long contexts, and proposes a multidimensional evaluation protocol to guide the selection and extension of positional encodings in large language models, offering both theoretical insights and practical recommendations.

Abstract

Self-attention models content-dependent interactions between tokens but does not by itself encode token order. Position encoding addresses this limitation by introducing absolute coordinates, relative distances, or position-dependent rotations into Transformer representations and attention scores. This technical survey develops a unified account of sinusoidal and learned absolute position embeddings, Shaw-style relative position representations, Transformer-XL, T5 relative position bias, ALiBi, and Rotary Position Embeddings (RoPE). We derive how RoPE converts absolute position indices into relative phase differences in Query-Key inner products and compare these methods in terms of where position is injected, computational cost, compatibility with KV caching, and length extrapolation. We then examine long-context extensions, including Position Interpolation, RoPE scaling laws, NTK-aware scaling, Dynamic NTK, NTK-by-parts, YaRN, LongRoPE, and LongRoPE2, with emphasis on frequency allocation, attention rescaling, training length, and target context length. We also summarize implementation considerations, evaluation protocols, and position-encoding choices in representative large language models. A central conclusion is that the ability to compute positional features beyond the training length does not imply reliable long-context generalization; context extension must be evaluated through short-context retention, position-wise perplexity, retrieval, reasoning, and long-context code tasks.

Problem

• Position Encoding
• Transformer
• Long-Context Modeling
• Self-Attention
• Sequence Order

Innovation

• Rotary Position Embeddings
• Position Encoding
• Long-Context Scaling
• Transformer
• Length Extrapolation