Disentangling the Expressivity of RoPE

2026-08-12

AI Summary

This work clarifies the fundamental distinction between two theoretical accounts of Rotary Position Embedding (RoPE)—its capacity for expressing periodic modular predicates versus its reliance on a position-anchor-based local offset mechanism—and investigates their respective implications for generalization. Through a formal analysis of RoPE’s behavior in finite-precision soft-attention Transformers, the study establishes, for the first time, a rigorous correspondence between RoPE’s expressivity and classes of languages definable in temporal logic with modular predicates. The findings reveal that periodic RoPE enables length generalization on modular language tasks, whereas standard non-repeating RoPE merely simulates precision-dependent, bounded lookback operations, exhibiting a strong locality bias that severely degrades performance on tasks requiring long-range, position-invariant memory access.

Abstract

Two accounts recur in explanations of the success of rotary position embeddings (RoPE). Expressivity studies associate periodic position information with modular predicates, whereas mechanistic and long-context studies emphasize positional anchors and local offsets. We formalize both accounts for fully uniform, finite-precision soft-attention transformers. We find that, if every rotary component is periodic, RoPE transformers recognize exactly the languages definable in past temporal logic with modular predicates. Conventional RoPE is different: The rotations it computes never repeat. This yields a precision-dependent bounded simulation of fixed-offset look-back operators, rather than an all-length modular characterization. Controlled experiments match this separation: Constructed periodic schedules length-generalize on modular languages, while conventional RoPE behaves more like a bounded locality bias and can impair tasks requiring position-invariant access to distant context. Altogether, our findings shed light on RoPE transformers, bringing theoretical expressivity characterizations closer to models used in practice.

Problem

• RoPE
• expressivity
• modular predicates
• positional encoding
• transformers

Innovation

• RoPE
• expressivity
• modular predicates
• temporal logic
• length generalization