🤖 AI Summary
In Transformer architectures, content (“what”) and position (“where”) representations are deeply entangled in mainstream positional encodings such as RoPE, inducing modeling bias—particularly degrading zero-shot length extrapolation. This work first identifies and formalizes the “what–where” coupling mechanism inherent in RoPE. To address it, we propose Polar Coordinate Positional Encoding (PoPE): it explicitly decouples content and position at the geometric level by encoding relative position as angular coordinates and content-dependent modulation as radial coordinates. PoPE is parameter-free, plug-and-play, and fully compatible with standard Transformers. Experiments across music, genomic, and language modeling tasks demonstrate consistent perplexity reduction across model scales (124M–774M parameters). Crucially, PoPE significantly improves zero-shot length extrapolation—enabling coherent generation far beyond training sequence lengths—without interpolation or fine-tuning.
📝 Abstract
The attention mechanism in a Transformer architecture matches key to query based on both content -- the what -- and position in a sequence -- the where. We present an analysis indicating that what and where are entangled in the popular RoPE rotary position embedding. This entanglement can impair performance particularly when decisions require independent matches on these two factors. We propose an improvement to RoPE, which we call Polar Coordinate Position Embeddings or PoPE, that eliminates the what-where confound. PoPE is far superior on a diagnostic task requiring indexing solely by position or by content. On autoregressive sequence modeling in music, genomic, and natural language domains, Transformers using PoPE as the positional encoding scheme outperform baselines using RoPE with respect to evaluation loss (perplexity) and downstream task performance. On language modeling, these gains persist across model scale, from 124M to 774M parameters. Crucially, PoPE shows strong zero-shot length extrapolation capabilities, whereas RoPE's performance degrades significantly on longer sequences at test time without fine tuning or the use of position-interpolation methods.