LeRoPE: Learnable RoPE Frequencies Improve Language Modeling

2026-07-11

AI Summary

This work addresses the limited expressiveness of traditional Rotary Position Embedding (RoPE), which relies on fixed base-frequency hyperparameters. The authors propose Learnable Rotary Position Embedding (LeRoPE), the first method to treat RoPE’s frequency components as learnable scalars, thereby significantly enhancing the model’s capacity for positional modeling. Experiments reveal an emergent phenomenon wherein high-norm, position-sensitive frequency bands arise during training. The approach is validated through from-scratch training of language models ranging from 52M to 2.5B parameters, consistently outperforming standard RoPE and its existing variants. At the largest scale, LeRoPE achieves both improved performance and a 3.4% reduction in computational cost.

Abstract

Rotary Positional Encodings (RoPE) are currently the most popular positional encodings used in modern language models. RoPE rotates two-dimensional chunks of query and key vectors, operating as a function of their relative positional offset. The position-wise rates of rotation in RoPE typically follow a geometric sequence specified by a fixed base-frequency hyperparameter. Prior work has improved performance by either increasing this parameter to slow rotation or by applying RoPE to only a subset of QK dimensions. In this work we modify RoPE by learning a scalar per frequency, treating frequencies as learnable parameters rather than hyperparameters. We validate Learned RoPE by training a ladder of language models from scratch, ranging from 52M to 2.5B parameters. We observe and analyze the emergence of a high-norm, positional LeRoPE band. LeRoPE consistently outperforms RoPE and partial RoPE across all scales, with RoPE requiring 3.4% more compute (FLOPs) to match LeRoPE at the largest scale.

Problem

• Rotary Positional Encodings
• language modeling
• position encoding
• frequency learning
• RoPE

Innovation

• Learnable RoPE
• Rotary Positional Encoding
• Language Modeling
• Positional Encoding
• Frequency Learning