Higher-Dimensional Rotary Position Embedding

2026-08-30

AI Summary

为解决RoPE在深度混合和跨通道鲁棒性上的局限，本文提出HD-RoPE方法，通过高维旋转和Paley-I正交基增强了通道耦合与旋转自由度。

Abstract

Transformers rely on position embedding mechanisms in long context modeling in most cases. Rotary Position Embedding (RoPE) embeds positional information with independent 2D rotations, forming relative position terms in self-attention. However, its pairwise, block-based, and decoupled structure limits deep mixing and robustness across channels. We propose HD-RoPE, which extends RoPE from independent 2D rotations to higher-dimensional rotations and introduces a Paley-I orthogonal basis to obtain balanced, isotropic, and dense phase mixing within each rotation subspace. This significantly enhances channel coupling and rotational degrees of freedom while maintaining orthogonal stability and the relative position closure property. Furthermore, HD-RoPE is easily optimized for engineering efficiency without introducing additional trainable parameters. We have conducted extensive evaluation results demonstrating that HD-RoPE achieves significant performance improvements over standard RoPE across various popular benchmarks and in both long and short contexts.

Problem

• Rotary Position Embedding
• long context modeling
• channel coupling
• rotational degrees of freedom
• orthogonal stability

Innovation

• HD-RoPE
• Higher-Dimensional Rotations
• Paley-I Orthogonal Basis
• Channel Coupling