ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices

2025-06-04

AI Summary

Traditional positional encodings suffer from poor robustness and limited flexibility; while Rotary Position Embedding (RoPE) alleviates these issues via rotation-based encoding, its handcrafted rotation matrices exhibit constrained representational capacity. This paper proposes Composable RoPE (ComRoPE), a learnable rotation-based positional encoding that parameterizes the rotation angles via trainable commutative matrices. We formally derive and axiomatize the RoPE equation for the first time, and design two novel, theoretically grounded architectures for learnable commutative angle matrices—ensuring mathematical rigor while substantially increasing modeling freedom. ComRoPE significantly improves stability under positional shifts and enhances generalization across resolutions. Evaluated on Vision Transformers (ViTs) trained on ImageNet-1K, ComRoPE outperforms state-of-the-art methods by 1.6% (standard resolution) and 2.9% (high-resolution), respectively. The implementation is publicly available.

Technology Category

• Computer Vision: Representation Learning for Vision
• Search and Optimization: Learning to Search
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

The Transformer architecture has revolutionized various regions since it was proposed, and its effectiveness largely depends on the ability to encode positional information. Traditional position encoding methods exhibit significant limitations due to lack of robustness and flexibility of position. Therefore, Rotary Positional Encoding (RoPE) was proposed to alleviate these issues, which integrates positional information by rotating the embeddings in the attention mechanism. However, RoPE requires manually defined rotation matrices with limited transformation space, constraining the model's capacity. In this work, we propose ComRoPE, which generalizes RoPE by defining it in terms of trainable commuting angle matrices. Specifically, we demonstrate that pairwise commutativity of these matrices is essential for RoPE to achieve scalability and positional robustness. We formally define the RoPE Equation, which is an essential condition that ensures consistent performance with position offsets. Based on the theoretical analysis, we present two types of trainable commuting angle matrices as sufficient solutions to the RoPE equation, which significantly improve performance, surpassing the current state-of-the-art method by 1.6% at training resolution and 2.9% at higher resolution on the ImageNet-1K dataset. Furthermore, our framework shows versatility in generalizing to existing RoPE formulations and offering new insights for future positional encoding research. To ensure reproducibility, the source code and instructions are available at https://github.com/Longin-Yu/ComRoPE

Problem

• Enhancing position encoding robustness and flexibility in Transformers
• Overcoming limitations of manually defined RoPE rotation matrices
• Improving model performance with trainable commuting angle matrices

Innovation

• Trainable commuting angle matrices generalize RoPE
• Pairwise commutativity ensures scalability and robustness
• Two matrix types solve RoPE equation effectively