Clifford Algebraic Rotor Embeddings : Maybe embeddings should start to CARE

2025-11-11

AI Summary

Existing RoPE methods struggle to preserve translational equivariance under high-dimensional scaling; notably, Spherical RoPE suffers from the non-commutativity of spherical rotations, leading to ambiguous definitions of rotated sequences. To address this, we propose QuatRo—the first quaternion-based rotational positional embedding framework—and generalize it to Clifford Algebra Rotational Embedding (CARE). CARE leverages Clifford spinors to uniformly model rotations in arbitrary dimensions and employs a multivector structure for hierarchical positional encoding. Within the geometric algebra framework, CARE rigorously guarantees translational equivariance while subsuming and extending both Mixed RoPE and Spherical RoPE. Experiments demonstrate that CARE achieves superior theoretical soundness, expressive flexibility, and empirical effectiveness in high-dimensional positional modeling.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Diffusion Models for Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Responsible Web: Algorithmic accountability and transparency on the web
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Rotary Positional Embeddings (RoPE) have demonstrated exceptional performance as a positional encoding method, consistently outperforming their baselines. While recent work has sought to extend RoPE to higher-dimensional inputs, many such extensions are non-commutative, thereby forfeiting RoPE's shift-equivariance property. Spherical RoPE is one such non-commutative variant, motivated by the idea of rotating embedding vectors on spheres rather than circles. However, spherical rotations are inherently non-commutative, making the choice of rotation sequence ambiguous. In this work, we explore a quaternion-based approach -- Quaternion Rotary Embeddings (QuatRo) -- in place of Euler angles, leveraging quaternions' ability to represent 3D rotations to parameterize the axes of rotation. We show Mixed RoPE and Spherical RoPE to be special cases of QuatRo. Further, we propose a generalization of QuatRo to Clifford Algebraic Rotary Embeddings (CARE) using geometric algebra. Viewing quaternions as the even subalgebra of Cl(3,0,0), we extend the notion of rotary embeddings from quaternions to Clifford rotors acting on multivectors. This formulation enables two key generalizations: (1) extending rotary embeddings to arbitrary dimensions, and (2) encoding positional information in multivectors of multiple grades, not just vectors. We present preliminary experiments comparing spherical, quaternion, and Clifford-based rotary embeddings.

Problem

• Extending rotary positional embeddings to higher-dimensional inputs while preserving equivariance
• Resolving non-commutative rotation ambiguities in spherical positional encodings
• Generalizing quaternion embeddings to arbitrary dimensions using geometric algebra

Innovation

• Clifford algebra extends rotary embeddings to multivectors
• Quaternions replace Euler angles for 3D rotation parameterization
• Geometric algebra enables positional encoding in arbitrary dimensions