Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encoding

2025-01-01

AI Summary

Existing positional encoding methods suffer from rigidity: fixed attention patterns, weak long-range modeling capability, and insufficient context awareness or task adaptability. This paper proposes TAPE—a dynamic, content-driven positional embedding framework. Its core innovation is the first **contextual equivariant positional encoding**, which generates layer-wise adaptive position representations conditioned on sequence content, while enforcing permutation and orthogonal equivariance constraints to ensure training stability. TAPE seamlessly integrates with standard Transformer architectures without backbone modification, enabling plug-and-play deployment via equivariant representation learning and parameter-efficient fine-tuning. Empirical evaluation on language modeling, arithmetic reasoning, and long-context retrieval tasks demonstrates that TAPE significantly outperforms mainstream methods—including RoPE and ALiBi—with zero additional parameters, validating the effectiveness of jointly designing content-awareness and equivariance.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Transformers rely on both content-based and position-based addressing mechanisms to make predictions, but existing positional encoding techniques often diminish the effectiveness of position-based addressing. Many current methods enforce rigid patterns in attention maps, limiting the ability to model long-range dependencies and adapt to diverse tasks. Additionally, most positional encodings are learned as general biases, lacking the specialization required for different instances within a dataset. To address this, we propose con$ extbf{T}$extualized equivari$ extbf{A}$nt $ extbf{P}$osition $ extbf{E}$mbedding ($ extbf{TAPE}$), a novel framework that enhances positional embeddings by incorporating sequence content across layers. TAPE introduces dynamic, context-aware positional encodings, overcoming the constraints of traditional fixed patterns. By enforcing permutation and orthogonal equivariance, TAPE ensures the stability of positional encodings during updates, improving robustness and adaptability. Our method can be easily integrated into pre-trained transformers, offering parameter-efficient fine-tuning with minimal overhead. Extensive experiments shows that TAPE achieves superior performance in language modeling, arithmetic reasoning, and long-context retrieval tasks compared to existing positional embedding techniques.

Problem

• Positional Encoding
• Attention Mechanism
• Adaptability

Innovation

• TAPE
• Dynamic Position Encoding
• Content-aware Adaptation