positional encoding design

To address the long-range dependency decay and computational overhead inherent in conventional position encodings for Transformers, this paper proposes Complex Position Encoding (CoPE). CoPE jointly models token semantics and positional information within a complex-valued embedding space: the real part encodes content, while the imaginary part encodes position; a phase-aware attention mechanism is further introduced to explicitly capture positional dependency patterns. Crucially, CoPE natively supports linear attention, circumventing the sequence-length extrapolation bottleneck caused by explicit position encoding injection. On the GLUE benchmark, CoPE consistently outperforms RoPE, sinusoidal encoding, and learned position encodings—achieving superior accuracy while maintaining lower computational complexity. These results empirically validate the effectiveness and efficiency of unifying content and position modeling in the complex domain.

Transformer positional encodings and attention mechanisms have long lacked a unified geometric and physical interpretation. Method: This paper introduces the first framework embedding Transformers within geometric field theory: discrete token positions are mapped to a continuous embedding manifold, and self-attention is formalized as a kernel-modulated integral operator defined on this manifold. By integrating manifold embedding, differential geometry, and field-theoretic principles, attention is recast as function modulation and transformation in continuous space. Contribution/Results: The framework provides an interpretable geometric semantics for core Transformer components—unifying the mathematical foundations of positional encoding and attention—and establishes a theoretical bridge between discrete neural architectures and continuous field theory. It enables principled design of next-generation attention mechanisms endowed with explicit geometric priors, advancing both interpretability and inductive bias engineering in deep learning.

This work addresses the suboptimal performance in multimodal representation alignment caused by modality gaps and data scarcity. To this end, the authors propose a disentangled representation learning framework based on shared and modality-specific codebooks. Leveraging a compositional vector quantization mechanism, the method decomposes multimodal features into shared semantic components and modality-unique components, and employs a progressive alignment strategy to optimize the alignment space without requiring fully paired data. The unified shared codebook effectively bridges the modality gap, while the modality-specific codebooks mitigate dominant-modality bias, enabling more balanced multimodal fusion. The approach achieves state-of-the-art performance across classification and retrieval tasks spanning nine modalities, including text, images, video, and audio.

This study systematically investigates how positional encoding affects the expressive power, generalization, and long-sequence extrapolation capability of Transformers. To address this, we propose a unified theoretical framework that, for the first time, incorporates linear bias methods—including ALiBi—into the formal modeling of positional encodings. We further introduce a novel orthogonal encoding scheme based on wavelet and Legendre polynomial transforms, and rigorously establish its superiority via function approximation theory and Rademacher complexity analysis. Empirical evaluation on synthetic sequence tasks demonstrates that our encoding reduces generalization error by 37% and improves extrapolation length by 2.1× compared to sinusoidal encoding, significantly enhancing out-of-distribution generalization to unseen sequence lengths. Our core contributions are (i) a unified analytical paradigm for positional encoding, grounded in learning theory, and (ii) a theoretically justified, orthogonal encoding design that achieves state-of-the-art empirical performance.

This study addresses the lack of theoretical stability guarantees for similarity-based positional encoding (simPE) under image rotation perturbations. By integrating Lipschitz continuity analysis with matrix perturbation theory, the work provides the first rigorous robustness guarantee for simPE: although simPE is not rotation-invariant, it exhibits bounded stability under small-angle rotations when satisfying a Lipschitz condition, and an explicit perturbation bound in Frobenius norm is derived. Controlled experiments on Arrow, Shapes, Digits, and FashionMNIST datasets validate the theoretical predictions, demonstrating that simPE consistently outperforms standard learnable positional encodings across multiple metrics—including accuracy, F1 score, precision, and recall—under small to moderate rotation angles.

This study investigates the influence of positional encoding on the internal spatial representation structure and robustness of Vision Transformers. To this end, the authors propose a Spatial Similarity Distance Correlation (SSDC) metric and systematically evaluate stability under content perturbations by combining multiple positional encoding schemes—learned absolute, sinusoidal, and rotary—with representational geometry analysis. The findings reveal that positional encoding establishes an indexing anchoring mechanism that shapes a stable internal spatial organization. Crucially, robustness primarily stems from this stable positional reference frame rather than the specific form of the encoding itself. Models without positional encoding exhibit fragile spatial structures highly susceptible to perturbations, whereas incorporating positional encoding significantly enhances robustness under distributional shifts.

Transformer models are inherently insensitive to word order and rely on positional encodings to inject sequential information, yet existing designs often lack a rigorous theoretical foundation. This work proposes a geometric framework for positional encoding, establishing its necessity and separability, and derives a minimally parameterized representation. Building upon the Hellinger distance and classical multidimensional scaling (MDS), the authors construct an information-theoretically optimal encoding scheme. By leveraging matrix rank analysis and neural tangent kernel (NTK) theory, they unify the evaluation of encoding quality into a single stress metric. Empirical validation on SST-2 and IMDB demonstrates that ALiBi encodings exhibit significantly lower stress compared to sinusoidal and RoPE encodings, corroborating their near rank-1 optimal structure.

This work addresses the lack of explicit, temporally consistent 3D spatial representations in video-based visual question answering grounded in the physical world. To this end, the authors propose a spatial encoding framework that transforms RGB videos into explicit 3D spatial representations by jointly performing 6D object pose estimation, multi-object tracking, and geometric prediction within a unified spatial encoder—marking the first integration of these three tasks. Building upon this representation, they introduce a spatial scoring reward mechanism to fine-tune large language models via reinforcement learning, enabling perspective-aware and geometry-grounded reasoning based on explicit 3D bounding boxes and semantic labels. The proposed method achieves state-of-the-art performance on the VSI-Bench benchmark, outperforming existing closed-source vision-language models.

Existing Transformer models struggle to effectively model long-range contexts and global document structure due to the entanglement of positional and semantic information. This work proposes the first tri-stream decoupled architecture that explicitly separates semantic, absolute positional, and relative positional representations, restricting the masked language modeling objective exclusively to the semantic stream. Through orthogonal subspace analysis and evaluation using the Flash-Holmes probing framework, we find that the absolute positional subspace spontaneously organizes into a low-frequency two-dimensional manifold reflecting document structure, while attention heads exhibit functional specialization between structural and semantic tasks. On the Flash-Holmes benchmark covering 65 linguistic phenomena, our approach significantly improves representational quality in 49 cases, demonstrating its enhanced capacity to preserve document-level structural information.

Transformer models inherently lack positional awareness and rely on positional encodings to capture sequence order. This work presents the first unified framework encompassing mainstream positional encoding methods—including sinusoidal and learnable absolute encodings, Shaw’s relative encoding, Transformer-XL, T5 bias, ALiBi, RoPE, and their long-context extensions such as YaRN and LongRoPE—and systematically analyzes their characteristics in terms of representation injection mechanisms, computational overhead, compatibility with KV caching, and length extrapolation capabilities. The study demonstrates that the computability of positional features does not necessarily imply strong generalization to long contexts, and proposes a multidimensional evaluation protocol to guide the selection and extension of positional encodings in large language models, offering both theoretical insights and practical recommendations.