Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings

2024-12-30

AI Summary

This work investigates how causal Transformers implicitly encode positional information in the absence of explicit positional encodings. Method: Through cosine similarity analysis in the token embedding space, we observe that similarity between adjacent token embeddings decays monotonically with distance—a pattern linearly decodable into accurate position representations. We conduct controlled ablation experiments and extensive hyperparameter sweeps across diverse architectures and training configurations to validate robustness. Contribution/Results: We identify and empirically verify “proximal embedding similarity decay” as a self-emergent positional mechanism—stable under both random initialization and post-training, and independent of any positional encoding design. This finding provides the first theoretical foundation and interpretable intrinsic mechanism for position-awareness in encoding-free Transformers, significantly advancing our understanding of how attention-based models acquire and utilize positional information without explicit inductive biases.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Causal Learning
• Computer Vision: Diffusion Models for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Transformers with causal attention can solve tasks that require positional information without using positional encodings. In this work, we propose and investigate a new hypothesis about how positional information can be stored without using explicit positional encoding. We observe that nearby embeddings are more similar to each other than faraway embeddings, allowing the transformer to potentially reconstruct the positions of tokens. We show that this pattern can occur in both the trained and the randomly initialized Transformer models with causal attention and no positional encodings over a common range of hyperparameters.

Problem

• Positional Information
• Causal Transformer
• Positional Encoding

Innovation

• Causal Transformers
• Position Information Learning
• No Position Encoding Dependency