Computation Mechanism Behind LLM Position Generalization

2025-03-17

AI Summary

This work investigates the intrinsic computational mechanisms underlying large language models’ (LLMs) positional generalization—i.e., their robustness to position-perturbed inputs and ability to extrapolate beyond training sequence lengths. Methodologically, we decompose attention logits, conduct empirical analyses across model depths and architectures, and develop a theoretical representation model. We discover, for the first time, a strong linear correlation (r = 0.959) between mid-layer attention logits and the sum of positional encoding and semantic importance scores, revealing a learnable decoupling pattern between position- and semantics-sensitive features. Based on this, we propose the first empirically verifiable computational criterion and interpretability framework for positional flexibility. Our framework establishes a causal link between positional generalization capability and internal representational structure—specifically, the geometry and decomposition of attention logits—thereby offering a novel paradigm for understanding the fundamental principles of sequence modeling in LLMs.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Most written natural languages are composed of sequences of words and sentences. Similar to humans, large language models (LLMs) exhibit flexibility in handling textual positions - a phenomenon we term position generalization. They can understand texts with position perturbations and generalize to longer texts than those encountered during training with the latest techniques. These phenomena suggest that LLMs handle positions tolerantly, but how LLMs computationally process positional relevance remains largely unexplored. This work connects the linguistic phenomenon with LLMs' computational mechanisms. We show how LLMs enforce certain computational mechanisms for the aforementioned tolerance in position perturbations. Despite the complex design of the self-attention mechanism, this work reveals that LLMs learn a counterintuitive disentanglement of attention logits. Their values show a 0.959 linear correlation with an approximation of the arithmetic sum of positional relevance and semantic importance. Furthermore, we identify a prevalent pattern in intermediate features, which we prove theoretically enables this effect. The pattern, which is different from how randomly initialized parameters would behave, suggests that it is a learned behavior rather than a natural result of the model architecture. Based on these findings, we provide computational explanations and criteria for LLMs' position flexibilities. This work takes a pioneering step in linking position generalization with modern LLMs' internal mechanisms.

Problem

• Explores how LLMs handle positional relevance in text.
• Investigates computational mechanisms for position generalization in LLMs.
• Links linguistic phenomena with LLMs' internal computational processes.

Innovation

• LLMs learn disentangled attention logits
• Linear correlation with positional relevance
• Intermediate features enable position flexibility