Positional Biases Shift as Inputs Approach Context Window Limits

2025-08-10

AI Summary

Large language models (LLMs) exhibit positional biases—such as the “lost-in-the-middle” phenomenon—in long-context processing, yet how performance systematically varies with input length remains poorly understood. Method: We propose a systematic analytical framework grounded in *relative context length*, enabling cross-model quantification of the dynamic evolution of primacy and recency effects. Contribution/Results: Experiments reveal that primacy effects diminish significantly when input length exceeds 50% of the model’s context window, whereas recency effects remain stable; crucially, model performance depends more on the target token’s distance from the input end than its absolute position. Further analysis uncovers that positional bias originates primarily in the *retrieval stage*: successful retrieval is a prerequisite for accurate reasoning, and reasoning biases inherit directly from retrieval preferences. This work establishes, for the first time, the *length-dependent mechanism* underlying positional bias, offering a new paradigm for long-context modeling and evaluation.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Models (LLMs) often struggle to use information across long inputs effectively. Prior work has identified positional biases, such as the Lost in the Middle (LiM) effect, where models perform better when information appears at the beginning (primacy bias) or end (recency bias) of the input, rather than in the middle. However, long-context studies have not consistently replicated these effects, raising questions about their intensity and the conditions under which they manifest. To address this, we conducted a comprehensive analysis using relative rather than absolute input lengths, defined with respect to each model's context window. Our findings reveal that the LiM effect is strongest when inputs occupy up to 50% of a model's context window. Beyond that, the primacy bias weakens, while recency bias remains relatively stable. This effectively eliminates the LiM effect; instead, we observe a distance-based bias, where model performance is better when relevant information is closer to the end of the input. Furthermore, our results suggest that successful retrieval is a prerequisite for reasoning in LLMs, and that the observed positional biases in reasoning are largely inherited from retrieval. These insights have implications for long-context tasks, the design of future LLM benchmarks, and evaluation methodologies for LLMs handling extended inputs.

Problem

• Analyzes positional biases in LLMs with long inputs
• Examines Lost in the Middle effect under varying context lengths
• Links retrieval success to reasoning biases in LLMs

Innovation

• Relative input lengths analysis for positional biases
• Distance-based bias replaces Lost in the Middle effect
• Retrieval as prerequisite for reasoning in LLMs