Do RAG Systems Suffer From Positional Bias?

2025-05-21

AI Summary

Prior work overestimates the practical impact of positional bias in Retrieval-Augmented Generation (RAG), assuming large language models (LLMs) inherently assign higher weight to top-ranked passages. Method: We conduct empirical analysis within end-to-end RAG pipelines and controlled experiments across three standard benchmarks, jointly evaluating retrieval quality and interference level. We further compare position-aware re-ranking, LLM-perceived re-ranking, and random shuffling. Results: Over 60% of queries exhibit high-interference fragments co-occurring with relevant ones among the top-10 retrieved passages, compromising positional “purity” and diminishing positional bias. Position-aware re-ranking fails to improve performance—and underperforms random ordering—demonstrating that real-world retrieval noise effectively nullifies positional bias. This is the first end-to-end RAG study to empirically refute the validity of the “re-ranking to adapt to positional preference” hypothesis.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Preferences
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Retrieval Augmented Generation enhances LLM accuracy by adding passages retrieved from an external corpus to the LLM prompt. This paper investigates how positional bias - the tendency of LLMs to weight information differently based on its position in the prompt - affects not only the LLM's capability to capitalize on relevant passages, but also its susceptibility to distracting passages. Through extensive experiments on three benchmarks, we show how state-of-the-art retrieval pipelines, while attempting to retrieve relevant passages, systematically bring highly distracting ones to the top ranks, with over 60% of queries containing at least one highly distracting passage among the top-10 retrieved passages. As a result, the impact of the LLM positional bias, which in controlled settings is often reported as very prominent by related works, is actually marginal in real scenarios since both relevant and distracting passages are, in turn, penalized. Indeed, our findings reveal that sophisticated strategies that attempt to rearrange the passages based on LLM positional preferences do not perform better than random shuffling.

Problem

• Investigates positional bias in RAG systems affecting passage utility
• Analyzes LLM susceptibility to distracting passages in retrieval pipelines
• Evaluates impact of positional bias on real-world RAG performance

Innovation

• Investigates positional bias in RAG systems
• Analyzes distracting passages in retrieval pipelines
• Tests rearrangement strategies against random shuffling