LLM-based Listwise Reranking under the Effect of Positional Bias

2026-04-04

AI Summary

This work addresses the susceptibility of large language models to position bias in listwise reranking, which hinders their ability to promote relevant but initially low-ranked passages. To mitigate this issue, the authors propose DebiasFirst, a method that incorporates inverse propensity scoring into the loss function during fine-tuning for positional calibration and introduces a position-aware data augmentation strategy to counteract biases arising from model architecture and imbalanced training data distributions. The approach substantially improves reranking performance—measured by NDCG@10—and enhances robustness against variations in initial rankings. Empirical results demonstrate its effectiveness across multiple retrievers in reducing the influence of documents’ original positions, and it is shown to be compatible with debiasing techniques applied at inference time.

Abstract

LLM-based listwise passage reranking has attracted attention for its effectiveness in ranking candidate passages. However, these models suffer from positional bias, where passages positioned towards the end of the input are less likely to be moved to top positions in the ranking. We hypothesize that there are two primary sources of positional bias: (1) architectural bias inherent in LLMs and (2) the imbalanced positioning of relevant documents. To address this, we propose DebiasFirst, a method that integrates positional calibration and position-aware data augmentation during fine-tuning. Positional calibration uses inverse propensity scoring to adjust for positional bias by re-weighting the contributions of different positions in the loss function when training. Position-aware augmentation augments training data to ensure that each passage appears equally across varied positions in the input list. This approach markedly enhances both effectiveness and robustness to the original ranking across diverse first-stage retrievers, reducing the dependence of NDCG@10 performance on the position of relevant documents. DebiasFirst also complements the inference-stage debiasing methods, offering a practical solution for mitigating positional bias in reranking.

Problem

• positional bias
• LLM-based reranking
• listwise ranking
• passage reranking
• ranking effectiveness

Innovation

• positional bias
• listwise reranking
• inverse propensity scoring
• position-aware augmentation
• LLM-based ranking