BracketRank: Large Language Model Document Ranking via Reasoning-based Competitive Elimination

2026-04-10

AI Summary

This work addresses the limitations of large language model (LLM)-based rerankers in complex semantic retrieval tasks, particularly their sensitivity to context length and ranking instability. To overcome these challenges, the authors propose a reasoning-driven tournament-style reranking framework that integrates adaptive grouping, a reasoning-augmented stepwise prompting mechanism, and a bracket-based elimination structure amenable to parallel processing. This approach enables efficient and robust document reranking by leveraging both logical inference and competitive selection. Evaluated on the BRIGHT benchmark, the method achieves an nDCG@10 of 26.56, and attains state-of-the-art results on TREC Deep Learning Track 2019 and 2020 with nDCG@5 scores of 77.90 and 75.85, respectively, significantly outperforming existing approaches and demonstrating the effectiveness of combining reasoning capabilities with tournament-style competition in reranking.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Reasoning-intensive retrieval requires deep semantic inference beyond surface-level keyword matching, posing a challenge for current LLM-based rerankers limited by context constraints and order sensitivity. We propose \textbf{\BracketRank}, a framework that treats document reranking as a reasoning-driven competitive tournament. Our approach introduces three key innovations: (1) adaptive grouping based on model context limits, (2) reasoning-enhanced prompts that mandate step-by-step relevance explanations, and (3) a bracket-style elimination structure with winner and loser tracks. This design ensures robust document advancement while enabling parallel processing across competition stages. Evaluation on the BRIGHT reasoning benchmark shows that \BracketRank achieves \textbf{26.56 nDCG@10}, significantly outperforming state-of-the-art baselines including RankGPT-4 (17.0) and Rank-R1-14B (20.5). On TREC datasets, BracketRank achieves 77.90 nDCG@5 on DL 19 and 75.85 nDCG@5 on DL 20, exceeding all baselines, establishing that explicit reasoning within competitive elimination is a powerful paradigm for complex, multi-step retrieval tasks. https://github.com/DataScienceUIBK/BracketRank

Problem

• reasoning-intensive retrieval
• LLM-based reranking
• context constraints
• order sensitivity
• semantic inference

Innovation

• reasoning-based retrieval
• competitive elimination
• adaptive grouping
• reasoning-enhanced prompting
• LLM reranking