LTRR: Learning To Rank Retrievers for LLMs

2025-06-16

AI Summary

Single, static retrievers in RAG systems exhibit weak generalization and struggle to adapt to diverse queries. Method: This paper proposes a dynamic query routing mechanism that frames retriever selection as a learning-to-rank (LTR) task optimized for large language model (LLM) performance gain, using Answer Correctness (AC) as the primary objective. It introduces a dual-path framework—comprising both a zero-shot heuristic router and a trainable router—trained via pairwise ranking on synthetically generated, controllable QA data. Contribution/Results: This work is the first to formalize retriever selection as an LLM-utility-driven LTR problem. Evaluated on the SIGIR 2025 LiveRAG Challenge, it achieves state-of-the-art performance in both answer correctness and factual consistency, significantly outperforming the best single-retriever baseline—especially on out-of-distribution queries, where its generalization capability is markedly superior.

Technology Category

• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Planning with Language Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

Retrieval-Augmented Generation (RAG) systems typically rely on a single fixed retriever, despite growing evidence that no single retriever performs optimally across all query types. In this paper, we explore a query routing approach that dynamically selects from a pool of retrievers based on the query, using both train-free heuristics and learned routing models. We frame routing as a learning-to-rank (LTR) problem and introduce LTRR, a framework that learns to rank retrievers by their expected utility gain to downstream LLM performance. Our experiments, conducted on synthetic QA data with controlled query type variations, show that routing-based RAG systems can outperform the best single-retriever-based systems. Performance gains are especially pronounced in models trained with the Answer Correctness (AC) metric and with pairwise learning approaches, especially with XGBoost. We also observe improvements in generalization to out-of-distribution queries. As part of the SIGIR 2025 LiveRAG challenge, our submitted system demonstrated the practical viability of our approach, achieving competitive performance in both answer correctness and faithfulness. These findings highlight the importance of both training methodology and metric selection in query routing for RAG systems.

Problem

• Dynamic retriever selection for diverse query types
• Learning-to-rank framework to optimize retriever utility
• Improving RAG system performance with query routing

Innovation

• Dynamic retriever selection using query routing
• Learning-to-rank framework for retriever utility
• XGBoost enhances performance with pairwise learning