RankFlow: A Multi-Role Collaborative Reranking Workflow Utilizing Large Language Models

2025-02-02

AI Summary

Existing end-to-end re-ranking methods struggle with fine-grained relevance modeling in information retrieval and exhibit limited semantic understanding and decision-making capabilities. To address this, we propose the first LLM-based multi-role collaborative re-ranking paradigm. Our approach decomposes the re-ranking task into four specialized, role-anchored components—query rewriting, pseudo-answer generation, paragraph summarization, and holistic re-ranking—orchestrated via role-specific prompting and a multi-stage pipeline to enable joint query–passage semantic modeling. Evaluated on standard benchmarks including TREC-DL, BEIR, and NovelEval, our method significantly outperforms state-of-the-art approaches. Ablation studies confirm the distinct and complementary contributions of each role. The core innovation lies in logically decoupling LLM capabilities into schedulable, interpretable, task-aligned roles—thereby overcoming the limitations of monolithic, unified modeling architectures.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Summarization
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

In an Information Retrieval (IR) system, reranking plays a critical role by sorting candidate passages according to their relevance to a specific query. This process demands a nuanced understanding of the variations among passages linked to the query. In this work, we introduce RankFlow, a multi-role reranking workflow that leverages the capabilities of Large Language Models (LLMs) and role specializations to improve reranking performance. RankFlow enlists LLMs to fulfill four distinct roles: the query Rewriter, the pseudo Answerer, the passage Summarizer, and the Reranker. This orchestrated approach enables RankFlow to: (1) accurately interpret queries, (2) draw upon LLMs' extensive pre-existing knowledge, (3) distill passages into concise versions, and (4) assess passages in a comprehensive manner, resulting in notably better reranking results. Our experimental results reveal that RankFlow outperforms existing leading approaches on widely recognized IR benchmarks, such as TREC-DL, BEIR, and NovelEval. Additionally, we investigate the individual contributions of each role in RankFlow. Code is available at https://github.com/jincan333/RankFlow.

Problem

• Information Retrieval
• Ranking Accuracy
• Query-Document Relationship

Innovation

• RankFlow
• Multi-Role Collaboration
• Super Language Model