OPERA: A Reinforcement Learning--Enhanced Orchestrated Planner-Executor Architecture for Reasoning-Oriented Multi-Hop Retrieval

2025-08-22

AI Summary

Existing methods face three critical bottlenecks in complex multi-hop reasoning retrieval: poor generalization of reasoning-guided planning, weak query reformulation leading to missed key documents, and insufficient fine-grained reasoning for effective noise filtering. To address these, we propose OPERA—a planning-execution co-architectural framework. Its Goal Planning module enables dynamic, multi-step reasoning guidance, while the Reasoning Execution module supports iterative query reformulation and fine-grained result filtering. We further introduce MAPGRPO, a novel reinforcement learning algorithm that explicitly models the dynamic coupling between retrieval and reasoning. OPERA integrates large language models with dense retrievers and employs a modular, multi-agent progressive optimization strategy. Extensive experiments on multiple challenging multi-hop benchmarks demonstrate significant improvements over state-of-the-art methods, validating the efficacy of deep reasoning-retrieval synergy.

Technology Category

• Multiagent Systems: Multiagent Planning
• Search and Optimization: Metareasoning and Metaheuristics
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Recent advances in large language models (LLMs) and dense retrievers have driven significant progress in retrieval-augmented generation (RAG). However, existing approaches face significant challenges in complex reasoning-oriented multi-hop retrieval tasks: 1) Ineffective reasoning-oriented planning: Prior methods struggle to generate robust multi-step plans for complex queries, as rule-based decomposers perform poorly on out-of-template questions. 2) Suboptimal reasoning-driven retrieval: Related methods employ limited query reformulation, leading to iterative retrieval loops that often fail to locate golden documents. 3) Insufficient reasoning-guided filtering: Prevailing methods lack the fine-grained reasoning to effectively filter salient information from noisy results, hindering utilization of retrieved knowledge. Fundamentally, these limitations all stem from the weak coupling between retrieval and reasoning in current RAG architectures. We introduce the Orchestrated Planner-Executor Reasoning Architecture (OPERA), a novel reasoning-driven retrieval framework. OPERA's Goal Planning Module (GPM) decomposes questions into sub-goals, which are executed by a Reason-Execute Module (REM) with specialized components for precise reasoning and effective retrieval. To train OPERA, we propose Multi-Agents Progressive Group Relative Policy Optimization (MAPGRPO), a novel variant of GRPO. Experiments on complex multi-hop benchmarks show OPERA's superior performance, validating both the MAPGRPO method and OPERA's design. Code is available at https://github.com/Ameame1/OPERA.

Problem

• Generates robust multi-step plans for complex reasoning queries
• Addresses iterative retrieval loops in multi-hop document retrieval
• Filters salient information from noisy retrieval results effectively

Innovation

• Reinforcement learning-enhanced orchestrated planner-executor architecture
• Goal Planning Module decomposes questions into sub-goals
• Reason-Execute Module with specialized reasoning and retrieval components