Talk to Right Specialists: Routing and Planning in Multi-agent System for Question Answering

2025-01-14

AI Summary

To address challenges in cross-domain question answering—including hallucination in single-source RAG, difficulty integrating multiple knowledge bases, slow retrieval, and weak handling of sensitive information—this paper proposes RopMura, a multi-agent system. Methodologically, it integrates LLM-driven multi-agent architecture, retrieval-augmented generation (RAG), and hierarchical task planning. Its core contributions are: (1) a lightweight, knowledge-boundary-aware dynamic router that enables domain-adaptive query routing and physical isolation of sensitive data; and (2) a multi-hop query decomposition planner supporting stepwise, collaborative reasoning. Experiments demonstrate significant improvements over baselines: the routing module boosts single-hop QA accuracy by 12.3%, while the joint routing-and-planning mechanism increases end-to-end accuracy on complex cross-domain queries by 28.6%.

Technology Category

• Multiagent Systems: Multiagent Planning
• Intelligent Robots: Multi-Robot Systems
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Community question answering

Abstract

Leveraging large language models (LLMs), an agent can utilize retrieval-augmented generation (RAG) techniques to integrate external knowledge and increase the reliability of its responses. Current RAG-based agents integrate single, domain-specific knowledge sources, limiting their ability and leading to hallucinated or inaccurate responses when addressing cross-domain queries. Integrating multiple knowledge bases into a unified RAG-based agent raises significant challenges, including increased retrieval overhead and data sovereignty when sensitive data is involved. In this work, we propose RopMura, a novel multi-agent system that addresses these limitations by incorporating highly efficient routing and planning mechanisms. RopMura features two key components: a router that intelligently selects the most relevant agents based on knowledge boundaries and a planner that decomposes complex multi-hop queries into manageable steps, allowing for coordinating cross-domain responses. Experimental results demonstrate that RopMura effectively handles both single-hop and multi-hop queries, with the routing mechanism enabling precise answers for single-hop queries and the combined routing and planning mechanisms achieving accurate, multi-step resolutions for complex queries.

Problem

• Cross-domain questions
• Information retrieval speed
• Sensitive information handling

Innovation

• Smart Router
• Planner Design
• Multi-domain Knowledge Fusion