Multi-Agent Deep Research: Training Multi-Agent Systems with M-GRPO

2025-11-17

AI Summary

In multi-agent systems, a unified large language model (LLM) struggles to accommodate heterogeneous agents’ divergent data distributions and execution frequencies; moreover, cross-server deployment disrupts end-to-end gradient flow and impairs credit assignment. Method: We propose M-GRPO (Hierarchical Group Relative Policy Optimization), which enables decoupled, co-training of agent-specific LLMs via heterogeneous trajectory alignment and hierarchical credit assignment. We further design a decentralized architecture with minimal sufficient statistics sharing to support efficient cross-server optimization. Contribution/Results: Evaluated on GAIA, XBench-DeepSearch, and WebWalkerQA, M-GRPO significantly outperforms single-agent and frozen-subagent multi-agent baselines—improving tool-augmented reasoning accuracy, stability, and sample efficiency. To our knowledge, it is the first framework to systematically resolve gradient fragmentation and optimization inconsistency in heterogeneous multi-agent training.

Technology Category

• Multiagent Systems: Multiagent Learning
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Multi-agent systems perform well on general reasoning tasks. However, the lack of training in specialized areas hinders their accuracy. Current training methods train a unified large language model (LLM) for all agents in the system. This may limit the performances due to different distributions underlying for different agents. Therefore, training multi-agent systems with distinct LLMs should be the next step to solve. However, this approach introduces optimization challenges. For example, agents operate at different frequencies, rollouts involve varying sub-agent invocations, and agents are often deployed across separate servers, disrupting end-to-end gradient flow. To address these issues, we propose M-GRPO, a hierarchical extension of Group Relative Policy Optimization designed for vertical Multi-agent systems with a main agent (planner) and multiple sub-agents (multi-turn tool executors). M-GRPO computes group-relative advantages for both main and sub-agents, maintaining hierarchical credit assignment. It also introduces a trajectory-alignment scheme that generates fixed-size batches despite variable sub-agent invocations. We deploy a decoupled training pipeline in which agents run on separate servers and exchange minimal statistics via a shared store. This enables scalable training without cross-server backpropagation. In experiments on real-world benchmarks (e.g., GAIA, XBench-DeepSearch, and WebWalkerQA), M-GRPO consistently outperforms both single-agent GRPO and multi-agent GRPO with frozen sub-agents, demonstrating improved stability and sample efficiency. These results show that aligning heterogeneous trajectories and decoupling optimization across specialized agents enhances tool-augmented reasoning tasks.

Problem

• Training multi-agent systems with distinct LLMs faces optimization challenges
• Variable sub-agent invocation frequencies disrupt standard gradient flow
• Separate agent deployment across servers prevents end-to-end training

Innovation

• Hierarchical credit assignment for multi-agent systems
• Trajectory alignment with variable sub-agent invocations
• Decoupled training pipeline across separate servers