Which LLM Multi-Agent Protocol to Choose?

2025-10-20

AI Summary

To address the lack of standardized evaluation and dynamic adaptation mechanisms for communication protocol selection in large-scale multi-agent systems, this paper introduces ProtocolBench—the first systematic benchmark framework for evaluating multi-agent communication protocols. It quantitatively assesses mainstream protocols (A2A, ACP, ANP, Agora) across four dimensions: task success rate, end-to-end latency, communication overhead, and fault tolerance. Furthermore, we propose ProtocolRouter, a learnable dynamic protocol routing mechanism that adaptively selects the optimal protocol based on runtime feedback (e.g., system load, fault signals). Experimental results demonstrate substantial protocol-level disparities: latency differences reach up to 3.48 seconds, and overall completion times vary by 36.5%. Under fault storms, ProtocolRouter recovers 18.1% faster than the best-performing static protocol; in complex scenarios such as GAIA, it significantly improves task success rates.

Technology Category

• Multiagent Systems: Agent Communication
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

As large-scale multi-agent systems evolve, the communication protocol layer has become a critical yet under-evaluated factor shaping performance and reliability. Despite the existence of diverse protocols (A2A, ACP, ANP, Agora, etc.), selection is often intuition-driven and lacks standardized guidance. We introduce ProtocolBench, a benchmark that systematically compares agent protocols along four measurable axes: task success, end-to-end latency, message or byte overhead, and robustness under failures. On ProtocolBench, protocol choice significantly influences system behavior. In the Streaming Queue scenario, overall completion time varies by up to 36.5% across protocols, and mean end-to-end latency differs by 3.48 s. Under Fail-Storm Recovery, resilience also differs consistently across protocols. Beyond evaluation, we present ProtocolRouter, a learnable protocol router that selects per-scenario (or per-module) protocols from requirement and runtime signals. ProtocolRouter reduces Fail-Storm recovery time by up to 18.1% versus the best single-protocol baseline, and achieves scenario-specific gains such as higher success in GAIA. We also release ProtocolRouterBench to standardize protocol evaluation and improve reliability at scale.

Problem

• Evaluating communication protocols in multi-agent systems
• Comparing protocol performance across measurable performance axes
• Selecting optimal protocols for specific scenarios automatically

Innovation

• Benchmark systematically compares agent protocols
• Learnable router selects protocols per scenario
• Router reduces recovery time versus baselines