Efficient Mixture-of-Agents Serving via Tree-Structured Routing, Adaptive Pruning, and Dependency-Aware Prefill-Decode Overlap

2025-12-19

AI Summary

To address high end-to-end latency in Mixture-of-Agents (MoA) inference—caused by intensive inter-agent communication and low hardware utilization—this paper proposes an algorithm-system co-optimization framework. First, it introduces a novel tree-structured agent topology to reduce communication complexity. Second, it designs a semantic-consistency-driven dynamic pruning mechanism that adaptively terminates low-confidence downstream agents at runtime. Third, it implements dependency-aware incremental pipelining of prefilling and decoding stages. Collectively, these techniques enhance structural sparsity and computational resource utilization. Experiments demonstrate up to 90% reduction in end-to-end latency while maintaining accuracy within ±1% of the baseline; notably, accuracy improves on certain tasks.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Multiagent Systems: Agent Communication
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Mixture-of-Agents (MoA) inference can suffer from dense inter-agent communication and low hardware utilization, which jointly inflate serving latency. We present a serving design that targets these bottlenecks through an algorithm-system co-design. First, we replace dense agent interaction graphs with a hierarchical tree topology that induces structured sparsity in inter-agent communication. Second, we introduce a runtime adaptive mechanism that selectively terminates or skips downstream agent invocations using semantic agreement and confidence signals from intermediate outputs. Third, we pipeline agent execution by overlapping incremental prefilling with decoding across dependency-related agents, improving utilization and reducing inference latency. Across representative tasks, this approach substantially reduces end-to-end latency (up to 90%) while maintaining comparable accuracy (within $pm$1%) relative to dense-connectivity MoA baselines, and can improve accuracy in certain settings.

Problem

• Reduces dense inter-agent communication in MoA inference
• Improves hardware utilization and reduces serving latency
• Maintains accuracy while optimizing agent execution pipeline

Innovation

• Hierarchical tree topology reduces agent communication density
• Adaptive pruning mechanism terminates redundant downstream agent calls
• Overlapping prefilling and decoding pipelines agent execution dependencies