Differentiable Mixture-of-Agents Incentivizes Swarm Intelligence of Large Language Models

2026-05-15

AI Summary

This work addresses the limitations of existing large language model–based multi-agent systems, which typically rely on fixed communication topologies and struggle to adapt to dynamic task requirements. The authors propose the Differentiable Mixture of Agents (DMoA) framework, which employs a context-aware, differentiable routing mechanism to dynamically activate a sparse subset of agents during inference. By incorporating a recursive architecture that fuses historical and current contextual information, DMoA enables more coherent and adaptive decision-making. Furthermore, the framework introduces a self-supervised signal based on predictive entropy, facilitating test-time adaptation and self-evolution without requiring labeled data. Evaluated across nine benchmark tasks, DMoA significantly outperforms current state-of-the-art methods, demonstrating superior efficiency, robustness, and ensemble capability.

Abstract

Recent advances in Large Language Models (LLMs) have catalyzed the development of multi-agent systems (MAS) for complex reasoning tasks. However, existing MAS typically rely on pre-defined or pre-compiled communication topologies, which limits their flexibility and adaptability to dynamic task requirements. In this work, we propose Differentiable Mixture-of-Agents (DMoA), a self-evolving multi-agent framework that enables elastic and adaptive agent collaboration during inference. Instead of statically constructing workflows, DMoA dynamically routes and activates agents at each reasoning step, allowing the system to implicitly simulate diverse communication topologies and adapt to evolving demands. To achieve this, we design a differentiable, context-aware routing mechanism that leverages recurrent structures to incorporate historical and contextual information, producing sparse agent activations in a step-wise manner. Furthermore, we introduce predictive entropy as self-supervised signals to optimize the routing process, enabling efficient test-time adaptation without external annotations. Extensive experiments across 9 benchmarks demonstrate that DMoA achieves state-of-the-art performance while exhibiting strong efficiency, robustness, and ensembling capabilities.

Problem

• multi-agent systems
• communication topologies
• adaptive collaboration
• Large Language Models
• dynamic task requirements

Innovation

• Differentiable Mixture-of-Agents
• dynamic routing
• swarm intelligence
• self-supervised adaptation
• multi-agent systems