Mo' Models, Mo' Problems: How to best select model pools when designing Multi-Agent Systems

2026-09-15

AI Summary

研究如何从大量开源模型中选择最优模型以设计多智能体系统，通过评估8种模型选择策略，发现单一模型家族内选择表现最佳。

Abstract

Multi-agent Systems (MAS) combine multiple model outputs to solve complex reasoning tasks. However, despite rapid growth of available open-source models, there is limited research on how to select optimal model candidates out of this massive pool. We systematically evaluate 8 model selection strategies (including model size, accuracy and answer diversity) across before-generation (routing) and after-generation (majority-voting, LLM-as-a-judge) MAS architectures on challenging scientific benchmarks. Our findings show a significant gap between theoretical oracle potential and actual performance: Expanding candidate pool sizes often degrades performance below that of the top performing base-model. We find that candidate selection within a single model family is the strategy that yields the best relative performance over a standalone model. These results demonstrate that adding arbitrary models to a heterogeneous MAS can introduce system instability, highlighting model selection as a critical design choice for multi-agent systems.

Problem

• Multi-Agent Systems
• model selection
• candidate pool
• system instability

Innovation

• model selection
• multi-agent systems
• candidate pool size
• single model family
• system instability