Mixture of Complementary Agents for Robust LLM Ensemble

2026-05-21

AI Summary

This work addresses the challenge of selecting complementary proposer models to enhance ensemble performance in multi-LLM collaboration, introducing model complementarity as the central selection criterion rather than relying solely on accuracy or diversity. To this end, the authors formulate proposer selection as a combinatorial optimization problem and propose a few-shot annotation–based complementarity metric paired with a scalable greedy selection algorithm, enabling efficient evaluation and identification of complementary models under limited labeled data. Experimental results demonstrate that the proposed approach significantly improves ensemble performance while achieving an optimal trade-off between computational cost and effectiveness.

Abstract

Multi-AI collaboration, such as ensembling or debating large language models (LLMs), is a promising paradigm for aggregating information and boosting performance. A foundational step in these pipelines is to feed the responses of several proposer LLMs into a summarizer LLM, which synthesizes a better answer. However, choosing which proposers to include is non-trivial. Existing approaches primarily focus either on accuracy (picking the strongest models) or diversity (ensuring variety), and often overlook the interactions among proposers and with the summarizer. We reframe proposer selection as a combinatorial selection problem akin to feature selection, where the value of an LLM lies in its complementarity with others. However, directly applying standard feature-selection algorithms is impractical in the LLM setting due to prohibitive time complexity. Motivated by this limitation, we explore an extensive range of computationally feasible, greedy-style selection algorithms that assess complementarity using a small labeled set. Our experiments validate complementarity as a guiding principle for proposer selection and identify methods that achieve the best performance-cost trade-offs in practice.

Problem

• LLM ensemble
• complementarity
• proposer selection
• multi-AI collaboration
• summarizer LLM

Innovation

• complementarity
• LLM ensemble
• proposer selection
• greedy algorithm
• multi-AI collaboration