Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?

2025-08-24

AI Summary

This work investigates the relative contributions of debate and voting to decision quality in multi-agent large language models (MLLMs), and elucidates the intrinsic mechanisms underlying debate effectiveness. We formalize multi-agent debate as a belief-evolution stochastic process and theoretically prove that its belief trajectory forms a martingale—implying that pure debate, without aggregation, does not improve the expected correctness rate; performance gains primarily stem from majority-voting ensembling. To enhance debate efficacy, we propose a bias-aware belief update intervention strategy. Extensive experiments across seven NLP benchmarks demonstrate that majority voting accounts for the vast majority of performance improvement, and simple ensemble methods consistently outperform complex debate protocols in robustness and stability. Our code is publicly available.

Technology Category

• Multiagent Systems: Agreement, Argumentation & Negotiation
• Machine Learning: Ensemble Methods
• Humans and AI: Voting

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Multi-Agent Debate~(MAD) has emerged as a promising paradigm for improving the performance of large language models through collaborative reasoning. Despite recent advances, the key factors driving MAD's effectiveness remain unclear. In this work, we disentangle MAD into two key components--Majority Voting and inter-agent Debate--and assess their respective contributions. Through extensive experiments across seven NLP benchmarks, we find that Majority Voting alone accounts for most of the performance gains typically attributed to MAD. To explain this, we propose a theoretical framework that models debate as a stochastic process. We prove that it induces a martingale over agents' belief trajectories, implying that debate alone does not improve expected correctness. Guided by these insights, we demonstrate that targeted interventions, by biasing the belief update toward correction, can meaningfully enhance debate effectiveness. Overall, our findings suggest that while MAD has potential, simple ensembling methods remain strong and more reliable alternatives in many practical settings. Code is released in https://github.com/deeplearning-wisc/debate-or-vote.

Problem

• Disentangling majority voting and debate contributions
• Modeling debate as a stochastic belief process
• Assessing debate effectiveness versus simple ensembling methods

Innovation

• Majority Voting accounts for most performance gains
• Debate modeled as stochastic belief martingale process
• Targeted interventions enhance debate effectiveness correction