Decision Protocols in Multi-Agent Large Language Model Conversations

2026-07-06

AI Summary

This work investigates the design of efficient decision protocols in multi-agent large language model systems to enhance task performance while balancing training and inference costs. The authors propose MALLM, a unified framework that systematically compares three structured decision protocols—voting, consensus, and adjudication—across knowledge-intensive benchmarks (e.g., MMLU, GPQA) and complex reasoning tasks (e.g., StrategyQA, Math-lvl-5). Empirical results demonstrate that consensus protocols achieve superior performance on knowledge-dense tasks, whereas voting and adjudication mechanisms are better suited for intricate logical reasoning. Furthermore, increasing response diversity among agents significantly improves overall decision quality, highlighting the importance of heterogeneity in multi-agent collaborative reasoning.

Abstract

Improving the task performance of Large Language Models (LLMs) is essential, yet scaling these models faces significant challenges such as diminishing returns and high costs. Multi-Agent Systems (MAS) offer a promising solution by distributing tasks among specialized agents to improve the overall task performance. This can reduce training costs at the expense of increased test time due to the discussion and decision-making process. The decision protocol is a critical component of MAS because it specifies how multiple agents collaborate to create a final solution. This thesis introduces the Multi-Agent LLM (MALLM) framework, which implements and evaluates various decision protocols, namely voting, consensus, and judge decision mechanisms, to simulate multi-agent discussions for conversational task solving. Unlike previous work that used a single decision protocol or tested them on limited datasets, this study systematically examines their impact on a diverse set of tasks, ranging from knowledge-based datasets (MMLU, MMLU-Pro, GPQA) and logic-based datasets (StrategyQA, MuSR, Math-lvl-5, SQuAD 2.0). The results indicate that consensus protocols excel in knowledge-intensive domains while voting and judge protocols are more effective for logic-based tasks. Increasing response diversity through independent solution generation improves decision quality, while changes in information access during the decision process have minimal impact.

Problem

• Multi-Agent Systems
• Large Language Models
• Decision Protocols
• Task Performance
• Conversational AI

Innovation

• Multi-Agent Systems
• Decision Protocols
• Large Language Models
• Consensus Mechanism
• Task Performance