Efficient Leave-one-out Approximation in LLM Multi-agent Debate Based on Introspection

2025-05-28

AI Summary

Existing methods for assessing individual contributions in large language model (LLM)-based multi-agent debates—particularly standard leave-one-out (LOO)—suffer from prohibitive computational cost and poor deployability due to the need for repeated full-debate reruns. To address this, we propose IntrospecLOO, a lightweight, prompt-engineering-based LOO approximation that eliminates the need for re-executing debates. Its core innovation lies in introspective querying: a single-round prompting strategy that instructs agents to autonomously disregard the output of a specified participant, thereby enabling efficient, decoupled contribution attribution. Evaluated across three benchmark datasets, IntrospecLOO achieves individual contribution estimation accuracy comparable to standard LOO while reducing query overhead by approximately 75%. This method significantly improves evaluation efficiency and scalability, establishing the first low-cost, high-fidelity contribution attribution framework for LLM multi-agent systems grounded entirely in prompt engineering.

Technology Category

• Multiagent Systems: Adversarial Agents
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Multi-agent systems based on large language models (LLMs) advance automatic task completion in various fields, where debate is a common cooperation form for agents to solve complicated problems with reasoning and cross-review to solidify answers. Assessing the individual contributions of agents within these debates is crucial for system refinement and outcome reliability. Traditional leave-one-out (LOO) method offers a clear framework for evaluating each agent's role but face challenges in LLM-based systems due to high computational costs and associated financial implications. This paper presents introspective-leave-one-out (IntrospecLOO), a simple yet effective prompting for approximation of LOO in LLM-powered multi-agent debates. IntrospecLOO introduces an additional querying round after standard debates, prompting agents to update their answers while ignoring responses from a designated agent. This strategy effectively isolates and gauges each participant's influence at a reduced query complexity compared to the original LOO approaches. Validation through experiments on three benchmark datasets confirms the effectiveness of IntrospecLOO.

Problem

• Evaluating individual agent contributions in LLM multi-agent debates
• Reducing computational costs of traditional leave-one-out methods
• Approximating LOO efficiently via introspective prompting

Innovation

• Introspective-leave-one-out for agent evaluation
• Reduced query complexity in multi-agent debates
• Prompting agents to update answers post-debate