Judging with Many Minds: Do More Perspectives Mean Less Prejudice?

2025-05-26

AI Summary

This work systematically uncovers the nonlinear amplification mechanisms of four intrinsic biases—position, redundancy, chain-of-thought, and conformity—in multi-agent LLM-as-Judge systems. Multi-agent debate frameworks exacerbate and accumulate these biases, undermining evaluation fairness and reliability. Method: We comparatively analyze multi-agent debate versus LLM-as-Meta-Judge architectures, and propose a novel paradigm embedding the single-agent debiasing method PINE into multi-agent collaboration to enable unbiased agent intervention in bias propagation. Contribution/Results: Experiments demonstrate that PINE effectively mitigates bias in debate-based evaluation but yields marginal gains in meta-judging settings. We further introduce a quantifiable, multidimensional bias assessment framework and provide both theoretical foundations and practical guidelines for designing fair, robust multi-agent judgment systems.

Technology Category

• Multiagent Systems: Agreement, Argumentation & Negotiation
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

LLM-as-Judge has emerged as a scalable alternative to human evaluation, enabling large language models (LLMs) to provide reward signals in trainings. While recent work has explored multi-agent extensions such as multi-agent debate and meta-judging to enhance evaluation quality, the question of how intrinsic biases manifest in these settings remains underexplored. In this study, we conduct a systematic analysis of four diverse bias types: position bias, verbosity bias, chain-of-thought bias, and bandwagon bias. We evaluate these biases across two widely adopted multi-agent LLM-as-Judge frameworks: Multi-Agent-Debate and LLM-as-Meta-Judge. Our results show that debate framework amplifies biases sharply after the initial debate, and this increased bias is sustained in subsequent rounds, while meta-judge approaches exhibit greater resistance. We further investigate the incorporation of PINE, a leading single-agent debiasing method, as a bias-free agent within these systems. The results reveal that this bias-free agent effectively reduces biases in debate settings but provides less benefit in meta-judge scenarios. Our work provides a comprehensive study of bias behavior in multi-agent LLM-as-Judge systems and highlights the need for targeted bias mitigation strategies in collaborative evaluation settings.

Problem

• Analyzing intrinsic biases in multi-agent LLM-as-Judge frameworks
• Evaluating bias amplification in debate versus meta-judge approaches
• Assessing effectiveness of debiasing methods in collaborative LLM evaluation

Innovation

• Multi-agent debate enhances LLM-as-Judge evaluation
• Meta-judge framework resists bias better than debate
• PINE debiasing method reduces bias in debates