Who can we trust? LLM-as-a-jury for Comparative Assessment

2026-02-18

AI Summary

This work addresses the inconsistency of large language models (LLMs) when used as automatic evaluators in pairwise text comparisons, which stems from varying reliability and probabilistic biases, compounded by the absence of human annotations for calibration. The authors propose BT-sigma, an extension of the Bradley–Terry model within the LLM-as-a-jury framework, which—under fully unsupervised conditions—explicitly models each LLM judge’s discriminative ability. Using only pairwise comparison data, BT-sigma jointly infers item rankings and judge reliability without external supervision. Experiments across multiple natural language generation benchmarks demonstrate that BT-sigma significantly outperforms simple averaging aggregation methods. Moreover, the learned discriminative parameters exhibit strong correlation with the cycle consistency of LLM judgments, effectively enabling adaptive self-calibration of evaluator reliability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large language models (LLMs) are increasingly applied as automatic evaluators for natural language generation assessment often using pairwise comparative judgements. Existing approaches typically rely on single judges or aggregate multiple judges assuming equal reliability. In practice, LLM judges vary substantially in performance across tasks and aspects, and their judgment probabilities may be biased and inconsistent. Furthermore, human-labelled supervision for judge calibration may be unavailable. We first empirically demonstrate that inconsistencies in LLM comparison probabilities exist and show that it limits the effectiveness of direct probability-based ranking. To address this, we study the LLM-as-a-jury setting and propose BT-sigma, a judge-aware extension of the Bradley-Terry model that introduces a discriminator parameter for each judge to jointly infer item rankings and judge reliability from pairwise comparisons alone. Experiments on benchmark NLG evaluation datasets show that BT-sigma consistently outperforms averaging-based aggregation methods, and that the learned discriminator strongly correlates with independent measures of the cycle consistency of LLM judgments. Further analysis reveals that BT-sigma can be interpreted as an unsupervised calibration mechanism that improves aggregation by modelling judge reliability.

Problem

• LLM-as-a-jury
• comparative assessment
• judge reliability
• bias and inconsistency
• unsupervised calibration

Innovation

• LLM-as-a-jury
• Bradley-Terry model
• judge reliability
• unsupervised calibration
• pairwise comparison