A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth

2026-01-29

AI Summary

This work addresses the challenge of evaluating large language models (LLMs) in open-ended tasks without ground-truth labels, where existing methods neglect the varying reliability among judge LLMs, leading to biased rankings and inaccurate uncertainty estimates. The authors propose a judge-reliability-aware ranking framework that extends the Bradley–Terry–Luce model by introducing judge-specific discrimination parameters to jointly estimate both the abilities of evaluated models and the reliability of judges. This is the first approach to explicitly model judge reliability within a reference-free evaluation paradigm, establishing identifiability theory and proving consistency and asymptotic normality of the maximum likelihood estimator—enabling statistically sound confidence intervals for rank differences. Experiments demonstrate that the method significantly improves alignment with human preferences across multiple benchmarks and a newly collected dataset, outperforms unweighted baselines in data efficiency, and yields well-calibrated uncertainty quantification.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Reasoning under Uncertainty: Relational Probabilistic Models
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Evaluating large language models (LLMs) on open-ended tasks without ground-truth labels is increasingly done via the LLM-as-a-judge paradigm. A critical but under-modeled issue is that judge LLMs differ substantially in reliability; treating all judges equally can yield biased leaderboards and misleading uncertainty estimates. More data can make evaluation more confidently wrong under misspecified aggregation. We propose a judge-aware ranking framework that extends the Bradley-Terry-Luce model by introducing judge-specific discrimination parameters, jointly estimating latent model quality and judge reliability from pairwise comparisons without reference labels. We establish identifiability up to natural normalizations and prove consistency and asymptotic normality of the maximum likelihood estimator, enabling confidence intervals for score differences and rank comparisons. Across multiple public benchmarks and a newly collected dataset, our method improves agreement with human preferences, achieves higher data efficiency than unweighted baselines, and produces calibrated uncertainty quantification for LLM rankings.

Problem

• LLM evaluation
• LLM-as-a-judge
• judge reliability
• ground-truth-free evaluation
• ranking bias

Innovation

• judge-aware ranking
• Bradley-Terry-Luce model
• LLM evaluation
• uncertainty quantification
• pairwise comparison