Ask the Right Comparison:Bias-Aware Bayesian Active Top-$k$ Ranking with LLM Judges

2026-07-02

AI Summary

This work addresses the challenge that large language models (LLMs) used as evaluators often exhibit systematic biases—such as verbosity preference or position effects—and are subject to stochastic noise, which undermines the reliability of conventional voting mechanisms for identifying top-k candidates under limited pairwise comparisons. To overcome this, the authors propose a bias-aware Bayesian inference framework that explicitly models covariate-induced biases and introduces a top-k–oriented active querying strategy, prioritizing comparisons that maximally reduce uncertainty about top-k membership. This approach is the first to enable adaptive correction of heterogeneous evaluator biases across diverse LLMs while focusing specifically on top-k identification rather than full ranking optimization. Evaluated on a benchmark encompassing 16 mainstream LLMs, the method achieves substantially improved top-k recall rates of 0.84–1.0 (up from 0.5–0.6) and attains optimal performance with fewer comparisons, demonstrating particularly effective bias correction for mid- to low-tier models.

Abstract

Large language models (LLMs) are increasingly used as cheap, scalable judges that compare candidate outputs pairwise -- to rank responses, select models, or triage papers. Yet LLM judges are both noisy and systematically biased: they favor verbose or well-formatted answers and exhibit position effects, so simply aggregating their votes recovers a ranking of presentation, not of true quality. We study the practical goal of identifying the \topk{} items under a fixed comparison budget, and make two contributions. First, we cast judging as Bayesian inference over latent quality with explicit, judge-specific bias covariates (verbosity, position), regularized by a shrinkage prior so that the data decide which biases a given judge actually exhibits. Second, we introduce a \topk-aware active acquisition rule that chooses the next comparison to maximally reduce uncertainty about \topk{} \emph{membership}, rather than about the full ranking. On a controlled benchmark with known ground-truth quality, judged by sixteen real LLMs spanning open and proprietary families (Llama, Qwen, Phi-4, GPT-4o-mini/5.1/5.5, Gemini, DeepSeek, and Claude Haiku/Sonnet/Opus), naive aggregation plateaus at a wrong \topk{} on biased judges regardless of budget, while our bias-aware model recovers it; \topk-aware acquisition reaches this ceiling with far fewer comparisons than round-robin or a global-uncertainty (D-optimal) rule. Bias is real but heterogeneous and capability-dependent: cheap and mid-tier judges carry a strong verbosity bias that our model corrects (lifting recall from $\sim$$0.5$--$0.6$ to $0.84$--$1.0$), whereas the frontier judges we tested show little bias and already rank accurately, so bias-aware modeling changes little there.

Problem

• LLM judges
• systematic bias
• top-k ranking
• comparison-based evaluation
• verbosity bias

Innovation

• Bayesian active learning
• top-k ranking
• LLM bias correction
• bias-aware inference
• active acquisition