CyclicJudge: Mitigating Judge Bias Efficiently in LLM-based Evaluation

2026-03-02

AI Summary

This work addresses the susceptibility of large language models used as evaluators (LLM-as-judge) to systematic biases, which undermines assessment reliability. The authors propose CyclicJudge, a novel method that introduces the first bias analysis framework based on variance decomposition, disentangling evaluation scores into distinct components attributable to the scenario, the generated response, the judge, and residual noise. By incorporating a round-robin assignment mechanism, CyclicJudge completely eliminates judge-induced bias without increasing the per-evaluation computational cost. Experimental results on MT-Bench demonstrate that the proposed approach effectively removes systematic bias, significantly enhancing both consistency and fairness in model evaluations.

Technology Category

• Machine Learning: Evaluation and Analysis
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Computer Vision: Bias, Fairness & Privacy

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

LLM-as-judge evaluation has become standard practice for open-ended model assessment; however, judges exhibit systematic biases that cannot be eliminated by increasing the number of scenarios or generations. These biases are often similar in magnitude to the model differences that benchmarks are designed to detect, resulting in unreliable rankings when single-judge evaluations are used. This work introduces a variance decomposition that partitions benchmark score variance into scenario, generation, judge, and residual components. Based on this analysis, CyclicJudge, a round-robin assignment of judges, is demonstrated to be the optimal allocation strategy. It eliminates bias precisely while requiring each judge only once per cycle, maintaining the cost of single-judge evaluation. Empirical validation on MT-Bench supports all theoretical predictions.

Problem

• judge bias
• LLM-based evaluation
• systematic bias
• unreliable ranking
• benchmark evaluation

Innovation

• CyclicJudge
• LLM-as-judge
• judge bias
• variance decomposition
• round-robin assignment