RoPoLL: Robust Panel of LLM Judges

2026-06-29

AI Summary

This work addresses the vulnerability of traditional LLM juries (PoLL) to unbounded bias under contamination from mode collapse, sycophancy, or safety refusals—biases that cannot be mitigated by simply scaling up jury size. To tackle this, the paper introduces RoPoLL, the first approach integrating the Huber contamination model and robust statistical theory into LLM evaluation. RoPoLL preserves the original jury architecture but replaces mean aggregation with geometric median estimation, achieving strong robustness against contamination rates up to 50%. The method enjoys an optimal finite-sample breakdown point of 1/2, requires no hyperparameter tuning, and attains the information-theoretic minimax lower bound. Experiments demonstrate that RoPoLL significantly outperforms PoLL across 13 open-source LLM judges, three reward model benchmarks, and multiple adversarial attacks; notably, a RoPoLL ensemble of just three 38B-parameter models surpasses the 675B Mistral-Large-3 by a factor of 1.31 on HelpSteer-2.

Abstract

The LLM Jury, a Panel of LLM Evaluators (PoLL) reporting consensus scores, has become a practical alternative to single-judge LLM evaluation, yet its statistical behavior remains poorly understood. We formalize the LLM Jury under the Huber contamination model and show that PoLL incurs unbounded bias under any positive contamination, regardless of jury size, whenever a single judge fails in a biased, LLM-typical way (mode collapse, sycophancy, safety refusal). Framing jury consensus as classical robust mean estimation, we propose RoPoLL (Robust Panel of LLM-as-Judge), which preserves the PoLL panel but replaces the aggregation function with a robust mean estimator, instantiated with the geometric median (GM): tuning-free, with the optimal finite-sample breakdown point 1/2. A finite-sample error bound and a matching information-theoretic minimax lower bound agree on the parametric rate sigma*sqrt(d/N) and differ on the breakdown floor by a factor of sqrt(d), a statistical-computational gap that polynomial-time RoPoLL pays relative to the intractable Tukey halfspace median. Across 13 open-weight judges (4B-675B), three reward-model benchmarks, and four corruption regimes at rates up to 50%, RoPoLL dominates PoLL on every biased corruption type: by about 19% on cross-dimensional attacks at matched compute, and by orders of magnitude on heavy-tailed Byzantine adversaries. A 3-judge RoPoLL committee at 38B beats Mistral-Large-3 (675B) by 1.31x on HelpSteer-2 under 30% bimodal-random corruption, an 18x parameter advantage at better accuracy; a Noisy-GT control confirms the premium is paid against biased contamination, not benign imprecision.

Problem

• LLM-as-Judge
• robustness
• bias
• contamination
• consensus evaluation

Innovation

• Robust Mean Estimation
• Geometric Median
• LLM-as-Judge
• Huber Contamination Model
• Breakdown Point