Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the data

2024-10-17

arXiv.org

AI Summary

LLM-as-judge evaluation—using large language models as automated annotators—reduces human labeling costs but suffers from distortion due to judge-model capability limitations or self-preference bias. Method: We establish the first theoretical framework grounded in information theory and statistical inference to characterize the fundamental annotation-efficiency ceiling of this paradigm, complemented by empirical validation across diverse model pairs and tasks. Contribution/Results: We prove that when the judge model is not strictly superior to the evaluated model, debiasing methods cannot reduce human annotation requirements by more than 50%, and observed gains are consistently far below this bound. This reveals an intrinsic limitation of LLM-as-judge for evaluating state-of-the-art models, establishing a critical theoretical boundary for trustworthy AI evaluation and affirming the irreplaceable role of high-quality human annotations.

Technology Category

• Machine Learning: Evaluation and Analysis
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

High quality annotations are increasingly a bottleneck in the explosively growing machine learning ecosystem. Scalable evaluation methods that avoid costly annotation have therefore become an important research ambition. Many hope to use strong existing models in lieu of costly labels to provide cheap model evaluations. Unfortunately, this method of using models as judges introduces biases, such as self-preferencing, that can distort model comparisons. An emerging family of debiasing tools promises to fix these issues by using a few high quality labels to debias a large number of model judgments. In this paper, we study how far such debiasing methods, in principle, can go. Our main result shows that when the judge is no more accurate than the evaluated model, no debiasing method can decrease the required amount of ground truth labels by more than half. Our result speaks to the severe limitations of the LLM-as-a-judge paradigm at the evaluation frontier where the goal is to assess newly released models that are possibly better than the judge. Through an empirical evaluation, we demonstrate that the sample size savings achievable in practice are even more modest than what our theoretical limit suggests. Along the way, our work provides new observations about debiasing methods for model evaluation, and points out promising avenues for future work.

Problem

• Scalable evaluation without costly annotation
• Biases in using models as judges
• Limits of debiasing methods in model evaluation

Innovation

• LLM as Judge paradigm
• Debiasing tools for judgments
• Scalable evaluation methods