Metric Match: A Subset Selection Approach to Evaluating LLM Judge Reliability

2026-06-12

AI Summary

Evaluating the reliability of large language models (LLMs) as judges typically relies on extensive human annotations, which are costly. This work proposes Metric Match, a novel approach that aligns subset selection with overall reliability estimation by integrating correlation-based subset selection, synthetic label utilization, and cost modeling to efficiently estimate agreement between LLMs and human raters. The method substantially reduces annotation requirements while improving estimation accuracy, achieving an average 18.7% reduction in estimation error and a 32.5% decrease in annotation volume across 15 datasets, with a win rate of 0.838. In a medical application scenario, it further demonstrates practical value by saving $1,041.67 in expert annotation costs.

Abstract

LLM judges are used to reduce the need for costly human labor in evaluating open-ended text generation. However, the reliability of these judges depends critically on their alignment with human raters -- a property that itself depends on costly human annotations. In this work, we develop a method (Metric Match) for estimating correlation-based reliability metrics of LLM judges from limited annotations. Metric Match selects a subset of samples for human annotation such that the subset matches the population reliability metric with respect to acquired synthetic labels. We empirically show that Metric Match achieves a win-rate of 0.838 against random subset selection across four different correlation metrics and 15 datasets, with an 18.7% decrease in average estimation error and reduces annotation needs by 32.5%. We provide a cost model and highlight a medical case study where our method saves $1,041.67 compared to random selection for expert annotation. Further, we shift our task from reliability estimation to reliability classification of whether a given judge is above a deployment threshold, outperforming random selection with Metric Match. All project code is publicly available, and we additionally provide an installable package for ease of use.

Problem

• LLM judge reliability
• human annotation cost
• correlation estimation
• subset selection
• evaluation metrics

Innovation

• subset selection
• LLM judge reliability
• correlation estimation
• annotation efficiency
• Metric Match