Conditional Evaluation of Language Models with Cheap Auxiliary Signals

2026-08-17

AI Summary

This study addresses the challenge of costly gold labels and biased auxiliary signals in conditional performance evaluation of language models by proposing LACE, a semi-supervised estimator. Integrating local centering with ridge-controlled variates, LACE leverages gold-label residuals and abundant cheap signals to achieve unbiased, efficient evaluation without calibration. Theoretically, we establish its uncalibrated identifiability and local oracle optimality, deriving an efficiency gain formula based on local R². Experiments across eight benchmarks, including MATH-500, demonstrate that LACE significantly enhances both evaluation efficiency and accuracy. These results validate LACE as a reliable paradigm for low-cost model assessment, effectively mitigating the trade-off between annotation expense and evaluation fidelity in large-scale language model benchmarking.

Abstract

Aggregate accuracy hides where models succeed and fail. Estimating conditional performance profiles from gold labels alone is expensive, while cheap auxiliary signals such as LLM-judge scores, pairwise comparisons, confidence scores, and judge-disagreement features can be collected for every benchmark item but are often biased or miscalibrated. We propose LACE (Local Augmented Control-Variate Evaluation), a semi-supervised estimator for conditional LLM evaluation. The key step is local centering: after subtracting the conditional mean of a cheap signal within the target profile region, any linear augmentation has zero conditional mean and therefore cannot change the estimand. The augmentation coefficient is used only for efficiency, and a local ridge control variate combines a gold-label residual mean from the labeled subset with a cheap-signal mean from the full item pool. We prove calibration-free identification, unbiasedness for grouped profiles, local oracle optimality within centered linear augmentations, and first-order adaptivity to the estimated coefficient. The resulting gain formula is governed by a population local $R^2$, which characterizes how the efficiency attainable from the cheap signals varies across profile values. We also derive corresponding estimators for direct paired model gaps and deployment-weighted scores. We empirically evaluate the primary performance-profile estimator on MATH-500, ScienceQA, MMLU, WinoGrande, HellaSwag, TruthfulQA, GSM8K, and ARC.

Problem

• Conditional Evaluation
• Language Models
• Cheap Auxiliary Signals
• Performance Profiles
• Biased Signals

Innovation

• LACE
• Local Centering
• Control Variate
• Conditional Evaluation
• Semi-supervised Estimator