Prediction-Powered Active Testing

2026-07-09

AI Summary

This work addresses the challenge of efficiently estimating model risk under high labeling costs, where existing active testing methods struggle to leverage predictions from black-box models effectively. We propose Prediction-driven Active Testing (PPAT), a novel framework that treats black-box predictions as control variates to residualize the loss—rather than using them as pseudo-labels—and integrates an unbiased LURE estimator to substantially reduce variance while preserving unbiasedness. We derive both theoretically optimal and practically feasible sample acquisition strategies for variance minimization and establish the asymptotic normality of the resulting estimator to enable valid confidence interval construction. Empirical results demonstrate that PPAT achieves more accurate risk estimates with fewer labeled samples across tabular regression and image classification tasks, yielding significantly narrower confidence intervals that still attain the desired coverage.

Abstract

Active testing provides a label--efficient approach to risk estimation by adaptively selecting which test points should be labelled. However, existing estimators fail to exploit the informative predictions of powerful black--box models, even though such predictions are increasingly available in settings where labels remain expensive. To address this, we propose \textbf{Prediction--Powered Active Testing (PPAT)}, a novel label--efficient risk estimation framework that combines the unbiased LURE estimator \citep{farquhar2021statistical} with a prediction--powered control variate. Rather than using proxy predictions as biased pseudo--labels, PPAT uses them to residualise the loss, preserving unbiasedness while reducing variance. Beyond the estimator itself, PPAT also changes which points should be acquired: we derive oracle and practical surrogate--based acquisition rules tailored to reducing the variance of our estimator. Moreover, we establish asymptotic normality for PPAT, yielding asymptotically valid confidence intervals and thus a principled estimate of the uncertainty around our estimates. Across tabular regression and image--classification tasks, PPAT outperforms existing methods in risk estimation, while its confidence intervals attain the target coverage with substantially fewer labels and smaller widths.

Problem

• active testing
• risk estimation
• black-box models
• label efficiency
• prediction-powered inference

Innovation

• Prediction-Powered Inference
• Active Testing
• Control Variates
• Label-Efficient Risk Estimation
• Confidence Intervals