Generalized Prediction-Powered Inference, with Application to Binary Classifier Evaluation

2026-02-10

AI Summary

This work extends Prediction-powered Inference (PPI) to arbitrary regular asymptotically linear estimators under partial observability and covariate distribution shift, transcending the limitations of classical M-estimation. The proposed framework establishes a general and computationally efficient semiparametric estimation approach that integrates missing data handling with semiparametric efficiency theory. It effectively corrects for three types of covariate shift and is successfully applied to the estimation of performance metrics such as true positive rate, false positive rate, and AUC. Numerical experiments demonstrate that the proposed estimator substantially outperforms baseline methods relying solely on labeled data.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Probabilistic Inference
• Intelligent Robots: State Estimation

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In the partially-observed outcome setting, a recent set of proposals known as"prediction-powered inference"(PPI) involve (i) applying a pre-trained machine learning model to predict the response, and then (ii) using these predictions to obtain an estimator of the parameter of interest with asymptotic variance no greater than that which would be obtained using only the labeled observations. While existing PPI proposals consider estimators arising from M-estimation, in this paper we generalize PPI to any regular asymptotically linear estimator. Furthermore, by situating PPI within the context of an existing rich literature on missing data and semi-parametric efficiency theory, we show that while PPI does not achieve the semi-parametric efficiency lower bound outside of very restrictive and unrealistic scenarios, it can be viewed as a computationally-simple alternative to proposals in that literature. We exploit connections to that literature to propose modified PPI estimators that can handle three distinct forms of covariate distribution shift. Finally, we illustrate these developments by constructing PPI estimators of true positive rate, false positive rate, and area under the curve via numerical studies.

Problem

• prediction-powered inference
• partially-observed outcomes
• covariate distribution shift
• asymptotically linear estimation
• semi-parametric efficiency

Innovation

• prediction-powered inference
• asymptotically linear estimator
• covariate shift
• semi-parametric efficiency
• binary classifier evaluation