Calibeating Prediction-Powered Inference

2026-04-23

AI Summary

This work addresses the efficiency loss in semi-supervised mean estimation caused by uncalibrated outputs from black-box prediction models. Under a setting with limited labeled samples and abundant unlabeled data, we propose a post-hoc calibration approach that requires no model retraining. We introduce linear and isotonic calibration strategies, establish the first-order optimality of isotonic calibration, and demonstrate the first-order equivalence between linear calibration and PPI++. This analysis clarifies the theoretical connections among PPI, AIPW, and PPI++. By integrating augmented inverse probability weighting (AIPW) with empirical efficiency maximization, our method substantially outperforms the original PPI on both simulated and real-world datasets, achieving performance comparable to or even surpassing that of AIPW and PPI++. We also release an open-source Python package, ppi_aipw, to facilitate reproducibility and adoption.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Probabilistic Programming
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We study semisupervised mean estimation with a small labeled sample, a large unlabeled sample, and a black-box prediction model whose output may be miscalibrated. A standard approach in this setting is augmented inverse-probability weighting (AIPW) [Robins et al., 1994], which protects against prediction-model misspecification but can be inefficient when the prediction score is poorly aligned with the outcome scale. We introduce Calibrated Prediction-Powered Inference, which post-hoc calibrates the prediction score on the labeled sample before using it for semisupervised estimation. This simple step requires no retraining and can improve the original score both as a predictor of the outcome and as a regression adjustment for semisupervised inference. We study both linear and isotonic calibration. For isotonic calibration, we establish first-order optimality guarantees: isotonic post-processing can improve predictive accuracy and estimator efficiency relative to the original score and simpler post-processing rules, while no further post-processing of the fitted isotonic score yields additional first-order gains. For linear calibration, we show first-order equivalence to PPI++. We also clarify the relationship among existing estimators, showing that the original PPI estimator is a special case of AIPW and can be inefficient when the prediction model is accurate, while PPI++ is AIPW with empirical efficiency maximization [Rubin et al., 2008]. In simulations and real-data experiments, our calibrated estimators often outperform PPI and are competitive with, or outperform, AIPW and PPI++. We provide an accompanying Python package, ppi_aipw, at https://larsvanderlaan.github.io/ppi-aipw/.

Problem

• semisupervised inference
• prediction-powered inference
• miscalibrated predictions
• mean estimation
• black-box model

Innovation

• calibrated prediction
• semisupervised inference
• isotonic calibration
• AIPW
• prediction-powered inference