Do More Predictions Improve Statistical Inference? Filtered Prediction-Powered Inference

2026-02-11

AI Summary

This work addresses the challenge of statistical inference in settings where ground-truth labels are scarce but abundant, yet noisy, predictions are available. Conventional prediction-powered inference methods often degrade accuracy by indiscriminately incorporating all predictions, regardless of their reliability. To overcome this limitation, we propose Filtered Prediction-Powered Inference (FPPI), a novel framework that introduces a data-adaptive filtering mechanism to selectively apply prediction-based corrections only in regions where predictions are informatively accurate. This approach explicitly rejects the prevailing “more predictions are better” assumption. Theoretical analysis demonstrates that FPPI achieves superior asymptotic efficiency and faster convergence rates. Empirical evaluations confirm that FPPI significantly enhances inference accuracy and robustness—particularly in tasks such as language model evaluation—while simultaneously reducing reliance on costly labeled data.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Probabilistic Inference
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks

Abstract

Recent advances in artificial intelligence have enabled the generation of large-scale, low-cost predictions with increasingly high fidelity. As a result, the primary challenge in statistical inference has shifted from data scarcity to data reliability. Prediction-powered inference methods seek to exploit such predictions to improve efficiency when labeled data are limited. However, existing approaches implicitly adopt a use-all philosophy, under which incorporating more predictions is presumed to improve inference. When prediction quality is heterogeneous, this assumption can fail, and indiscriminate use of unlabeled data may dilute informative signals and degrade inferential accuracy. In this paper, we propose Filtered Prediction-Powered Inference (FPPI), a framework that selectively incorporates predictions by identifying a data-adaptive filtered region in which predictions are informative for inference. We show that this region can be consistently estimated under a margin condition, achieving fast rates of convergence. By restricting the prediction-powered correction to the estimated filtered region, FPPI adaptively mitigates the impact of biased or noisy predictions. We establish that FPPI attains strictly improved asymptotic efficiency compared with existing prediction-powered inference methods. Numerical studies and a real-data application to large language model evaluation demonstrate that FPPI substantially reduces reliance on expensive labels by selectively leveraging reliable predictions, yielding accurate inference even in the presence of heterogeneous prediction quality.

Problem

• prediction-powered inference
• heterogeneous prediction quality
• statistical inference
• data reliability
• filtered region

Innovation

• Filtered Prediction-Powered Inference
• prediction-powered inference
• data-adaptive filtering
• heterogeneous prediction quality
• statistical efficiency