Do We Really Even Need Data? A Modern Look at Drawing Inference with Predicted Data

2025-12-05

AI Summary

When pretrained models impute missing data, even highly accurate predictions induce bias in causal inference by ignoring prediction uncertainty, thereby invalidating standard statistical inference. Method: We develop a unified framework attributing inference failure to the dual effects—bias and variance—introduced by prediction; conduct error propagation analysis to explicitly model how prediction uncertainty propagates to final causal estimates; and reestablish theoretical links between individual participant data (IPD) analysis and classical statistical theory. Contribution/Results: We demonstrate that direct imputation distorts standard errors and confidence intervals. Our framework yields statistically principled, actionable guidelines for using predicted values in causal estimation, enabling proper uncertainty quantification. Empirical validation confirms substantial improvements in the reliability of causal inference based on machine learning–preprocessed data, bridging the gap between modern predictive modeling and rigorous causal inference.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Causality
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

As artificial intelligence and machine learning tools become more accessible, and scientists face new obstacles to data collection (e.g., rising costs, declining survey response rates), researchers increasingly use predictions from pre-trained algorithms as substitutes for missing or unobserved data. Though appealing for financial and logistical reasons, using standard tools for inference can misrepresent the association between independent variables and the outcome of interest when the true, unobserved outcome is replaced by a predicted value. In this paper, we characterize the statistical challenges inherent to drawing inference with predicted data (IPD) and show that high predictive accuracy does not guarantee valid downstream inference. We show that all such failures reduce to statistical notions of (i) bias, when predictions systematically shift the estimand or distort relationships among variables, and (ii) variance, when uncertainty from the prediction model and the intrinsic variability of the true data are ignored. We then review recent methods for conducting IPD and discuss how this framework is deeply rooted in classical statistical theory. We then comment on some open questions and interesting avenues for future work in this area, and end with some comments on how to use predicted data in scientific studies that is both transparent and statistically principled.

Problem

• Addresses statistical challenges of using predicted data for inference
• Identifies bias and variance issues in predicted data analysis
• Reviews methods for transparent and principled use of predicted data

Innovation

• Using predicted data as substitutes for missing observations
• Characterizing inference failures via bias and variance issues
• Reviewing methods rooted in classical statistical theory