PCS-UQ: Uncertainty Quantification via the Predictability-Computability-Stability Framework

2025-05-13

AI Summary

Existing uncertainty quantification (UQ) methods for machine learning in high-stakes settings face critical limitations: traditional approaches rely on correctly specified generative models and are sensitive to model misspecification, while conformal prediction ignores model selection and yields overly conservative prediction intervals. Method: We propose a novel UQ framework grounded in the Predictability-Computability-Stability (PCS) paradigm—first systematically integrating PCS principles into UQ. Our approach selects optimal models via predictive checking, evaluates algorithmic and sample variability using multi-Bootstrap, and introduces a local adaptive calibration mechanism. Contribution/Results: Theoretically, our method guarantees valid coverage probability. Empirically, across 17 regression and 6 classification datasets, plus 3 vision benchmarks, it reduces average prediction interval/set width by 20% compared to split conformal prediction, while substantially improving subgroup robustness and compactness.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

As machine learning (ML) models are increasingly deployed in high-stakes domains, trustworthy uncertainty quantification (UQ) is critical for ensuring the safety and reliability of these models. Traditional UQ methods rely on specifying a true generative model and are not robust to misspecification. On the other hand, conformal inference allows for arbitrary ML models but does not consider model selection, which leads to large interval sizes. We tackle these drawbacks by proposing a UQ method based on the predictability, computability, and stability (PCS) framework for veridical data science proposed by Yu and Kumbier. Specifically, PCS-UQ addresses model selection by using a prediction check to screen out unsuitable models. PCS-UQ then fits these screened algorithms across multiple bootstraps to assess inter-sample variability and algorithmic instability, enabling more reliable uncertainty estimates. Further, we propose a novel calibration scheme that improves local adaptivity of our prediction sets. Experiments across $17$ regression and $6$ classification datasets show that PCS-UQ achieves the desired coverage and reduces width over conformal approaches by $approx 20%$. Further, our local analysis shows PCS-UQ often achieves target coverage across subgroups while conformal methods fail to do so. For large deep-learning models, we propose computationally efficient approximation schemes that avoid the expensive multiple bootstrap trainings of PCS-UQ. Across three computer vision benchmarks, PCS-UQ reduces prediction set size over conformal methods by $20%$. Theoretically, we show a modified PCS-UQ algorithm is a form of split conformal inference and achieves the desired coverage with exchangeable data.

Problem

• Addresses unreliable uncertainty quantification in ML models
• Improves model selection and stability via PCS framework
• Reduces prediction interval size compared to conformal methods

Innovation

• UQ method based on PCS framework
• Prediction check screens unsuitable models
• Novel calibration improves local adaptivity