Model uncertainty quantification using feature confidence sets for outcome excursions

2025-04-28

AI Summary

In high-stakes domains—such as healthcare, finance, and autonomous driving—there is a critical need to identify feature subspaces responsible for model predictions exceeding safety-critical thresholds. Method: This paper introduces the “outcome excursion confidence set” framework, the first model-agnostic, data-driven approach for uncertainty quantification targeting threshold-exceeding behavior. It constructs inner and outer confidence sets to precisely localize feature subregions that drive continuous or binary outputs beyond a user-specified threshold, integrating nonparametric statistical inference with threshold-driven subspace identification. Contribution/Results: The method provides both finite-sample and asymptotic theoretical guarantees. Evaluated on real-world tasks—including house price prediction and early sepsis diagnosis—it achieves empirically verified coverage rates strictly satisfying nominal confidence levels, while substantially enhancing interpretability and reliability of high-risk decisions.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Multiagent Systems: Multiagent Systems under Uncertainty

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

When implementing prediction models for high-stakes real-world applications such as medicine, finance, and autonomous systems, quantifying prediction uncertainty is critical for effective risk management. Traditional approaches to uncertainty quantification, such as confidence and prediction intervals, provide probability coverage guarantees for the expected outcomes $f(oldsymbol{x})$ or the realized outcomes $f(oldsymbol{x})+epsilon$. Instead, this paper introduces a novel, model-agnostic framework for quantifying uncertainty in continuous and binary outcomes using confidence sets for outcome excursions, where the goal is to identify a subset of the feature space where the expected or realized outcome exceeds a specific value. The proposed method constructs data-dependent inner and outer confidence sets that aim to contain the true feature subset for which the expected or realized outcomes of these features exceed a specified threshold. We establish theoretical guarantees for the probability that these confidence sets contain the true feature subset, both asymptotically and for finite sample sizes. The framework is validated through simulations and applied to real-world datasets, demonstrating its utility in contexts such as housing price prediction and time to sepsis diagnosis in healthcare. This approach provides a unified method for uncertainty quantification that is broadly applicable across various continuous and binary prediction models.

Problem

• Quantify prediction uncertainty for high-stakes applications
• Develop model-agnostic confidence sets for outcome excursions
• Validate framework with theoretical guarantees and real-world datasets

Innovation

• Model-agnostic framework for uncertainty quantification
• Confidence sets for outcome excursions
• Theoretical guarantees for feature subset containment