Generative Conformal Prediction with Vectorized Non-Conformity Scores

2024-10-17

AI Summary

Traditional conformal prediction suffers from overly conservative uncertainty sets in multivariate settings due to its reliance on scalar nonconformity scores. To address this, we propose a vectorized nonconformity scoring framework: leveraging generative models—such as conditional VAEs or diffusion models—to sample multiple predictions, and integrating density estimation with hierarchical empirical quantiles to construct density-aware, adaptive uncertainty balls. This approach transcends the limitations of scalar scoring by enabling dynamic uncertainty allocation—expanding high-confidence regions, contracting low-confidence ones, and even excluding implausible regions. The method is theoretically guaranteed to maintain statistical validity (i.e., marginal coverage). Empirical evaluation on synthetic and real-world datasets demonstrates an average reduction of 23%–41% in the volume of uncertainty sets while strictly preserving target coverage, significantly outperforming existing state-of-the-art methods.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Multiagent Systems: Multiagent Systems under Uncertainty

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Large-scale security measurements

Abstract

Conformal prediction (CP) provides model-agnostic uncertainty quantification with guaranteed coverage, but conventional methods often produce overly conservative uncertainty sets, especially in multi-dimensional settings. This limitation arises from simplistic non-conformity scores that rely solely on prediction error, failing to capture the prediction error distribution's complexity. To address this, we propose a generative conformal prediction framework with vectorized non-conformity scores, leveraging a generative model to sample multiple predictions from the fitted data distribution. By computing non-conformity scores across these samples and estimating empirical quantiles at different density levels, we construct adaptive uncertainty sets using density-ranked uncertainty balls. This approach enables more precise uncertainty allocation -- yielding larger prediction sets in high-confidence regions and smaller or excluded sets in low-confidence regions -- enhancing both flexibility and efficiency. We establish theoretical guarantees for statistical validity and demonstrate through extensive numerical experiments that our method outperforms state-of-the-art techniques on synthetic and real-world datasets.

Problem

• Addresses overly conservative uncertainty sets
• Introduces vectorized non-conformity scores
• Enhances precision in uncertainty allocation

Innovation

• Generative conformal prediction
• Vectorized non-conformity scores
• Density-ranked uncertainty balls