Valid Selection among Conformal Sets

2025-06-25

AI Summary

Conformal prediction provides statistically valid prediction sets, yet directly selecting the optimal set—e.g., the smallest—among multiple valid candidates violates the nominal coverage guarantee. This work proposes a stability-based set selection framework that robustly identifies the most compact prediction set while strictly preserving the target coverage level. We further extend this strategy to the online learning setting for the first time, introducing a structured update mechanism to adapt to evolving data streams. Theoretically, we prove that the selected sets retain distribution-free marginal coverage guarantees. Empirically, our method significantly improves prediction set tightness across diverse models and datasets, consistently maintaining high coverage and strong robustness to distribution shifts and model misspecification.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Stochastic Optimization
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Conformal prediction offers a distribution-free framework for constructing prediction sets with coverage guarantees. In practice, multiple valid conformal prediction sets may be available, arising from different models or methodologies. However, selecting the most desirable set, such as the smallest, can invalidate the coverage guarantees. To address this challenge, we propose a stability-based approach that ensures coverage for the selected prediction set. We extend our results to the online conformal setting, propose several refinements in settings where additional structure is available, and demonstrate its effectiveness through experiments.

Problem

• Selecting optimal conformal prediction sets without losing coverage guarantees
• Ensuring validity when choosing smallest prediction set among options
• Extending stability-based selection to online conformal prediction settings

Innovation

• Stability-based approach ensures coverage guarantees
• Extends results to online conformal setting
• Refinements for structured settings enhance effectiveness