CAP: A General Algorithm for Online Selective Conformal Prediction with FCR Control

2024-03-12

arXiv.org

AI Summary

This paper addresses the time-multiplicity problem arising from dynamic screening in online selective prediction, proposing CAP—a novel framework that strictly controls the real-time false coverage rate (FCR). CAP is the first method to achieve exact conditional coverage guarantees under finite samples and distribution-free assumptions. It integrates adaptive selection rules, post-selection calibration, and online conformal prediction, augmented with a dynamic distribution-shift adaptation mechanism to ensure long-term FCR convergence to the target level. Experiments on synthetic and real-world datasets demonstrate that CAP precisely maintains the prescribed FCR while yielding substantially narrower prediction intervals than state-of-the-art baselines, thereby reconciling statistical rigor with predictive efficiency.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

We study the problem of post-selection predictive inference in an online fashion. To avoid devoting resources to unimportant units, a preliminary selection of the current individual before reporting its prediction interval is common and meaningful in online predictive tasks. Since the online selection causes a temporal multiplicity in the selected prediction intervals, it is important to control the real-time false coverage-statement rate (FCR) which measures the overall miscoverage level. We develop a general framework named CAP (Calibration after Adaptive Pick) that performs an adaptive pick rule on historical data to construct a calibration set if the current individual is selected and then outputs a conformal prediction interval for the unobserved label. We provide tractable procedures for constructing the calibration set for popular online selection rules. We proved that CAP can achieve an exact selection-conditional coverage guarantee in the finite-sample and distribution-free regimes. To account for the distribution shift in online data, we also embed CAP into some recent dynamic conformal prediction algorithms and show that the proposed method can deliver long-run FCR control. Numerical results on both synthetic and real data corroborate that CAP can effectively control FCR around the target level and yield more narrowed prediction intervals over existing baselines across various settings.

Problem

• Controls false coverage rate in online prediction tasks
• Adapts to distribution shifts in streaming data
• Ensures accurate, narrow prediction intervals post-selection

Innovation

• Adaptive pick rule for calibration set
• Exact selection-conditional coverage guarantee
• Dynamic conformal prediction for distribution shift