Online Conformal Inference with Retrospective Adjustment for Faster Adaptation to Distribution Shift

2025-11-06

AI Summary

Online conformal prediction faces challenges from data non-exchangeability: distribution shift causes coverage failure in conventional methods, and forward-only update schemes hinder rapid adaptation. To address this, we propose an online conformal inference framework with a retrospective recalibration mechanism—upon arrival of new data, it dynamically refines historical predictions via an efficient leave-one-out update formula, overcoming the limitation of purely forward updates. Within a regression setting, our method integrates conformal prediction while theoretically guaranteeing asymptotic coverage validity under non-exchangeable data. Experiments on synthetic and real-world datasets demonstrate that our approach significantly accelerates coverage probability recalibration, improves predictive set validity and stability, and outperforms existing online conformal prediction methods.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Conformal prediction has emerged as a powerful framework for constructing distribution-free prediction sets with guaranteed coverage assuming only the exchangeability assumption. However, this assumption is often violated in online environments where data distributions evolve over time. Several recent approaches have been proposed to address this limitation, but, typically, they slowly adapt to distribution shifts because they update predictions only in a forward manner, that is, they generate a prediction for a newly observed data point while previously computed predictions are not updated. In this paper, we propose a novel online conformal inference method with retrospective adjustment, which is designed to achieve faster adaptation to distributional shifts. Our method leverages regression approaches with efficient leave-one-out update formulas to retroactively adjust past predictions when new data arrive, thereby aligning the entire set of predictions with the most recent data distribution. Through extensive numerical studies performed on both synthetic and real-world data sets, we show that the proposed approach achieves faster coverage recalibration and improved statistical efficiency compared to existing online conformal prediction methods.

Problem

• Addresses slow adaptation to distribution shifts in online conformal prediction
• Retroactively adjusts past predictions when new data becomes available
• Achieves faster coverage recalibration in evolving data environments

Innovation

• Retroactively adjusts past predictions using new data
• Leverages regression with leave-one-out update formulas
• Aligns all predictions with recent data distribution