Relevance-Aware Thresholding in Online Conformal Prediction for Time Series

2025-10-03

AI Summary

Existing online conformal prediction (OCP) methods update the conformity threshold solely based on binary coverage signals, neglecting the continuous relationship between prediction intervals and true values—leading to abrupt threshold adjustments and overly conservative intervals. This work proposes a correlation-aware OCP framework that replaces discrete coverage feedback with a differentiable, bounded correlation scoring function (e.g., normalized distance or calibrated likelihood), enabling smooth, adaptive threshold updates. The method dynamically calibrates prediction intervals under nonstationary time-series distributions while strictly maintaining the target coverage level. Experiments demonstrate that, compared to state-of-the-art OCP approaches, our method reduces average interval width by 12.7%–28.4%, while keeping coverage deviation below 0.5%. This significantly improves the precision–reliability trade-off in uncertainty quantification.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Uncertainty quantification has received considerable interest in recent works in Machine Learning. In particular, Conformal Prediction (CP) gains ground in this field. For the case of time series, Online Conformal Prediction (OCP) becomes an option to address the problem of data distribution shift over time. Indeed, the idea of OCP is to update a threshold of some quantity (whether the miscoverage level or the quantile) based on the distribution observation. To evaluate the performance of OCP methods, two key aspects are typically considered: the coverage validity and the prediction interval width minimization. Recently, new OCP methods have emerged, offering long-run coverage guarantees and producing more informative intervals. However, during the threshold update step, most of these methods focus solely on the validity of the prediction intervals~--~that is, whether the ground truth falls inside or outside the interval~--~without accounting for their relevance. In this paper, we aim to leverage this overlooked aspect. Specifically, we propose enhancing the threshold update step by replacing the binary evaluation (inside/outside) with a broader class of functions that quantify the relevance of the prediction interval using the ground truth. This approach helps prevent abrupt threshold changes, potentially resulting in narrower prediction intervals. Indeed, experimental results on real-world datasets suggest that these functions can produce tighter intervals compared to existing OCP methods while maintaining coverage validity.

Problem

• Enhancing threshold updates in online conformal prediction
• Quantifying prediction interval relevance using ground truth
• Achieving narrower intervals while maintaining coverage validity

Innovation

• Replaces binary evaluation with relevance functions
• Prevents abrupt threshold changes in updates
• Produces narrower prediction intervals while valid