Optimal training-conditional regret for online conformal prediction

2026-02-18

AI Summary

This work addresses the degradation of online conformal prediction performance under non-stationary data streams caused by unknown distribution shifts, including both abrupt and gradual drifts. To tackle this challenge, the paper proposes two novel algorithms—one tailored for pre-trained settings and the other for online training—both incorporating drift detection mechanisms to dynamically update either the calibration set or the underlying model. Departing from the conventional assumption of exchangeability, the approach leverages model stability and, for the first time, establishes a non-asymptotic analysis within a training-conditioned regret framework. Theoretically, the proposed algorithms achieve minimax-optimal regret bounds, and empirical evaluations demonstrate their effectiveness in maintaining valid prediction sets across diverse drift scenarios.

Technology Category

• Machine Learning: Online Learning & Bandits
• Reasoning under Uncertainty: Stochastic Optimization
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

We study online conformal prediction for non-stationary data streams subject to unknown distribution drift. While most prior work studied this problem under adversarial settings and/or assessed performance in terms of gaps of time-averaged marginal coverage, we instead evaluate performance through training-conditional cumulative regret. We specifically focus on independently generated data with two types of distribution shift: abrupt change points and smooth drift. When non-conformity score functions are pretrained on an independent dataset, we propose a split-conformal style algorithm that leverages drift detection to adaptively update calibration sets, which provably achieves minimax-optimal regret. When non-conformity scores are instead trained online, we develop a full-conformal style algorithm that again incorporates drift detection to handle non-stationarity; this approach relies on stability - rather than permutation symmetry - of the model-fitting algorithm, which is often better suited to online learning under evolving environments. We establish non-asymptotic regret guarantees for our online full conformal algorithm, which match the minimax lower bound under appropriate restrictions on the prediction sets. Numerical experiments corroborate our theoretical findings.

Problem

• online conformal prediction
• distribution drift
• non-stationary data
• training-conditional regret
• change points

Innovation

• online conformal prediction
• distribution drift
• training-conditional regret
• drift detection
• minimax regret