Adaptive Calibration in Non-Stationary Environments

2026-05-12

AI Summary

This work addresses the challenge of adapting calibration error to the degree of non-stationarity in online prediction settings. To this end, it proposes a novel online algorithm that integrates a time-scheduling mechanism with a non-uniform partitioning of the prediction space—allocating higher resolution around the true outcomes. This approach achieves, for the first time, a smooth interpolation and unified theoretical guarantee for calibration error across the spectrum from i.i.d. to fully adversarial environments. Over $T$ rounds, the method attains an $\ell_1$ calibration error of $\widetilde{O}(\sqrt{T} + (TC)^{1/3})$, and both $\ell_2$ and pseudo-KL calibration errors of $\widetilde{O}((1+C)^{1/3})$, simultaneously matching the optimal rates known in both stationary and adversarial regimes.

Abstract

Making calibrated online predictions is a central challenge in modern AI systems. Much of the existing literature focuses on fully adversarial environments where outcomes may be arbitrary, leading to conservative algorithms that can perform suboptimally in more benign settings, such as when outcomes are nearly stationary. This gap raises a natural question: can we design online prediction algorithms whose calibration error automatically adapts to the degree of non-stationarity in the environment, smoothly interpolating between i.i.d. and adversarial regimes? We answer this question in the affirmative and develop a suite of algorithms that achieve adaptive calibration guarantees under multiple calibration measures. Specifically, with $T$ being the number of rounds and $C\in[0,T]$ being an unknown non-stationary measure defined as the minimal $\ell_1$ deviation of the mean outcomes, our algorithms attain $\widetilde{O}(\sqrt{T}+(TC)^{\frac{1}{3}})$ for $\ell_1$ calibration error and $\widetilde{O}((1+C)^{\frac{1}{3}})$ for both $\ell_2$ and pseudo KL calibration error. These bounds match the optimal rates in the stationary case ($C=0$) and recover known guarantees in the fully adversarial regime ($C=T$). Our approach builds on and extends prior work [Hu et al., 2026, Luo et al., 2025], introducing an epoch-based scheduling together with a novel non-uniform partition of the prediction space that allocates finer resolution near the underlying ground truth.

Problem

• adaptive calibration
• non-stationary environments
• online prediction
• calibration error
• distribution shift

Innovation

• adaptive calibration
• non-stationary environments
• online prediction
• epoch-based scheduling
• non-uniform partition