The Importance of Being Smoothly Calibrated

2026-03-16

AI Summary

This work addresses the challenge of achieving low regret calibrated prediction under unknown loss functions. It introduces the notion of smooth calibration and constructs a universal prediction framework that competes with any benchmark predictor across all bounded proper losses. The core contribution establishes a tight, unimprovable equivalence bound between smooth calibration error and universal prediction error. Furthermore, it reveals that the upper calibration distance is inherently constrained by sample complexity—specifically, it cannot be estimated within a quadratic factor without dependence on the support of the predictor’s outputs. The paper also provides a clear characterization of smooth calibration via the Earth Mover’s Distance, thereby unifying and extending existing theories of universal prediction.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Recent work has highlighted the centrality of smooth calibration [Kakade and Foster, 2008] as a robust measure of calibration error. We generalize, unify, and extend previous results on smooth calibration, both as a robust calibration measure, and as a step towards omniprediction, which enables predictions with low regret for downstream decision makers seeking to optimize some proper loss unknown to the predictor. We present a new omniprediction guarantee for smoothly calibrated predictors, for the class of all bounded proper losses. We smooth the predictor by adding some noise to it, and compete against smoothed versions of any benchmark predictor on the space, where we add some noise to the predictor and then post-process it arbitrarily. The omniprediction error is bounded by the smooth calibration error of the predictor and the earth mover's distance from the benchmark. We exhibit instances showing that this dependence cannot, in general, be improved. We show how this unifies and extends prior results [Foster and Vohra, 1998; Hartline, Wu, and Yang, 2025] on omniprediction from smooth calibration. We present a crisp new characterization of smooth calibration in terms of the earth mover's distance to the closest perfectly calibrated joint distribution of predictions and labels. This also yields a simpler proof of the relation to the lower distance to calibration from [Blasiok, Gopalan, Hu, and Nakkiran, 2023]. We use this to show that the upper distance to calibration cannot be estimated within a quadratic factor with sample complexity independent of the support size of the predictions. This is in contrast to the distance to calibration, where the corresponding problem was known to be information-theoretically impossible: no finite number of samples suffice [Blasiok, Gopalan, Hu, and Nakkiran, 2023].

Problem

• smooth calibration
• omniprediction
• calibration error
• earth mover's distance
• proper loss

Innovation

• smooth calibration
• omniprediction
• earth mover's distance
• proper loss
• calibration distance