$L_2$-Regularized Empirical Risk Minimization Guarantees Small Smooth Calibration Error

2025-10-15

AI Summary

This paper investigates whether standard ℓ₂-regularized empirical risk minimization (ERM) can achieve good probabilistic calibration *intrinsically*, without relying on post-hoc calibration or dedicated calibration regularizers. Methodologically, it establishes the first generalization analysis framework for smooth calibration error (smCE), explicitly linking calibration performance to optimization error, regularization strength, and Rademacher complexity. In reproducing kernel Hilbert spaces (RKHS), it derives finite-sample upper bounds on smCE for kernel ridge regression and logistic regression. Experimentally, standard ℓ₂-ERM alone achieves calibration performance competitive with state-of-the-art calibration-specific methods. The core contribution is the theoretical revelation that ℓ₂ regularization inherently provides calibration guarantees, accompanied by the first interpretable and quantifiable generalization theory for calibration—bridging regularization, optimization, and statistical learning theory in a unified framework.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness
• Reasoning under Uncertainty: Graphical Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Calibration of predicted probabilities is critical for reliable machine learning, yet it is poorly understood how standard training procedures yield well-calibrated models. This work provides the first theoretical proof that canonical $L_{2}$-regularized empirical risk minimization directly controls the smooth calibration error (smCE) without post-hoc correction or specialized calibration-promoting regularizer. We establish finite-sample generalization bounds for smCE based on optimization error, regularization strength, and the Rademacher complexity. We then instantiate this theory for models in reproducing kernel Hilbert spaces, deriving concrete guarantees for kernel ridge and logistic regression. Our experiments confirm these specific guarantees, demonstrating that $L_{2}$-regularized ERM can provide a well-calibrated model without boosting or post-hoc recalibration. The source code to reproduce all experiments is available at https://github.com/msfuji0211/erm_calibration.

Problem

• Theoretical proof that L2-regularized empirical risk minimization controls smooth calibration error
• Establishes generalization bounds for calibration error based on optimization and regularization
• Demonstrates well-calibrated models without post-hoc correction or specialized regularizers

Innovation

• L2-regularized empirical risk minimization controls calibration error
• Theoretical proof for smooth calibration without post-hoc correction
• Generalization bounds based on optimization error and regularization