Information-theoretic Generalization Analysis for Expected Calibration Error

2024-05-24

Neural Information Processing Systems

AI Summary

This work addresses the lack of theoretical characterization of binning-induced bias in estimating the Expected Calibration Error (ECE). We systematically analyze estimation bias under two standard binning schemes—uniform mass and uniform width—deriving improved upper bounds on bias with tighter convergence rates and identifying the optimal number of bins that minimizes bias. Innovatively, we establish an information-theoretic framework for analyzing the generalization error of ECE estimation, revealing for the first time a quantitative relationship between estimation bias and the number of bins. This yields a computable, distribution-free upper bound on the true calibration performance of unknown data. Experiments on models including ResNet and CNN empirically validate the tightness and effectiveness of the bound. Our results provide a verifiable theoretical foundation for calibration in trustworthy machine learning.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Computer Vision: Bias, Fairness & Privacy

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Security and Privacy: Large-scale security measurements

Abstract

While the expected calibration error (ECE), which employs binning, is widely adopted to evaluate the calibration performance of machine learning models, theoretical understanding of its estimation bias is limited. In this paper, we present the first comprehensive analysis of the estimation bias in the two common binning strategies, uniform mass and uniform width binning. Our analysis establishes upper bounds on the bias, achieving an improved convergence rate. Moreover, our bounds reveal, for the first time, the optimal number of bins to minimize the estimation bias. We further extend our bias analysis to generalization error analysis based on the information-theoretic approach, deriving upper bounds that enable the numerical evaluation of how small the ECE is for unknown data. Experiments using deep learning models show that our bounds are nonvacuous thanks to this information-theoretic generalization analysis approach.

Problem

• Analyzes estimation bias in ECE binning strategies
• Establishes optimal bin number to minimize bias
• Extends bias analysis to generalization error bounds

Innovation

• Analyzes bias in uniform mass and width binning
• Establishes upper bounds with improved convergence rates
• Extends to information-theoretic generalization error analysis