Combining Priors with Experience: Confidence Calibration Based on Binomial Process Modeling

2024-12-14

arXiv.org

AI Summary

Existing confidence calibration methods predominantly rely on statistical fitting, neglecting the underlying prior distribution governing calibration curves. This work proposes a novel calibration framework based on the Binomial Process (BPM), the first to model calibration data as a binomial process. We theoretically establish its Lipschitz continuity and high sample efficiency—requiring only $3/B$ samples compared to histogram-based methods (where $B$ is the number of bins). Our approach jointly incorporates prior knowledge and empirical observations, fitting a continuous calibration curve via maximum likelihood estimation and joint optimization. We further introduce the Total Calibration Error (TCE$_{ ext{pm}}$), a consistent and unbiased metric for calibration error assessment. Extensive experiments on both synthetic and real-world datasets demonstrate that our method significantly outperforms state-of-the-art approaches in calibration accuracy, robustness under limited samples, and consistency of error estimation.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Confidence calibration of classification models is a technique to estimate the true posterior probability of the predicted class, which is critical for ensuring reliable decision-making in practical applications. Existing confidence calibration methods mostly use statistical techniques to estimate the calibration curve from data or fit a user-defined calibration function, but often overlook fully mining and utilizing the prior distribution behind the calibration curve. However, a well-informed prior distribution can provide valuable insights beyond the empirical data under the limited data or low-density regions of confidence scores. To fill this gap, this paper proposes a new method that integrates the prior distribution behind the calibration curve with empirical data to estimate a continuous calibration curve, which is realized by modeling the sampling process of calibration data as a binomial process and maximizing the likelihood function of the binomial process. We prove that the calibration curve estimating method is Lipschitz continuous with respect to data distribution and requires a sample size of $3/B$ of that required for histogram binning, where $B$ represents the number of bins. Also, a new calibration metric ($TCE_{bpm}$), which leverages the estimated calibration curve to estimate the true calibration error (TCE), is designed. $TCE_{bpm}$ is proven to be a consistent calibration measure. Furthermore, realistic calibration datasets can be generated by the binomial process modeling from a preset true calibration curve and confidence score distribution, which can serve as a benchmark to measure and compare the discrepancy between existing calibration metrics and the true calibration error. The effectiveness of our calibration method and metric are verified in real-world and simulated data.

Problem

• Estimates true posterior probability for reliable decision-making
• Integrates prior distribution with empirical data for calibration
• Proposes a new consistent calibration metric (TCE_bpm)

Innovation

• Binomial process modeling for calibration
• Integration of prior and empirical data
• New consistent calibration metric TCE_bpm