An Entropic Metric for Measuring Calibration of Machine Learning Models

2025-02-20

AI Summary

This paper addresses the challenge of disentangling underconfidence from overconfidence in evaluating confidence calibration for binary classification models. We propose the Entropy Calibration Difference (ECD), a novel metric that, for the first time, adapts state estimation principles from object tracking to calibration assessment. ECD quantifies directional calibration errors by modeling the divergence between predicted and true distributions via information entropy, and integrates binning with expected error decomposition to yield interpretable, sign-aware measurements of both under- and overconfidence. Unlike conventional metrics such as Expected Calibration Error (ECE) and Ensemble Soft Calibration Error (ESCE), ECD offers unambiguous semantic interpretation and demonstrates superior sensitivity and directional discrimination across multiple real-world and synthetic experiments. It thus enables joint characterization of model safety (robustness to miscalibration) and statistical efficiency (calibration fidelity), providing a more principled foundation for trustworthy uncertainty quantification.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Multiagent Systems: Multiagent Systems under Uncertainty

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements

Abstract

Understanding the confidence with which a machine learning model classifies an input datum is an important, and perhaps under-investigated, concept. In this paper, we propose a new calibration metric, the Entropic Calibration Difference (ECD). Based on existing research in the field of state estimation, specifically target tracking (TT), we show how ECD may be applied to binary classification machine learning models. We describe the relative importance of under- and over-confidence and how they are not conflated in the TT literature. Indeed, our metric distinguishes under- from over-confidence. We consider this important given that algorithms that are under-confident are likely to be 'safer' than algorithms that are over-confident, albeit at the expense of also being over-cautious and so statistically inefficient. We demonstrate how this new metric performs on real and simulated data and compare with other metrics for machine learning model probability calibration, including the Expected Calibration Error (ECE) and its signed counterpart, the Expected Signed Calibration Error (ESCE).

Problem

• Measure machine learning model calibration
• Introduce Entropic Calibration Difference metric
• Distinguish under-confidence from over-confidence

Innovation

• Introduces Entropic Calibration Difference (ECD)
• Applies ECD to binary classification models
• Distinguishes under-confidence from over-confidence