A Ranking Approach for Measuring Calibration

2026-09-11

AI Summary

本文提出了一种新的度量方法rankECE，通过比较预测概率相近的点来更准确地估计模型的校准误差，以解决现有ECE估计方法不准确的问题。

Abstract

When providing forecasted probabilities with a predictive model, the ideal model offers perfect calibration: the true probability of the outcome (i.e., the probability that $Y=1$) exactly matches the forecasted probability $f(X)$. In practice, models inevitably exhibit calibration error, and it is therefore important to be able to measure this miscalibration to assess a model's reliability. The Expected Calibration Error (ECE) is the most widely used measure of miscalibration, but is known to be impossible to estimate the ECE with guaranteed accuracy in an assumption-free setting. In this work, we propose an alternative measure, the rankECE, that is based on comparing points with neighboring values of the predicted probability $f(X)$. Our theoretical guarantees and empirical results establish that rankECE provides a better proxy for ECE as compared to binned approximations to ECE, which are the most commonly-used approximations in practice.

Problem

• Calibration
• Miscalibration
• Expected Calibration Error (ECE)

Innovation

• rankECE
• calibration error
• Expected Calibration Error (ECE)
• predicted probability