Quantile Adaptive Temperature Scaling for Confidence Calibration

2026-06-19

AI Summary

This work addresses the poor calibration of deep neural networks, which often leads to substantial errors in low-confidence regions. Conventional temperature scaling struggles to correct heterogeneous miscalibration across the confidence spectrum due to its global, uniform adjustment. To overcome this limitation, the authors propose a quantile-aware adaptive temperature scaling method that maps predicted confidences into quantile space and constructs a monotonic temperature function adaptively varying with empirical confidence quantiles. This approach explicitly models confidence heterogeneity and naturally aligns with a reparameterized expected calibration error (ECE) objective. Experimental results demonstrate that the method consistently outperforms existing post-hoc calibration techniques across diverse datasets, model architectures, and distribution shift scenarios, yielding more reliable confidence estimates without altering the original predictions.

Abstract

Deep neural networks often produce poorly calibrated confidence estimates, overstating their certainty even when predictions are incorrect. Temperature Scaling remains the most widely used posthoc calibration method due to its simplicity and effectiveness, yet its global, uniform rescaling of logits fails to correct the highly heterogeneous structure of miscalibration observed across the confidence spectrum. In particular, the largest correctness confidence discrepancies arise in different quantile regions depending on the setting, low confidence predictions, where uncertainty matters most, tend to exhibit the largest correctness confidence discrepancies, which standard TS leaves largely unaddressed. We introduce Quantile Adaptive Temperature Scaling (QaTS), a simple and efficient post hoc calibration method that adapts the temperature as a function of a predictions empirical confidence quantile. By mapping confidences into the quantile space, QaTS normalizes the calibration problem, makes the structure of miscalibration explicit and enables a monotone temperature function that adapts across quantiles while leaving well calibrated high confidence predictions largely unchanged. preserving high confidence behavior. This quantile aware formulation aligns naturally with a reparameterized Expected Calibration Error (ECE) objective and yields a sample wise temperature that is robust across a variety of challenging scenarios, such as class imbalance and distributional shifts. Across a broad range of datasets, architectures, evaluation scenarios and diverse tasks, QaTS consistently, and substantially, outperforms state of the art post hoc calibration methods, delivering more reliable and trustworthy confidence estimates without modifying model predictions.

Problem

• confidence calibration
• temperature scaling
• quantile
• miscalibration
• expected calibration error

Innovation

• Quantile Adaptive Temperature Scaling
• confidence calibration
• Expected Calibration Error
• post-hoc calibration
• temperature scaling