🤖 AI Summary
Deep neural network regressors often suffer from pointwise miscalibration—i.e., a mismatch between the predicted and true conditional distributions—hindering reliable single-sample uncertainty quantification. Existing calibration metrics (e.g., Expected Calibration Error, ECE) assess only marginal probability alignment and lack input-level interpretability. To address this, we propose *conditional congruence*, a stricter probabilistic fidelity criterion requiring alignment of predicted and ground-truth conditional distributions *at each input*. We define the Conditional Congruence Error (CCE) as an interpretable, pointwise reliability diagnostic. Methodologically, we develop a nonparametric distance estimation framework based on conditional kernel mean embeddings, enabling scalable application to high-dimensional image regression. Experiments demonstrate that CCE precisely quantifies distributional miscalibration on synthetic data, scales effectively to real-world image regression tasks, and substantially improves discrimination of reliability for out-of-distribution samples.
📝 Abstract
While significant progress has been made in specifying neural networks capable of representing uncertainty, deep networks still often suffer from overconfidence and misaligned predictive distributions. Existing approaches for addressing this misalignment are primarily developed under the framework of calibration, with common metrics such as Expected Calibration Error (ECE). However, calibration can only provide a strictly marginal assessment of probabilistic alignment. Consequently, calibration metrics such as ECE are distribution-wise measures and cannot diagnose the point-wise reliability of individual inputs, which is important for real-world decision-making. We propose a stronger condition, which we term conditional congruence, for assessing probabilistic fit. We also introduce a metric, Conditional Congruence Error (CCE), that uses conditional kernel mean embeddings to estimate the distance, at any point, between the learned predictive distribution and the empirical, conditional distribution in a dataset. We show that using CCE to measure congruence 1) accurately quantifies misalignment between distributions when the data generating process is known, 2) effectively scales to real-world, high dimensional image regression tasks, and 3) can be used to gauge model reliability on unseen instances.