CLUE: Neural Networks Calibration via Learning Uncertainty-Error alignment

2025-05-28

AI Summary

Neural network uncertainty estimation often suffers from poor calibration; existing post-hoc or binning-based methods are non-differentiable, poorly scalable, and generalize weakly. This paper proposes CLUE, a framework that enables end-to-end, differentiable, domain-agnostic calibration without post-processing by explicitly aligning predicted uncertainty with actual prediction error during training. Its core contributions are: (1) the first formalization of the uncertainty–error alignment principle; (2) a differentiable, statistically grounded joint loss function that jointly optimizes both predictive accuracy and calibration quality; and (3) seamless integration into standard training pipelines. Extensive experiments across vision, regression, language modeling, and out-of-distribution/domain-shift benchmarks demonstrate that CLUE achieves state-of-the-art calibration performance while maintaining competitive predictive accuracy and incurring negligible computational overhead.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Uncertainty Representations
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Reliable uncertainty estimation is critical for deploying neural networks (NNs) in real-world applications. While existing calibration techniques often rely on post-hoc adjustments or coarse-grained binning methods, they remain limited in scalability, differentiability, and generalization across domains. In this work, we introduce CLUE (Calibration via Learning Uncertainty-Error Alignment), a novel approach that explicitly aligns predicted uncertainty with observed error during training, grounded in the principle that well-calibrated models should produce uncertainty estimates that match their empirical loss. CLUE adopts a novel loss function that jointly optimizes predictive performance and calibration, using summary statistics of uncertainty and loss as proxies. The proposed method is fully differentiable, domain-agnostic, and compatible with standard training pipelines. Through extensive experiments on vision, regression, and language modeling tasks, including out-of-distribution and domain-shift scenarios, we demonstrate that CLUE achieves superior calibration quality and competitive predictive performance with respect to state-of-the-art approaches without imposing significant computational overhead.

Problem

• Aligns neural network uncertainty with empirical error during training
• Improves calibration without post-hoc adjustments or binning methods
• Ensures scalable, differentiable uncertainty estimation across diverse domains

Innovation

• Aligns predicted uncertainty with observed error
• Uses novel joint loss function for optimization
• Fully differentiable and domain-agnostic calibration