Calibrated Sampling-Free Uncertainty Estimation in Bayesian Deep Learning

2026-06-15

AI Summary

This work addresses the unreliability of deep learning models in high-stakes scenarios due to overconfidence, a limitation inadequately mitigated by existing Bayesian approaches that require multiple forward passes at test time and incur substantial computational overhead. To overcome this, the authors propose Calibrated Variance Propagation (CVP), an efficient method that estimates predictive uncertainty in a single forward pass and is readily applicable to modern architectures such as Transformers and CNNs. CVP introduces a novel variance propagation mechanism through normalization layers, incorporates approximations for activation functions, and applies a lightweight posterior calibration step to absorb residual errors. Experiments demonstrate that CVP substantially improves prediction coverage: on NLVR2 and VQAv2, BEiT-3 achieves coverage of 14.6% at a 0.5% risk level, up from 8.2%, while ViLT improves from 2.6% to 10.8%.

Abstract

Modern deep learning models remain notoriously prone to overconfidence, limiting their reliability in high-stakes applications. Bayesian methods aim to counter this by learning a distribution over model parameters, and recent advances now make this feasible for large-scale architectures at costs comparable to AdamW. However, a challenge remains at test time: predictions must be averaged across many forward passes with weights sampled from the posterior, which is prohibitively expensive. Variance propagation offers an efficient alternative, computing layer-wise analytical approximations of uncertainty in a single forward pass. While such techniques are effective for MLPs, their extension to modern architectures remains challenging, due to increased depth and diversity of layer types. To fill this gap, we propose Calibrated Variance Propagation (CVP), which introduces a new propagation method for normalization layers, combines it with recent techniques for handling activation functions, and absorbs residual error through a light calibration step. CVP yields comparably accurate uncertainty estimates to MC sampling across transformers and CNNs, at a fraction of the cost. Against prior variance propagation work, CVP improves coverage at $0.5\%$ risk from $8.2\%$ to $14.6\%$ with BEiT-3 on Visual Reasoning (NLVR2) and from $2.6\%$ to $10.8\%$ with ViLT on VQAv2, with gains extending to convolutional architectures.

Problem

• Bayesian Deep Learning
• Uncertainty Estimation
• Variance Propagation
• Overconfidence
• Modern Architectures

Innovation

• Calibrated Variance Propagation
• Bayesian Deep Learning
• Uncertainty Estimation
• Sampling-Free Inference
• Normalization Layers