Position: The Time for Sampling Is Now! Charting a New Course for Bayesian Deep Learning

2026-05-20

AI Summary

Sampling-based inference in Bayesian neural networks has long been undervalued due to misconceptions about its computational efficiency, limiting its application in uncertainty quantification and model averaging. This work demonstrates that modern sampling methods—such as Stochastic Adaptive Inference (SAI)—now match or even surpass optimization-based inference in computational efficiency, advocating for their adoption as the core paradigm in Bayesian deep learning. The study emphasizes two critical future directions: efficient exploration of complex posterior distributions and high-fidelity distillation of posterior samples. By establishing a theoretical foundation for the practical deployment of sampling-based inference, this research significantly enhances predictive performance and reliability in downstream tasks.

Abstract

The practical adoption of sampling-based inference (SAI) in Bayesian neural networks (BNNs) remains limited, partly due to persistent misconceptions about the feasibility and efficiency of sampling. This position paper argues that SAI has achieved computational parity with optimization-based methods and is at the verge of superseding such methods for effective and efficient inference in BNNs. This development should be in the interest of the whole community, promoting BNNs as a principled paradigm with its long-standing yet unfulfilled promise of providing principled uncertainty quantification for neural networks. SAI can even do more -- yielding superior prediction performance through model averaging, serving as the foundation for a plethora of possible downstream tasks, and providing crucial insights into the landscape of BNNs. In order to make such a change happen and unfold the potential of sampling, overcoming current misconceptions is a necessary first step. The next step is to realign research efforts toward addressing remaining challenges in SAI. In particular, the community must focus on two core problems: sufficient exploration of the posterior landscape and high-fidelity distillation of posterior samples for efficient downstream inference. By addressing conceptual and practical obstacles, we can unlock the full potential of SAI and establish it as a central tool in Bayesian deep learning.

Problem

• sampling-based inference
• Bayesian neural networks
• posterior exploration
• uncertainty quantification
• posterior distillation

Innovation

• sampling-based inference
• Bayesian neural networks
• posterior exploration
• model averaging
• posterior distillation