Walking on the Fiber: A Simple Geometric Approximation for Bayesian Neural Networks

2025-12-01

AI Summary

Exact posterior inference in Bayesian neural networks (BNNs) is computationally intractable, while mainstream approximations—e.g., Laplace approximation—suffer from poor scalability and inaccurate posterior estimation in deep networks. To address this, we propose a geometric parameter-space deformation framework for efficient posterior approximation. Leveraging the low-dimensional manifold structure of loss minima in overparameterized networks, our method learns an explicit, invertible coordinate transformation that maps the complex posterior to a standard distribution—enabling direct sampling without iterative optimization. Theoretically grounded and empirically scalable, our approach achieves uncertainty calibration and predictive accuracy on par with or superior to state-of-the-art methods (e.g., Laplace, SWAG, K-FAC), at significantly lower computational cost across multiple benchmarks. This work introduces a novel paradigm for scalable deep Bayesian inference.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Probabilistic Programming
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Bayesian Neural Networks provide a principled framework for uncertainty quantification by modeling the posterior distribution of network parameters. However, exact posterior inference is computationally intractable, and widely used approximations like the Laplace method struggle with scalability and posterior accuracy in modern deep networks. In this work, we revisit sampling techniques for posterior exploration, proposing a simple variation tailored to efficiently sample from the posterior in over-parameterized networks by leveraging the low-dimensional structure of loss minima. Building on this, we introduce a model that learns a deformation of the parameter space, enabling rapid posterior sampling without requiring iterative methods. Empirical results demonstrate that our approach achieves competitive posterior approximations with improved scalability compared to recent refinement techniques. These contributions provide a practical alternative for Bayesian inference in deep learning.

Problem

• Efficiently sample Bayesian posterior in over-parameterized networks
• Leverage low-dimensional structure of loss minima for scalability
• Enable rapid posterior sampling without iterative methods

Innovation

• Sampling posterior via low-dimensional loss minima
• Deforming parameter space for rapid sampling
• Scalable Bayesian inference without iterative methods