Analog Bayesian neural networks are insensitive to the shape of the weight distribution

2025-01-09

AI Summary

This work addresses the challenge of complex device-level stochastic error modeling when implementing Bayesian neural networks (BNNs) in analog hardware. Specifically, it investigates how the shape of the weight distribution—under mean-field variational inference (MFVI)—affects predictive performance. Through extensive MFVI training and uncertainty quantification experiments under realistic analog device noise (Gaussian, uniform, and impulse noise), we empirically demonstrate for the first time that the BNN’s predictive distribution depends solely on the first two moments (mean and variance) of the weight posterior, irrespective of the specific parametric form of the variational approximation. This finding substantially relaxes the requirement for precise characterization of noise distribution morphology in analog hardware design. Experimental results show that prediction errors across diverse noise distributions differ by less than 0.5%, significantly enhancing the feasibility and robustness of analog BNN deployment.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Cognitive Modeling & Cognitive Systems: Analogy

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Recent work has demonstrated that Bayesian neural networks (BNN's) trained with mean field variational inference (MFVI) can be implemented in analog hardware, promising orders of magnitude energy savings compared to the standard digital implementations. However, while Gaussians are typically used as the variational distribution in MFVI, it is difficult to precisely control the shape of the noise distributions produced by sampling analog devices. This paper introduces a method for MFVI training using real device noise as the variational distribution. Furthermore, we demonstrate empirically that the predictive distributions from BNN's with the same weight means and variances converge to the same distribution, regardless of the shape of the variational distribution. This result suggests that analog device designers do not need to consider the shape of the device noise distribution when hardware-implementing BNNs performing MFVI.

Problem

• Bayesian Neural Networks
• Random Errors
• Prediction Accuracy

Innovation

• Bayesian Neural Networks
• Inherent Randomness
• Low-power Analog Devices