On weight and variance uncertainty in neural networks for regression tasks

2025-01-08

AI Summary

In regression tasks, inaccurate uncertainty quantification and poor model calibration often arise from inadequate modeling of predictive variance in neural networks. To address this, we propose a Bayesian neural network framework that jointly models dual uncertainties—weight uncertainty and output variance uncertainty. This work introduces, for the first time in Bayesian regression, systematic posterior inference over variance parameters, leveraging a hybrid prior combining Gaussian and spike-and-slab distributions, and employs dropout-based variational inference for scalable approximation. We evaluate the method on function approximation benchmarks and the riboflavin gene expression dataset. Results demonstrate substantial improvements in predictive accuracy over standard Bayesian neural networks, alongside better-calibrated uncertainty estimates and enhanced generalization—particularly under small-sample regimes and in noise-sensitive settings.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We consider the problem of weight uncertainty proposed by [Blundell et al. (2015). Weight uncertainty in neural network. In International conference on machine learning, 1613-1622, PMLR.] in neural networks {(NNs)} specialized for regression tasks. {We further} investigate the effect of variance uncertainty in {their model}. We show that including the variance uncertainty can improve the prediction performance of the Bayesian {NN}. Variance uncertainty enhances the generalization of the model {by} considering the posterior distribution over the variance parameter. { We examine the generalization ability of the proposed model using a function approximation} example and {further illustrate it with} the riboflavin genetic data set. {We explore fully connected dense networks and dropout NNs with} Gaussian and spike-and-slab priors, respectively, for the network weights.

Problem

• Neural Networks
• Prediction Variability
• Uncertainty Quantification

Innovation

• Neural Network
• Uncertainty Quantification
• Weight Prior Distribution