Gaussian Mean Field Variational Inference can Overestimate Predictive Variance

2026-06-24

AI Summary

Conventional wisdom holds that mean-field variational inference (MFVI) invariably underestimates posterior uncertainty; however, this work demonstrates that MFVI can overestimate predictive variance—particularly along directions where the training data distribution is concentrated. Within a conjugate Bayesian linear regression framework, we theoretically analyze the discrepancy between MFVI’s behavior in parameter space and predictive space, revealing for the first time the phenomenon of predictive variance overestimation and its dependence on the data covariance structure. We further establish a connection between this effect and the cold posterior phenomenon. Empirically, we show that MFVI can yield predictive variances on in-distribution data that are worse than those of the prior, while temperature scaling effectively mitigates this bias, substantially improving predictive performance and better approximating the exact posterior.

Abstract

Mean Field Variational Inference (MFVI) is widely understood to underestimate posterior variance. By analysing conjugate Bayesian Linear Regression (BLR), we show that this characterization is incomplete: while MFVI underestimates the variance in parameter space, it can overestimate the predictive variance compared to the exact posterior. We show that if the MFVI posterior underestimates predictive variances in some directions, it necessarily overestimates them in others. Crucially, this overestimation occurs in directions where the training data concentrates. This leads to the surprising result that, for a test point drawn from the training distribution, MFVI's expected predictive variance exceeds that of the exact posterior. We demonstrate a pathological case of this effect, where the MFVI posterior fails to reduce predictive variance compared to the prior on in distribution data. We connect these results to the Cold Posterior Effect, arguing that varying the temperature can correct this overestimation, yielding predictions closer to those of the exact posterior. We validate our theory on synthetic and real-world regression tasks.

Problem

• Mean Field Variational Inference
• Predictive Variance
• Bayesian Linear Regression
• Posterior Overestimation
• Cold Posterior Effect

Innovation

• Mean Field Variational Inference
• Predictive Variance Overestimation
• Bayesian Linear Regression
• Cold Posterior Effect
• Temperature Scaling