Variational Inference with Mixtures of Isotropic Gaussians

2025-06-16

AI Summary

Balancing multimodal posterior modeling and computational efficiency remains a key challenge in Bayesian variational inference. To address this, we propose a novel variational inference framework based on isotropic Gaussian mixtures—where all components share identical variances and equal weights. We establish, for the first time, a rigorous optimization theory for this variational family, introducing a synergistic optimization scheme that couples entropy-regularized mirror descent (for mean updates) with Bures metric descent (for variance updates), enabling efficient joint minimization of the KL divergence to the true posterior. Unlike standard Gaussian approximations, our method achieves significantly improved fidelity in capturing multimodal posteriors while retaining low memory footprint and rapid convergence. Numerical experiments across benchmark models demonstrate consistent gains in both accuracy and computational efficiency.

Technology Category

• Machine Learning: Bayesian Learning
• Reasoning under Uncertainty: Relational Probabilistic Models
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Variational inference (VI) is a popular approach in Bayesian inference, that looks for the best approximation of the posterior distribution within a parametric family, minimizing a loss that is typically the (reverse) Kullback-Leibler (KL) divergence. In this paper, we focus on the following parametric family: mixtures of isotropic Gaussians (i.e., with diagonal covariance matrices proportional to the identity) and uniform weights. We develop a variational framework and provide efficient algorithms suited for this family. In contrast with mixtures of Gaussian with generic covariance matrices, this choice presents a balance between accurate approximations of multimodal Bayesian posteriors, while being memory and computationally efficient. Our algorithms implement gradient descent on the location of the mixture components (the modes of the Gaussians), and either (an entropic) Mirror or Bures descent on their variance parameters. We illustrate the performance of our algorithms on numerical experiments.

Problem

• Develop efficient variational inference for isotropic Gaussian mixtures
• Balance accuracy and computational efficiency in posterior approximations
• Optimize mixture component locations and variances via gradient descent

Innovation

• Uses mixtures of isotropic Gaussians for approximation
• Implements gradient descent on Gaussian modes
• Applies Mirror or Bures descent on variances