Adaptive Exponential Integration for Stable Gaussian Mixture Black-Box Variational Inference

2026-01-21

AI Summary

This work addresses the instability and inefficiency of standard optimization methods in Gaussian mixture black-box variational inference (BBVI), which often arise from non-positive-definite covariance matrices and noisy gradients. The authors propose a stable and efficient optimization framework that, for the first time, integrates natural gradient preconditioning, exponential integrators, and adaptive time stepping into Gaussian mixture BBVI. This approach guarantees positive definiteness of the covariance matrices and enables stage-wise optimization. The method establishes theoretical connections to manifold optimization and mirror descent, and under noise-free conditions, it is proven to achieve exponential convergence. Empirical evaluations on multimodal distributions, the Neal funnel, and the Darcy flow inverse problem demonstrate superior stability and convergence efficiency, while the theory ensures almost sure convergence.

Technology Category

• Search and Optimization: Mixed Discrete/Continuous Search
• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Optimization

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Black-box variational inference (BBVI) with Gaussian mixture families offers a flexible approach for approximating complex posterior distributions without requiring gradients of the target density. However, standard numerical optimization methods often suffer from instability and inefficiency. We develop a stable and efficient framework that combines three key components: (1) affine-invariant preconditioning via natural gradient formulations, (2) an exponential integrator that unconditionally preserves the positive definiteness of covariance matrices, and (3) adaptive time stepping to ensure stability and to accommodate distinct warm-up and convergence phases. The proposed approach has natural connections to manifold optimization and mirror descent. For Gaussian posteriors, we prove exponential convergence in the noise-free setting and almost-sure convergence under Monte Carlo estimation, rigorously justifying the necessity of adaptive time stepping. Numerical experiments on multimodal distributions, Neal's multiscale funnel, and a PDE-based Bayesian inverse problem for Darcy flow demonstrate the effectiveness of the proposed method.

Problem

• Black-box variational inference
• Gaussian mixture
• numerical instability
• optimization efficiency
• covariance positivity

Innovation

• exponential integrator
• adaptive time stepping
• natural gradient
• Gaussian mixture variational inference
• affine-invariant preconditioning