Nearly Dimension-Independent Convergence of Mean-Field Black-Box Variational Inference

2025-05-27

AI Summary

This work studies the high-dimensional convergence of black-box variational inference (BBVI) under the mean-field location-scale variational family. For strongly log-concave and log-smooth target distributions, we establish, for the first time, a nearly dimension-free iteration complexity of $O(log d)$, substantially mitigating the “curse of dimensionality.” Our theoretical analysis reveals that the gradient variance bound cannot be improved via Hessian spectral bounds; achieves $O(log d)$ convergence under sub-Gaussian approximating families; eliminates explicit dimension dependence entirely in the constant-Hessian case; and yields an $O(d^{2/k})$ bound for heavy-tailed distribution families. The analysis integrates reparameterization gradients, stochastic optimization theory, high-dimensional probability inequalities, and log-concave distribution theory. This provides the first tight, dimension-adaptive convergence guarantee for BBVI.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Optimization
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Security and Privacy: Large-scale security measurements

Abstract

We prove that, given a mean-field location-scale variational family, black-box variational inference (BBVI) with the reparametrization gradient converges at an almost dimension-independent rate. Specifically, for strongly log-concave and log-smooth targets, the number of iterations for BBVI with a sub-Gaussian family to achieve an objective $epsilon$-close to the global optimum is $mathrm{O}(log d)$, which improves over the $mathrm{O}(d)$ dependence of full-rank location-scale families. For heavy-tailed families, we provide a weaker $mathrm{O}(d^{2/k})$ dimension dependence, where $k$ is the number of finite moments. Additionally, if the Hessian of the target log-density is constant, the complexity is free of any explicit dimension dependence. We also prove that our bound on the gradient variance, which is key to our result, cannot be improved using only spectral bounds on the Hessian of the target log-density.

Problem

• Proves dimension-independent convergence rate for BBVI
• Analyzes convergence for strongly log-concave and log-smooth targets
• Investigates gradient variance bounds for variational inference

Innovation

• BBVI with reparametrization gradient achieves dimension-independent convergence
• Strongly log-concave targets require O(log d) iterations
• Constant Hessian eliminates explicit dimension dependence