Optimization Guarantees for Square-Root Natural-Gradient Variational Inference

2025-07-10

AI Summary

Natural-gradient variational inference (NGVI) exhibits fast empirical convergence but lacks rigorous theoretical convergence guarantees—even under idealized settings such as concave log-likelihoods and Gaussian approximations. Method: We introduce a square-root parameterization of the covariance matrix, embedding the natural gradient flow into a Euclidean geometric framework. This circumvents analytical challenges arising from singularity of the Fisher information matrix and complex Riemannian curvature inherent in standard parameterizations. Contribution/Results: We establish, for the first time, global convergence of the continuous-time natural gradient flow for Gaussian variational inference and derive explicit convergence rate bounds for its discrete-time counterpart. Our analysis unifies Euclidean and Wasserstein geometric perspectives. Experiments demonstrate that the proposed method significantly outperforms both standard Euclidean-gradient descent and Wasserstein natural-gradient methods in terms of convergence speed and numerical stability.

Technology Category

• Machine Learning: Learning with Manifolds
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

Variational inference with natural-gradient descent often shows fast convergence in practice, but its theoretical convergence guarantees have been challenging to establish. This is true even for the simplest cases that involve concave log-likelihoods and use a Gaussian approximation. We show that the challenge can be circumvented for such cases using a square-root parameterization for the Gaussian covariance. This approach establishes novel convergence guarantees for natural-gradient variational-Gaussian inference and its continuous-time gradient flow. Our experiments demonstrate the effectiveness of natural gradient methods and highlight their advantages over algorithms that use Euclidean or Wasserstein geometries.

Problem

• Theoretical convergence guarantees for natural-gradient variational inference
• Square-root parameterization for Gaussian covariance in optimization
• Advantages of natural gradient over Euclidean or Wasserstein methods

Innovation

• Square-root parameterization for Gaussian covariance
• Novel convergence guarantees for variational-Gaussian inference
• Natural gradient methods outperform Euclidean alternatives