Extending Mean-Field Variational Inference via Entropic Regularization: Theory and Computation

2024-04-14

arXiv.org

AI Summary

Mean-field variational inference (MF-VI) fails to capture posterior dependency structures in high-dimensional Bayesian models. Method: We propose Ξ-variational inference (Ξ-VI), the first VI framework incorporating entropy regularization, establishing a theoretical connection to entropic optimal transport and enabling efficient polynomial-time computation via the Sinkhorn algorithm. Contributions/Results: Theoretically, Ξ-VI guarantees high-dimensional consistency, asymptotic normality, and algorithmic stability. Methodologically, it reveals an explicit trade-off between statistical accuracy and computational efficiency. Empirically, Ξ-VI significantly outperforms MF-VI on both synthetic and real-world datasets, accurately recovering posterior dependencies while maintaining scalability and statistical reliability.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Reasoning under Uncertainty: Probabilistic Inference
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Variational inference (VI) has emerged as a popular method for approximate inference for high-dimensional Bayesian models. In this paper, we propose a novel VI method that extends the naive mean field via entropic regularization, referred to as $Xi$-variational inference ($Xi$-VI). $Xi$-VI has a close connection to the entropic optimal transport problem and benefits from the computationally efficient Sinkhorn algorithm. We show that $Xi$-variational posteriors effectively recover the true posterior dependency, where the dependence is downweighted by the regularization parameter. We analyze the role of dimensionality of the parameter space on the accuracy of $Xi$-variational approximation and how it affects computational considerations, providing a rough characterization of the statistical-computational trade-off in $Xi$-VI. We also investigate the frequentist properties of $Xi$-VI and establish results on consistency, asymptotic normality, high-dimensional asymptotics, and algorithmic stability. We provide sufficient criteria for achieving polynomial-time approximate inference using the method. Finally, we demonstrate the practical advantage of $Xi$-VI over mean-field variational inference on simulated and real data.

Problem

• Extending mean-field variational inference with entropic regularization
• Improving posterior dependency recovery in Bayesian models
• Analyzing statistical-computational trade-offs in variational approximation

Innovation

• Extends mean-field VI via entropic regularization
• Uses Sinkhorn algorithm for efficient computation
• Recovers posterior dependency with regularization parameter