Robust and Scalable Variational Bayes

2025-04-16

AI Summary

This paper addresses robust and scalable Bayesian inference for large-scale datasets subject to arbitrary contamination. The proposed method partitions the data into subsets, performs independent variational approximations on each subset, and aggregates the resulting subposterior distributions via the Wasserstein geometric median—yielding the variational median posterior (VM-Posterior). This is the first application of the Wasserstein geometric median for variational Bayes aggregation. We rigorously establish its posterior contraction and statistical robustness, and derive a variational Bernstein–von Mises theorem applicable to general covariance Gaussian models and the mean-field family. The framework supports distributed computation and scales to datasets with millions of samples. Under arbitrary contamination, it maintains the optimal $O(1/sqrt{n})$ convergence rate—substantially outperforming standard variational Bayes and state-of-the-art robust Bayesian methods.

Technology Category

• Machine Learning: Bayesian Learning
• Reasoning under Uncertainty: Relational Probabilistic Models
• Intelligent Robots: State Estimation

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

We propose a robust and scalable framework for variational Bayes (VB) that effectively handles outliers and contamination of arbitrary nature in large datasets. Our approach divides the dataset into disjoint subsets, computes the posterior for each subset, and applies VB approximation independently to these posteriors. The resulting variational posteriors with respect to the subsets are then aggregated using the geometric median of probability measures, computed with respect to the Wasserstein distance. This novel aggregation method yields the Variational Median Posterior (VM-Posterior) distribution. We rigorously demonstrate that the VM-Posterior preserves contraction properties akin to those of the true posterior, while accounting for approximation errors or the variational gap inherent in VB methods. We also provide provable robustness guarantee of the VM-Posterior. Furthermore, we establish a variational Bernstein-von Mises theorem for both multivariate Gaussian distributions with general covariance structures and the mean-field variational family. To facilitate practical implementation, we adapt existing algorithms for computing the VM-Posterior and evaluate its performance through extensive numerical experiments. The results highlight its robustness and scalability, making it a reliable tool for Bayesian inference in the presence of complex, contaminated datasets.

Problem

• Handles outliers in large datasets via variational Bayes
• Aggregates subset posteriors using geometric median method
• Ensures robustness and scalability in Bayesian inference

Innovation

• Divide dataset into subsets for VB
• Aggregate posteriors using geometric median
• Prove robustness guarantees for VM-Posterior