Robust Variational Bayes by Min-Max Median Aggregation

2025-12-14

AI Summary

This paper addresses robust inference for high-dimensional data contaminated by outliers and adversarial corruption. Methodologically, it proposes a scalable variational Bayes (VB) framework that partitions the data into $m$ disjoint subsets, computes local variational posteriors, and aggregates them via a novel min-max median rule—replacing conventional KL-divergence-based averaging—to rigorously unify KL minimization and ELBO maximization. It introduces an aggregate-and-rescale strategy for local latent variables and establishes a non-asymptotic Bernstein–von Mises theorem accommodating diverging $m$. Theoretically, the framework achieves near-optimal statistical rates and asymptotically recovers the ideal posterior mean. Its two-stage design substantially reduces approximation error. Empirical results demonstrate strong robustness against outliers and data contamination, alongside excellent scalability to high dimensions and large sample sizes.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Intelligent Robots: State Estimation
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

We propose a robust and scalable variational Bayes (VB) framework designed to effectively handle contamination and outliers in dataset. Our approach partitions the data into $m$ disjoint subsets and formulates a joint optimization problem based on robust aggregation principles. A key insight is that the full posterior distribution is equivalent to the minimizer of the mean Kullback-Leibler (KL) divergence from the $m$-powered local posterior distributions. To enhance robustness, we replace the mean KL divergence with a min-max median formulation. The min-max formulation not only ensures consistency between the KL minimizer and the Evidence Lower Bound (ELBO) maximizer but also facilitates the establishment of improved statistical rates for the mean of variational posterior. We observe a notable discrepancy in the $m$-powered marginal log likelihood function contingent on the presence of local latent variables. To address this, we treat these two scenarios separately to guarantee the consistency of the aggregated variational posterior. Specifically, when local latent variables are present, we introduce an aggregate-and-rescale strategy. Theoretically, we provide a non-asymptotic analysis of our proposed posterior, incorporating a refined analysis of Bernstein-von Mises (BvM) theorem to accommodate a diverging number of subsets $m$. Our findings indicate that the two-stage approach yields a smaller approximation error compared to directly aggregating the $m$-powered local posteriors. Furthermore, we establish a nearly optimal statistical rate for the mean of the proposed posterior, advancing existing theories related to min-max median estimators. The efficacy of our method is demonstrated through extensive simulation studies.

Problem

• Develops robust variational Bayes for contaminated datasets
• Introduces min-max median aggregation to handle outliers
• Ensures posterior consistency with local latent variables

Innovation

• Robust variational Bayes using min-max median aggregation
• Data partitioning with joint optimization for outlier handling
• Aggregate-and-rescale strategy for local latent variables