Fast Mixing of Data Augmentation Algorithms: Bayesian Probit, Logit, and Lasso Regression

2024-12-11

AI Summary

This work establishes non-asymptotic theoretical guarantees for the convergence of Bayesian data augmentation (DA) algorithms. Focusing on three canonical DA Markov chains—Probit, Logit, and Lasso—we derive the first explicit polynomial upper bounds on their mixing times: $O(nd log(log eta/varepsilon))$ for Probit/Logit DA and $O(d^2(d log d + n log n)^2 log(eta/varepsilon))$ for Lasso DA. Under sub-Gaussian or log-concave distributional assumptions, high-probability convergence rates achieve quasi-linear $ ilde{O}(n+d)$, substantially improving upon existing Langevin-based methods. Our analysis integrates Markov chain conductance theory, isoperimetric inequalities, and Bayesian computation principles, accommodating both warm starts and feasible initializations. The framework is robust to high-dimensional settings, large sample sizes, and imbalanced response distributions.

Technology Category

• Machine Learning: Bayesian Learning
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Security and Privacy: Data transparency and provenance

Abstract

Despite the widespread use of the data augmentation (DA) algorithm, the theoretical understanding of its convergence behavior remains incomplete. We prove the first non-asymptotic polynomial upper bounds on mixing times of three important DA algorithms: DA algorithm for Bayesian Probit regression (Albert and Chib, 1993, ProbitDA), Bayesian Logit regression (Polson, Scott, and Windle, 2013, LogitDA), and Bayesian Lasso regression (Park and Casella, 2008, Rajaratnam et al., 2015, LassoDA). Concretely, we demonstrate that with $eta$-warm start, parameter dimension $d$, and sample size $n$, the ProbitDA and LogitDA require $mathcal{O}left(ndlog left(frac{log eta}{epsilon} ight) ight)$ steps to obtain samples with at most $epsilon$ TV error, whereas the LassoDA requires $mathcal{O}left(d^2(dlog d +n log n)^2 log left(frac{eta}{epsilon} ight) ight)$ steps. The results are generally applicable to settings with large $n$ and large $d$, including settings with highly imbalanced response data in the Probit and Logit regression. The proofs are based on the Markov chain conductance and isoperimetric inequalities. Assuming that data are independently generated from either a bounded, sub-Gaussian, or log-concave distribution, we improve the guarantees for ProbitDA and LogitDA to $ ilde{mathcal{O}}(n+d)$ with high probability, and compare it with the best known guarantees of Langevin Monte Carlo and Metropolis Adjusted Langevin Algorithm. We also discuss the mixing times of the three algorithms under feasible initialization.

Problem

• Analyzing mixing times of data augmentation algorithms
• Proving convergence bounds for Bayesian regression models
• Comparing performance with Langevin Monte Carlo methods

Innovation

• ProbitDA and LogitDA mix in O(nd log(logη/ε)) steps
• LassoDA mixes in O(d²(d log d + n log n)² log(η/ε)) steps
• Improved guarantees to O~(n+d) for ProbitDA and LogitDA