Spike-and-Slab Posterior Sampling in High Dimensions

2025-03-04

AI Summary

Efficient posterior sampling under spike-and-slab priors remains challenging for high-dimensional sparse linear regression, especially when signal-to-noise ratio (SNR) is arbitrary and the number of measurements $n$ is sublinear in dimension $d$. Method: We propose the first provably efficient sampling algorithm that operates under arbitrary SNR and requires only $n gg k^3 cdot mathrm{polylog}(d)$ measurements. Our approach leverages restricted isometry property (RIP) matrix analysis to design both a polynomial-time Markov chain Monte Carlo (MCMC) sampler and a near-linear stochastic gradient Langevin dynamics (SGLD) variant, compatible with Gaussian and Laplace diffuse components. Contribution/Results: Under $k$-sparsity, our algorithm achieves $mathcal{O}(nd)$ time complexity with rigorous statistical convergence guarantees. It eliminates the need for strong SNR assumptions or $Omega(d)$ measurements required by prior methods, thereby substantially broadening the applicability of Bayesian sparse inference in high dimensions.

Technology Category

• Machine Learning: Bayesian Learning
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Posterior sampling with the spike-and-slab prior [MB88], a popular multimodal distribution used to model uncertainty in variable selection, is considered the theoretical gold standard method for Bayesian sparse linear regression [CPS09, Roc18]. However, designing provable algorithms for performing this sampling task is notoriously challenging. Existing posterior samplers for Bayesian sparse variable selection tasks either require strong assumptions about the signal-to-noise ratio (SNR) [YWJ16], only work when the measurement count grows at least linearly in the dimension [MW24], or rely on heuristic approximations to the posterior. We give the first provable algorithms for spike-and-slab posterior sampling that apply for any SNR, and use a measurement count sublinear in the problem dimension. Concretely, assume we are given a measurement matrix $mathbf{X} in mathbb{R}^{n imes d}$ and noisy observations $mathbf{y} = mathbf{X}mathbf{ heta}^star + mathbf{xi}$ of a signal $mathbf{ heta}^star$ drawn from a spike-and-slab prior $pi$ with a Gaussian diffuse density and expected sparsity k, where $mathbf{xi} sim mathcal{N}(mathbb{0}_n, sigma^2mathbf{I}_n)$. We give a polynomial-time high-accuracy sampler for the posterior $pi(cdot mid mathbf{X}, mathbf{y})$, for any SNR $sigma^{-1}$>0, as long as $n geq k^3 cdot ext{polylog}(d)$ and $X$ is drawn from a matrix ensemble satisfying the restricted isometry property. We further give a sampler that runs in near-linear time $approx nd$ in the same setting, as long as $n geq k^5 cdot ext{polylog}(d)$. To demonstrate the flexibility of our framework, we extend our result to spike-and-slab posterior sampling with Laplace diffuse densities, achieving similar guarantees when $sigma = O(frac{1}{k})$ is bounded.

Problem

• Develop provable algorithms for spike-and-slab posterior sampling.
• Enable sampling with sublinear measurement count in high dimensions.
• Achieve polynomial-time high-accuracy sampling for any signal-to-noise ratio.

Innovation

• Polynomial-time high-accuracy posterior sampler
• Near-linear time sampler for spike-and-slab
• Flexible framework for Laplace diffuse densities