A variational Bayes approach to debiased inference for low-dimensional parameters in high-dimensional linear regression

2024-06-18

arXiv.org

AI Summary

This paper addresses the challenge of unbiased statistical inference for low-dimensional target parameters—such as individual regression coefficients—in high-dimensional sparse linear regression. We propose a scalable variational Bayesian method that innovatively combines mean-field approximation with conditional posterior modeling of target versus nuisance variables, achieving both computational efficiency and accurate uncertainty quantification. Crucially, we establish the first Bernstein–von Mises-type asymptotic theory for this class of variational inference, guaranteeing asymptotic normality of the variational posterior and alignment with frequentist properties. Theoretically, our estimator is proven consistent and asymptotically normal, enabling valid confidence interval construction. Numerical experiments demonstrate that our method matches the inferential accuracy of state-of-the-art approaches while substantially outperforming standard variational Bayes in coverage and calibration.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Probabilistic Inference
• Intelligent Robots: State Estimation

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Security and Privacy: Large-scale security measurements

Abstract

We propose a scalable variational Bayes method for statistical inference for a single or low-dimensional subset of the coordinates of a high-dimensional parameter in sparse linear regression. Our approach relies on assigning a mean-field approximation to the nuisance coordinates and carefully modelling the conditional distribution of the target given the nuisance. This requires only a preprocessing step and preserves the computational advantages of mean-field variational Bayes, while ensuring accurate and reliable inference for the target parameter, including for uncertainty quantification. We investigate the numerical performance of our algorithm, showing that it performs competitively with existing methods. We further establish accompanying theoretical guarantees for estimation and uncertainty quantification in the form of a Bernstein--von Mises theorem.

Problem

• Debiased inference for high-dimensional sparse regression
• Scalable variational Bayes for low-dimensional parameters
• Accurate uncertainty quantification in target parameters

Innovation

• Variational Bayes for high-dimensional regression
• Mean-field approximation for nuisance coordinates
• Ensures accurate target parameter inference