Parametric Mean-Field empirical Bayes in high-dimensional linear regression

2026-01-23

AI Summary

This study addresses the problem of empirical Bayes estimation for parameters in high-dimensional linear regression with i.i.d. priors, particularly under the challenging regime where the design matrix is random and the dimension $p$ approaches or exceeds the sample size $n$. We propose a variational empirical Bayes (vEB) approach that estimates the prior by maximizing a variational lower bound on the marginal likelihood and establish its asymptotic theory. Our key contributions include revealing a phase transition phenomenon wherein vEB achieves information-theoretically optimal performance when $p = o(n^{2/3})$, introducing a debiasing correction to enhance inference accuracy in higher dimensions, and proving that the method matches oracle posterior performance both coordinate-wise and in non-local inference. Theoretical analysis and numerical experiments jointly confirm the efficacy of the proposed method and the existence of the identified phase transition threshold.

Technology Category

• Machine Learning: Bayesian Learning
• Reasoning under Uncertainty: Relational Probabilistic Models
• Intelligent Robots: State Estimation

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In this paper, we consider the problem of parametric empirical Bayes estimation of an i.i.d. prior in high-dimensional Bayesian linear regression, with random design. We obtain the asymptotic distribution of the variational Empirical Bayes (vEB) estimator, which approximately maximizes a variational lower bound of the intractable marginal likelihood. We characterize a sharp phase transition behavior for the vEB estimator -- namely that it is information theoretically optimal (in terms of limiting variance) up to $p=o(n^{2/3})$ while it suffers from a sub-optimal convergence rate in higher dimensions. In the first regime, i.e., when $p=o(n^{2/3})$, we show how the estimated prior can be calibrated to enable valid coordinate-wise and delocalized inference, both under the \emph{empirical Bayes posterior} and the oracle posterior. In the second regime, we propose a debiasing technique as a way to improve the performance of the vEB estimator beyond $p=o(n^{2/3})$. Extensive numerical experiments corroborate our theoretical findings.

Problem

• empirical Bayes
• high-dimensional linear regression
• variational estimation
• prior estimation
• statistical inference

Innovation

• variational Empirical Bayes
• high-dimensional linear regression
• phase transition
• debiased estimation
• empirical Bayes posterior