Fast Debiasing of the LASSO Estimator

2025-02-27

AI Summary

To address the inherent bias of LASSO estimators in high-dimensional sparse regression and the high computational cost of conventional debiasing methods—which require iterative approximation of the inverse matrix—this paper proposes a novel paradigm: directly parameterizing the debiasing matrix $W = AM^ op$ and solving it in closed form. Under the mild condition that rows of the design matrix are approximately uncorrelated, we derive a unique, computationally efficient analytical solution, eliminating costly iterative optimization. The proposed method rigorously preserves the statistical guarantees of debiased LASSO, including $sqrt{n}$-consistency and asymptotic normality. Its theoretical time complexity is reduced from $O(nplog(1/varepsilon))$ for iterative approaches to $O(np)$. Numerical experiments confirm its effectiveness in bias correction and accurate coverage of confidence intervals.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

In high-dimensional sparse regression, the extsc{Lasso} estimator offers excellent theoretical guarantees but is well-known to produce biased estimates. To address this, cite{Javanmard2014} introduced a method to ``debias"the extsc{Lasso} estimates for a random sub-Gaussian sensing matrix $oldsymbol{A}$. Their approach relies on computing an ``approximate inverse"$oldsymbol{M}$ of the matrix $oldsymbol{A}^ op oldsymbol{A}/n$ by solving a convex optimization problem. This matrix $oldsymbol{M}$ plays a critical role in mitigating bias and allowing for construction of confidence intervals using the debiased extsc{Lasso} estimates. However the computation of $oldsymbol{M}$ is expensive in practice as it requires iterative optimization. In the presented work, we re-parameterize the optimization problem to compute a ``debiasing matrix"$oldsymbol{W} := oldsymbol{AM}^{ op}$ directly, rather than the approximate inverse $oldsymbol{M}$. This reformulation retains the theoretical guarantees of the debiased extsc{Lasso} estimates, as they depend on the emph{product} $oldsymbol{AM}^{ op}$ rather than on $oldsymbol{M}$ alone. Notably, we provide a simple, computationally efficient, closed-form solution for $oldsymbol{W}$ under similar conditions for the sensing matrix $oldsymbol{A}$ used in the original debiasing formulation, with an additional condition that the elements of every row of $oldsymbol{A}$ have uncorrelated entries. Also, the optimization problem based on $oldsymbol{W}$ guarantees a unique optimal solution, unlike the original formulation based on $oldsymbol{M}$. We verify our main result with numerical simulations.

Problem

• Debiasing LASSO estimator in high-dimensional regression
• Computing debiasing matrix directly for efficiency
• Ensuring unique optimal solution in optimization

Innovation

• Reparameterizes optimization for debiasing
• Provides closed-form solution efficiently
• Ensures unique optimal solution directly