Approaching Optimality for Solving Dense Linear Systems with Low-Rank Structure

2025-07-15

AI Summary

This work addresses the efficient and high-accuracy solution of dense linear systems and regression problems exhibiting low-rank structure: given a $d imes d$ positive-definite matrix or an $n imes d$ design matrix $A$ with $k$ large singular values, we propose the first randomized algorithm achieving high-probability convergence at nearly optimal computational complexity—$widetilde{O}(d^2 + k^omega)$ for linear systems and $widetilde{O}(mathrm{nnz}(A) + d^2 + k^omega)$ for regression. Our method introduces a unified recursive preconditioning framework comprising three variants, integrating matrix sketching with low-rank updates to adapt to problem structure. It breaks classical time–accuracy trade-offs by relaxing statistical assumptions to a weaker generalized mean condition. Moreover, we deliver the first near-linear-time multiplicative approximation to the nuclear norm of arbitrary dense matrices. Compared to prior approaches, our algorithm achieves significant improvements in accuracy, runtime, and broad applicability across structured dense problems.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

We provide new high-accuracy randomized algorithms for solving linear systems and regression problems that are well-conditioned except for $k$ large singular values. For solving such $d imes d$ positive definite system our algorithms succeed whp. and run in time $ ilde O(d^2 + k^ω)$. For solving such regression problems in a matrix $mathbf{A} in mathbb{R}^{n imes d}$ our methods succeed whp. and run in time $ ilde O(mathrm{nnz}(mathbf{A}) + d^2 + k^ω)$ where $ω$ is the matrix multiplication exponent and $mathrm{nnz}(mathbf{A})$ is the number of non-zeros in $mathbf{A}$. Our methods nearly-match a natural complexity limit under dense inputs for these problems and improve upon a trade-off in prior approaches that obtain running times of either $ ilde O(d^{2.065}+k^ω)$ or $ ilde O(d^2 + dk^{ω-1})$ for $d imes d$ systems. Moreover, we show how to obtain these running times even under the weaker assumption that all but $k$ of the singular values have a suitably bounded generalized mean. Consequently, we give the first nearly-linear time algorithm for computing a multiplicative approximation to the nuclear norm of an arbitrary dense matrix. Our algorithms are built on three general recursive preconditioning frameworks, where matrix sketching and low-rank update formulas are carefully tailored to the problems' structure.

Problem

• Solving dense linear systems with low-rank structure efficiently
• Improving runtime for regression problems with large singular values
• Computing nuclear norm approximation for dense matrices nearly-linearly

Innovation

• Randomized algorithms for linear systems
• Tailored matrix sketching techniques
• Recursive preconditioning frameworks