Randomized Kaczmarz Methods with Beyond-Krylov Convergence

2025-01-20

AI Summary

Solving ill-conditioned linear systems—particularly overdetermined and underdetermined cases—remains challenging due to slow convergence of traditional Krylov methods (e.g., CG, GMRES) and their inability to exploit outlier singular value structures. Method: This paper proposes two novel iterative algorithms—Kaczmarz++ and CD++—featuring adaptive momentum acceleration, Tikhonov-regularized projections, singular-value-aware sampling, and an equation-block information reuse memoization mechanism. Contribution/Results: We theoretically establish that both methods efficiently capture and leverage outlier singular values, achieving Krylov-level convergence for both overdetermined and underdetermined systems within a unified framework—the first such result. Experiments demonstrate that Kaczmarz++ significantly outperforms GMRES and CG across diverse ill-conditioned systems; CD++, while matching CG/GMRES in arithmetic complexity for positive semidefinite systems, exhibits strong competitive performance in benchmark tests.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Randomized Kaczmarz methods form a family of linear system solvers which converge by repeatedly projecting their iterates onto randomly sampled equations. While effective in some contexts, such as highly over-determined least squares, Kaczmarz methods are traditionally deemed secondary to Krylov subspace methods, since this latter family of solvers can exploit outliers in the input's singular value distribution to attain fast convergence on ill-conditioned systems. In this paper, we introduce Kaczmarz++, an accelerated randomized block Kaczmarz algorithm that exploits outlying singular values in the input to attain a fast Krylov-style convergence. Moreover, we show that Kaczmarz++ captures large outlying singular values provably faster than popular Krylov methods, for both over- and under-determined systems. We also develop an optimized variant for positive semidefinite systems, called CD++, demonstrating empirically that it is competitive in arithmetic operations with both CG and GMRES on a collection of benchmark problems. To attain these results, we introduce several novel algorithmic improvements to the Kaczmarz framework, including adaptive momentum acceleration, Tikhonov-regularized projections, and a memoization scheme for reusing information from previously sampled equation~blocks.

Problem

• Linear Problem Solving
• Krylov Subspace Methods
• Efficiency Improvement

Innovation

• Kaczmarz++
• CD++
• Optimized Randomized Kaczmarz Method