Towards Universal Convergence of Backward Error in Linear System Solvers

2026-04-17

AI Summary

This work addresses the limitations of classical iterative methods, which rely on forward error and are constrained by the condition number of the matrix. It introduces a new paradigm using backward error as the convergence criterion. The key contributions include the first proof that Richardson iteration achieves a condition-number-independent $O(1/k)$ convergence rate in backward error for any positive semidefinite linear system. Building on this, the authors design an accelerated algorithm, MINBERR, attaining an $O(1/k^2)$ convergence rate. They further integrate backward error minimization into Krylov subspace methods and extend the approach to general linear systems. The resulting general-purpose solver has complexity $O(n^2/\varepsilon)$, while MINBERR achieves $O(n^2/\sqrt{\varepsilon})$, demonstrating superior numerical performance in benchmark experiments.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

The quest for an algorithm that solves an $n\times n$ linear system in $O(n^2)$ time complexity, or $O(n^2 \text{poly}(1/ε))$ when solving up to $ε$ relative error, is a long-standing open problem in numerical linear algebra and theoretical computer science. There are two predominant paradigms for measuring relative error: forward error (i.e., distance from the output to the optimum solution) and backward error (i.e., distance to the nearest problem solved by the output). In most prior studies, convergence of iterative linear system solvers is measured via various notions of forward error, and as a result, depends heavily on the conditioning of the input. Yet, the numerical analysis literature has long advocated for backward error as the more practically relevant notion of approximation. In this work, we show that -- surprisingly -- the classical and simple Richardson iteration incurs at most $1/k$ (relative) backward error after $k$ iterations on any positive semidefinite (PSD) linear system, irrespective of its condition number. This universal convergence rate implies an $O(n^2/ε)$ complexity algorithm for solving a PSD linear system to $ε$ backward error, and we establish similar or better complexity when using a variety of Krylov solvers beyond Richardson. Then, by directly minimizing backward error over a Krylov subspace, we attain an even faster $O(1/k^2)$ universal rate, and we turn this into an efficient algorithm, MINBERR, with complexity $O(n^2/\sqrtε)$. We extend this approach via normal equations to solving general linear systems, for which we empirically observe $O(1/k)$ convergence. We report strong numerical performance of our algorithms on benchmark problems.

Problem

• backward error
• linear system solvers
• universal convergence
• Richardson iteration
• Krylov subspace

Innovation

• backward error
• universal convergence
• Richardson iteration
• Krylov subspace
• MINBERR