Last-Iterate Convergence of Randomized Kaczmarz and SGD with Greedy Step Size

2026-04-10

AI Summary

This work investigates the last-iterate convergence of stochastic gradient descent (SGD) and the randomized Kaczmarz algorithm under interpolation conditions with greedy stepsize strategies. Focusing on smooth quadratic objectives, the paper introduces an analytical framework based on stochastic contraction processes, which integrates the dynamics of eigenvalue evolution with a refined discrete-to-continuous reduction technique. This approach yields a significant improvement in the convergence rate for the last iterate, advancing it from the previously known $O(1/t^{1/2})$ to $O(1/t^{3/4})$. The result provides a unified theoretical analysis for these two classical algorithms and substantially strengthens existing convergence guarantees in the interpolation regime.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Non-convex Optimization
• Machine Learning: Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Models for Web evolution

Abstract

We study last-iterate convergence of SGD with greedy step size over smooth quadratics in the interpolation regime, a setting which captures the classical Randomized Kaczmarz algorithm as well as other popular iterative linear system solvers. For these methods, we show that the $t$-th iterate attains an $O(1/t^{3/4})$ convergence rate, addressing a question posed by Attia, Schliserman, Sherman, and Koren, who gave an $O(1/t^{1/2})$ guarantee for this setting. In the proof, we introduce the family of stochastic contraction processes, whose behavior can be described by the evolution of a certain deterministic eigenvalue equation, which we analyze via a careful discrete-to-continuous reduction.

Problem

• last-iterate convergence
• randomized Kaczmarz
• SGD
• greedy step size
• interpolation regime

Innovation

• last-iterate convergence
• greedy step size
• stochastic contraction processes
• Randomized Kaczmarz
• interpolation regime