New Bounds for the Last Iterate of the Stochastic subGradient Method

2026-06-23

AI Summary

This study investigates the optimization error bound of the last iterate in stochastic subgradient methods for one-dimensional convex Lipschitz functions. Under a fixed step-size strategy, the analysis distinguishes between two noise settings: independent and identically distributed (i.i.d.) noise and general noise with uniformly bounded variance. By integrating tools from stochastic optimization theory, subgradient analysis, and concentration inequalities, the paper introduces a refined error decomposition technique. The main contributions are twofold: first, under the i.i.d. noise assumption, it establishes an optimal $1/\sqrt{n}$ error bound for the last iterate, eliminating the logarithmic factor present in prior results; second, under the weaker assumption of merely bounded variance (without i.i.d.), it proves a $(\log n)/\sqrt{n}$ error bound, thereby revealing the inherent suboptimality of the last iterate in this setting and resolving in the negative an open question posed by Koren and Segal.

Abstract

We study the last iterate of the stochastic subgradient method for one-dimensional convex Lipschitz objectives. For a fixed horizon $n$, we consider the standard fixed stepsizes $η=Θ(1/\sqrt n)$. We prove that, for such stepsize policies, under additive i.i.d. subgradient noise with uniformly bounded variance, the last iterate features an optimization error of order $1/\sqrt n$, thereby removing the extra $(\log n)$ factor present in existing generic bounds. On the other hand, we show that without the i.i.d. assumption, the optimization error can be of order $(\log n)/\sqrt n$. Thus, under the uniformly bounded variance assumption alone, the last iterate of SsGM is suboptimal even in dimension one, resolving negatively an open problem posed in Koren and Segal, COLT, 2020.

Problem

• stochastic subgradient method
• last iterate
• optimization error
• convex optimization
• convergence rate

Innovation

• stochastic subgradient method
• last iterate convergence
• convex optimization
• i.i.d. noise
• optimization error bound