Optimal Asymptotic Rates for (Stochastic) Gradient Descent under the Local PL-Condition: A Geometric Approach

2026-05-14

AI Summary

This work investigates the asymptotic convergence behavior of gradient descent (GD) and stochastic gradient descent (SGD) in non-convex optimization problems satisfying a local Polyak–Łojasiewicz (PL) condition. By introducing a geometric perspective that integrates the PL inequality, a multiplicative gradient noise model, and analyses from both continuous-time dynamical systems and discrete iterative processes, the paper provides the first geometric interpretation of the local PL condition. The main contribution lies in establishing that, under this non-convex setting, the asymptotic convergence rates of (S)GD match those of the strongly convex quadratic case—thereby achieving optimality—and formally proving the optimal asymptotic convergence rates for (S)GD under the local PL condition.

Abstract

Stochastic gradient descent (SGD) has been studied extensively over the past decades due to its simplicity and broad applicability in machine learning. In this work, we analyze the local behavior of gradient descent and stochastic gradient descent for minimizing $C^2$-functions that satisfy the Polyak-Lojasiewicz (PL) inequality and under a multiplicative gradient noise model motivated by overparameterized neural networks. Using a geometric interpretation of the PL-condition, we prove a simple yet surprising fact: in this possibly non-convex setting, the asymptotic convergence rate of (S)GD matches the rate obtained for strongly convex quadratics.

Problem

• Stochastic Gradient Descent
• Polyak-Lojasiewicz condition
• asymptotic convergence rate
• non-convex optimization
• gradient noise

Innovation

• Polyak-Lojasiewicz condition
• stochastic gradient descent
• asymptotic convergence rate
• geometric approach
• non-convex optimization