🤖 AI Summary
This work investigates the convergence properties of standard stochastic gradient descent (SGD) and its momentum variant under heavy-tailed gradient noise, without resorting to stabilization techniques such as gradient clipping or normalization. For strongly convex, convex, and non-convex objective functions, it establishes the first comprehensive theoretical convergence guarantees for vanilla SGD with momentum in the presence of heavy-tailed noise. Leveraging probabilistic inequalities and tools from stochastic optimization analysis, the study demonstrates that while the method does converge, its convergence rate is significantly slower compared to variants employing gradient clipping or normalization. These theoretical findings are corroborated by experiments on synthetic functions, revealing the inherent limitations of momentum SGD in heavy-tailed stochastic environments.
📝 Abstract
Stochastic gradient descent (SGD) is a cornerstone of modern optimization. While its performance under heavy-tailed noise is often addressed through specialized modifications such as gradient clipping or normalization, we investigate a more fundamental question: how does vanilla SGD, particularly with momentum, perform in the presence of heavy-tailed noise? In this paper, we refine existing convergence results for vanilla SGD and, more importantly, provide the first comprehensive convergence analysis of vanilla SGD with momentum for strongly convex, convex, and nonconvex objectives, without employing any gradient control mechanisms. Our results demonstrate that the obtained convergence rates are inferior to the optimal rates achieved by clipped or normalized variants of SGD, thereby revealing inherent limitations of vanilla methods under heavy-tailed noise. The theoretical findings are supported by experiments on synthetic functions.