Problem
Research questions and friction points this paper is trying to address.
stochastic min-max optimization
heavy-tailed noise
gradient modification
convergence
The Role of Gradient Modification in Heavy-Tailed Nonconvex Stochastic Min-Max Optimization
🤖 AI Summary
研究在重尾噪声下，使用未经修改的随机梯度下降上升法解决非凸-强凹和非凸-凹优化问题，并提出无需梯度裁剪的新算法。
📝 Abstract
Stochastic min-max optimization has attracted increasing attention due to its applications in modern machine learning, while existing theoretical studies mainly rely on the bounded variance assumption for stochastic gradients. Under heavy-tailed noise, where stochastic gradients only possess a finite $p$-th moment for $p\in(1,2]$, gradient clipping or normalization is commonly believed to be necessary to guarantee convergence. In this work, we revisit stochastic min-max optimization under heavy-tailed noise and provide a comprehensive theoretical study of stochastic gradient descent ascent (SGDA). We first show that vanilla SGDA, without any modification to its update rule, can converge under heavy-tailed noise in both nonconvex-strongly-concave (NC-SC) and nonconvex-concave (NC-C) settings, establishing the first convergence guarantees for SGDA in these regimes. Beyond unregularized problems, we further investigate regularized stochastic min-max optimization, where directly incorporating gradient normalization into proximal updates is nontrivial due to the incompatibility between normalization and proximal structures. We overcome this difficulty by developing new clipping-free algorithms, i.e., Stoc-TRGDAM and Stoc-TRGDmax, and they both can achieve the optimal dependence on the target accuracy without using gradient clipping.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Gradient Modification
Heavy-Tailed Noise
Stochastic Min-Max Optimization
Clipping-Free Algorithms
🔎 Similar Papers
No similar papers found.
