On the Convergence of Adam, Revisited

2026-07-03

AI Summary

This study investigates whether Adam and its variants can guarantee convergence with vanishing average regret under arbitrary momentum decay parameters. By constructing a specific three-period linear loss sequence—with slopes \(c\), \(-1\), and \(-1\), where \(c\) is slightly greater than 2—and leveraging tools from online convex optimization together with the projected Adam algorithm, the analysis overcomes the conventional restriction requiring \(\beta_1 < \sqrt{\beta_2}\). The results demonstrate that even under more permissive parameter settings, several widely used optimizers—including Adam, AdamW, RMSProp, and NAdam—can still incur non-vanishing average regret. In fact, their average regret is shown to be bounded below by a positive constant, thereby exposing a fundamental convergence limitation of these methods in worst-case scenarios.

Abstract

We show that projected Adam for online optimization with arbitrary moment decay parameters $β_1,β_2\in[0,1)$ can have average regret bounded away from zero. A similar result of Reddi-Kale-Kumar from 2018 required $β_1<\sqrt{β_2}$. Similar to their result, we use a three-periodic sequence of linear functions on $[-1,1]$ with slopes $c,-1,-1$, though we use $c$ slightly larger than $2$. This nonzero average regret result extends to Adam variants such as AdamW, RMSProp, NAdam, Adan, AdaMax, Muon, and to an i.i.d. variant of the three-periodic sequence of slopes for Adam.

Problem

• Adam
• convergence
• average regret
• online optimization
• moment decay parameters

Innovation

• Adam convergence
• average regret
• online optimization
• moment decay parameters
• non-convergence