Open Problem: Is AdamW Effective Under Heavy-Tailed Noise?

2026-06-22

AI Summary

This work addresses the lack of theoretical convergence guarantees for the AdamW optimizer under heavy-tailed gradient noise, providing the first systematic analysis of its convergence properties in such settings. By introducing a weighted metric space and a novel lower-bound analytical framework—integrating stochastic optimization theory with heavy-tailed distribution analysis—the study reveals that AdamW’s denominator momentum mechanism may suppress large gradients, thereby posing a fundamental theoretical obstacle. The authors establish baseline convergence of AdamW under a specific weighted metric and develop a new lower-bound mechanism that elucidates the behavior of adaptive optimizers in non-light-tailed noise environments. This contribution lays a rigorous theoretical foundation and opens pathways for future advances in understanding and improving adaptive optimization methods under realistic, heavy-tailed stochastic conditions.

Abstract

AdamW is the de facto optimizer for training large language models (LLMs), yet the theory behind it still lives mostly in finite-variance regimes. This is increasingly unsatisfying, as empirical evidence indicates that stochastic gradient noise in LLM pretraining is typically heavy-tailed. Recent work shows that sign-based optimizers such as Lion and Muon achieve sharp heavy-tailed rates, and that AdaGrad can also converge under heavy-tailed noise. However, no rigorous convergence theory for AdamW has yet been established in this regime. Can AdamW converge under the same heavy-tailed assumptions, or does its second-moment accumulator create a genuine obstruction? We formulate this as an open problem, prove a positive weighted-metric benchmark, and give a corridor lower-bound mechanism showing how denominator memory can hide large gradients.

Problem

• AdamW
• heavy-tailed noise
• convergence
• stochastic gradient
• optimization

Innovation

• heavy-tailed noise
• AdamW
• convergence theory
• second-moment accumulator
• corridor lower-bound