Adam-HNAG: A Convergent Reformulation of Adam with Accelerated Rate

2026-04-09

AI Summary

This work addresses the lack of convergence guarantees for the Adam optimizer in deterministic full-batch settings by introducing a continuous-time Adam-HNAG dynamical system, which integrates variable and operator splitting with curvature-aware gradient correction. From this framework, two discrete algorithms—Adam-HNAG and its synchronous variant Adam-HNAG-s—are derived. Within a unified Lyapunov analysis framework, the study establishes, for the first time, convergence and accelerated convergence rates for Adam-type methods in smooth convex optimization. Theoretical findings are corroborated by numerical experiments that demonstrate the efficacy of the proposed methods and highlight distinct practical behaviors between the two discretization strategies.

Technology Category

• Search and Optimization: Non-convex Optimization
• Natural Language Processing: Learning & Optimization for NLP
• Machine Learning: Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Adam has achieved strong empirical success, but its theory remains incomplete even in the deterministic full-batch setting, largely because adaptive preconditioning and momentum are tightly coupled. In this work, a convergent reformulation of full-batch Adam is developed by combining variable and operator splitting with a curvature-aware gradient correction. This leads to a continuous-time Adam-HNAG flow with an exponentially decaying Lyapunov function, as well as two discrete methods: Adam-HNAG, and Adam-HNAG-s, a synchronous variant closer in form to Adam. Within a unified Lyapunov analysis framework, convergence guarantees are established for both methods in the convex smooth setting, including accelerated convergence. Numerical experiments support the theory and illustrate the different empirical behavior of the two discretizations. To the best of our knowledge, this provides the first convergence proof for Adam-type methods in convex optimization.

Problem

• Adam
• convergence
• optimization
• adaptive preconditioning
• momentum

Innovation

• Adam optimization
• convergence proof
• accelerated rate
• Lyapunov analysis
• operator splitting