Central limit theorem for the averaged Adam optimizer

2026-06-19

AI Summary

This work investigates the statistical behavior and convergence rate of averaged Adam as it approaches the attracting zero of its associated vector field. Leveraging stochastic approximation theory and dynamical systems analysis, the study establishes, for the first time, a central limit theorem for averaged Adam, rigorously demonstrating its classical \(n^{-1/2}\) convergence rate and explicitly characterizing the covariance structure of the limiting distribution. These results confirm the asymptotic efficiency equivalence between averaged Adam and classical stochastic approximation algorithms, thereby providing a solid theoretical foundation for understanding the statistical properties of adaptive optimization methods.

Abstract

In this article, we analyse convergence of the averaged Adam optimizer to an attracting zero of the Adam vector field. We provide a central limit theorem that, in particular, quantifies exactly the speed of convergence. The order of convergence is $n^{-1/2}$ in the number of steps of the algorithm which coincides with the order observed for classical stochastic approximation algorithms. The covariance in the central limit theorem is given in terms of properties of the Adam algorithm in the state of the attractor.

Problem

• central limit theorem
• Adam optimizer
• convergence rate
• stochastic approximation
• attracting zero

Innovation

• central limit theorem
• averaged Adam optimizer
• convergence rate
• stochastic approximation
• covariance structure