On the Principles Behind Neural Network Optimizers

2026-08-17

AI Summary

This study addresses the limited theoretical foundations and high memory overhead of the Adam optimizer by integrating random matrix theory with Hessian block-diagonal structure analysis to elucidate its convergence phase transition mechanisms and underlying effectiveness. Building upon these theoretical insights, we propose Adam-mini, a novel optimizer that employs targeted design strategies to reduce training memory consumption by 50% without compromising model performance. This work not only clarifies the operational mechanics of Adam from a theoretical perspective but also provides a practically viable, low-memory optimization solution grounded in rigorous theory for the efficient training of large-scale models.

Abstract

Reliable optimization is central to neural network (NN) training, yet Adam, the default optimizer for modern LLMs, rests on a fragile foundation. This thesis develops a principled grounding for Adam and motivates new designs. First, we revisit Adam's divergence--convergence debate and show the existence of a problem-dependent phase transition: with properly chosen, batch-size-dependent hyperparameters, Adam converges, whereas under small-$β_2$ regimes it can diverge. Second, we investigate why Adam substantially outperforms SGD on Transformers through Hessian structure. We find that the Hessian evolves toward a near-block-diagonal form along training, accompanied by strong block heterogeneity. We prove that this structure makes Adam's diagonal preconditioner effective. We further show that this special Hessian structure originates from consecutive multiplications of large matrix variables, and we provide a rigorous analysis based on random matrix theory. Finally, these insights motivate Adam-mini, a new optimizer that reduces Adam's memory footprint by 50\% while preserving its performance. Our results also have broader implications beyond Adam: they reveal new local structures in matrix-based nonconvex problems, and also help understand and improve recent NN optimizers, such as Muon.

Problem

• Neural Network Optimizers
• Adam Convergence
• Hessian Structure
• Transformer Optimization
• Nonconvex Optimization

Innovation

• Adam optimizer
• Hessian structure
• phase transition
• Adam-mini
• diagonal preconditioner