🤖 AI Summary
The rapid proliferation and evolution of deep learning optimizers necessitate a systematic synthesis to clarify their design principles, applicability domains, and shared challenges. Method: We propose the first unified mathematical framework for classifying and comparatively analyzing optimizers—from SGD and AdamW to recent methods such as Sophia and Muon—formally characterizing their update rules, hyperparameter semantics, and convergence properties. Contribution/Results: Through theoretical unification and empirical validation, we identify a coherent evolutionary trajectory: from adaptive learning rates, to second-order approximations, to sparse gradient correction. We explicitly delineate open challenges in generalization, large-model scalability, and computational efficiency. Our work establishes a structured knowledge base for optimizers, providing both theoretical foundations and practical guidance for algorithm selection, refinement, and novel paradigm design.
📝 Abstract
Deep learning optimizers are optimization algorithms that enable deep neural networks to learn. The effectiveness of learning is highly dependent on the optimizer employed in the training process. Alongside the rapid advancement of deep learning, a wide range of optimizers with different approaches have been developed. This study aims to provide a review of various optimizers that have been proposed and received attention in the literature. From Stochastic gradient descent to the most recent ones such as Momentum, AdamW, Sophia, and Muon in chronological order, optimizers are examined individually, and their distinctive features are highlighted in the study. The update rule of each optimizer is presented in detail, with an explanation of the associated concepts and variables. The techniques applied by these optimizers, their contributions to the optimization process, and their default hyperparameter settings are also discussed. In addition, insights are offered into the open challenges encountered in the optimization of deep learning models. Thus, a comprehensive resource is provided both for understanding the current state of optimizers and for identifying potential areas of future development.