Evolution of Optimization Methods: Algorithms, Scenarios, and Evaluations

2026-04-14

AI Summary

This work addresses the multifaceted trade-offs faced by deep learning optimizers in convergence speed, generalization, computational efficiency, and privacy preservation. It presents the first unified framework that systematically elucidates the underlying principles and applicable scenarios of first-order (e.g., SGD, Adam), second-order, and zeroth-order optimization methods. Through large-scale empirical evaluations across diverse model architectures and practical constraints—including distributed training and differential privacy—the study combines theoretical analysis with extensive experimentation to distill fundamental trade-offs and evolutionary patterns in optimizer design. Building on these insights, the paper proposes actionable guidelines for developing optimizers that enable efficient, robust, and trustworthy training. The accompanying benchmarking codebase is publicly released to foster reproducibility and further research.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Non-convex Optimization
• Computer Vision: Learning & Optimization for CV

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

Balancing convergence speed, generalization capability, and computational efficiency remains a core challenge in deep learning optimization. First-order gradient descent methods, epitomized by stochastic gradient descent (SGD) and Adam, serve as the cornerstone of modern training pipelines. However, large-scale model training, stringent differential privacy requirements, and distributed learning paradigms expose critical limitations in these conventional approaches regarding privacy protection and memory efficiency. To mitigate these bottlenecks, researchers explore second-order optimization techniques to surpass first-order performance ceilings, while zeroth-order methods reemerge to alleviate memory constraints inherent to large-scale training. Despite this proliferation of methodologies, the field lacks a cohesive framework that unifies underlying principles and delineates application scenarios for these disparate approaches. In this work, we retrospectively analyze the evolutionary trajectory of deep learning optimization algorithms and present a comprehensive empirical evaluation of mainstream optimizers across diverse model architectures and training scenarios. We distill key emerging trends and fundamental design trade-offs, pinpointing promising directions for future research. By synthesizing theoretical insights with extensive empirical evidence, we provide actionable guidance for designing next-generation highly efficient, robust, and trustworthy optimization methods. The code is available at https://github.com/APRIL-AIGC/Awesome-Optimizer.

Problem

• deep learning optimization
• convergence speed
• generalization capability
• computational efficiency
• privacy protection

Innovation

• optimization algorithms
• second-order methods
• zeroth-order optimization
• empirical evaluation
• deep learning