Gradient Descent Algorithm Survey

2025-11-25

AI Summary

The absence of systematic, practice-oriented guidelines for selecting and tuning optimization algorithms hinders efficient deep learning deployment. Method: We conduct a comprehensive literature review coupled with multi-scenario empirical evaluation—assessing SGD, Mini-batch SGD, Momentum, Adam, and Lion across convergence speed, training stability, and generalization performance—and perform rigorous parameter sensitivity analysis. Contribution/Results: We propose novel, task- and scale-agnostic algorithm selection principles and hyperparameter tuning paradigms, bridging a critical gap between optimization theory and engineering practice. Our standardized, empirically grounded guidelines significantly improve training efficiency and robustness, and have been successfully adopted in multiple academic studies and industrial-scale deep learning deployments.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Algorithm Configuration
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Focusing on the practical configuration needs of optimization algorithms in deep learning, this article concentrates on five major algorithms: SGD, Mini-batch SGD, Momentum, Adam, and Lion. It systematically analyzes the core advantages, limitations, and key practical recommendations of each algorithm. The research aims to gain an in-depth understanding of these algorithms and provide a standardized reference for the reasonable selection, parameter tuning, and performance improvement of optimization algorithms in both academic research and engineering practice, helping to solve optimization challenges in different scales of models and various training scenarios.

Problem

• Surveying five major optimization algorithms for deep learning configuration needs
• Analyzing core advantages and limitations of each algorithm systematically
• Providing standardized reference for algorithm selection and parameter tuning

Innovation

• Surveying five major gradient descent algorithms
• Analyzing core advantages and practical limitations
• Providing standardized reference for algorithm selection