BADM: Batch ADMM for Deep Learning

2024-06-30

arXiv.org

AI Summary

To address the slow convergence and limited generalization of stochastic gradient descent (SGD) in deep learning, this paper proposes Batched Alternating Direction Method of Multipliers (BADM)—the first ADMM-based optimization algorithm tailored for batch-driven deep neural network training. BADM introduces hierarchical batching (batch + sub-batch), joint primal-dual variable updates, and global parameter aggregation, enabling efficient distributed optimization without second-order information. Theoretical analysis establishes its convergence guarantee under standard assumptions. Empirical evaluation across graph modeling, computer vision, image generation, and natural language processing tasks demonstrates that BADM significantly accelerates convergence compared to SGD and Adam, while achieving superior test accuracy. Crucially, it simultaneously improves optimization efficiency and generalization performance—resolving a key trade-off in deep learning optimization.

Technology Category

• Machine Learning: Optimization
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Stochastic gradient descent-based algorithms are widely used for training deep neural networks but often suffer from slow convergence. To address the challenge, we leverage the framework of the alternating direction method of multipliers (ADMM) to develop a novel data-driven algorithm, called batch ADMM (BADM). The fundamental idea of the proposed algorithm is to split the training data into batches, which is further divided into sub-batches where primal and dual variables are updated to generate global parameters through aggregation. We evaluate the performance of BADM across various deep learning tasks, including graph modelling, computer vision, image generation, and natural language processing. Extensive numerical experiments demonstrate that BADM achieves faster convergence and superior testing accuracy compared to other state-of-the-art optimizers.

Problem

• Deep Learning
• Neural Networks
• Training Efficiency

Innovation

• BADM
• ADMM
• Deep Learning Acceleration