Data-Parallel Neural Network Training via Nonlinearly Preconditioned Trust-Region Method

2025-02-07

AI Summary

Gradient-based optimizers (e.g., SGD, Adam) in distributed training of large-scale deep neural networks suffer from manual hyperparameter tuning—especially learning rate—and limited scalability. Method: This paper proposes a data-parallel nonlinear preconditioned trust-region optimization method. It is the first to integrate nonlinear preconditioning into the trust-region framework, enabling implicit adaptive step-size control without explicit learning rate specification. The method employs efficient data parallelism and communication-aware optimizations to ensure practical feasibility. Contribution/Results: Evaluated on MNIST and CIFAR-10, the method achieves validation accuracy comparable to SGD and Adam while significantly improving training scalability and robustness across varying batch sizes and network architectures. Crucially, it eliminates the need for learning-rate tuning and other optimizer-specific hyperparameters, offering a new paradigm for hyperparameter-free, highly scalable distributed deep learning.

Technology Category

• Machine Learning: Optimization
• Computer Vision: Learning & Optimization for CV
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Parallel training methods are increasingly relevant in machine learning (ML) due to the continuing growth in model and dataset sizes. We propose a variant of the Additively Preconditioned Trust-Region Strategy (APTS) for training deep neural networks (DNNs). The proposed APTS method utilizes a data-parallel approach to construct a nonlinear preconditioner employed in the nonlinear optimization strategy. In contrast to the common employment of Stochastic Gradient Descent (SGD) and Adaptive Moment Estimation (Adam), which are both variants of gradient descent (GD) algorithms, the APTS method implicitly adjusts the step sizes in each iteration, thereby removing the need for costly hyperparameter tuning. We demonstrate the performance of the proposed APTS variant using the MNIST and CIFAR-10 datasets. The results obtained indicate that the APTS variant proposed here achieves comparable validation accuracy to SGD and Adam, all while allowing for parallel training and obviating the need for expensive hyperparameter tuning.

Problem

• Data-parallel training for deep neural networks
• Nonlinear preconditioner in optimization strategy
• Eliminates need for hyperparameter tuning

Innovation

• Nonlinearly Preconditioned Trust-Region Method
• Data-Parallel Approach for DNNs
• Implicit Step Size Adjustment