Training in reverse: How iteration order influences convergence and stability in deep learning

2025-02-03

AI Summary

This work identifies the gradient update order itself as a critical factor influencing deep learning optimization dynamics. To address this, we propose backward stochastic gradient descent (backward-SGD), which applies parameter updates in reverse mini-batch order. Theoretically, we prove that within contraction mapping regions, backward-SGD converges deterministically to a unique fixed point, whereas standard SGD only converges in distribution—yielding superior deterministic convergence and stability without requiring learning rate scheduling or data augmentation. Empirically, under constant learning rates and limited-data regimes, backward-SGD significantly improves training robustness and final test accuracy. Crucially, this study establishes update ordering as an independent, controllable optimization dimension—introducing “order modulation” as a lightweight, hyperparameter-free paradigm for enhancing training stability.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Despite exceptional achievements, training neural networks remains computationally expensive and is often plagued by instabilities that can degrade convergence. While learning rate schedules can help mitigate these issues, finding optimal schedules is time-consuming and resource-intensive. This work explores theoretical issues concerning training stability in the constant-learning-rate (i.e., without schedule) and small-batch-size regime. Surprisingly, we show that the order of gradient updates affects stability and convergence in gradient-based optimizers. We illustrate this new line of thinking using backward-SGD, which processes batch gradient updates like SGD but in reverse order. Our theoretical analysis shows that in contractive regions (e.g., around minima) backward-SGD converges to a point while the standard forward-SGD generally only converges to a distribution. This leads to improved stability and convergence which we demonstrate experimentally. While full backward-SGD is computationally intensive in practice, it highlights opportunities to exploit reverse training dynamics (or more generally alternate iteration orders) to improve training. To our knowledge, this represents a new and unexplored avenue in deep learning optimization.

Problem

• Deep Learning Optimization
• Training Stability
• Learning Efficiency

Innovation

• Gradient Calculation Order
• Reversed SGD
• Deep Learning Optimization