🤖 AI Summary
To address the degradation of adaptive optimizers (e.g., Adam) under regularization, this paper establishes a unified mathematical framework integrating preconditioning and regularization. It reveals that algorithms such as AdaGrad, RMSProp, and Adam inherently accelerate convergence via implicit Hessian conditioning improvement. We propose a novel paradigm—gradient regularization compatible with preconditioning—and systematically analyze how ℓ₂ regularization, weight decay (AdamW), and normalization affect preconditioning efficacy. For the first time, we reinterpret normalization’s acceleration mechanism through the lens of Hessian condition number, proving it enhances optimization stability by alleviating Hessian ill-conditioning. Building on theoretical analysis, we derive design principles for regularization schemes tailored to preconditioned optimization and devise a condition-number-aware preconditioned training algorithm. Our work provides a unifying explanation for diverse acceleration techniques and enables scalable, principled optimization.
📝 Abstract
Accelerated training algorithms, such as adaptive learning rates and various normalization methods, are widely used but not fully understood. When regularization is introduced, standard optimizers like adaptive learning rates may not perform effectively. This raises the need for alternative regularization approaches and the question of how to properly combine regularization with preconditioning. In this paper, we address these challenges using the theory of preconditioning as follows: (1) We explain how preconditioning with AdaGrad, RMSProp, and Adam accelerates training; (2) We explore the interaction between regularization and preconditioning, outlining different options for selecting the variables for regularization, and in particular we discuss how to implement that for the gradient regularization; and (3) We demonstrate how normalization methods accelerate training by improving Hessian conditioning, and discuss how this perspective can lead to new preconditioning training algorithms. Our findings offer a unified mathematical framework for understanding various acceleration techniques and deriving appropriate regularization schemes.