Cautious Optimizers: Improving Training with One Line of Code

2024-11-25

arXiv.org

AI Summary

To address the slow convergence and poor stability of momentum-based optimizers (e.g., AdamW) in Transformer pretraining, this paper proposes the “Cautious Optimizer” framework. Grounded in Hamiltonian dynamics and Lyapunov stability theory, it first identifies intrinsic structural deficiencies in momentum optimizers and then introduces a joint gradient clipping and step-size calibration mechanism. The framework enables theoretical convergence guarantees with minimal implementation overhead—requiring only a single line of PyTorch code to transform any momentum optimizer into a certified variant (e.g., C-AdamW, C-Lion). Empirically, it accelerates Llama and MAE pretraining by 1.47× and significantly improves performance across LLM post-training tasks. The open-source implementation has been widely adopted in the community.

Technology Category

• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV
• Machine Learning: Optimization

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

AdamW has been the default optimizer for transformer pretraining. For many years, our community searched for faster and more stable optimizers with only constrained positive outcomes. In this work, we propose a single-line modification in Pytorch to any momentum-based optimizer, which we rename cautious optimizer, e.g. C-AdamW and C-Lion. Our theoretical result shows that this modification preserves Adam's Hamiltonian function and it does not break the convergence guarantee under the Lyapunov analysis. In addition, a whole new family of optimizers is revealed by our theoretical insight. Among them, we pick the simplest one for empirical experiments, showing not only speed-up on Llama and MAE pretraining up to $1.47$ times, but also better results in LLM post-training tasks. Code is available at https://github.com/kyleliang919/C-Optim.

Problem

• Optimizer Improvement
• Transformer Model Training
• Momentum-based Optimization

Innovation

• Pytorch
• Cautious Optimizer
• Performance Enhancement