Gradient Methods with Online Scaling Part I. Theoretical Foundations

2025-05-29

AI Summary

This paper addresses the challenge of adaptive step-size selection in first-order optimization by proposing the Online Scaled Gradient Method (OSGM), which learns dynamic step sizes online via a convergence-driven feedback function, yielding theoretically guaranteed acceleration. Its main contributions are threefold: (1) It introduces the first first-order algorithm with non-asymptotic superlinear convergence; (2) It unifies and theoretically justifies heuristic approaches such as hypergradient descent; (3) For smooth convex problems, it ensures trajectory-dependent global convergence; for smooth strongly convex problems, it achieves improved iteration complexity over classical methods and attains local superlinear convergence—its asymptotic performance matching or exceeding that of the optimal fixed step size.

Technology Category

• Search and Optimization: Non-convex Optimization
• Machine Learning: Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Social networks and social learning

Abstract

This paper establishes the theoretical foundations of the online scaled gradient methods (OSGM), a framework that utilizes online learning to adapt stepsizes and provably accelerate first-order methods. OSGM quantifies the effectiveness of a stepsize by a feedback function motivated from a convergence measure and uses the feedback to adjust the stepsize through an online learning algorithm. Consequently, instantiations of OSGM achieve convergence rates that are asymptotically no worse than the optimal stepsize. OSGM yields desirable convergence guarantees on smooth convex problems, including 1) trajectory-dependent global convergence on smooth convex objectives; 2) an improved complexity result on smooth strongly convex problems, and 3) local superlinear convergence. Notably, OSGM constitutes a new family of first-order methods with non-asymptotic superlinear convergence, joining the celebrated quasi-Newton methods. Finally, OSGM explains the empirical success of the popular hypergradient-descent heuristic in optimization for machine learning.

Problem

• Establishes theoretical foundations for online scaled gradient methods (OSGM)
• Achieves asymptotically optimal convergence rates via adaptive stepsizes
• Provides non-asymptotic superlinear convergence for first-order methods

Innovation

• Online learning adapts stepsizes for acceleration
• Feedback function quantifies stepsize effectiveness
• Achieves non-asymptotic superlinear convergence rates