Simple Stepsize for Quasi-Newton Methods with Global Convergence Guarantees

2025-08-27

AI Summary

To address the lack of global convergence guarantees for quasi-Newton methods in general convex optimization, this paper proposes a simple step-size scheduling strategy that neither requires strong convexity nor conventional line search. By controlling the relative error in the Hessian approximation, the method is rigorously shown to achieve an $O(1/k)$ global convergence rate for convex objectives. An adaptive variant further attains an accelerated $O(1/k^2)$ rate. This constitutes the first quasi-Newton framework achieving both global and accelerated convergence using only fixed or adaptive step sizes—bypassing line search entirely. The approach significantly simplifies implementation while broadening theoretical applicability. Both theoretical analysis and empirical evaluation confirm its effectiveness and superiority over existing quasi-Newton and first-order methods.

Technology Category

• Search and Optimization: Non-convex Optimization
• Machine Learning: Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

Quasi-Newton methods are widely used for solving convex optimization problems due to their ease of implementation, practical efficiency, and strong local convergence guarantees. However, their global convergence is typically established only under specific line search strategies and the assumption of strong convexity. In this work, we extend the theoretical understanding of Quasi-Newton methods by introducing a simple stepsize schedule that guarantees a global convergence rate of ${O}(1/k)$ for the convex functions. Furthermore, we show that when the inexactness of the Hessian approximation is controlled within a prescribed relative accuracy, the method attains an accelerated convergence rate of ${O}(1/k^2)$ -- matching the best-known rates of both Nesterov's accelerated gradient method and cubically regularized Newton methods. We validate our theoretical findings through empirical comparisons, demonstrating clear improvements over standard Quasi-Newton baselines. To further enhance robustness, we develop an adaptive variant that adjusts to the function's curvature while retaining the global convergence guarantees of the non-adaptive algorithm.

Problem

• Develops simple stepsize for quasi-Newton global convergence
• Extends theoretical understanding with O(1/k) convergence rate
• Achieves accelerated O(1/k²) rate with controlled Hessian approximation

Innovation

• Simple stepsize schedule for global convergence
• Inexact Hessian approximation for accelerated rate
• Adaptive variant enhancing curvature robustness