Adaptive backtracking line search

2024-08-23

International Conference on Learning Representations

AI Summary

Traditional backtracking line search employs a fixed scaling factor, requiring multiple trial steps to satisfy the Armijo condition—leading to inefficiency. This paper proposes a lightweight adaptive step-size mechanism that dynamically links the step-size scaling factor to the current violation magnitude of the descent criterion, without additional gradient or function evaluations, while preserving theoretical convergence guarantees. By integrating the descent lemma with an accelerated gradient framework, our method rigorously ensures convergence rates for both convex and nonconvex smooth optimization problems. Experiments across 15 real-world datasets demonstrate that, for convex problems, the number of backtracking iterations does not increase; for nonconvex problems, existing convergence guarantees are fully maintained; and overall optimization speed is significantly improved.

Technology Category

• Search and Optimization: Non-convex Optimization
• Machine Learning: Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Backtracking line search is foundational in numerical optimization. The basic idea is to adjust the step-size of an algorithm by a constant factor until some chosen criterion (e.g. Armijo, Descent Lemma) is satisfied. We propose a novel way to adjust step-sizes, replacing the constant factor used in regular backtracking with one that takes into account the degree to which the chosen criterion is violated, with no additional computational burden. This light-weight adjustment leads to significantly faster optimization, which we confirm by performing a variety of experiments on over fifteen real world datasets. For convex problems, we prove adaptive backtracking requires no more adjustments to produce a feasible step-size than regular backtracking does. For nonconvex smooth problems, we prove adaptive backtracking enjoys the same guarantees of regular backtracking. Furthermore, we prove adaptive backtracking preserves the convergence rates of gradient descent and its accelerated variant.

Problem

• Proposes adaptive step-size adjustment for faster optimization
• Ensures same guarantees as regular backtracking for nonconvex problems
• Preserves convergence rates of gradient descent methods

Innovation

• Adaptive step-size adjustment based on criterion violation
• No additional computational cost required
• Faster optimization with proven convergence guarantees