A short proof of near-linear convergence of adaptive gradient descent under fourth-order growth and convexity

2026-04-14

AI Summary

This work proposes a concise Lyapunov-based analytical framework to establish local approximate linear convergence of adaptive-stepsize gradient descent for smooth functions satisfying fourth-order growth conditions and convexity. Departing from conventional approaches that rely on intricate arguments involving “valley manifolds,” the analysis constructs a novel Lyapunov function to deliver a direct and rigorous convergence proof. Leveraging this insight, the authors design an enhanced adaptive variant of the algorithm. The resulting theoretical treatment is notably more streamlined, and numerical experiments demonstrate the superior convergence speed and stability of the proposed method.

Technology Category

• Search and Optimization: Non-convex Optimization
• Machine Learning: Learning with Manifolds
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Security and Privacy: Large-scale security measurements

Abstract

Davis, Drusvyatskiy, and Jiang showed that gradient descent with an adaptive stepsize converges locally at a nearly-linear rate for smooth functions that grow at least quartically away from their minimizers. The argument is intricate, relying on monitoring the performance of the algorithm relative to a certain manifold of slow growth -- called the ravine. In this work, we provide a direct Lyapunov-based argument that bypasses these difficulties when the objective is in addition convex and a has a unique minimizer. As a byproduct of the argument, we obtain a more adaptive variant than the original algorithm with encouraging numerical performance.

Problem

• adaptive gradient descent
• near-linear convergence
• fourth-order growth
• convexity
• unique minimizer

Innovation

• adaptive gradient descent
• Lyapunov analysis
• near-linear convergence
• fourth-order growth
• convex optimization