Faster Acceleration for Steepest Descent

2024-09-28

arXiv.org

AI Summary

This work addresses smooth convex optimization under the ℓₚ norm, breaking the traditional reliance of acceleration methods on Euclidean geometry and achieving, for the first time, dimension-adaptive first-order acceleration. We propose a novel framework coupling heterogeneous-norm duality iteration with implicit interpolation, built upon primal-dual sequence design, non-Euclidean smoothness analysis, and ℓₚ-specific first-order oracle optimization. For d-dimensional ℓₚ-smooth functions, our method reduces the first-order oracle complexity to O(d^{1−2/p}). This bound strictly improves upon classical accelerated rates—e.g., O(d^{1/2})—and resolves a long-standing bottleneck in dimension dependence. Our result establishes the first universal, tight acceleration theory for non-Euclidean optimization, providing both conceptual clarity and practical efficiency gains for high-dimensional ℓₚ-structured problems.

Technology Category

• Search and Optimization: Non-convex Optimization
• Machine Learning: Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Security and Privacy: Large-scale security measurements

Abstract

Recent advances (Sherman, 2017; Sidford and Tian, 2018; Cohen et al., 2021) have overcome the fundamental barrier of dimension dependence in the iteration complexity of solving $ell_infty$ regression with first-order methods. Yet it remains unclear to what extent such acceleration can be achieved for general $ell_p$ smooth functions. In this paper, we propose a new accelerated first-order method for convex optimization under non-Euclidean smoothness assumptions. In contrast to standard acceleration techniques, our approach uses primal-dual iterate sequences taken with respect to $ extit{differing}$ norms, which are then coupled using an $ extit{implicitly}$ determined interpolation parameter. For $ell_p$ norm smooth problems in $d$ dimensions, our method provides an iteration complexity improvement of up to $O(d^{1-frac{2}{p}})$ in terms of calls to a first-order oracle, thereby allowing us to circumvent long-standing barriers in accelerated non-Euclidean steepest descent.

Problem

• Accelerate steepest descent
• Overcome dimension dependence
• Improve iteration complexity

Innovation

• Non-Euclidean smoothness assumptions
• Primal-dual iterate sequences
• Implicit interpolation parameter