A Regularized Newton Method for Nonconvex Optimization with Global and Local Complexity Guarantees

2025-02-07

AI Summary

This work addresses efficient computation of ε-stationary points in nonconvex optimization problems with Lipschitz-continuous Hessians. We propose an adaptive quadratic regularization Newton method: it introduces, for the first time, a gradient-difference-driven adaptive regularization term that eliminates the need to know the Hessian Lipschitz constant a priori; subproblems are solved via a negative-curvature-aware linear conjugate gradient method. Theoretically, the algorithm achieves a global second-order oracle complexity of O(ε⁻³⁄²) and an Hessian-vector product complexity of Õ(ε⁻⁷⁄⁴). Moreover, it attains local quadratic convergence when restricted to regions where the Hessian is positive definite. Numerical experiments demonstrate superior performance over state-of-the-art nonconvex optimizers.

Technology Category

• Search and Optimization: Non-convex Optimization
• Machine Learning: Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

We consider the problem of finding an $epsilon$-stationary point of a nonconvex function with a Lipschitz continuous Hessian and propose a quadratic regularized Newton method incorporating a new class of regularizers constructed from the current and previous gradients. The method leverages a recently developed linear conjugate gradient approach with a negative curvature monitor to solve the regularized Newton equation. Notably, our algorithm is adaptive, requiring no prior knowledge of the Lipschitz constant of the Hessian, and achieves a global complexity of $O(epsilon^{-frac{3}{2}}) + ilde O(1)$ in terms of the second-order oracle calls, and $ ilde O(epsilon^{-frac{7}{4}})$ for Hessian-vector products, respectively. Moreover, when the iterates converge to a point where the Hessian is positive definite, the method exhibits quadratic local convergence. Preliminary numerical results illustrate the competitiveness of our algorithm.

Problem

• Nonconvex optimization with global guarantees
• Adaptive quadratic regularized Newton method
• Achieves efficient second-order oracle complexity

Innovation

• Quadratic regularized Newton method
• Linear conjugate gradient approach
• Adaptive algorithm no Lipschitz constant