🤖 AI Summary
This work proposes an adaptive scalar-step gradient descent method for non-convex optimization that overcomes the restrictive assumptions of traditional approaches. Conventional methods rely on strong regularity conditions—such as global Lipschitz or Hölder continuity of the full gradient—which lead to overly conservative step sizes. In contrast, the proposed algorithm leverages one-sided Hölder regularity to estimate local curvature along the descent direction and dynamically adjusts the step size via a sufficient decrease condition. This strategy relaxes the need for global regularity assumptions while permitting larger steps in flat regions without compromising convergence. Theoretically, the method guarantees optimal stationarity of iterates for non-convex objectives. Empirical results demonstrate superior performance over existing scalar-step gradient methods in binary classification and non-convex Hölder regression tasks, achieving lower final loss, smaller gradient norms, and wider classification margins.
📝 Abstract
We study adaptive gradient descent for continuously differentiable, possibly nonconvex objectives under one-sided Hölder regularity. Unlike classical Hölder- or Lipschitz-gradient assumptions, which control the full gradient variation, our condition bounds only the directional term appearing in the descent inequality. This can allow less conservative step sizes when large gradient changes are orthogonal to, or favorable along, the update direction. We propose an adaptive scalar-step method based on an estimate of positive one-sided Hölder curvature, combined with a simple sufficient-decrease safeguard. For nonconvex objectives on a convex region containing the accepted update segments, we prove an explicit best-iterate stationarity bound with a rate determined by the Hölder exponent. Unlike predetermined diminishing step-size schemes, the method adapts to the local descent geometry. We evaluate the approach on two full-batch benchmarks designed to separate directional curvature from full gradient variation. On a binary classification problem, the method achieves the lowest final cross-entropy, objective value, and gradient norm, together with the largest classification margin among the compared scalar gradient methods. On a nonconvex Hölder regression problem, it attains the lowest final objective gap and gradient norm. These results indicate that one-sided Hölder curvature is an effective adaptive step-size signal when full-gradient variation is inflated by directions that do not hinder descent.