🤖 AI Summary
In deep learning, the sharpness of a loss function—quantified by the largest eigenvalue of its Hessian—constrains the stability of gradient step sizes, yet conventional estimation methods are computationally prohibitive. This work reveals for the first time that the step size accepted by a single Armijo backtracking line search implicitly encodes directional curvature information, enabling an efficient approximation of sharpness. Building on this insight, we propose a low-overhead probe (≈1% computational cost) along the Adam update direction, combined with a fixed safety factor κ=2, to automatically set a calibration-free upper bound on the learning rate for Adam and AdamW. Experiments demonstrate strong negative correlations (−0.91 to −0.95) between logα and logλ₁ across CIFAR-10, Fashion-MNIST, and Imagenette, and show robust prevention of optimization divergence over initial learning rates spanning three orders of magnitude (10⁻³–3.0) without any additional hyperparameter tuning.
📝 Abstract
The local sharpness of the loss, the top Hessian eigenvalue $λ_1$, determines the largest stable gradient step, but measuring it normally requires Lanczos or Hessian-vector iterations. We observe that a single Armijo backtracking line search already carries this information at the cost of a few forward passes: the accepted step $α$ brackets the \emph{directional} curvature $q = g^\top H g/\|g\|^2$ within the multiplicative band set by the backtracking factor. Across CIFAR-10, Fashion-MNIST and Imagenette, $\logα$ tracks $\logλ_1$ at Pearson $-0.91$ to $-0.95$, giving a low-cost online Edge-of-Stability reading. Used once at initialisation, this measurement yields a learning-rate cap (a safeguard, not a faster optimiser) that makes Adam robust to a too-large initial learning rate across more than three orders of magnitude ($10^{-3}$ to $3.0$), at about one percent overhead, and it is a no-op when the chosen rate is already safe. One probe is enough: periodic in-training probing adds no robust benefit. The raw-gradient probe exposes the mechanism but needs a safety factor calibrated to the architecture by a one-minute divergence sweep. Probing along Adam's own update direction removes this calibration: a single fixed safety factor $κ= 2$ avoids divergence on all nine architectures we test and across the full learning-rate grids of all four benchmarks, and the recipe transfers to AdamW unchanged.