AdaGrad does not adapt to Hölder-smoothness for composite objectives

2026-06-29

AI Summary

This work reveals that standard AdaGrad fails to achieve the classical convergence rate in Hölder smooth composite convex optimization due to excessive decay of its cumulative step size, caused by non-vanishing gradients at the optimum. By constructing a one-dimensional deterministic counterexample, the study explicitly demonstrates, for the first time, a fundamental mismatch between AdaGrad’s gradient accumulation mechanism and the optimality conditions inherent to composite optimization. To address this limitation, the paper proposes alternative accumulation strategies based on either the gradient mapping or gradient differences, which effectively circumvent the identified deficiency. Through a theoretical analysis grounded in convex analysis, convergence theory, and Hölder smoothness assumptions, the authors prove that AdaGrad’s convergence rate on certain composite problems is provably worse than known lower bounds, thereby providing a rigorous foundation for designing improved adaptive algorithms.

Abstract

We exhibit a simple deterministic one-dimensional convex composite optimization problem for which AdaGrad scheme does not achieve the classical convergence rate $\mathcal{O}(n^{-(1+ν)/2})$ associated with Hölder-smooth objectives. The example highlights a basic mismatch between classical AdaGrad accumulation and composite optimality. A main insight is that the gradient of the smooth term may not vanish at the optimum, causing AdaGrad to keep reducing its stepsize excessively and converge more slowly. We also discuss why alternative accumulation mechanisms based on gradient mappings or on successive gradient differences, avoid this pathology.

Problem

• AdaGrad
• Hölder-smoothness
• composite optimization
• convergence rate

Innovation

• AdaGrad
• Hölder-smoothness
• composite optimization
• convergence rate
• adaptive stepsize