Stochastic Adaptive Gradient Descent Without Descent

2025-09-18

AI Summary

To address the reliance of stochastic gradient descent (SGD) on manually tuned step sizes in convex stochastic optimization, this paper proposes a fully hyperparameter-free adaptive step-size strategy. The method accesses only first-order stochastic gradient information via an oracle and dynamically captures the local geometric structure of the objective function—marking the first successful extension of “descent-free” adaptive gradient ideas to the stochastic optimization setting. Theoretically, under standard convexity and gradient variance assumptions, we establish a rigorous $O(1/sqrt{T})$ convergence rate. Empirically, the method matches the performance of carefully tuned baseline algorithms across diverse tasks—including logistic regression, neural network training, and robust optimization—while requiring zero hyperparameter configuration. This eliminates manual tuning overhead, significantly enhancing algorithmic robustness, reproducibility, and practical applicability.

Technology Category

• Search and Optimization: Non-convex Optimization
• Machine Learning: Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

We introduce a new adaptive step-size strategy for convex optimization with stochastic gradient that exploits the local geometry of the objective function only by means of a first-order stochastic oracle and without any hyper-parameter tuning. The method comes from a theoretically-grounded adaptation of the Adaptive Gradient Descent Without Descent method to the stochastic setting. We prove the convergence of stochastic gradient descent with our step-size under various assumptions, and we show that it empirically competes against tuned baselines.

Problem

• Introduces adaptive step-size for stochastic convex optimization
• Uses first-order oracle without hyper-parameter tuning
• Proves convergence and competes against tuned baselines

Innovation

• Stochastic adaptive step-size strategy
• No hyper-parameter tuning required
• First-order stochastic oracle utilization