Accelerated Gradient Descent for Faster Convergence with Minimal Overhead

2026-05-15

AI Summary

This work addresses the slow convergence of non-convex optimization in deep learning by proposing CT-AGD, a novel method that, for the first time, effectively incorporates local curvature information into a lightweight first-order accelerated framework. CT-AGD explicitly estimates curvature via finite differences and introduces a noise-robust heuristic strategy to mitigate bias and perturbations induced by mini-batch training. While maintaining computational and memory overhead comparable to adaptive optimizers such as Adam, CT-AGD achieves significantly faster convergence, reducing the number of training epochs by 33% on average to reach the same accuracy level.

Abstract

In this paper, we present CT-AGD (Curvature-Tuned Accelerated Gradient Descent), an optimization method for non-convex optimization problems in deep learning training tasks. CT-AGD is a general boosting procedure that accelerates first-order methods by explicitly capturing the local curvature using finite-difference quotients, and the development of heuristics aimed at mitigating noise and bias introduced by stochastic mini-batch training. CT-AGD has a comparable storage and computational overhead as adaptive gradient methods such as Adam. Our extensive experiments demonstrate that CT-AGD achieves the same level of accuracy as the baseline first-order methods, yet reduces the required training epochs by 33% on average.

Problem

• non-convex optimization
• accelerated gradient descent
• deep learning training
• convergence acceleration
• first-order methods

Innovation

• Curvature estimation
• Accelerated gradient descent
• Non-convex optimization
• Stochastic mini-batch training
• First-order methods