Randomized Subspace Nesterov Accelerated Gradient

2026-05-01

AI Summary

This work proposes a Nesterov-accelerated gradient method based on random subspaces for settings where only low-dimensional projected gradients are accessible—such as in forward-mode automatic differentiation or communication-constrained environments. The approach introduces an innovative three-sequence acceleration framework tailored to matrix smoothness, applicable to both smooth convex and strongly convex optimization problems under mild assumptions on matrix smoothness and general sketching moment conditions. Theoretical analysis demonstrates that the proposed method achieves oracle complexity superior to that of classical full-dimensional Nesterov acceleration. Moreover, it unifies the convergence analysis of subspace sketching methods, revealing the critical roles of matrix smoothness and sketch distribution in optimization efficiency, and provides a unified theoretical foundation for comparing the performance of different sketch families.

Abstract

Randomized-subspace methods reduce the cost of first-order optimization by using only low-dimensional projected-gradient information, a feature that is attractive in forward-mode automatic differentiation and communication-limited settings. While Nesterov acceleration is well understood for full-gradient and coordinate-based methods, obtaining accelerated methods for general subspace sketches that use only projected-gradient information and can improve over full-dimensional Nesterov acceleration in oracle complexity is technically nontrivial. We develop randomized-subspace Nesterov accelerated gradient methods for smooth convex and smooth strongly convex optimization under matrix smoothness and generic sketch moment assumptions. The key technical ingredient is a three-sequence formulation tailored to matrix smoothness, which recovers the corresponding classical Nesterov methods in the full-dimensional case. The resulting theory establishes accelerated oracle-complexity guarantees and makes explicit how matrix smoothness and the sketch distribution enter the complexity. It also provides a unified basis for comparing sketch families and identifying when randomized-subspace acceleration improves over full-dimensional Nesterov acceleration in oracle complexity.

Problem

• randomized subspace
• Nesterov acceleration
• projected gradient
• oracle complexity
• matrix smoothness

Innovation

• randomized subspace
• Nesterov acceleration
• matrix smoothness
• oracle complexity
• sketching