SHANG++: Robust Stochastic Acceleration under Multiplicative Noise

2026-03-10

AI Summary

本文针对乘性噪声下Nesterov加速的不稳定性问题，通过离散化Hessian驱动的加速梯度流提出SHANG及改进版SHANG++方法，增强了鲁棒性和收敛速度。

Abstract

Under the multiplicative noise scaling (MNS) condition, original Nesterov acceleration is provably sensitive to noise and may diverge when gradient noise overwhelms the signal. In this paper, we develop two accelerated stochastic gradient descent methods by discretizing the Hessian-driven Nesterov accelerated gradient flow. We first derive SHANG, a direct Gauss-Seidel-type discretization that already improves stability under MNS. We then introduce SHANG++, which adds a damping correction and achieves faster convergence with stronger noise robustness. We establish convergence guarantees for both convex and strongly convex objectives under MNS, together with explicit parameter choices. In our experiments, SHANG++ performs consistently well across convex problems and applications in deep learning. In a dedicated noise experiment on ResNet-34, a single hyperparameter configuration attains accuracy within 1% of the noise-free setting. Across all experiments, SHANG++ outperforms existing accelerated methods in robustness and efficiency, with minimal parameter sensitivity.

Problem

• multiplicative noise scaling
• Nesterov acceleration
• stochastic gradient descent
• noise robustness

Innovation

• SHANG++
• Hessian-driven Nesterov accelerated gradient flow
• Multiplicative Noise Scaling (MNS)
• Damping Correction
• Robustness