🤖 AI Summary
This work addresses the poorly understood implicit smoothing mechanism of stochastic gradient descent (SGD) in non-convex optimization. We provide the first theoretical characterization of how the SGD noise—determined jointly by learning rate, batch size, and gradient variance—induces a quantifiable smoothing effect on the objective function, and establish intrinsic links among smoothness, sharpness, and generalization performance. We propose a progressive optimization algorithm featuring joint scheduling of learning rate and batch size to dynamically modulate noise intensity for optimal implicit regularization. Empirical validation on ResNet-based image classification demonstrates significantly improved convergence stability and test accuracy; moreover, smoothness strongly correlates with generalization accuracy. Our core contributions are threefold: (i) an explicit, interpretable smoothing interpretation of SGD noise; (ii) a sharpness-driven theoretical framework for generalization; and (iii) the first noise-adaptive optimization paradigm with rigorous theoretical justification.
📝 Abstract
The graduated optimization approach is a heuristic method for finding global optimal solutions for nonconvex functions by using a function smoothing operation with stochastic noise. We show that stochastic noise in stochastic gradient descent (SGD) has the effect of smoothing the objective function, the degree of which is determined by the learning rate, batch size, and variance of the stochastic gradient. Using this finding, we propose and analyze a new graduated optimization algorithm that varies the degree of smoothing by varying the learning rate and batch size, and provide experimental results on image classification tasks with ResNets that support our theoretical findings. We further show that there is an interesting correlation between the degree of smoothing by SGD's stochastic noise, the well-studied ``sharpness'' indicator, and the generalization performance of the model.