Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks

2026-06-26

AI Summary

This work addresses the challenge of improving neural network generalization by analyzing optimization dynamics through the lens of data distribution and parameter structure, with a focus on steering training toward flat minima. Leveraging the Wolkowicz–Styan inequality, the authors derive, for the first time, a closed-form expression for the gradient of the largest eigenvalue of the Hessian of the cross-entropy loss—enabling explicit optimization of its spectral upper bound without numerical approximation. By updating parameters along the steepest descent direction defined by this gradient, the method effectively compresses the Hessian eigenvalue spectrum in three-layer networks, thereby avoiding sharp minima and saddle points. This approach guides convergence toward flatter minima that exhibit superior generalization, offering a novel theoretical and algorithmic pathway for understanding and promoting flatness in deep learning optimization.

Abstract

The flatness hypothesis suggests that flatness of the loss landscape, as measured by the eigenvalues of the loss Hessian, correlates with better neural network generalization. While various algorithms reduce these eigenvalues, most focus on procedural design, leaving it unclear how data distributions and NN parameters structurally determine directions toward flat minima. Characterizing these directions analytically is generally intractable. To overcome this mathematical difficulty, recent studies derived the Wolkowicz-Styan (WS) upper bound on the maximum eigenvalue of the cross-entropy loss Hessian in three-layer NNs. Although this upper bound is differentiable, its gradient was not derived. Therefore, we analytically derive the gradient of the WS upper bound to characterize directions leading to flat minima. Based on this, we propose Hessian Spectral Range (HSR) Regularization, which updates parameters along the steepest descent direction of the WS bound. Experiments demonstrate that HSR Regularization narrows the Hessian eigenvalue spectrum, avoids sharp minima and saddle points, and promotes convergence to flat minima. Although the applicability of this method is currently limited to cross-entropy loss and three-layer architectures, to the best of the authors' knowledge, this is the first study to report a closed-form gradient that promotes convergence to flat minima without numerical approximations. Therefore, the theoretical analysis of this gradient is expected to contribute to the further development of NNs.

Problem

• flat minima
• loss Hessian
• eigenspectrum
• generalization
• neural networks

Innovation

• closed-form gradient
• flat minima
• Hessian eigenvalue spectrum
• WS upper bound
• HSR Regularization