How the Hessian-Spectrum of Neural Networks Depends on Data

2026-07-15

AI Summary

This work investigates how the sharpness of neural network loss landscapes in classification tasks is influenced by the distribution of data across classes. By analytically deriving the Hessian spectrum of linear networks under mean squared error loss for arbitrary width, depth, and dataset size, and corroborating the findings with large-scale numerical experiments, the study establishes—for the first time under minimal simplifying assumptions—an exact analytical relationship between the largest eigenvalue of the Hessian and the proportion of samples in the dominant class. The theory demonstrates that solution sharpness is governed primarily by the prevalence of the most frequent class and exhibits remarkable robustness across diverse nonlinear architectures and realistic settings, thereby systematically elucidating the critical role of data structure in shaping optimization landscapes.

Abstract

The Hessian matrix is an important quantity of interest when it comes to studying the loss landscape and optimization dynamics in deep learning, as well as designing measures of generalization, second-order learning algorithms, etc. Prior works have focused on empirical results or pursued a theoretical treatment under overly simplified settings. In this work, we derive the eigenvalues of the Hessian of linear networks with arbitrary widths and depths, and datasets with an arbitrary number of samples, features, and labels. Importantly, for classification tasks with MSE loss, we identify that the sharpness of the solution is directly related to the maximum proportion of samples belonging to any class. We empirically validate our predictions and systematically analyze the effects of shedding the impractical assumptions one at a time, as well as incorporating nonlinearities. We observe that our predictions are considerably robust in most cases, allowing us to extend our conclusions to more practical learning setups.

Problem

• Hessian spectrum
• neural networks
• loss landscape
• data dependence
• sharpness

Innovation

• Hessian spectrum
• linear neural networks
• loss landscape sharpness
• MSE classification
• theoretical deep learning