Towards Quantifying the Hessian Structure of Neural Networks

2025-05-05

AI Summary

The origins of the approximate block-diagonal structure observed in neural network Hessian matrices remain theoretically unexplained. Method: We propose a dual explanatory framework—“static forces” (architectural design) and “dynamic forces” (training dynamics)—and rigorously disentangle their effects for the first time. Using random matrix theory, linear models, and single-hidden-layer networks under both MSE and cross-entropy losses, we analyze the spectral behavior of diagonal versus off-diagonal Hessian blocks under random initialization. Contribution/Results: We prove that as the number of classes (C o infty), the spectra of diagonal and off-diagonal Hessian blocks spectrally separate, causing block diagonality to emerge intrinsically; (C) is the dominant structural parameter. This result holds across loss functions and network scales, providing the first falsifiable theoretical explanation for the strongly block-diagonal Hessians empirically observed in large language models ((C sim 10^4)–(10^5)).

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Empirical studies reported that the Hessian matrix of neural networks (NNs) exhibits a near-block-diagonal structure, yet its theoretical foundation remains unclear. In this work, we reveal two forces that shape the Hessian structure: a ``static force'' rooted in the architecture design, and a ``dynamic force'' arisen from training. We then provide a rigorous theoretical analysis of ``static force'' at random initialization. We study linear models and 1-hidden-layer networks with the mean-square (MSE) loss and the Cross-Entropy (CE) loss for classification tasks. By leveraging random matrix theory, we compare the limit distributions of the diagonal and off-diagonal Hessian blocks and find that the block-diagonal structure arises as $C ightarrow infty$, where $C$ denotes the number of classes. Our findings reveal that $C$ is a primary driver of the near-block-diagonal structure. These results may shed new light on the Hessian structure of large language models (LLMs), which typically operate with a large $C$ exceeding $10^4$ or $10^5$.

Problem

• Theoretical foundation of Hessian matrix structure in NNs
• Impact of static and dynamic forces on Hessian
• Role of class count C in block-diagonal structure

Innovation

• Analyzed Hessian matrix block-diagonal structure in NNs
• Identified static and dynamic forces shaping Hessian
• Used random matrix theory for theoretical analysis