Characterizing Optimizer-Dependent Training Dynamics Through Hessian Eigenvector Displacement and Localization

2026-06-29

AI Summary

This study investigates how different optimizers influence the evolution of the leading eigenvectors of the Hessian during neural network training. By integrating concepts from spin-glass physics—specifically displacement metrics and the inverse participation ratio—with Hessian spectral analysis, eigenvector trajectory tracking, and architecture-aware random null models, the work reveals for the first time how optimizers modulate the stability of curvature directions and the degree of parameter localization. The findings demonstrate that SGD promotes stabilization of dominant curvature directions, whereas Adam induces pronounced eigenvector reconfiguration and stronger parameter localization, thereby underscoring the fundamental role of optimizer choice in shaping the dynamics of loss landscape exploration.

Abstract

Hessian spectral properties are a standard tool in analysing neural-network training, with eigenvalues linked to sharpness, generalization, and optimization dynamics. Eigenvalues quantify curvature magnitude, while eigenvectors identify which parameters generate that curvature. In this work, we study how the leading Hessian eigenvectors evolve during training and how they affect the learning trajectories. We track the training dynamics of multilayer perceptrons on a classification problem and measure eigenvector dynamics through two complementary statistics: (i) displacement over time, inspired by analyses of glassy systems, and (ii) localization via the inverse participation ratio. The metrics are compared against a random null model of the Hessian induced by the architecture. Our results reveal clear optimizer-dependent behaviour. SGD leads to progressively more stable leading curvature directions, while Adam exhibits substantially stronger reorganization of eigenvectors throughout training. We also observe a localization phenomenon under Adam, where a small subset of parameters contributes disproportionately to the leading curvature directions. These results suggest that Hessian eigenvector dynamics capture key differences in optimizer behaviour and the resulting training trajectories.

Problem

• Hessian eigenvectors
• optimizer-dependent dynamics
• training dynamics
• localization
• neural network optimization

Innovation

• Hessian eigenvector dynamics
• optimizer-dependent training
• eigenvector displacement
• localization
• inverse participation ratio