Scalar Representations of Neural Network Training Dynamics

2026-06-29

AI Summary

This work addresses the challenge of analytically characterizing the dynamics of high-dimensional neural network training trajectories. To this end, it introduces—for the first time—the scalar embedding methodology from time-series analysis into the study of training dynamics, constructing low-dimensional representations that capture essential dynamical features. By defining a Lyapunov-like characteristic timescale, estimating Lyapunov exponents, performing trajectory sensitivity analyses, and examining inter-trajectory distance statistics, the study reveals the decorrelation scale and asymptotic behavior of the training process. Experimental results demonstrate that the embedded trajectories faithfully reproduce the dynamics of the original parameter space, and that the asymptotic distribution of trajectory distances consistently follows a skewed log-normal form across diverse settings, thereby validating both the efficacy and theoretical significance of the proposed approach.

Abstract

Training in artificial neural networks can be viewed as a trajectory evolving through a high-dimensional loss landscape. However, the large number of trainable parameters makes the direct analysis of these dynamics challenging. In this work, we treat such training trajectories as temporal networks and apply recently proposed strategies for the scalar embedding of temporal networks. We investigate whether such a scalar embedding provides a meaningful low-dimensional representation of neural network training dynamics. Using a multilayer perceptron trained on the MNIST classification task, we show that the embedding preserves the main dynamical features observed in the original parameter space, including the emergence of sensitivity to initial conditions for specific learning rate regimes and an accurate reconstruction of the network's maximum Lyapunov exponent. We then use the embedded scalar trajectory to define a characteristic time, analogous to a Lyapunov time, after which the exponential separation between initially close embedded trajectories saturates. This characteristic time captures the typical decorrelation time between initially close network trajectories in the original high-dimensional system. Finally, we investigate the statistical organization of asymptotic training states through a spacing observable defined in the embedded space. We find that the distributions of rescaled asymptotic spacings collapse onto a common form across initial conditions and are compatible with a skew lognormal distribution. Altogether, our results suggest that scalar low-dimensional embeddings provide a useful framework for studying and visualizing the dynamical properties of neural network optimization trajectories.

Problem

• training dynamics
• scalar embedding
• temporal networks
• loss landscape
• Lyapunov exponent

Innovation

• scalar embedding
• temporal networks
• training dynamics
• Lyapunov exponent
• low-dimensional representation