Spectral Lens: Activation and Gradient Spectra as Diagnostics of LLM Optimization

2026-05-07

AI Summary

Conventional training metrics often fail to reveal differences in the internal representations and optimization dynamics of large language models. This work proposes a dual-perspective diagnostic protocol that combines activation covariance spectra with single-sample gradient SVD spectra to systematically analyze the evolution of representational geometry and learning dynamics under varying training configurations. For the first time, activation and gradient spectra are leveraged as predictive diagnostic tools for downstream efficiency and for distinguishing between architectural improvement types. Through mechanistic modeling, the study elucidates how spectral features relate to task-relevant feature learning. Experiments on 12–48 layer decoder-only models demonstrate that batch size implicitly shapes representational geometry, that early tail components of activation spectra predict token efficiency, and that coordinated changes in spectral heads and gradient spectra effectively differentiate optimization gains arising from learning-side versus execution-side modifications.

Abstract

Training loss and throughput can hide distinct internal representation in language-model training. To examine these hidden mechanics, we use spectral measurements as practical and operational diagnostics. Using a controlled family of decoder-only models adapted from the modded NanoGPT codebase, we introduce an empirical protocol based on activation covariance and per-sample gradient SVD spectra. This dual-view reveals three empirical findings and one mechanistic explanation. First, batch size acts as a latent determinant of representation geometry: runs that reach equal loss settle into systematically distinct activation spectra. Second, the activation covariance tail measured early in training reliably forecasts downstream token efficiency. Third, movement of the activation spectrum head (leading modes), together with gradient spectra, characterizes underlying learning-dynamics changes, separating learning-side architectural improvements from primarily execution-side gains. These predictive and diagnostic signals persist across the 12-, 36-, and 48-layer model tiers. Finally, a mechanistic model proves the main observations and explains how activation covariance spectra correlate with task-aligned feature learning.

Problem

• spectral diagnostics
• activation spectra
• gradient spectra
• representation geometry
• LLM optimization

Innovation

• spectral analysis
• activation covariance
• gradient SVD
• representation geometry
• learning dynamics