A Training-Time Diagnostic for Generalization via the Log-Alignment Ratio

2026-05-27

AI Summary

This work proposes a validation-free, real-time generalization diagnostic method by reformulating the Logit Alignment Ratio (LAR) as the spectral overlap between weight and activation spectra, enabling dynamic tracking of alignment evolution between parameters and activations during training to distinguish memorization from generalization. For the first time, it establishes a direct link between LAR, effective functional dimensionality, and the generalization gap, allowing efficient monitoring using only forward-pass data. Leveraging singular value decomposition–based spectral analysis and parameterization theory, the method accurately predicts effective dimensionality in grokking tasks and effectively tracks the generalization gap during pretraining of a 3-billion-parameter language model, all with negligible computational overhead.

Abstract

We study the log-alignment ratio (LAR), a measure of parameter-activation alignment, introduced in parameterization theory. We reformulate it as the overlap between a weight spectrum $p$ of the normalized squared singular values of a matrix and an activation spectrum $q$ of the normalized squared projections of inputs onto its singular directions. We show that unembedding LAR tracks the transition between memorization and generalization in two different settings by capturing the spread of $p$ and $q$ during training. In grokking, LAR predicts the effective dimension of the learned function: $k \approx n^{2(1-\text{LAR})}$, where $n$ is the input dimension of the matrix. In 3B-parameter language model pre-training, its deviation from a non-overfitting baseline tracks the generalization gap, and its rate of decline increases as overfitting approaches. LAR is computable from quantities available during the forward pass with negligible computational overhead, and requires no held-out validation data.

Problem

• generalization
• memorization
• log-alignment ratio
• grokking
• overfitting

Innovation

• log-alignment ratio
• generalization
• parameter-activation alignment
• grokking
• overfitting detection