Neural Rank Collapse: Weight Decay and Small Within-Class Variability Yield Low-Rank Bias
This work investigates the origin of low-rank bias in deep neural networks and its connection to neural collapse. For general feedforward networks with nonlinear activations, we propose the “neural rank collapse” mechanism: weight decay jointly with intra-class variance in hidden layers drives rapid singular value decay across weight matrices, inducing progressive rank reduction. We establish, for the first time in nonlinear deep networks, a quantitative theoretical link between low-rank bias and neural collapse—extending beyond existing linear-network analyses. Our theory proves that the rank decay rate is proportional to the intra-class variance of the preceding layer’s hidden representations. Using singular value analysis, statistical modeling of latent-space distributions, and extensive experiments across architectures (ResNet, CNN), we empirically validate the mechanism. Furthermore, leveraging this insight, we achieve controllable rank compression of weight matrices by over 30% without sacrificing accuracy.