Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning

2026-05-13

AI Summary

This work elucidates how deep neural networks learn effective internal representations through hierarchical structures. To this end, it introduces the Neural LoFi framework, which, in a simplified limit of gradient-based training, reformulates deep feature learning as a layerwise iterative spectral method that selects directions maximally correlated with the labels at low orders. This study establishes the first analytically tractable theory of deep feature learning beyond lazy training regimes, demonstrating that deep networks construct novel features layer by layer via low-order compositionality and enabling predictions of the sample complexity required for concept emergence. Combining spectral analysis, kernel space interpretations, and a simplified model of gradient dynamics, the theory is validated on both fully connected and convolutional architectures: Neural LoFi substantially outperforms random feature baselines, recovers semantically structured filters, and yields predicted representations closely matching those obtained from early-stage gradient descent on real datasets.

Abstract

Understanding how deep neural networks learn useful internal representations from data remains a central open problem in the theory of deep learning. We introduce Neural Low-Degree Filtering (Neural LoFi), a stylized limit of gradient-based training in which hierarchical feature learning becomes an explicit iterative spectral procedure. In this limit, the dynamics at each layer decouple: given the current representation, the next layer selects directions with maximal accessible low-degree correlation to the label. This yields a tractable surrogate mechanism for deep learning, together with a natural kernel-space interpretation. Neural LoFi provides a mathematically explicit framework for studying multi-layer feature learning beyond the lazy regime. It predicts how representations are selected layer by layer, explains how emergence of concepts arises with given sample complexity,and gives a concrete mechanism by which depth progressively constructs new features from old ones through low-degree compositionality. We complement the theory with mechanistic experiments on fully connected and convolutional architectures, showing that Neural LoFi improves over lazy random-feature baselines, recovers meaningful structured filters, and predicts representations aligned with early gradient-descent feature discovery with real datasets.

Problem

• deep learning
• feature learning
• spectral theory
• neural networks
• representation learning

Innovation

• Neural Low-Degree Filtering
• spectral theory
• hierarchical feature learning
• low-degree correlation
• beyond lazy training