Sublinearly Structured Deep Neural Networks Achieve Feature Learning Consistency for Compositional Functions

2026-06-22

AI Summary

This work investigates whether deep neural networks can achieve feature learning and predictive consistency comparable to classical models when learning target functions with hierarchical compositional structure. For deep networks whose input/output dimensions and hidden layer widths grow sublinearly with sample size, the study establishes the first theoretical guarantee of feature learning consistency and demonstrates that such architectures possess universal approximation capability for hierarchical compositional functions. Theoretical analysis reveals that mainstream CNN architectures inherently satisfy this sublinear structural condition. Empirical results corroborate the asymptotic consistency of these networks in large-sample regimes and show that their predictive performance matches or even surpasses that of significantly wider networks.

Abstract

Over the past decade, deep neural networks (DNNs) have achieved remarkable success on complex machine-learning tasks, yet the theoretical foundations of their performance remain incomplete. From a statistical viewpoint, a natural question is: can DNNs attain feature-learning and prediction consistency comparable to that of classical models? While a full characterization is open, we provide positive results for a broad subclass. We establish feature-learning consistency guarantees for sublinearly structured DNNs-architectures whose input/output dimensions and number of hidden neurons grow sublinearly with the sample size-when learning hierarchically compositional target functions. Importantly, this consistency still holds even in the conventional "over-parameterized" regime where the total number of parameters exceeds the number of training samples. Empirically, sublinearly structured DNNs match or surpass wide DNNs in prediction. A structural audit further indicates that widely used convolutional neural networks (CNNs), including AlexNet, VGGNet, ResNet, GoogLeNet, are sublinearly structured on their image classification benchmarks. We further prove that the sublinearly structured DNNs achieve universal approximation for hierarchically compositional functions in the large-sample limit. Moreover, images exhibit an inherent hierarchical, compositional structure. Taken together, these results explain, through a statistical lens, why many large-scale deep learning models succeed after adequate training on massive image datasets.

Problem

• feature learning consistency
• compositional functions
• deep neural networks
• statistical consistency
• hierarchical structure

Innovation

• sublinearly structured DNNs
• feature-learning consistency
• compositional functions
• over-parameterization
• universal approximation