How Width and Data Shape Generalization Scaling Laws in Quadratic Neural Networks

2026-06-26

AI Summary

This work investigates how model width and training sample size jointly influence the generalization performance of finite-width, two-layer quadratic neural networks with ℓ² regularization under structured, finite-sample data regimes. Leveraging spectral analysis and finite-sample generalization theory, the study establishes—for the first time in finite-width networks capable of feature learning—an explicit, data-dependent expression for generalization error dominated by the spectral structure of the target function. This expression reveals power-law relationships governing how generalization error scales with network width, sample size, and regularization strength, identifies multiple scaling regimes and their phase-transition boundaries (such as the interpolation threshold), and demonstrates that the spectral structure of the data fundamentally determines the exponent of the generalization power law.

Abstract

Understanding how performance scales jointly with model size and data is a central problem in modern machine learning. Existing theoretical works on scaling laws typically describe generalization as a function of data or compute, often in fixed-feature or infinite-width regimes and for online SGD. Here, we instead study how generalization scales with the number of trainable parameters and the number of samples in a feature-learning model. We analyze $\ell_2$-regularized empirical test error minimization in a quadratic two-layer network in a finite-sample setting with structured data. This setting allows for an explicit characterization of the generalization error as a function of the number of samples, model width, and regularization. Our results reveal a phase diagram with distinct scaling regimes as the number of parameters varies. In particular, the generalization error follows data-dependent power laws controlled by the spectral structure of the target. We further characterize the transitions between regimes, including the onset of interpolation, and their impact on generalization.

Problem

• scaling laws
• generalization
• quadratic neural networks
• model width
• structured data

Innovation

• scaling laws
• quadratic neural networks
• generalization error
• finite-width analysis
• structured data