Representation Costs in Data Science: Foundations and the Quasi-Banach Spaces of Deep Neural Networks

2026-06-12

AI Summary

This work proposes a unified framework that characterizes the representation cost and the induced native function space of parametric data-fitting methods through regularization in parameter space. By axiomatizing the representation cost, it for the first time brings classical models—such as kernel methods, wavelets, and shallow networks—under a common theoretical lens and rigorously establishes the equivalence between parametric and nonparametric approaches in the over-parameterized regime. The central contribution is the proof that deep ReLU networks induce a native space that is a p-quasi-Banach space with \( p = 2/L \), revealing that when \( L > 2 \), their inductive bias cannot be captured by conventional norms. This result elucidates the fundamental role depth plays in shaping the structure of the associated function space.

Abstract

We develop a general framework for analyzing representation costs of parametric data-fitting methods through their parameter-space regularizers. From this abstract perspective, we define representation costs for arbitrary parametric models and reveal their induced (native) function spaces. This unifies recent function-space views of data-fitting methods. We also prove that many natural results hold in this abstract setting, including representer theorems for parametric methods on their native spaces. The framework also rigorously connects parametric methods with their equivalent nonparametric descriptions under sufficient overparameterization. Classical methods and their native spaces, such as kernel methods / reproducing kernel Hilbert spaces, wavelets / Besov spaces, and shallow neural networks / variation spaces emerge as special cases of our abstract framework. A byproduct of "axiomatizing" the study of representation costs is that we also immediately obtain new results for deep neural networks: For depth-$L$ feedforward ReLU networks, their induced native spaces are $p$-normable quasi-Banach spaces with $p = 2/L$. This reveals that the inductive bias of deep neural networks (as given by the representation cost) cannot be captured by norms for depths $L > 2$.

Problem

• representation cost
• inductive bias
• function spaces
• deep neural networks
• quasi-Banach spaces

Innovation

• representation cost
• quasi-Banach space
• deep neural networks
• inductive bias
• native function space