🤖 AI Summary
Existing theoretical frameworks for the function spaces realized by deep neural networks (DNNs) remain confined to shallow models, failing to characterize their practical inductive biases and inherent sparse structures. This work establishes, for the first time, a reproducing kernel Banach space (RKBS) intrinsically associated with DNNs, introducing a sparsity-inducing norm to explicitly model their inductive bias and deriving a representation theorem applicable to finite-depth architectures. Methodologically, it integrates reproducing kernel theory, variational analysis, and functional analysis—transcending conventional shallow-network theory. The main contributions are: (1) a rigorous correspondence between DNNs and RKBSs; (2) revelation of an intrinsic sparse functional structure underlying DNN representations; and (3) theoretical justification for finite-width/depth networks, supporting their capacity to adaptively model latent input structures—thereby advancing foundational theory for interpretable deep learning.
📝 Abstract
Studying the function spaces defined by neural networks helps to understand the corresponding learning models and their inductive bias. While in some limits neural networks correspond to function spaces that are reproducing kernel Hilbert spaces, these regimes do not capture the properties of the networks used in practice. In contrast, in this paper we show that deep neural networks define suitable reproducing kernel Banach spaces. These spaces are equipped with norms that enforce a form of sparsity, enabling them to adapt to potential latent structures within the input data and their representations. In particular, leveraging the theory of reproducing kernel Banach spaces, combined with variational results, we derive representer theorems that justify the finite architectures commonly employed in applications. Our study extends analogous results for shallow networks and can be seen as a step towards considering more practically plausible neural architectures.