🤖 AI Summary
Why do deep neural networks (DNNs) generalize well despite residing in high-dimensional parameter spaces—a phenomenon unexplained by classical model selection theory? This paper proposes a novel theoretical framework integrating singular differential geometry and information geometry. Specifically, it defines the local effective dimension of the parameter manifold via spectral analysis of the Fisher information matrix and constructs a short-description-length measure aligned with the network’s intrinsic singularity. For the first time, this approach unifies singular semi-Riemannian geometry with locally varying-dimensional information theory, reconstructing model complexity from the geometric singularity of the parameter manifold and thereby providing a rigorous geometric foundation for Occam’s razor. Experiments demonstrate that the proposed measure quantitatively captures the low generalization error of DNNs even under high parametric complexity, substantially enhancing the interpretability of deep learning generalization mechanisms.
📝 Abstract
Why do deep neural networks (DNNs) benefit from very high dimensional parameter spaces? Their huge parameter complexities vs. stunning performances in practice is all the more intriguing and not explainable using the standard theory of model selection for regular models. In this work, we propose a geometrically flavored information-theoretic approach to study this phenomenon. Namely, we introduce the locally varying dimensionality of the parameter space of neural network models by considering the number of significant dimensions of the Fisher information matrix, and model the parameter space as a manifold using the framework of singular semi-Riemannian geometry. We derive model complexity measures which yield short description lengths for deep neural network models based on their singularity analysis thus explaining the good performance of DNNs despite their large number of parameters.