Representation Gap: Explaining the Unreasonable Effectiveness of Neural Networks from a Geometric Perspective

2026-05-20

AI Summary

This work aims to precisely characterize the asymptotic generalization error of neural networks and proposes a practical, efficiently estimable proxy metric to guide model design. To this end, the authors introduce the “representation gap” from a geometric perspective as a surrogate for generalization error and derive its asymptotic behavior by integrating equivariant diffusion models, optimal quantization theory, and point process theory. The theoretical analysis reveals that the representation gap is governed by the intrinsic dimensionality of the task, establishing a rigorous connection between the two. Experiments on both synthetic and real-world datasets validate the accuracy of the derived asymptotic law, and the estimated intrinsic dimensions align well with existing literature, demonstrating the method’s strong interpretability, estimability, and broad applicability.

Abstract

Characterizing precisely the asymptotic generalization error of neural networks using parameters that can be estimated efficiently is a crucial problem in machine learning, which relies heavily on heuristics and practitioners' intuition to make key design choices. In order to mitigate this issue, we introduce the Representation Gap, a metric closely related to the generalization error, but admitting better-behaved asymptotic dynamics. Focusing on equivariant diffusion models and leveraging results from optimal quantization and point-process theory, we derive a precise asymptotic equivalent of the Representation Gap and show that it is governed by a single parameter, the \textit{intrinsic dimension} of the task, which is easy to interpret, efficient to estimate, and can be linked to the equivariances of common neural network architectures. We show that this asymptotic dynamic also extends to a broader range of tasks and training algorithms. Finally, we demonstrate empirically that our asymptotic law and intrinsic dimension estimation are accurate on a wide range of synthetic datasets, where these quantities are known, as well as on more realistic datasets, where we obtain results consistent with the related literature.

Problem

• generalization error
• asymptotic analysis
• intrinsic dimension
• representation gap
• neural networks

Innovation

• Representation Gap
• intrinsic dimension
• equivariant diffusion models
• asymptotic generalization error
• optimal quantization