Statistical Properties of Training & Generalization

2026-06-18

AI Summary

Deep learning models exhibit generalization capabilities in real-world tasks that surpass predictions from classical statistical learning theory, yet the underlying mechanisms remain poorly understood. This work addresses this gap by integrating statistical learning theory with physics-inspired priors to systematically analyze neural scaling laws under physically constrained scenarios. It uncovers novel scaling behaviors and explicitly models the interplay between inductive biases and architectural design choices. Leveraging a physics-informed machine learning framework, the study elucidates the statistical mechanisms responsible for the exceptional generalization of deep learning in physics-related tasks, thereby providing theoretical foundations and practical guidance for designing models tailored to scientific computing applications.

Abstract

Deep learning has managed to evade numerous intuitions from classical statistics to achieve unprecedented performance on a number of real-world tasks. In this article, we investigate the key features and surprises of deep learning from a physics-informed perspective, taking care to point out and justify where possible the many choices inherent in constructing a deep learning model. In particular, we review the phenomenon of neural scaling laws and discuss their interplay with the constraints and inductive biases which may be present when applying machine learning to problems in physics.

Problem

• deep learning
• statistical properties
• neural scaling laws
• generalization
• inductive biases

Innovation

• neural scaling laws
• physics-informed machine learning
• inductive bias
• generalization
• statistical properties