A Qualitative Test-Risk Mechanism for Scaling Behavior in Normalized Residual Networks

2026-05-08

AI Summary

This work investigates the theoretical impact of depth extension on test risk in normalized residual networks. By inserting new residual blocks into a trained network, it uniquely decomposes the improvement in test risk due to depth extension into three components: representation gain, optimization gain, and generalization transfer, and establishes a unified theoretical framework. Leveraging assumptions of first-order gradient descent, neighborhood analysis under zero initialization, norm control enabled by post-normalization architecture, and Rademacher complexity bounds, the paper proves that the expanded hypothesis class contains an auxiliary model with lower population risk. Consequently, two complementary upper bounds on test risk are derived—one applicable when a positive population margin exists and another for degenerate cases. The findings reveal that the efficacy of depth extension arises from the interplay among depth, width, and data.

Abstract

The scaling behavior, in which test performance often improves as model size and data increase, is a central empirical phenomenon in modern deep learning, yet its theoretical basis remains incomplete. In this paper, we study depth expansion in normalized residual networks: starting from a trained model in an old hypothesis class, we insert a new residual block at an intermediate layer and ask when such an expansion can yield a provable improvement in test risk. We develop a unified framework that decomposes this question into representational gain, optimization gain, and generalization transfer. First, under a first-order descent condition near zero initialization, we prove that the expanded hypothesis class contains an auxiliary jumpboard model with strictly smaller population risk than the original model. Second, under norm control tailored to post-normalized residual architectures, we establish a norm-based Rademacher complexity bound for the expanded model class. These ingredients lead to two complementary test-risk guarantees: one route passes through population risk and is tighter when a positive population margin is available, while the other works directly at the train/test level, avoids Hoeffding transfer, and is more robust in degenerate regimes. Together, these results provide a theorem-driven mechanism under which residual depth expansion can improve test performance in normalized residual networks. More broadly, they suggest that scaling is inherently joint: depth creates new improving directions, width enhances the finite-sample observability of weak signals, and data determines whether the statistical cost of expansion can be controlled.

Problem

• scaling behavior
• normalized residual networks
• test risk
• depth expansion
• generalization

Innovation

• normalized residual networks
• scaling behavior
• test-risk guarantee
• Rademacher complexity
• depth expansion