Sharp Stability Threshold and Certification for Designing Stable Residual Architectures

2026-07-16

AI Summary

This work addresses the lack of theoretical stability guarantees in deep residual networks, whose current designs rely heavily on empirical tuning. We propose a sublinear growth principle that ensures stable training without normalization layers by constraining the input-amplitude exponent \( q \leq 1 \) of the residual block’s velocity field, establishing the first certifiable architectural framework with provable stability. This condition is shown to be both necessary and sufficient, directly linking stability to the input-amplitude exponent of architectural primitives for the first time. Leveraging ordinary differential equations and Hamilton–Jacobi–Bellman optimal control theory, we construct a function space for velocity fields and develop an exponent calculus encompassing five fundamental operations. Experiments demonstrate that reducing the supercritical Mamba block’s exponent from \( q = 5 \) to \( q = 1 \) yields consistently stable and efficient training on both Mamba and PatchTST architectures.

Abstract

We propose \emph{the sublinear-growth principle} for deep residual architectures -- a sharp stability threshold on the input-magnitude exponent of every residual block's velocity field: $$\|v(x, t)\| \leq c\,\|x\|^q + b, \qquad q \in [0, 1].$$ The threshold $q = 1$ is established via two independent arguments. Classical ODE theory gives a global forward flow on $[0, T]$ at $q \le 1$ and exhibits divergent velocity fields at any $q > 1$. The optimal-control analysis, via the Hamilton-Jacobi-Bellman equation, sharpens this to a selection statement: the training optimum is bang-bang on the boundary of the admissible class, so the optimum at $q > 1$ blows up while the optimum at $q \le 1$ is safe by construction. The exponent criterion $q \le 1$ is thereby a necessary and sufficient condition for stable training. It clarifies architectural placements that ensure the stability of training and inference, explaining, for instance, the stabilizing role of layer normalization. The sublinear-growth velocity fields form \emph{the right function space} on which forward dynamics, adjoint sensitivity, and architectural composition are all well-controlled. An arithmetic of input-magnitude exponents under the five operations that build residual blocks enables efficient certification of $q_k \le 1$ at the level of architectural primitives, in place of ad hoc trial and error in the search for stable neural architectural designs. A parameter-free modification reduces the supercritical Mamba block from $q = 5$ to $q = 1$ without layer normalization, demonstrating this point. Experiments on Mamba and PatchTST confirm that the $q \le 1$ variants train stably: the criterion is the input-magnitude exponent, not the presence of a normalization layer.

Problem

• stability threshold
• residual architectures
• velocity field
• input-magnitude exponent
• stable training

Innovation

• sublinear-growth principle
• stability threshold
• residual architectures
• input-magnitude exponent
• Hamilton-Jacobi-Bellman equation