An optimal control approach for neural network architecture adaptation with a posteriori error estimation

2026-07-08

AI Summary

This work addresses the challenge of improving generalization by dynamically optimizing neural network depth in response to task complexity. The training process is formulated as a continuous-time optimal control problem, and a dual-weighted residual method is introduced to construct a computable posterior error upper bound. This enables layer-wise error decomposition and precise, adaptive architectural expansion. By integrating piecewise-linear parameterized network structures with optimal control theory, the proposed approach significantly outperforms existing architecture-adaptive methods on scientific computing benchmarks such as the Navier–Stokes equations, demonstrating superior generalization performance and computational efficiency.

Abstract

This work presents a novel approach for adapting neural network architecture along the depth based on a posteriori error estimation. By formulating neural network training as a continuous-time optimal control problem, we derive rigorous error estimates that quantify how approximation error distributes across network layers. This error decomposition enables a principled depth adaptation strategy: new layers are inserted at locations of maximum estimated error, allowing the network to efficiently capture complex, nonlinear variations in the underlying problem. Our framework introduces a novel network architecture that treats weights and biases as piecewise linear functions varying across layers, with the error estimator bounding the discrepancy between this discrete representation and the true continuous optimal control solution. The approach leverages dual weighted residual methodology from finite element analysis to derive computable upper bounds on the functional error. A key theoretical contribution is the derivation of explicit error bounds that decompose the total approximation error into interval-wise contributions, providing a rigorous basis for targeted architecture refinement. We demonstrate the effectiveness of our method on scientific datasets, including learning the observable-to-parameter map for the Navier-Stokes equation. Numerical results reveal that our approach consistently outperforms existing architecture adaptation methods in terms of generalization performance.

Problem

• neural network architecture adaptation
• a posteriori error estimation
• optimal control
• depth adaptation
• approximation error

Innovation

• optimal control
• a posteriori error estimation
• neural architecture adaptation
• dual weighted residual
• continuous-time formulation