Layerwise goal-oriented adaptivity for neural ODEs: an optimal control perspective

2026-01-12

AI Summary

This work proposes a goal-oriented adaptive framework for constructing neural networks by formulating Neural Ordinary Differential Equations (Neural ODEs) as optimal control problems constrained by ordinary differential equations. Leveraging a dual-weighted residual estimator as an error indicator, the method dynamically adjusts network depth during training by combining explicit Euler time stepping with steepest descent optimization of control parameters. This approach uniquely integrates goal-oriented adaptivity into the Neural ODE paradigm, enabling the automatic construction of compact yet highly accurate models. Experimental results across multiple benchmark classification tasks demonstrate that the proposed method achieves superior performance and computational efficiency compared to conventional fixed-depth architectures.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

In this work, we propose a novel layerwise adaptive construction method for neural network architectures. Our approach is based on a goal--oriented dual-weighted residual technique for the optimal control of neural differential equations. This leads to an ordinary differential equation constrained optimization problem with controls acting as coefficients and a specific loss function. We implement our approach on the basis of a DG(0) Galerkin discretization of the neural ODE, leading to an explicit Euler time marching scheme. For the optimization we use steepest descent. Finally, we apply our method to the construction of neural networks for the classification of data sets, where we present results for a selection of well known examples from the literature.

Problem

• neural ODEs
• goal-oriented adaptivity
• optimal control
• neural architecture construction
• layerwise adaptation

Innovation

• neural ODEs
• goal-oriented adaptivity
• optimal control
• dual-weighted residual
• DG(0) Galerkin