A Simultaneous Approach for Training Neural Differential-Algebraic Systems of Equations

2025-04-07

AI Summary

Training neural differential-algebraic equations (Neural DAEs) suffers from high computational cost and difficulty in strictly satisfying algebraic constraints. Method: This paper proposes a fully discrete synchronous optimization framework that unifies DAE numerical integration and neural network parameter learning into a single nonlinear programming (NLP) problem—abandoning the conventional nested optimization paradigm. Key innovations include: (i) implicit DAE solving via full temporal discretization on a time mesh; (ii) hybrid modeling integrating physical laws and data-driven components; (iii) an NLP-structured Hessian approximation strategy; and (iv) cold-start initialization via auxiliary problems. Results: Experiments demonstrate substantial improvements in accuracy, generalization, and computational efficiency under sparse data, unobservable states, and multi-trajectory scenarios. The method exhibits strong robustness and practicality in real-world scientific modeling tasks.

Technology Category

• Search and Optimization: Mixed Discrete/Continuous Search
• Natural Language Processing: Learning & Optimization for NLP
• Machine Learning: Deep Neural Architectures and Foundation Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Scientific machine learning is an emerging field that broadly describes the combination of scientific computing and machine learning to address challenges in science and engineering. Within the context of differential equations, this has produced highly influential methods, such as neural ordinary differential equations (NODEs). Recent works extend this line of research to consider neural differential-algebraic systems of equations (DAEs), where some unknown relationships within the DAE are learned from data. Training neural DAEs, similarly to neural ODEs, is computationally expensive, as it requires the solution of a DAE for every parameter update. Further, the rigorous consideration of algebraic constraints is difficult within common deep learning training algorithms such as stochastic gradient descent. In this work, we apply the simultaneous approach to neural DAE problems, resulting in a fully discretized nonlinear optimization problem, which is solved to local optimality and simultaneously obtains the neural network parameters and the solution to the corresponding DAE. We extend recent work demonstrating the simultaneous approach for neural ODEs, by presenting a general framework to solve neural DAEs, with explicit consideration of hybrid models, where some components of the DAE are known, e.g. physics-informed constraints. Furthermore, we present a general strategy for improving the performance and convergence of the nonlinear programming solver, based on solving an auxiliary problem for initialization and approximating Hessian terms. We achieve promising results in terms of accuracy, model generalizability and computational cost, across different problem settings such as sparse data, unobserved states and multiple trajectories. Lastly, we provide several promising future directions to improve the scalability and robustness of our approach.

Problem

• Training neural DAEs is computationally expensive
• Algebraic constraints complicate deep learning training
• Hybrid models require explicit consideration of known components

Innovation

• Simultaneous approach for neural DAE training
• Fully discretized nonlinear optimization problem
• Hybrid models with physics-informed constraints