An Optimisation Framework for the Well-Conditioned Training of Physics-Informed Neural Networks

2026-07-02

AI Summary

This work addresses the optimization challenges and limited accuracy of physics-informed neural networks (PINNs) caused by the severe ill-conditioning of their loss functions. The authors propose the DSGNAR framework, which introduces, for the first time, a double-sketch Gauss–Newton method into PINN training, augmented with adaptive regularization and step-size control to effectively mitigate ill-conditioning. The approach supports both single- and double-precision floating-point arithmetic and achieves relative ℓ² errors as low as 3×10⁻¹⁶ on benchmark problems such as the Burgers equation and high-dimensional Poisson equations—improving upon existing methods by five to eight orders of magnitude. Notably, in single precision, it approaches the round-off error limit within ten seconds, thereby substantially overcoming longstanding accuracy and efficiency bottlenecks in PINN-based solvers.

Abstract

Physics-informed neural networks (PINNs) have emerged as a promising route to solve partial differential equations, yet they have struggled to reach the precision of classical solvers. The obstacle is increasingly understood to be one of optimisation, owing to the severely ill-conditioned loss landscape. We present $\textbf{DSGNAR}$: Doubly-Sketched Gauss-Newton with Adaptive Ratio, a scalable second-order optimisation framework that confronts this ill-conditioning and, in doing so, obtains unprecedented accuracy and speed. $\textbf{DSGNAR}$ couples a doubly-sketched Gauss-Newton model with a novel strategy that carefully controls both regularisation and step length. Across a suite of problems spanning nonlinear, chaotic, multi-scale, high-dimensional, and Navier-Stokes, the framework greatly improves on the state of the art: able to attain relative $\ell_2$ errors as low as $3\times10^{-16}$ in double precision, improve contemporary results by five orders of magnitude on the canonical Burgers' equation, and as much as eight orders on a high-dimensional Poisson problem, while remaining markedly faster. We further show that, in single precision, solutions at the limit of round-off error can be obtained very quickly: Burgers' equation to $\ell_2^{\text{rel}} = 4.75 \times 10^{-7}$ in under ten seconds. The framework is also robust to the choice of architecture, arithmetic precision, and initial hyperparameters. The code is available at https://www.github.com/wephy/physics-informed-neural-networks

Problem

• Physics-Informed Neural Networks
• Ill-conditioning
• Optimization
• Partial Differential Equations
• Loss Landscape

Innovation

• Physics-Informed Neural Networks
• Second-order Optimization
• Ill-conditioning
• Gauss-Newton
• Adaptive Regularization