Convergence of Stochastic Gradient Methods for Wide Two-Layer Physics-Informed Neural Networks

2025-08-29

AI Summary

This work addresses the convergence of wide two-layer physics-informed neural networks (PINNs) under stochastic gradient descent (SGD) and stochastic gradient flow (SGF), a longstanding challenge unaddressed by prior deterministic analyses. Method: Leveraging overparameterization, we establish high-probability linear convergence by jointly ensuring Gram matrix positive definiteness throughout training and rigorously characterizing the stochastic dynamics induced by SGD/SGF—thereby controlling random perturbations across diverse common activation functions. Contribution/Results: Our analysis extends PINN convergence theory to stochastic optimization for the first time, revealing an intrinsic linear convergence mechanism of SGD/SGF in solving partial differential equations. It provides the first rigorous theoretical foundation for training PINNs via stochastic algorithms, bridging a critical gap between practical stochastic training and theoretical guarantees in scientific machine learning.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Probabilistic Circuits and Graphical Models
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks
• Web Mining and Content Analysis: Models for Web evolution

Abstract

Physics informed neural networks (PINNs) represent a very popular class of neural solvers for partial differential equations. In practice, one often employs stochastic gradient descent type algorithms to train the neural network. Therefore, the convergence guarantee of stochastic gradient descent is of fundamental importance. In this work, we establish the linear convergence of stochastic gradient descent / flow in training over-parameterized two layer PINNs for a general class of activation functions in the sense of high probability. These results extend the existing result [18] in which gradient descent was analyzed. The challenge of the analysis lies in handling the dynamic randomness introduced by stochastic optimization methods. The key of the analysis lies in ensuring the positive definiteness of suitable Gram matrices during the training. The analysis sheds insight into the dynamics of the optimization process, and provides guarantees on the neural networks trained by stochastic algorithms.

Problem

• Analyzing convergence of stochastic gradient methods
• Training over-parameterized physics-informed neural networks
• Ensuring positive definiteness of Gram matrices

Innovation

• Stochastic gradient descent for PINNs
• Linear convergence in over-parameterized networks
• Positive definiteness of Gram matrices