Evolutionary Two-Stage Hyperparameter Optimization Strategies for Physics-Informed Neural Networks

2026-06-18

AI Summary

This work addresses the challenges of unstable convergence, training stagnation, and hyperparameter sensitivity commonly encountered in Physics-Informed Neural Networks (PINNs), which stem from their highly non-convex and multi-objective loss landscapes. To overcome these issues, the authors propose a two-stage evolutionary hyperparameter optimization framework: the first stage rapidly screens candidate configurations using low-fidelity training, while the second stage refines the most promising candidates through high-fidelity evaluation. By integrating population-based evolutionary algorithms with gradient-based optimizers, the method effectively navigates mixed and non-differentiable hyperparameter spaces. Under a fixed computational budget, this approach significantly enhances solution accuracy and robustness. Experimental results on the Advection, Klein-Gordon, and Helmholtz equations demonstrate that the proposed method achieves lower average errors compared to standard training strategies under resource constraints.

Abstract

Physics-Informed Neural Networks (PINNs) solve Partial Differential Equations (PDEs) by embedding physical laws into neural network training. However, their performance suffers from unstable convergence, training plateaus, and strong sensitivity to architectural and optimization hyperparameters due to the highly non-convex and multi-term structure of the physics-informed loss. In this setting, the outer-loop hyperparameter search is a noisy and black-box optimization problem over heterogeneous parameters, where classical local or gradient-based strategies are easily trapped in suboptimal regions. Evolutionary algorithms, with their population-based exploration and ability to handle mixed, non-differentiable search spaces, provide a more robust mechanism for discovering promising configurations. We propose and investigate a two-stage approach based on evolutionary algorithms that combines exploration and exploitation parts of PINNs training to improve solution accuracy and robustness under fixed computational budgets. In the first stage, we perform low-fidelity training runs with truncated epochs to rapidly screen candidate configurations, treating hyperparameter selection as a black-box outer-loop problem. In the second stage, only the most promising candidates are fully trained with standard gradient-based optimizers to refine the solution. Evaluated on three popular problems, namely Advection, Klein-Gordon and Helmholtz equations, our method consistently outperforms standard training and achieves significantly lower mean error within constrained computational resources.

Problem

• Physics-Informed Neural Networks
• Hyperparameter Optimization
• Partial Differential Equations
• Non-convex Optimization
• Training Stability

Innovation

• Physics-Informed Neural Networks
• Evolutionary Algorithms
• Hyperparameter Optimization
• Two-Stage Optimization
• Black-Box Optimization