ConFIG: Towards Conflict-free Training of Physics Informed Neural Networks

2024-08-20

arXiv.org

AI Summary

In Physics-Informed Neural Networks (PINNs), gradients of initial/boundary condition losses and physics-informed PDE residuals often point in conflicting directions, impeding optimization and slowing convergence. To address this, we propose Conflict-Aware Gradient Fusion (CAGF): a novel gradient alignment framework that enforces positive dot products between all loss gradients and the unified parameter update direction via a multi-objective gradient alignment constraint; integrates dynamic norm-based scaling and alternating backpropagation; and incorporates momentum acceleration for enhanced stability. We provide theoretical convergence guarantees for the method. Extensive experiments on challenging PINN benchmarks—including problems with high-gradient regimes and strongly nonlinear PDEs—demonstrate that CAGF consistently outperforms state-of-the-art baselines: training speed improves by 32%–68%, while prediction error reduces by 41%–79%. Moreover, CAGF exhibits strong generalization on standard multi-task learning benchmarks.

Technology Category

• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

The loss functions of many learning problems contain multiple additive terms that can disagree and yield conflicting update directions. For Physics-Informed Neural Networks (PINNs), loss terms on initial/boundary conditions and physics equations are particularly interesting as they are well-established as highly difficult tasks. To improve learning the challenging multi-objective task posed by PINNs, we propose the ConFIG method, which provides conflict-free updates by ensuring a positive dot product between the final update and each loss-specific gradient. It also maintains consistent optimization rates for all loss terms and dynamically adjusts gradient magnitudes based on conflict levels. We additionally leverage momentum to accelerate optimizations by alternating the back-propagation of different loss terms. We provide a mathematical proof showing the convergence of the ConFIG method, and it is evaluated across a range of challenging PINN scenarios. ConFIG consistently shows superior performance and runtime compared to baseline methods. We also test the proposed method in a classic multi-task benchmark, where the ConFIG method likewise exhibits a highly promising performance. Source code is available at https://tum-pbs.github.io/ConFIG

Problem

• Resolves conflicting updates in PINN training.
• Ensures consistent optimization rates for all losses.
• Dynamically adjusts gradient magnitudes based on conflict.

Innovation

• Conflict-free updates via positive dot product
• Consistent optimization rates for loss terms
• Dynamic gradient magnitude adjustment