Neural Tangent Kernel of Neural Networks with Loss Informed by Differential Operators

2025-03-14

AI Summary

This work investigates the Neural Tangent Kernel (NTK) theory for deep neural networks trained under physics-informed losses involving differential operators, focusing on initialization, training dynamics, convergence, and explicit NTK structure. Method: We establish the first analytical framework for NTK under differential-operator-driven losses, integrating NTK theory, spectral analysis, and Physics-Informed Neural Network (PINN) modeling to rigorously derive the closed-form expression and spectral properties of the NTK. Contribution/Results: We theoretically prove that physics-informed losses do not universally accelerate eigenvalue decay or exacerbate spectral bias; instead, convergence behavior is jointly governed by how differential operators are embedded and the loss structure. Experiments validate the predicted spectral decay rates and bias patterns. This work provides the first systematic NTK-level theoretical explanation for generalization and optimization dynamics in PINNs, extending NTK analysis beyond conventional supervised learning settings.

Technology Category

• Machine Learning: Deep Learning Theory
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Spectral bias is a significant phenomenon in neural network training and can be explained by neural tangent kernel (NTK) theory. In this work, we develop the NTK theory for deep neural networks with physics-informed loss, providing insights into the convergence of NTK during initialization and training, and revealing its explicit structure. We find that, in most cases, the differential operators in the loss function do not induce a faster eigenvalue decay rate and stronger spectral bias. Some experimental results are also presented to verify the theory.

Problem

• Develop NTK theory for physics-informed neural networks.
• Analyze convergence and structure of NTK during training.
• Investigate spectral bias impact from differential operators.

Innovation

• Develop NTK theory for physics-informed loss networks
• Analyze NTK convergence during initialization and training
• Investigate differential operators' impact on spectral bias