Physics-Informed Neural Network with Squeeze-Excitation-like Attention

2026-06-18

AI Summary

This work addresses the challenges of poor convergence and limited representational capacity in physics-informed neural networks (PINNs) when tackling high-dimensional and high-frequency problems. To overcome these limitations, the authors propose a lightweight Squeeze-Excitation Attention (SEA) module integrated into the PINN architecture to dynamically modulate neuron weights. This approach enables near-deterministic initialization, substantially reducing initial residuals and enhancing nonlinear expressivity—effectively handling high-frequency solutions without relying on Fourier features or periodic activation functions. The resulting SEA-PINN seamlessly integrates into existing frameworks such as TSA-PINN, achieving exceptionally low variance and initial loss in 17 out of 20 benchmark tests. Compared to standard FNN-PINNs, SEA-PINN improves performance by 83% on high-frequency cases, with an additional 42.49% gain when combined with TSA-PINN.

Abstract

We introduce SEA-PINN, a novel architecture that incorporates a Squeeze-Excitation-like attention mechanism into physics-informed neural networks to dynamically recalibrate the importance of neurons across layers. A key feature of SEA-PINN is its highly stable initialization. On 17 out of 20 benchmark problems, SEA-PINN exhibit nearly negligible variance and significantly reduced initial loss, establishing a quasi-deterministic and favorable starting point for optimization. Notably, without employing Fourier feature embeddings or periodic activation functions, SEA-PINN attained competitive accuracy (83\% vs. 90\% improvement relative to FNN-PINN on the high-frequency case 7) as compared with TSA-PINN-a model specifically engineered for high-frequency problems via learnable frequencies in sinusoidal activations. Furthermore, integrating SEA-PINN into TSA-PINN boosted performance by 42.49\%. These results underscore SEA-PINN as a lightweight plug-in module that enhances nonlinear representation power, promotes more robust and efficient convergence, and strengthens the overall reliability of physics-informed learning.

Problem

• Physics-Informed Neural Networks
• High-frequency problems
• Training instability
• Convergence efficiency
• Initialization variance

Innovation

• Physics-Informed Neural Networks
• Squeeze-Excitation Attention
• Stable Initialization
• High-Frequency Learning
• Lightweight Plug-in Module