🤖 AI Summary
This work addresses the high computational cost and memory consumption of automatic differentiation (AD) in physics-informed neural networks (PINNs), as well as its susceptibility to silent errors in architectures involving inter-sample dependencies such as BatchNorm or self-attention. The study presents the first systematic evaluation of finite differences (FD) as an alternative for derivative computation in PINNs, introducing a calibrated step-size strategy and a stochastic FD variant. It further proposes a sample-wise gradient approximation method that requires only forward passes. Experiments on three benchmark partial differential equations demonstrate that, in full-batch settings, FD achieves comparable accuracy to AD while being faster and using less memory. The proposed stochastic FD excels particularly in steady-state problems, and crucially, FD yields derivative errors an order of magnitude lower than AD in models with inter-sample dependencies.
📝 Abstract
We systematically investigate finite-difference (FD) derivative computation in Physics-Informed Neural Networks (PINNs) as an alternative to automatic differentiation (AD). On three benchmark PDEs we show that, with a properly calibrated step size, FD matches AD in accuracy on every problem while running faster across the full tested batch-size range and using substantially less GPU memory, and that a stochastic variant we propose outperforms AD on a stationary problem. We further show that for neural architectures with inter-sample dependencies (e.g. BatchNorm, self-attention) the standard PyTorch autograd idiom is silently incorrect; the correct per-sample alternative is computationally infeasible at PINN-relevant batch sizes, while FD provides a forward-only approximation that is empirically an order of magnitude closer to the true per-sample derivative.