PINN-Cast: Exploring the Role of Continuous-Depth NODE in Transformers and Physics Informed Loss as Soft Physical Constraints in Short-term Weather Forecasting

2026-04-29

AI Summary

This work addresses the high computational cost of traditional numerical weather prediction and the limited physical consistency of existing data-driven Transformer approaches, which rely on discrete layer updates that fail to capture smooth latent-space dynamics. To overcome these limitations, the authors propose a continuous-depth Transformer architecture that integrates Neural Ordinary Differential Equations (Neural ODEs) into the encoder, replacing discrete residual connections with adaptive numerical integration. The model further incorporates a dual-branch attention mechanism and a physics-informed soft-constraint loss function. This approach uniquely combines continuous-depth modeling, dual-branch attention, and physical regularization, achieving significantly improved forecast accuracy and physical consistency over standard discrete Transformers and existing Neural ODE methods in short-term weather prediction tasks.

Abstract

Operational weather prediction has long relied on physics-based numerical weather prediction (NWP), whose accuracy comes at the cost of substantial compute and complex simulation workflows. Recent transformer-based forecasters offer efficient data-driven alternatives, however transformers are physics-agnostic models. Additionally, standard transformer encoders evolve representations through discrete layer updates that may be less suited to modeling smooth latent dynamics. In this work, we propose a continuous-depth transformer encoder for weather forecasting that integrates Neural Ordinary Differential Equation (Neural ODE) dynamics within each encoder block. Specifically, we replace discrete residual updates with ODE-based updates solved using adaptive numerical integration. We also introduce a two-branch attention module that combines conventional patch-wise self-attention with an auxiliary branch that applies a derivative operator to attention logits, providing an additional change-sensitive interaction signal. To further align forecasts with governing principles, we propose a customized physics-informed training objective that enforces physical consistency as a soft constraint. We evaluate the proposed method against a standard discrete transformer baseline and an existing continuous-time Neural ODE forecasting variant, demonstrating the importance of PINN-Cast in short term weather forecasting.

Problem

• weather forecasting
• physics-informed
• continuous-depth
• Transformer
• Neural ODE

Innovation

• Neural ODE
• continuous-depth Transformer
• physics-informed loss
• soft physical constraints
• two-branch attention