An Attention-based Spatio-Temporal Neural Operator for Evolving Physics

2025-06-12

AI Summary

Modeling unknown dynamic physical processes across spatiotemporal scales in scientific machine learning remains challenging—especially in manufacturing settings where known control parameters coexist with unknown environmental disturbances, severely limiting model generalizability and interpretability. Method: We propose a disentangled spatiotemporal attention neural operator architecture. It innovatively couples a BDF-inspired temporal Transformer with an external forcing decoupling mechanism, enabling explicit separation between historical state evolution and exogenous loading. The design integrates separable attention, neural operators, and numerical differentiation priors to jointly capture long-range spatiotemporal dependencies and enhance physical discoverability. Contribution/Results: Our method significantly improves adaptability to unseen environmental conditions. It consistently outperforms state-of-the-art models across multiple scientific ML benchmarks, demonstrating superior accuracy in engineering prediction, implicit physical mechanism identification, and interpretable AI—while maintaining robustness and physical consistency.

Technology Category

• Machine Learning: Neuro-Symbolic Learning
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

In scientific machine learning (SciML), a key challenge is learning unknown, evolving physical processes and making predictions across spatio-temporal scales. For example, in real-world manufacturing problems like additive manufacturing, users adjust known machine settings while unknown environmental parameters simultaneously fluctuate. To make reliable predictions, it is desired for a model to not only capture long-range spatio-temporal interactions from data but also adapt to new and unknown environments; traditional machine learning models excel at the first task but often lack physical interpretability and struggle to generalize under varying environmental conditions. To tackle these challenges, we propose the Attention-based Spatio-Temporal Neural Operator (ASNO), a novel architecture that combines separable attention mechanisms for spatial and temporal interactions and adapts to unseen physical parameters. Inspired by the backward differentiation formula (BDF), ASNO learns a transformer for temporal prediction and extrapolation and an attention-based neural operator for handling varying external loads, enhancing interpretability by isolating historical state contributions and external forces, enabling the discovery of underlying physical laws and generalizability to unseen physical environments. Empirical results on SciML benchmarks demonstrate that ASNO outperforms over existing models, establishing its potential for engineering applications, physics discovery, and interpretable machine learning.

Problem

• Learning unknown evolving physical processes in SciML
• Predicting across spatio-temporal scales with environmental changes
• Enhancing interpretability and generalizability in physics discovery

Innovation

• Attention-based Spatio-Temporal Neural Operator (ASNO)
• Separable attention for spatial-temporal interactions
• Adapts to unseen physical parameters