Interpretable Spatial-Temporal Fusion Transformers: Multi-Output Prediction for Parametric Dynamical Systems with Time-Varying Inputs

2025-05-01

AI Summary

This work addresses the challenge of modeling multi-output parametric dynamical systems subject to time-varying external inputs and physical parameters. We propose a novel, interpretable multi-output Transformer architecture specifically designed for such systems. Our method jointly captures temporal dynamics and multi-channel spatial correlations via a generalized attention weight matrix—departing from conventional single-output paradigms. It incorporates spatiotemporal fusion encoding, an extended multi-head attention mechanism, and interpretable attention weight analysis, trained end-to-end for multi-output regression. Evaluated on strongly nonlinear, high-dimensional parametric systems, the model achieves significant improvements in predictive accuracy across all output channels. Crucially, the learned attention weights reveal physically meaningful spatiotemporal coupling patterns—e.g., input–state and inter-channel dependencies—thereby providing mechanistic insights alongside state-of-the-art performance. This integration of high fidelity and structural interpretability advances both predictive modeling and physics-informed learning for complex dynamical systems.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We explore the promising performance of a transformer model in predicting outputs of parametric dynamical systems with external time-varying input signals. The outputs of such systems vary not only with physical parameters but also with external time-varying input signals. Accurately catching the dynamics of such systems is challenging. We have adapted and extended an existing transformer model for single output prediction to a multiple-output transformer that is able to predict multiple output responses of these systems. The multiple-output transformer generalizes the interpretability of the original transformer. The generalized interpretable attention weight matrix explores not only the temporal correlations in the sequence, but also the interactions between the multiple outputs, providing explanation for the spatial correlation in the output domain. This multiple-output transformer accurately predicts the sequence of multiple outputs, regardless of the nonlinearity of the system and the dimensionality of the parameter space.

Problem

• Predicting outputs of parametric dynamical systems with time-varying inputs
• Extending transformer models for multi-output prediction in nonlinear systems
• Enhancing interpretability of spatial-temporal correlations in output domains

Innovation

• Transformer model for multi-output prediction
• Interpretable attention weight matrix analysis
• Handles nonlinearity and high-dimensional parameters