Video Prediction of Dynamic Physical Simulations With Pixel-Space Spatiotemporal Transformers

2025-07-22

IEEE Transactions on Neural Networks and Learning Systems

AI Summary

This work addresses the insufficient causal modeling in existing video generation models for long-term prediction of dynamic physical systems. We propose a pure Transformer architecture that performs end-to-end spatiotemporal reasoning directly in continuous pixel space—eliminating reliance on implicit feature learning or elaborate training strategies. Our method integrates spatiotemporal self-attention, autoregressive modeling, and a probe mechanism, enabling unsupervised training on physics simulation datasets while supporting physical object tracking and PDE parameter estimation. Experiments demonstrate that, while maintaining competitive mainstream video quality metrics (e.g., PSNR, SSIM), our approach improves physically accurate prediction horizon by 50%. Moreover, the model exhibits strong interpretability and out-of-distribution generalization, successfully transferring to unseen PDE parameter estimation tasks.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Inspired by the performance and scalability of autoregressive large language models (LLMs), transformer-based models have seen recent success in the visual domain. This study investigates a transformer adaptation for video prediction with a simple end-to-end approach, comparing various spatiotemporal self-attention layouts. Focusing on causal modeling of physical simulations over time; a common shortcoming of existing video-generative approaches, we attempt to isolate spatiotemporal reasoning via physical object tracking metrics and unsupervised training on physical simulation datasets. We introduce a simple yet effective pure transformer model for autoregressive video prediction, utilizing continuous pixel-space representations for video prediction. Without the need for complex training strategies or latent feature-learning components, our approach significantly extends the time horizon for physically accurate predictions by up to 50% when compared with existing latent-space approaches, while maintaining comparable performance on common video quality metrics. In addition, we conduct interpretability experiments to identify network regions that encode information useful to perform accurate estimations of PDE simulation parameters via probing models, and find that this generalizes to the estimation of out-of-distribution simulation parameters. This work serves as a platform for further attention-based spatiotemporal modeling of videos via a simple, parameter efficient, and interpretable approach.

Problem

• Autoregressive video prediction of physical simulations using transformers
• Improving long-term physical accuracy in video generation models
• Enabling interpretable spatiotemporal reasoning without complex latent features

Innovation

• Pixel-space transformers for video prediction
• Unsupervised training on physical simulations
• Interpretable spatiotemporal modeling without latent features