Time-Delayed Transformers for Data-Driven Modeling of Low-Dimensional Dynamics

2026-02-09

AI Summary

This work addresses the challenge of effectively modeling nonlinear and even chaotic low-dimensional spatiotemporal dynamical systems while preserving the efficiency and interpretability of linear models. The authors propose the Time-Delay Transformer (TD-TF), an extremely minimalist single-layer, single-head Transformer architecture that extends Time-Delay Dynamic Mode Decomposition (TD-DMD) to the nonlinear regime for the first time. TD-TF comprises only one self-attention layer—with a single query per time step—and one feedforward layer, achieving linear sequence complexity and a low parameter count. Experiments on synthetic signals, unsteady aerodynamic flows, the Lorenz '63 system, and reaction–diffusion dynamics demonstrate that TD-TF matches classical methods in linear regimes and substantially outperforms linear baselines in nonlinear and chaotic settings, accurately capturing long-term dynamical behavior.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Data Mining & Knowledge Management: Mining of Spatial, Temporal or Spatio-Temporal Data
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

We propose the time-delayed transformer (TD-TF), a simplified transformer architecture for data-driven modeling of unsteady spatio-temporal dynamics. TD-TF bridges linear operator-based methods and deep sequence models by showing that a single-layer, single-head transformer can be interpreted as a nonlinear generalization of time-delayed dynamic mode decomposition (TD-DMD). The architecture is deliberately minimal, consisting of one self-attention layer with a single query per prediction and one feedforward layer, resulting in linear computational complexity in sequence length and a small parameter count. Numerical experiments demonstrate that TD-TF matches the performance of strong linear baselines on near-linear systems, while significantly outperforming them in nonlinear and chaotic regimes, where it accurately captures long-term dynamics. Validation studies on synthetic signals, unsteady aerodynamics, the Lorenz'63 system, and a reaction-diffusion model show that TD-TF preserves the interpretability and efficiency of linear models while providing substantially enhanced expressive power for complex dynamics.

Problem

• data-driven modeling
• low-dimensional dynamics
• nonlinear dynamics
• chaotic systems
• spatio-temporal dynamics

Innovation

• Time-Delayed Transformer
• Dynamic Mode Decomposition
• Data-Driven Modeling
• Nonlinear Dynamics
• Spatio-Temporal Modeling