Parallel-in-Time Training of Recurrent Neural Networks for Dynamical Systems Reconstruction

2026-05-12

AI Summary

This work addresses the challenge that conventional sequential models, constrained by linear time complexity, struggle to efficiently reconstruct nonlinear dynamical systems from extremely long sequences. To overcome this limitation, the authors propose the GTF-DEER framework, which integrates Generalized Teacher Forcing (GTF) with a parallel associative scan mechanism, leveraging state-space models and parallelized backpropagation to bypass the sequence-length constraints inherent in recurrent training. This enables stable and efficient learning of nonlinear dynamics over arbitrarily long sequences. Empirical evaluations on sequences exceeding \( T = 10^4 \) demonstrate substantial improvements in reconstruction accuracy, underscoring the critical importance of long-sequence modeling for capturing complex dynamical behaviors.

Abstract

Reconstructing nonlinear dynamical systems (DS) from data (DSR) is a fundamental challenge in science and engineering, but it inherently relies on sequential models. Recent breakthroughs for sequential models have produced algorithms that parallelize computation along sequence length $T$, achieving logarithmic time complexity, $\mathcal{O}(\log T)$. Since sequence lengths have been practically limited due to the linear runtime complexity $\mathcal{O}(T)$ of classical backpropagation through time, this opens new avenues for DSR. This paper studies two prominent classes of parallel-in-time algorithms for this task, both of which leverage parallel associative scans as their core computational primitive. The first class comprises models with linear yet non-autonomous dynamics and a nonlinear readout, such as modern State Space Models (SSMs), while the second consists of general nonlinear models which can be parallelized using the DEER framework. We find that the linear training-time recurrence of the first class of models imposes limitations that often hinder learning of accurate nonlinear dynamics. To address this, we augment DEER with Generalized Teacher Forcing (GTF), a novel variant within the more general nonlinear framework that ensures stable and effective learning of nonlinear dynamics across arbitrary sequence lengths. Using GTF-DEER, we investigate the benefits of training on extremely long sequences ($T>10^4$) for DSR. Our results show that access to such long trajectories significantly improves DSR if the data features long time scales. This work establishes GTF-DEER as a robust tool for data-driven discovery and underscores the largely untapped potential of long-sequence learning in modeling complex DS.

Problem

• dynamical systems reconstruction
• nonlinear dynamics
• long-sequence learning
• parallel-in-time training
• recurrent neural networks

Innovation

• parallel-in-time training
• nonlinear dynamical systems
• Generalized Teacher Forcing
• DEER framework
• long-sequence learning