DSSMs: State Space Models with Explicit Memory via Delay Differential Equations

2026-07-11

AI Summary

This work addresses the limitation of state space models (SSMs) in capturing long-range dependencies due to their fixed-dimensional hidden states. The authors propose the Delayed State Space Model (DSSM), which introduces delay differential equations into the SSM framework for the first time, enhancing memory of historical information through an explicit delay feedback mechanism. Key contributions include a delay-agnostic stability-preserving parameterization, an efficient strategy for managing historical states, a frequency-domain transfer function–based kernel contour shifting technique to mitigate aliasing, and FFT-accelerated training. DSSM significantly outperforms baseline models on the targeted delay retrieval task and surpasses S4D on most standard sequence modeling benchmarks while maintaining near-optimal performance on other metrics.

Abstract

State Space Models (SSMs) have emerged as a powerful paradigm for efficient long-sequence modeling, offering parallel training and fast linear-time recurrent inference. However, like other recurrent architectures, SSMs must compress an unbounded history into a fixed-size state, which limits context retention and makes precise retrieval over long-range context inherently difficult. To overcome this limitation, we propose Delay State Space Models (DSSMs), a delay differential equation (DDE)-inspired extension of diagonal SSMs that augments discrete SSM recurrences with explicit delayed-state feedback. Making explicit delayed feedback practical requires new stability parameterization, history management, and FFT-training tools. We address these challenges with a practical discretization and parameterization grounded in a simple delay-independent stability condition. To bypass direct time-domain kernel construction, we derive the DSSM transfer function and compute kernels in the frequency domain, using a kernel contour shift to suppress aliasing and recover accurate FFT training. Empirically, DSSMs substantially improve targeted delayed-retrieval tasks while outperforming S4D on most standard sequence metrics and remaining close on the others.

Problem

• State Space Models
• long-range context
• context retention
• delayed retrieval
• recurrent architectures

Innovation

• Delay State Space Models
• Delay Differential Equations
• Explicit Memory
• Frequency-domain Training
• Long-range Retrieval