Anti-Collapse Dynamics and the Emergence of Multi-Time-Scale Learning in Recurrent Neural Networks

2026-06-28

AI Summary

Recurrent neural networks trained with stochastic gradient descent struggle to learn long-term dependencies due to exponentially decaying memory over time lags, which necessitates an exponential increase in data volume. This work reveals, through analysis of the coupling between state and parameter dynamics, that networks can spontaneously enter an “anti-collapse” regime wherein memory decay transitions from exponential to power-law, thereby enabling efficient long-range learning. We identify heavy-tailed fluctuations as a key mechanism sustaining multi-timescale learning and introduce a spectral exponent β—derived from a coarse-grained stochastic process model—to characterize the distribution of temporal scales. Theoretically, we show that a single β governs both the rate of forgetting and the breadth of temporal scales, and that effective long-range learning requires synergistic support from both architecture and optimizer to sustain broad-spectrum dynamics.

Abstract

Long-range learning is hard for recurrent networks trained with stochastic gradient descent, because the influence of a past input fades with the lag $\ell$, and if it fades too fast the dependence cannot be learned from finite data. This fade is captured by an envelope $f(\ell)$. An exponential fade makes the data needed to learn a lag-$\ell$ dependence grow exponentially, putting long horizons out of reach; a power-law fade keeps the cost polynomial. We show that the asymptotic decay class of $f(\ell)$ is not fixed by the architecture. Instead, it emerges from the coupling between the state dynamics and parameter dynamics, settling into either a collapsed regime (fast, exponential forgetting) or an extended, anti-collapsed regime (slow, power-law forgetting). The intuition is a competition within these coupled dynamics. Training drives the network's effective time scales toward short ones, while rare, heavy-tailed fluctuations of the learning dynamics push a few of them to very long values. The extended regime survives only when these heavy-tailed pushes are strong enough to balance the pull. We make this mathematically precise with a coarse-grained stochastic process and prove exactly when the extended regime exists. A single exponent, the spectral exponent~$β$, then governs both the spread of time scales and how slowly the network forgets. Realizing the regime in practice needs one more ingredient: the joint action of the architecture and the optimizer must be able to hold such a broad spread. A network whose capacity to generate broad time-scale spectra is severely constrained still collapses, even when supplied with strong heavy-tailed forcing. Heavy-tailed fluctuations thus act not as noise to be suppressed, but as the mechanism that sustains long-range learning.

Problem

• long-range learning
• recurrent neural networks
• forgetting dynamics
• time-scale spectrum
• heavy-tailed fluctuations

Innovation

• anti-collapse dynamics
• multi-time-scale learning
• heavy-tailed fluctuations
• power-law forgetting
• spectral exponent