Rethinking State Tracking in Recurrent Models Through Error Control Dynamics

2026-05-08

AI Summary

Existing recurrent models often suffer from hidden state drift in state tracking due to inadequate error control, undermining robust long-term performance. This work identifies error control as a central factor for tracking robustness and, for the first time, analyzes from a dynamical systems perspective why affine recurrent networks fail to correct errors within state-separating subspaces. Through theoretical analysis and empirical experiments, we integrate state-space models with linear attention architectures and introduce two key metrics—the discriminability ratio and the decoder readability threshold—to demonstrate the predictability of tracking collapse. We show that sharp drops in accuracy consistently coincide with error accumulation surpassing the readability threshold, revealing a clear and quantifiable precursor to failure.

Abstract

The theory of state tracking in recurrent architectures has predominantly focused on expressive capacity: whether a fixed architecture can theoretically realize a set of symbolic transition rules. We argue that equally important is error control, the dynamics governing hidden-state drift along the directions that distinguish symbolic states. We prove that affine recurrent networks, a class of models encompassing State-Space Models and Linear Attention, cannot correct errors along state-separating subspaces once they preserve state representations. Consequently, practical affine trackers do not learn robust state tracking; rather, they learn finite horizon solutions governed by accumulated state-relevant error. We characterize the mechanics of this failure, showing that tracking remains readable only while the accumulating within-class spread remains small relative to the initial between-class separation. We demonstrate empirically on group state-tracking tasks that this breakdown is predictable: tracking collapses when the distinguishability ratio crosses the readability threshold of the trained decoder. Across trained models, the point of this crossing predicts the horizon at which downstream accuracy fails. These results establish that robust state tracking is determined not only by an architecture's theoretical expressivity but crucially by its error control.

Problem

• state tracking
• error control
• recurrent models
• hidden-state drift
• robustness

Innovation

• error control
• state tracking
• affine recurrent networks
• state-space models
• linear attention