🤖 AI Summary
This study investigates task-dependent functional redundancy and approximate invariance in recurrent neural networks. By applying the real Schur decomposition, the recurrent weight matrix is decoupled into spectral blocks and non-normal coupling structures. Structured ablation is then performed while preserving the input–output mapping to identify weight perturbations that do not significantly affect task performance. The work introduces the notion of “task-dependent approximate functional invariance,” revealing non-universal yet task-specific symmetries within recurrent architectures. Experiments across dynamic tasks—including copying, flip-flop triggering, sine wave generation, and context-dependent integration—demonstrate that certain non-normal Schur couplings can be safely removed, while others are essential for autonomous replay. These findings validate the existence of task-constrained symmetries and provide an interpretable diagnostic framework for analyzing recurrent network dynamics.
📝 Abstract
Recurrent networks can contain substantial functional redundancy in
weight space: changing a recurrent matrix may leave the input-output
rollout nearly unchanged on a task distribution, while similar-scale
changes can destroy the same behavior. We study this redundancy in
one-layer tanh RNNs using ordered real Schur coordinates. The Schur
form separates spectral blocks from directed nonnormal couplings,
giving a diagnostic basis for structured ablations that keep the input
and readout maps fixed. In a fixed-length copy task, selected
nonnormal Schur couplings can be removed with little loss in some
trained solutions, whereas other couplings are necessary for accurate
autonomous replay. Across flip-flop, sine generation, and
context-dependent integration, the loss-preserving ablation profile
varies across tasks and trained solutions. These results identify
candidate approximate functional invariances, not universal symmetries
of recurrent weight space. Schur-coordinate ablations provide a
practical diagnostic for which structured perturbations preserve a
trained recurrent solution and which ones disrupt its computation.