Parity Requires Unified Input Dependence and Negative Eigenvalues in SSMs

2025-08-10

AI Summary

This work investigates the fundamental capability boundaries of diagonal state space models (SSMs) in solving basic state-tracking tasks such as parity. Addressing the central question—“Are both input dependence and negative eigenvalues simultaneously necessary?”—the authors provide a theoretical proof and empirical validation: stacking input-independent or non-negative-eigenvalue SSM layers (e.g., S4D or Mamba variants) provably fails on parity tasks. Only recursive dynamics that jointly incorporate input dependence and negative eigenvalues can model the parity function. Through carefully designed multi-layer SSM architectures, rigorous theoretical analysis, and controlled ablation experiments, the study establishes for the first time the strict necessity of *both* properties—neither alone suffices. This reveals a foundational mechanism underlying SSMs’ state-tracking expressivity and provides critical theoretical grounding for designing interpretable, principled architectures.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Reasoning under Uncertainty: Relational Probabilistic Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Recent work has shown that LRNN models such as S4D, Mamba, and DeltaNet lack state-tracking capability due to either time-invariant transition matrices or restricted eigenvalue ranges. To address this, input-dependent transition matrices, particularly those that are complex or non-triangular, have been proposed to enhance SSM performance on such tasks. While existing theorems demonstrate that both input-independent and non-negative SSMs are incapable of solving simple state-tracking tasks, such as parity, regardless of depth, they do not explore whether combining these two types in a multilayer SSM could help. We investigate this question for efficient SSMs with diagonal transition matrices and show that such combinations still fail to solve parity. This implies that a recurrence layer must both be input-dependent and include negative eigenvalues. Our experiments support this conclusion by analyzing an SSM model that combines S4D and Mamba layers.

Problem

• Investigates state-tracking capability limitations in structured state space models
• Tests whether combining input-independent and non-negative SSMs solves parity
• Determines recurrence layers require both input-dependence and negative eigenvalues

Innovation

• Input-dependent transition matrices enhance SSM
• Combining input-independent and non-negative SSMs fails
• Recurrence layer requires negative eigenvalues