The Expressive Limits of Diagonal SSMs for State-Tracking

2026-03-02

AI Summary

This study investigates the expressive limits of diagonal state space models (SSMs) in sequential state tracking tasks, with a focus on their inability to model states governed by non-abelian groups. By integrating group theory and representation theory, the work precisely characterizes the class of state tracking tasks realizable by a k-layer complex-valued diagonal SSM: namely, those corresponding to solvable groups admitting a subnormal series of abelian factors of length at most k. Theoretically, it is proven that a single-layer SSM cannot track arbitrary non-abelian group states under finite precision. Although deeper architectures theoretically extend expressivity, empirical results demonstrate persistent practical learning limitations. This work establishes a fundamental connection between SSM expressiveness and the algebraic structure of underlying state spaces, revealing an intrinsic architectural constraint in current SSM designs.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Intelligent Robots: State Estimation
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

State-Space Models (SSMs) have recently been shown to achieve strong empirical performance on a variety of long-range sequence modeling tasks while remaining efficient and highly-parallelizable. However, the theoretical understanding of their expressive power remains limited. In this work, we study the expressivity of input-Dependent Complex-valued Diagonal (DCD) SSMs on sequential state-tracking tasks. We show that single-layer DCD SSMs cannot express state-tracking of any non-Abelian group at finite precision. More generally, we show that $k$-layer DCD SSMs can express state-tracking of a group if and only if that group has a subnormal series of length $k$, with Abelian factors. That is, we identify the precise expressivity range of $k$-layer DCD SSMs within the solvable groups. Empirically, we find that multi-layer models often fail to learn state-tracking for non-Abelian groups, highlighting a gap between expressivity and learnability.

Problem

• State-Space Models
• Expressivity
• State-Tracking
• Non-Abelian Groups
• Diagonal SSMs

Innovation

• State-Space Models
• Expressivity
• Non-Abelian Groups
• Solvable Groups
• Diagonal SSMs