On the Expressive Power and Limitations of Multi-Layer SSMs

2026-04-15

AI Summary

This work investigates the expressive limitations of multilayer state space models (SSMs) on compositional tasks and their capability gap relative to streaming algorithms. Through a theoretical analysis grounded in computational complexity, state space modeling, and a finite-precision computation framework, the study establishes for the first time that an online chain-of-thought (CoT) mechanism enables multilayer SSMs to attain the expressive power of streaming algorithms. Furthermore, it delineates how the interplay between model width and numerical precision differs markedly depending on the presence or absence of online CoT, thereby systematically clarifying the trade-off dynamics between these two resources in determining model expressivity.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We study the expressive power and limitations of multi-layer state-space models (SSMs). First, we show that multi-layer SSMs face fundamental limitations in compositional tasks, revealing an inherent gap between SSMs and streaming models. Then, we examine the role of chain-of-thought (CoT), showing that offline CoT does not fundamentally increase the expressiveness, while online CoT can substantially increase its power. Indeed, with online CoT, multi-layer SSMs become equivalent in power to streaming algorithms. Finally, we investigate the tradeoff between width and precision, showing that these resources are not interchangeable in the base model, but admit a clean equivalence once online CoT is allowed. Overall, our results offer a unified perspective on how depth, finite precision, and CoT shape the power and limits of SSMs.

Problem

• state-space models
• expressive power
• compositional tasks
• chain-of-thought
• streaming algorithms

Innovation

• state-space models
• expressive power
• chain-of-thought
• streaming algorithms
• finite precision