On the Expressiveness of State Space Models via Temporal Logics

2026-01-27

AI Summary

This study investigates the expressive power of state space models (SSMs) as alternatives to Transformers in large language models. By integrating linear temporal logic (LTL) over finite traces and its extensions with formal language theory, the work provides the first formal characterization of SSM expressiveness. It distinguishes between quantized and high-precision SSMs: the former are limited to recognizing regular languages, while the latter can capture counting properties and certain non-regular languages. The paper establishes a theoretical framework for comparing the expressive capabilities of SSMs and Transformers, revealing fundamental similarities and differences between the two architectures in the context of language modeling.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We investigate the expressive power of state space models (SSM), which have recently emerged as a potential alternative to transformer architectures in large language models. Building on recent work, we analyse SSM expressiveness through fragments and extensions of linear temporal logic over finite traces. Our results show that the expressive capabilities of SSM vary substantially depending on the underlying gating mechanism. We further distinguish between SSM operating over fixed-width arithmetic (quantised models), whose expressive power remains within regular languages, and SSM with unbounded precision, which can capture counting properties and non-regular languages. In addition, we provide a systematic comparison between these different SSM variants and known results on transformers, thereby clarifying how the two architectures relate in terms of expressive power.

Problem

• state space models
• expressive power
• temporal logic
• transformers
• regular languages

Innovation

• State Space Models
• Expressive Power
• Temporal Logic
• Regular Languages
• Transformers