Selection Mechanisms for Sequence Modeling using Linear State Space Models

2025-05-23

AI Summary

To address weak state selection capability, training instability, and lack of interpretability in selective state space models (SSMs), this paper proposes a control-theoretic residual selection mechanism. Instead of resorting to complex linear time-varying (LTV) modeling, it pioneers the adoption of linear time-invariant (LTI) system fault detection principles into SSM selection: multiple parallel LTI subsystems generate state residuals, and the dominant state path is dynamically selected based on these residuals. The mechanism inherits the training stability of LTI systems while achieving strong, adaptive state selection. Empirically, it matches Mamba’s performance on multiple synthetic selectivity benchmarks, with significant improvements in selection accuracy and robustness. This work establishes the first SSM selection paradigm that simultaneously offers theoretical interpretability, foundations for controllability analysis, and competitive empirical performance.

Technology Category

• Intelligent Robots: State Estimation
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Recent advancements in language modeling tasks have been driven by architectures such as Transformers and, more recently, by Selective State Space Models (SSMs). In this paper, we introduce an alternative selection mechanism inspired by control theory methodologies. Specifically, we propose a novel residual generator for selection, drawing an analogy to fault detection strategies in Linear Time-Invariant (LTI) systems. Unlike Mamba, which utilizes Linear Time-Varying (LTV) systems, our approach combines multiple LTI systems, preserving their beneficial properties during training while achieving comparable selectivity. To evaluate the effectiveness of the proposed architecture, we test its performance on synthetic tasks. While these tasks are not inherently critical, they serve as benchmarks to test the selectivity properties of different cores architecture. This work highlights the potential of integrating theoretical insights with experimental advancements, offering a complementary perspective to deep learning innovations at the intersection of control theory and machine learning.

Problem

• Proposing a novel selection mechanism for sequence modeling inspired by control theory
• Combining multiple Linear Time-Invariant systems to enhance selectivity in models
• Evaluating the architecture on synthetic tasks to benchmark selectivity properties

Innovation

• Novel residual generator for selection
• Combines multiple Linear Time-Invariant systems
• Inspired by control theory methodologies