Analysis of Long Range Dependency Understanding in State Space Models

2026-01-19

AI Summary

This work addresses the limited interpretability of existing state space models regarding their long-range dependency mechanisms, particularly the unclear relationship between modeling capacity and architectural design in real-world tasks. Focusing on the S4D model, we present the first systematic analysis of its kernel behavior in the context of source code vulnerability detection. By integrating time-domain and frequency-domain analyses, we demonstrate that S4D can function as a low-pass, band-pass, or high-pass filter depending on its architectural configuration. This finding reveals that the model’s ability to capture long-range dependencies is profoundly influenced by its architecture, thereby offering both theoretical insights and concrete guidance for designing more effective state space models.

Technology Category

• Machine Learning: Kernel Methods
• Reasoning under Uncertainty: Relational Probabilistic Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Web Mining and Content Analysis: Models for Web evolution
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Although state-space models (SSMs) have demonstrated strong performance on long-sequence benchmarks, most research has emphasized predictive accuracy rather than interpretability. In this work, we present the first systematic kernel interpretability study of the diagonalized state-space model (S4D) trained on a real-world task (vulnerability detection in source code). Through time and frequency domain analysis of the S4D kernel, we show that the long-range modeling capability of S4D varies significantly under different model architectures, affecting model performance. For instance, we show that the depending on the architecture, S4D kernel can behave as low-pass, band-pass or high-pass filter. The insights from our analysis can guide future work in designing better S4D-based models.

Problem

• state-space models
• long-range dependency
• interpretability
• S4D
• kernel analysis

Innovation

• state-space models
• S4D
• long-range dependency
• kernel interpretability
• frequency domain analysis