Hankel Singular Value Regularization for Highly Compressible State Space Models

2025-10-26

AI Summary

To address the poor compressibility and difficulty in balancing accuracy and parameter count of State Space Models (SSMs) for long-sequence modeling in deep neural networks, this paper proposes Hankel Singular Value Regularization (HSVR). HSVR explicitly enforces exponential decay of the Hankel matrix singular values, encouraging the learned SSM dynamics to be low-rank and highly compressible. We further introduce a block-diagonal parameterization and a structure-preserving iterative optimization algorithm that ensures training stability and computational efficiency. On the Long Range Arena benchmark, our method achieves up to 10× model compression without accuracy loss—often yielding marginal gains—outperforming existing SSM compression approaches. The core contribution is the first integration of Hankel system theory into SSM regularization design, establishing a novel paradigm for compressible long-sequence modeling.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Semi-Supervised Learning
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Social recommender systems and personalization for the social Web
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Deep neural networks using state space models as layers are well suited for long-range sequence tasks but can be challenging to compress after training. We use that regularizing the sum of Hankel singular values of state space models leads to a fast decay of these singular values and thus to compressible models. To make the proposed Hankel singular value regularization scalable, we develop an algorithm to efficiently compute the Hankel singular values during training iterations by exploiting the specific block-diagonal structure of the system matrices that is we use in our state space model parametrization. Experiments on Long Range Arena benchmarks demonstrate that the regularized state space layers are up to 10$ imes$ more compressible than standard state space layers while maintaining high accuracy.

Problem

• Compressing state space models in neural networks
• Regularizing Hankel singular values for model compression
• Developing scalable algorithm for efficient training iterations

Innovation

• Regularizes Hankel singular values for model compression
• Develops efficient algorithm for scalable training iterations
• Uses block-diagonal structure in state space parametrization