MambaAD: Exploring State Space Models for Multi-class Unsupervised Anomaly Detection

2024-04-09

Neural Information Processing Systems

AI Summary

To address the challenge of jointly modeling long-range dependencies and ensuring computational efficiency in multi-class unsupervised anomaly detection—where CNNs and Transformers exhibit inherent trade-offs—this paper proposes MambaAD, the first framework to introduce state space models (SSMs), specifically the Mamba architecture, into this task. We design a Locality-Enhanced State Space (LSS) decoder that integrates five scanning strategies, eight-directional Hilbert curve serialization, and multi-kernel convolution to enable synergistic global-local representation learning. Additionally, MambaAD leverages a pre-trained encoder and a multi-scale feature fusion mechanism. Extensive experiments across six benchmark datasets and seven evaluation metrics demonstrate consistent superiority over CNN- and Transformer-based baselines, achieving new state-of-the-art performance. The code and pre-trained models are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Adversarial Attacks & Robustness
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent advancements in anomaly detection have seen the efficacy of CNN- and transformer-based approaches. However, CNNs struggle with long-range dependencies, while transformers are burdened by quadratic computational complexity. Mamba-based models, with their superior long-range modeling and linear efficiency, have garnered substantial attention. This study pioneers the application of Mamba to multi-class unsupervised anomaly detection, presenting MambaAD, which consists of a pre-trained encoder and a Mamba decoder featuring (Locality-Enhanced State Space) LSS modules at multi-scales. The proposed LSS module, integrating parallel cascaded (Hybrid State Space) HSS blocks and multi-kernel convolutions operations, effectively captures both long-range and local information. The HSS block, utilizing (Hybrid Scanning) HS encoders, encodes feature maps into five scanning methods and eight directions, thereby strengthening global connections through the (State Space Model) SSM. The use of Hilbert scanning and eight directions significantly improves feature sequence modeling. Comprehensive experiments on six diverse anomaly detection datasets and seven metrics demonstrate state-of-the-art performance, substantiating the method's effectiveness. The code and models are available at https://lewandofskee.github.io/projects/MambaAD.

Problem

• Addresses limitations of CNNs and transformers in anomaly detection
• Introduces MambaAD for multi-class unsupervised anomaly detection
• Proposes LSS module to capture long-range and local information

Innovation

• MambaAD uses Mamba-based models for anomaly detection.
• Integrates LSS modules with HSS blocks and convolutions.
• Employs Hilbert scanning for enhanced feature modeling.