VADMamba: Exploring State Space Models for Fast Video Anomaly Detection

2025-03-27

AI Summary

To address the longstanding trade-off between detection accuracy and inference speed in video anomaly detection, this paper pioneers the integration of state space models—specifically Mamba—into this task. We propose VQ-Mamba UNet, a novel architecture that synergistically combines vector quantization (VQ) with a non-negative visual state space (NVSS) module, enabling joint frame prediction and optical flow reconstruction within a multi-task learning framework. Additionally, we introduce a clip-level dual-branch evaluation strategy to enhance anomaly localization and scoring robustness. Our method achieves state-of-the-art inference speed on three mainstream benchmarks—outperforming CNN- and Transformer-based baselines by several-fold—while maintaining competitive detection accuracy. Experimental results demonstrate that state space models offer superior spatiotemporal modeling capacity and significant potential for real-time, high-fidelity video anomaly detection.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Calibration & Uncertainty Quantification
• Intelligent Robots: State Estimation

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Video anomaly detection (VAD) methods are mostly CNN-based or Transformer-based, achieving impressive results, but the focus on detection accuracy often comes at the expense of inference speed. The emergence of state space models in computer vision, exemplified by the Mamba model, demonstrates improved computational efficiency through selective scans and showcases the great potential for long-range modeling. Our study pioneers the application of Mamba to VAD, dubbed VADMamba, which is based on multi-task learning for frame prediction and optical flow reconstruction. Specifically, we propose the VQ-Mamba Unet (VQ-MaU) framework, which incorporates a Vector Quantization (VQ) layer and Mamba-based Non-negative Visual State Space (NVSS) block. Furthermore, two individual VQ-MaU networks separately predict frames and reconstruct corresponding optical flows, further boosting accuracy through a clip-level fusion evaluation strategy. Experimental results validate the efficacy of the proposed VADMamba across three benchmark datasets, demonstrating superior performance in inference speed compared to previous work. Code is available at https://github.com/jLooo/VADMamba.

Problem

• Improving video anomaly detection speed using state space models
• Combining Mamba model with multi-task learning for VAD
• Enhancing accuracy via clip-level fusion and optical flow

Innovation

• Uses Mamba model for fast video anomaly detection
• Incorporates Vector Quantization and NVSS block
• Employs multi-task learning for frame and flow