Sherlock: Towards Multi-scene Video Abnormal Event Extraction and Localization via a Global-local Spatial-sensitive LLM

2025-02-26

AI Summary

Current video anomaly detection (VAD) methods operate solely at the frame level, lacking the capability to model and spatiotemporally localize structured semantic elements of anomalous events—such as subjects, event types, objects, and scenes. To address this limitation, we introduce the novel task of Multi-scenario Video Anomalous Event Extraction and Localization (M-VAE). We construct the first M-VAE instruction-tuning dataset and propose a global-local spatially aware Video Large Language Model (Video-LLM). Our architecture incorporates a Spatially Enhanced Mixture-of-Experts (GSM) module and a Spatial Imbalance Regulator (SIR), enabling fine-grained semantic parsing and precise spatiotemporal localization. Experiments demonstrate that our method achieves 12.7%–18.3% improvements over state-of-the-art video LLMs in both quadruple extraction accuracy and spatiotemporal localization precision, significantly advancing the structured semantic understanding of video anomalies.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Video Understanding & Activity Analysis
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Prior studies on Video Anomaly Detection (VAD) mainly focus on detecting whether each video frame is abnormal or not in the video, which largely ignore the structured video semantic information (i.e., what, when, and where does the abnormal event happen). With this in mind, we propose a new chat-paradigm extbf{M}ulti-scene Video Abnormal Event Extraction and Localization (M-VAE) task, aiming to extract the abnormal event quadruples (i.e., subject, event type, object, scene) and localize such event. Further, this paper believes that this new task faces two key challenges, i.e., global-local spatial modeling and global-local spatial balancing. To this end, this paper proposes a Global-local Spatial-sensitive Large Language Model (LLM) named Sherlock, i.e., acting like Sherlock Holmes to track down the criminal events, for this M-VAE task. Specifically, this model designs a Global-local Spatial-enhanced MoE (GSM) module and a Spatial Imbalance Regulator (SIR) to address the two challenges respectively. Extensive experiments on our M-VAE instruction dataset show the significant advantages of Sherlock over several advanced Video-LLMs. This justifies the importance of global-local spatial information for the M-VAE task and the effectiveness of Sherlock in capturing such information.

Problem

• Multi-scene Video Abnormal Event Extraction
• Global-local Spatial-sensitive Modeling
• Event Localization and Quadruple Extraction

Innovation

• Global-local Spatial-sensitive LLM
• Global-local Spatial-enhanced MoE
• Spatial Imbalance Regulator