Learning to Tell Apart: Weakly Supervised Video Anomaly Detection via Disentangled Semantic Alignment

2025-11-13

AI Summary

Existing weakly supervised video anomaly detection methods rely on multimodal foundation models (e.g., CLIP) and multiple instance learning, making them susceptible to saliency bias, limiting their ability to discover diverse normal patterns, and suffering from category confusion due to visual similarity—thus hindering fine-grained classification. To address these issues, we propose the Decoupled Semantic Alignment Network (DSAN). First, we introduce a self-guided normality modeling branch that leverages normal prototypes to drive feature reconstruction, explicitly uncovering intrinsic normal patterns. Second, we design an event-background decoupled contrastive semantic alignment mechanism to separate anomalous and normal representations, enhancing class discriminability. DSAN integrates frame-level scoring, temporal decomposition, vision-language contrastive learning, and reconstruction. Extensive experiments demonstrate that DSAN significantly outperforms state-of-the-art methods on XD-Violence and UCF-Crime, achieving concurrent improvements in both anomaly detection accuracy and fine-grained classification performance.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Machine Learning: Multimodal Learning

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Web Mining and Content Analysis: Normalization, clustering, classification, and summarization of Web text
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Recent advancements in weakly-supervised video anomaly detection have achieved remarkable performance by applying the multiple instance learning paradigm based on multimodal foundation models such as CLIP to highlight anomalous instances and classify categories. However, their objectives may tend to detect the most salient response segments, while neglecting to mine diverse normal patterns separated from anomalies, and are prone to category confusion due to similar appearance, leading to unsatisfactory fine-grained classification results. Therefore, we propose a novel Disentangled Semantic Alignment Network (DSANet) to explicitly separate abnormal and normal features from coarse-grained and fine-grained aspects, enhancing the distinguishability. Specifically, at the coarse-grained level, we introduce a self-guided normality modeling branch that reconstructs input video features under the guidance of learned normal prototypes, encouraging the model to exploit normality cues inherent in the video, thereby improving the temporal separation of normal patterns and anomalous events. At the fine-grained level, we present a decoupled contrastive semantic alignment mechanism, which first temporally decomposes each video into event-centric and background-centric components using frame-level anomaly scores and then applies visual-language contrastive learning to enhance class-discriminative representations. Comprehensive experiments on two standard benchmarks, namely XD-Violence and UCF-Crime, demonstrate that DSANet outperforms existing state-of-the-art methods.

Problem

• Separates abnormal and normal video features at coarse-grained level
• Enhances fine-grained classification via disentangled semantic alignment
• Addresses category confusion in weakly supervised anomaly detection

Innovation

• Disentangled Semantic Alignment Network separates abnormal and normal features
• Self-guided normality modeling branch reconstructs video using normal prototypes
• Decoupled contrastive semantic alignment enhances class-discriminative representations