VADER: Towards Causal Video Anomaly Understanding with Relation-Aware Large Language Models

2025-11-10

AI Summary

Existing video anomaly understanding methods largely neglect deep causal relationships and dynamic interactions among objects, limiting interpretable modeling of anomalous behaviors. To address this, we propose the first causal understanding framework for video anomaly events, integrating relation-aware modeling with a Context-Aware Event Sampling (CAES) mechanism to jointly extract object-relational features and visual cues from key frames. We further design a Contrastive Object-Relation Encoder (CORE) to strengthen causal structure learning and incorporate a large language model (LLM) to generate fine-grained, causally coherent anomaly descriptions and support causal question-answering reasoning. Evaluated on multiple real-world benchmarks, our method achieves significant improvements in anomaly localization, explanatory description generation, and causal reasoning performance. It marks the first successful transition from appearance-based detection to causal-level understanding in video anomaly analysis, advancing the frontier of interpretable video anomaly understanding.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Reasoning under Uncertainty: Causality
• Machine Learning: Causal Learning

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Video anomaly understanding (VAU) aims to provide detailed interpretation and semantic comprehension of anomalous events within videos, addressing limitations of traditional methods that focus solely on detecting and localizing anomalies. However, existing approaches often neglect the deeper causal relationships and interactions between objects, which are critical for understanding anomalous behaviors. In this paper, we propose VADER, an LLM-driven framework for Video Anomaly unDErstanding, which integrates keyframe object Relation features with visual cues to enhance anomaly comprehension from video. Specifically, VADER first applies an Anomaly Scorer to assign per-frame anomaly scores, followed by a Context-AwarE Sampling (CAES) strategy to capture the causal context of each anomalous event. A Relation Feature Extractor and a COntrastive Relation Encoder (CORE) jointly model dynamic object interactions, producing compact relational representations for downstream reasoning. These visual and relational cues are integrated with LLMs to generate detailed, causally grounded descriptions and support robust anomaly-related question answering. Experiments on multiple real-world VAU benchmarks demonstrate that VADER achieves strong results across anomaly description, explanation, and causal reasoning tasks, advancing the frontier of explainable video anomaly analysis.

Problem

• Understanding causal relationships in video anomalies
• Modeling dynamic object interactions for anomaly comprehension
• Generating causally grounded descriptions of anomalous events

Innovation

• Integrates object relation features with visual cues
• Uses contrastive relation encoder for dynamic interactions
• Combines relational representations with large language models