SENSE-VAD: Sentient and Semantic Video Anomaly Detection for Autonomous Driving

2026-06-30

AI Summary

This work addresses the challenge of detecting complex semantic anomalies in videos that arise from social relationships—such as a child wandering away from guardians or a pedestrian being chased—which existing methods struggle to identify. To this end, we propose the first decoupling of social-semantic anomalies from motion- and appearance-based anomalies and introduce the first synthetic benchmark for video anomaly detection tailored to autonomous driving scenarios. Leveraging CARLA and Unreal Engine, we generate high-fidelity anomalous scenes encompassing individual behaviors, group interactions, and human-object interactions, accompanied by frame-level annotations and real-world videos to facilitate simulation-to-reality transfer validation. Evaluations reveal that state-of-the-art methods perform substantially worse on these social-semantic anomalies, underscoring their significance as a new frontier in anomaly detection. The dataset, annotations, and generation code are publicly released.

Abstract

Autonomous vehicles (AVs) must navigate not only motion-based hazards but also socially complex situations whose danger is constituted by inter-agent relationships rather than movement statistics alone. A child running away from a guardian, a person being carried by another, or a pursuer chasing a pedestrian across a sidewalk are all anomalous in social context, yet none produces an obvious motion signal that current anomaly detectors are equipped to flag. We introduce SENSE-VAD, the first synthetic video anomaly detection benchmark for autonomous driving explicitly designed around socially complex anomalies. Using the CARLA simulator and Unreal Engine (UE), we generate distinct anomaly scenarios across multiple categories: individual behaviors, group behaviors, person--object interactions, cyclist interactions, vehicle & agent, each annotated with per-frame binary labels. A key design principle is the separation of social anomaly from motion-based or appearance-based anomaly: many scenarios involve motion of objects that appears unremarkable in isolation but is anomalous in relational context. We additionally provide real-world normal and anomalous videos as a sim-to-real transfer probe. We evaluate state-of-the-art video anomaly detection baselines and demonstrate that socially complex anomalies constitute a distinct and currently unsolved challenge. Our dataset, annotations, and generation code are publicly available.

Problem

• video anomaly detection
• autonomous driving
• socially complex anomalies
• relational context
• synthetic benchmark

Innovation

• social anomaly detection
• autonomous driving
• synthetic video benchmark
• relational context
• sim-to-real transfer