Unlocking Vision-Language Models for Video Anomaly Detection via Fine-Grained Prompting

2025-10-02

AI Summary

Existing prompt-based frozen vision-language models (VLMs) exhibit weak generalization in video anomaly detection (VAD), primarily due to overly abstract prompts that fail to model fine-grained anomaly cues—such as human-object interactions and action semantics. To address this, we propose ASK-Hint: the first action-centered, knowledge-guided structured prompting framework for VAD. ASK-Hint introduces fine-grained grouped prompts, guided question design, and semantic consistency reasoning, enabling high-accuracy and interpretable anomaly detection without VLM fine-tuning. Our method achieves strong cross-dataset and cross-model generalization, setting new state-of-the-art performance on UCF-Crime and XD-Violence. Crucially, it delivers transparent, step-by-step reasoning paths—enhancing both diagnostic utility and trustworthiness.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Prompting has emerged as a practical way to adapt frozen vision-language models (VLMs) for video anomaly detection (VAD). Yet, existing prompts are often overly abstract, overlooking the fine-grained human-object interactions or action semantics that define complex anomalies in surveillance videos. We propose ASK-Hint, a structured prompting framework that leverages action-centric knowledge to elicit more accurate and interpretable reasoning from frozen VLMs. Our approach organizes prompts into semantically coherent groups (e.g. violence, property crimes, public safety) and formulates fine-grained guiding questions that align model predictions with discriminative visual cues. Extensive experiments on UCF-Crime and XD-Violence show that ASK-Hint consistently improves AUC over prior baselines, achieving state-of-the-art performance compared to both fine-tuned and training-free methods. Beyond accuracy, our framework provides interpretable reasoning traces towards anomaly and demonstrates strong generalization across datasets and VLM backbones. These results highlight the critical role of prompt granularity and establish ASK-Hint as a new training-free and generalizable solution for explainable video anomaly detection.

Problem

• Adapting vision-language models for video anomaly detection using fine-grained prompts
• Addressing overly abstract prompts that miss human-object interaction details
• Improving interpretable reasoning for complex anomalies in surveillance videos

Innovation

• Fine-grained prompting framework for video anomaly detection
• Action-centric knowledge guides frozen vision-language models
• Semantic prompt groups improve interpretable reasoning traces