SAVANT: Semantic Analysis with Vision-Augmented Anomaly deTection

2025-10-20

AI Summary

Autonomous driving systems exhibit insufficient robustness against rare semantic anomalies (out-of-distribution scenarios), and existing zero-shot vision-language model (VLM)-based prompting methods suffer from unstable performance and reliance on proprietary large models, hindering practical deployment. This paper proposes SAVANT: a structured framework that decomposes VLM reasoning via hierarchical visual-semantic analysis—categorizing scenes into four levels (street, infrastructure, dynamic objects, and environment)—and a two-stage detection pipeline. SAVANT integrates lightweight open-source VLMs, structured scene description extraction, and multimodal evaluation to enable efficient, low-cost local deployment. Evaluated on real-world driving data, SAVANT achieves 90.8% recall and 93.8% precision—substantially outperforming baseline methods—and automatically annotates over 9,640 anomalous images, effectively mitigating the scarcity of labeled anomaly data.

Technology Category

• Computer Vision: Vision for Robotics & Autonomous Driving
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Autonomous driving systems remain critically vulnerable to the long-tail of rare, out-of-distribution scenarios with semantic anomalies. While Vision Language Models (VLMs) offer promising reasoning capabilities, naive prompting approaches yield unreliable performance and depend on expensive proprietary models, limiting practical deployment. We introduce SAVANT (Semantic Analysis with Vision-Augmented Anomaly deTection), a structured reasoning framework that achieves high accuracy and recall in detecting anomalous driving scenarios from input images through layered scene analysis and a two-phase pipeline: structured scene description extraction followed by multi-modal evaluation. Our approach transforms VLM reasoning from ad-hoc prompting to systematic analysis across four semantic layers: Street, Infrastructure, Movable Objects, and Environment. SAVANT achieves 89.6% recall and 88.0% accuracy on real-world driving scenarios, significantly outperforming unstructured baselines. More importantly, we demonstrate that our structured framework enables a fine-tuned 7B parameter open-source model (Qwen2.5VL) to achieve 90.8% recall and 93.8% accuracy - surpassing all models evaluated while enabling local deployment at near-zero cost. By automatically labeling over 9,640 real-world images with high accuracy, SAVANT addresses the critical data scarcity problem in anomaly detection and provides a practical path toward reliable, accessible semantic monitoring for autonomous systems.

Problem

• Detecting rare semantic anomalies in autonomous driving scenarios
• Overcoming unreliable VLM prompting for practical anomaly detection
• Addressing data scarcity in autonomous system anomaly detection

Innovation

• Structured reasoning framework for anomaly detection
• Layered scene analysis across four semantic categories
• Open-source model achieving high accuracy and recall