TAU-R1: Visual Language Model for Traffic Anomaly Understanding

2026-03-19

AI Summary

Traffic anomaly understanding (TAU) has been hindered by the absence of dedicated benchmarks and tailored methodologies, limiting its applicability in intelligent transportation systems. To address this gap, this work introduces Roundabout-TAU, the first TAU-specific dataset focused on roundabout scenarios, along with TAU-R1, a two-stage vision-language framework. The first stage employs a lightweight classifier for coarse-grained anomaly detection, while the second leverages large language models to generate fine-grained event summaries. By integrating decomposed question-answering fine-tuning with a novel reinforcement-based post-training strategy, TAU-GRPO, the proposed approach achieves state-of-the-art performance in both anomaly classification and reasoning tasks, effectively balancing accuracy and deployment efficiency. The dataset and code are publicly released.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: Safety and Robustness
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Traffic Anomaly Understanding (TAU) is important for traffic safety in Intelligent Transportation Systems. Recent vision-language models (VLMs) have shown strong capabilities in video understanding. However, progress on TAU remains limited due to the lack of benchmarks and task-specific methodologies. To address this limitation, we introduce Roundabout-TAU, a dataset constructed from real-world roundabout videos collected in collaboration with the City of Carmel, Indiana. The dataset contains 342 clips and is annotated with more than 2,000 question-answer pairs covering multiple aspects of traffic anomaly understanding. Building on this benchmark, we propose TAU-R1, a two-layer vision-language framework for TAU. The first layer is a lightweight anomaly classifier that performs coarse anomaly categorisation, while the second layer is a larger anomaly reasoner that generates detailed event summaries. To improve task-specific reasoning, we introduce a two-stage training strategy consisting of decomposed-QA-enhanced supervised fine-tuning followed by TAU-GRPO, a GRPO-based post-training method with TAU-specific reward functions. Experimental results show that TAU-R1 achieves strong performance on both anomaly classification and reasoning tasks while maintaining deployment efficiency. The dataset and code are available at: https://github.com/siri-rouser/TAU-R1

Problem

• Traffic Anomaly Understanding
• Vision-Language Model
• Intelligent Transportation Systems
• Benchmark Dataset

Innovation

• Traffic Anomaly Understanding
• Vision-Language Model
• Two-stage Training
• Roundabout-TAU Dataset
• TAU-GRPO