OmniTraffic: A Controllable Generation Pipeline and Benchmark for Spatio-Temporal Traffic Reasoning

2026-06-14

AI Summary

Existing multimodal traffic benchmarks struggle to evaluate models’ structural awareness and spatiotemporal reasoning under controlled conditions. This work proposes a controllable generation and evaluation framework built upon 3D reconstructions of 12 real-world intersections and surveillance footage from multiple countries. It enables, for the first time, metadata-driven synthesis of structured, multi-view visual question answering (VQA) tasks, supporting a three-tiered task hierarchy spanning scene understanding, spatiotemporal reasoning, and decision support. The framework allows configurable intersection layouts, signal phases, and rare-event scenarios, generating 8 million VQA samples alongside 3,000 human-verified test instances. Experiments reveal significant deficiencies in topological and spatiotemporal reasoning among 11 state-of-the-art multimodal large models, while lightweight models fine-tuned on synthetic data demonstrate markedly improved performance in real-world settings.

Abstract

Traffic scene understanding requires models to reason beyond object recognition, including lane topology, multi-view geometry, temporal evolution, and signal-phase semantics. However, existing traffic-oriented multimodal benchmarks largely emphasize passive visual recognition or isolated video understanding, offering limited support for evaluating structure-aware traffic reasoning under controlled conditions. We introduce OmniTraffic, a controllable generation pipeline and benchmark for spatio-temporal traffic reasoning. Built around 12 real-world intersections reconstructed into editable 3D traffic environments and complemented by surveillance footage from two countries, OmniTraffic supports both controlled and natural-condition evaluation. It defines a three-level task hierarchy spanning scene perception, multi-view and temporal reasoning, and decision support. Using structured traffic metadata, OmniTraffic generates synchronized multi-view VQA samples covering vehicle states, lane functions, view--BEV correspondence, temporal dynamics, and signal-phase analysis, resulting in 8M VQA samples and a 3K human-verified test set. Evaluation of eleven frontier MLLMs reveals a large human--model gap, with the most pronounced failures in topology-grounded and spatio-temporal reasoning tasks. Fine-tuning a lightweight MLLM on simulated OmniTraffic data further improves performance on real-world traffic scenes, demonstrating the value of simulation-generated supervision for traffic-specific multimodal reasoning. Beyond a fixed dataset, OmniTraffic provides an extensible pipeline with configurable intersections, camera views, traffic demands, signal phases, visual conditions, and rare events.

Problem

• spatio-temporal reasoning
• traffic scene understanding
• multimodal benchmark
• structure-aware reasoning
• controlled evaluation

Innovation

• controllable generation
• spatio-temporal reasoning
• 3D traffic simulation
• multimodal VQA
• structured traffic metadata