Spatio-Temporal Data Enhanced Vision-Language Model for Traffic Scene Understanding

2025-11-12

AI Summary

Existing traffic scene understanding (TSU) methods predominantly treat the task as static image interpretation, neglecting spatiotemporal dynamics and semantic correlations among heterogeneous traffic elements. To address this, we propose the first spatiotemporal-aware vision-language model for TSU, introducing a novel dual-level spatiotemporal prompting mechanism: (i) dynamic spatiotemporal context fusion at the feature level, and (ii) ST-Aware multi-faceted prompt design at the prompt level, jointly modeling low-level visual and high-level semantic features. Built upon the CLIP framework, our method explicitly encodes fine-grained traffic semantics—including objects, events, and states—and their spatiotemporal dependencies. Extensive experiments on two real-world traffic datasets demonstrate that our approach achieves substantial performance gains with only few-shot supervision, particularly excelling in temporal reasoning and cross-modal alignment—outperforming prior methods by significant margins.

Technology Category

• Computer Vision: Language and Vision
• Planning, Routing, and Scheduling: Planning with Language Models
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Nowadays, navigation and ride-sharing apps have collected numerous images with spatio-temporal data. A core technology for analyzing such images, associated with spatiotemporal information, is Traffic Scene Understanding (TSU), which aims to provide a comprehensive description of the traffic scene. Unlike traditional spatio-temporal data analysis tasks, the dependence on both spatio-temporal and visual-textual data introduces distinct challenges to TSU task. However, recent research often treats TSU as a common image understanding task, ignoring the spatio-temporal information and overlooking the interrelations between different aspects of the traffic scene. To address these issues, we propose a novel SpatioTemporal Enhanced Model based on CILP (ST-CLIP) for TSU. Our model uses the classic vision-language model, CLIP, as the backbone, and designs a Spatio-temporal Context Aware Multiaspect Prompt (SCAMP) learning method to incorporate spatiotemporal information into TSU. The prompt learning method consists of two components: A dynamic spatio-temporal context representation module that extracts representation vectors of spatio-temporal data for each traffic scene image, and a bi-level ST-aware multi-aspect prompt learning module that integrates the ST-context representation vectors into word embeddings of prompts for the CLIP model. The second module also extracts low-level visual features and image-wise high-level semantic features to exploit interactive relations among different aspects of traffic scenes. To the best of our knowledge, this is the first attempt to integrate spatio-temporal information into visionlanguage models to facilitate TSU task. Experiments on two realworld datasets demonstrate superior performance in the complex scene understanding scenarios with a few-shot learning strategy.

Problem

• Integrating spatio-temporal data with vision-language models for traffic analysis
• Addressing overlooked interrelations between traffic scene aspects
• Enhancing traffic scene understanding beyond common image interpretation

Innovation

• Integrates spatio-temporal data into vision-language model
• Uses dynamic context representation for traffic scenes
• Implements bi-level multi-aspect prompt learning method