Vision-Based Natural Language Scene Understanding for Autonomous Driving: An Extended Dataset and a New Model for Traffic Scene Description Generation

2026-01-20

AI Summary

This work addresses the challenge of generating comprehensive and natural language descriptions of traffic scenes from monocular images in autonomous driving systems, a task hindered by the absence of high-quality, domain-specific datasets. To this end, the authors construct an extended traffic scene captioning dataset based on BDD100K and propose an image encoder–language decoder architecture incorporating a hybrid attention mechanism. This model jointly optimizes spatial layout and semantic relational features to produce rich, driving-relevant scene descriptions. Experimental results demonstrate that the proposed approach significantly outperforms baseline methods across automatic metrics—including CIDEr and SPICE—as well as human evaluations, thereby advancing the capability of language understanding and generation in complex traffic scenarios.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Traffic scene understanding is essential for enabling autonomous vehicles to accurately perceive and interpret their environment, thereby ensuring safe navigation. This paper presents a novel framework that transforms a single frontal-view camera image into a concise natural language description, effectively capturing spatial layouts, semantic relationships, and driving-relevant cues. The proposed model leverages a hybrid attention mechanism to enhance spatial and semantic feature extraction and integrates these features to generate contextually rich and detailed scene descriptions. To address the limited availability of specialized datasets in this domain, a new dataset derived from the BDD100K dataset has been developed, with comprehensive guidelines provided for its construction. Furthermore, the study offers an in-depth discussion of relevant evaluation metrics, identifying the most appropriate measures for this task. Extensive quantitative evaluations using metrics such as CIDEr and SPICE, complemented by human judgment assessments, demonstrate that the proposed model achieves strong performance and effectively fulfills its intended objectives on the newly developed dataset.

Problem

• traffic scene understanding
• natural language description
• autonomous driving
• vision-based scene interpretation
• scene description generation

Innovation

• vision-language modeling
• hybrid attention mechanism
• traffic scene description
• autonomous driving
• dataset extension