Structured Labeling Enables Faster Vision-Language Models for End-to-End Autonomous Driving

2025-06-05

AI Summary

End-to-end autonomous driving faces two key bottlenecks with vision-language models (VLMs): semantic redundancy arising from unstructured natural language annotations, and excessive model size hindering real-time deployment. Method: We propose a structured annotation paradigm and FastDrive—a lightweight VLM with 0.9B parameters—designed specifically for driving. Contribution/Results: (1) We introduce NuScenes-S, the first machine-friendly, structured dataset for autonomous driving; (2) FastDrive incorporates scene-aware architecture design, including structured semantic modeling, efficient cross-modal fusion, multimodal instruction tuning, and conditional encoding of environmental metadata (e.g., weather, time-of-day). Experiments demonstrate that FastDrive achieves ~20% higher accuracy on structured decision-making tasks and runs over 10× faster than 7B+ VLMs, validating the critical role of structured linguistic representations and contextual metadata in driving decision-making.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Vision-Language Models (VLMs) offer a promising approach to end-to-end autonomous driving due to their human-like reasoning capabilities. However, troublesome gaps remains between current VLMs and real-world autonomous driving applications. One major limitation is that existing datasets with loosely formatted language descriptions are not machine-friendly and may introduce redundancy. Additionally, high computational cost and massive scale of VLMs hinder the inference speed and real-world deployment. To bridge the gap, this paper introduces a structured and concise benchmark dataset, NuScenes-S, which is derived from the NuScenes dataset and contains machine-friendly structured representations. Moreover, we present FastDrive, a compact VLM baseline with 0.9B parameters. In contrast to existing VLMs with over 7B parameters and unstructured language processing(e.g., LLaVA-1.5), FastDrive understands structured and concise descriptions and generates machine-friendly driving decisions with high efficiency. Extensive experiments show that FastDrive achieves competitive performance on structured dataset, with approximately 20% accuracy improvement on decision-making tasks, while surpassing massive parameter baseline in inference speed with over 10x speedup. Additionally, ablation studies further focus on the impact of scene annotations (e.g., weather, time of day) on decision-making tasks, demonstrating their importance on decision-making tasks in autonomous driving.

Problem

• Reduces redundancy in loosely formatted language descriptions for autonomous driving
• Addresses high computational cost and slow inference speed of VLMs
• Improves decision-making accuracy with structured dataset and compact model

Innovation

• Structured benchmark dataset NuScenes-S
• Compact VLM baseline FastDrive 0.9B
• Machine-friendly structured descriptions processing