Large Language Models for Autonomous Driving (LLM4AD): oncept, Benchmark, Experiments, and Challenges

2024-10-20

AI Summary

This study systematically investigates the deep integration of large language models (LLMs) into autonomous driving (AD), addressing critical challenges including latency, safety, interpretability, privacy, and personalization, while aiming to enhance perception understanding, natural language interaction, and decision support. Method: We propose the “LLM4AD” paradigm and introduce the first AD-specific LLM instruction-following benchmark, incorporating multimodal perception–language joint modeling, simulation-based training, instruction tuning, and alignment techniques, validated across diverse scenarios on real-world vehicle platforms. Contributions/Results: (1) We formally define the LLM4AD conceptual framework; (2) we release the first instruction-following evaluation benchmark tailored to AD; (3) empirical results demonstrate that LLMs significantly improve human–vehicle collaboration efficiency and cross-task generalization, establishing a systematic pathway toward trustworthy, deployable onboard LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

With the broader usage and highly successful development of Large Language Models (LLMs), there has been a growth of interest and demand for applying LLMs to autonomous driving technology. Driven by their natural language understanding and reasoning ability, LLMs have the potential to enhance various aspects of autonomous driving systems, from perception and scene understanding to language interaction and decision-making. In this paper, we first introduce the novel concept of designing LLMs for autonomous driving (LLM4AD). Then, we propose a comprehensive benchmark for evaluating the instruction-following abilities of LLM4AD in simulation. Furthermore, we conduct a series of experiments on real-world vehicle platforms, thoroughly evaluating the performance and potential of our LLM4AD systems. Finally, we envision the main challenges of LLM4AD, including latency, deployment, security and privacy, safety, trust and transparency, and personalization. Our research highlights the significant potential of LLMs to enhance various aspects of autonomous vehicle technology, from perception and scene understanding to language interaction and decision-making.

Problem

• Applying LLMs to autonomous driving
• Enhancing perception and decision-making
• Addressing latency and security challenges

Innovation

• LLMs enhance autonomous driving systems
• Comprehensive benchmark for LLM4AD evaluation
• Real-world experiments on vehicle platforms