Towards Natural Language Communication for Cooperative Autonomous Driving via Self-Play

2025-05-23

AI Summary

This work addresses the lack of semantic interpretability and collaborative capability in vehicle-to-vehicle (V2V) communication under mixed human–autonomous driving scenarios. Methodologically, it proposes the first natural-language-based, multi-agent dialogue-driven V2V coordination framework, integrating large language models (LLMs) with a post-hoc debrief mechanism to enable explainable, task-oriented natural-language message generation and closed-loop decision-making. Training is conducted via multi-agent self-play within a custom gym-like driving simulation environment. Contributions include: (i) the first LLM-powered natural-language V2V communication framework jointly optimizing safety and efficiency; (ii) explicit support for human driver comprehension and intervention; and (iii) significant improvements over zero-shot LLM baselines across diverse traffic scenarios—achieving +18.7% collaborative success rate, +23.4% average traffic throughput, and high instruction accuracy, semantic clarity, and executability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Agent Communication
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Past work has demonstrated that autonomous vehicles can drive more safely if they communicate with one another than if they do not. However, their communication has often not been human-understandable. Using natural language as a vehicle-to-vehicle (V2V) communication protocol offers the potential for autonomous vehicles to drive cooperatively not only with each other but also with human drivers. In this work, we propose a suite of traffic tasks in autonomous driving where vehicles in a traffic scenario need to communicate in natural language to facilitate coordination in order to avoid an imminent collision and/or support efficient traffic flow. To this end, this paper introduces a novel method, LLM+Debrief, to learn a message generation and high-level decision-making policy for autonomous vehicles through multi-agent discussion. To evaluate LLM agents for driving, we developed a gym-like simulation environment that contains a range of driving scenarios. Our experimental results demonstrate that LLM+Debrief is more effective at generating meaningful and human-understandable natural language messages to facilitate cooperation and coordination than a zero-shot LLM agent. Our code and demo videos are available at https://talking-vehicles.github.io/.

Problem

• Enabling autonomous vehicles to communicate via natural language
• Improving cooperative driving through human-understandable V2V messages
• Learning message generation policies via multi-agent discussion

Innovation

• Uses natural language for V2V communication
• LLM+Debrief method for policy learning
• Gym-like simulation for driving scenarios