Modular Autonomy with Conversational Interaction: An LLM-driven Framework for Decision Making in Autonomous Driving

2026-01-09

arXiv.org

AI Summary

This work addresses the semantic gap between natural language commands and the action space of modular autonomous driving systems by proposing a safety-aware interaction framework grounded in large language models (LLMs). Integrated within the Autoware platform, the framework establishes an autonomous-driving-oriented interaction taxonomy, designs a domain-specific language (DSL) to map natural instructions to structured actions, and employs a two-stage LLM architecture augmented with a safety verification layer to ensure both transparency and reliability. Experimental results demonstrate that the system robustly and efficiently executes all five categories of natural language commands in simulation, establishing a scalable foundation for natural language interaction in safety-critical autonomous driving applications.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recent advancements in Large Language Models (LLMs) offer new opportunities to create natural language interfaces for Autonomous Driving Systems (ADSs), moving beyond rigid inputs. This paper addresses the challenge of mapping the complexity of human language to the structured action space of modular ADS software. We propose a framework that integrates an LLM-based interaction layer with Autoware, a widely used open-source software. This system enables passengers to issue high-level commands, from querying status information to modifying driving behavior. Our methodology is grounded in three key components: a taxonomization of interaction categories, an application-centric Domain Specific Language (DSL) for command translation, and a safety-preserving validation layer. A two-stage LLM architecture ensures high transparency by providing feedback based on the definitive execution status. Evaluation confirms the system's timing efficiency and translation robustness. Simulation successfully validated command execution across all five interaction categories. This work provides a foundation for extensible, DSL-assisted interaction in modular and safety-conscious autonomy stacks.

Problem

• Autonomous Driving
• Large Language Models
• Natural Language Interaction
• Modular Autonomy
• Command Translation

Innovation

• Large Language Models
• Domain Specific Language
• Modular Autonomy
• Conversational Interaction
• Safety Validation