Towards Human-Centric Autonomous Driving: A Fast-Slow Architecture Integrating Large Language Model Guidance with Reinforcement Learning

2025-05-11

AI Summary

Current autonomous driving systems lack interactive capabilities and personalized adaptability. This paper proposes a human-centered dual-mode (fast/slow) decision-making framework: a large language model (LLM) interprets multimodal user instructions to generate high-level driving intentions, while a deep reinforcement learning (DRL) agent executes low-latency, safety-constrained real-time motion control. Crucially, this work achieves the first synergistic yet decoupled integration of LLMs and RL within a closed-loop decision architecture. User preferences are deeply embedded across the entire perception–decision–execution pipeline. Evaluated in complex urban scenarios, the framework attains a millisecond-scale response latency while ensuring high safety—reducing collision rates significantly and improving alignment between driving behavior and user preferences by 42%.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Autonomous driving has made significant strides through data-driven techniques, achieving robust performance in standardized tasks. However, existing methods frequently overlook user-specific preferences, offering limited scope for interaction and adaptation with users. To address these challenges, we propose a"fast-slow"decision-making framework that integrates a Large Language Model (LLM) for high-level instruction parsing with a Reinforcement Learning (RL) agent for low-level real-time decision. In this dual system, the LLM operates as the"slow"module, translating user directives into structured guidance, while the RL agent functions as the"fast"module, making time-critical maneuvers under stringent latency constraints. By decoupling high-level decision making from rapid control, our framework enables personalized user-centric operation while maintaining robust safety margins. Experimental evaluations across various driving scenarios demonstrate the effectiveness of our method. Compared to baseline algorithms, the proposed architecture not only reduces collision rates but also aligns driving behaviors more closely with user preferences, thereby achieving a human-centric mode. By integrating user guidance at the decision level and refining it with real-time control, our framework bridges the gap between individual passenger needs and the rigor required for safe, reliable driving in complex traffic environments.

Problem

• Existing autonomous driving methods neglect user-specific preferences and interaction
• Need for integrating high-level user guidance with real-time decision-making
• Balancing personalized driving behaviors with safety in complex traffic

Innovation

• Fast-slow architecture combines LLM and RL
• LLM parses high-level user instructions
• RL agent handles low-level real-time decisions