Enhancing Autonomous Driving Systems with On-Board Deployed Large Language Models

2025-04-15

AI Summary

To address the limited generalization capability of autonomous driving systems in edge cases and their inability to comprehensively cover long-tail risks, this paper proposes a dual-track decision-making framework integrating knowledge-driven and data-driven paradigms. We innovatively design a synergistic on-vehicle architecture coupling a large language model (LLM) with model predictive control (MPC): DecisionxLLM enables natural-language-driven behavioral decision-making, while MPCxLLM dynamically adapts control parameters under safety constraints to enhance environmental adaptability. Technically, we integrate retrieval-augmented generation (RAG), LoRA fine-tuning, 4-bit quantization, and local inference to enable real-time deployment on low-compute platforms. Experiments demonstrate a 10.45% improvement in reasoning accuracy, a 52.2% gain in control adaptability, and a 10.5× speedup in token throughput (tokens/s). To our knowledge, this is the first work achieving high-robustness, high-real-time closed-loop LLM-MPC control.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Neural Networks (NNs) trained through supervised learning struggle with managing edge-case scenarios common in real-world driving due to the intractability of exhaustive datasets covering all edge-cases, making knowledge-driven approaches, akin to how humans intuitively detect unexpected driving behavior, a suitable complement to data-driven methods. This work proposes a hybrid architecture combining low-level Model Predictive Controller (MPC) with locally deployed Large Language Models (LLMs) to enhance decision-making and Human Machine Interaction (HMI). The DecisionxLLM module evaluates robotic state information against natural language instructions to ensure adherence to desired driving behavior. The MPCxLLM module then adjusts MPC parameters based on LLM-generated insights, achieving control adaptability while preserving the safety and constraint guarantees of traditional MPC systems. Further, to enable efficient on-board deployment and to eliminate dependency on cloud connectivity, we shift processing to the on-board computing platform: We propose an approach that exploits Retrieval Augmented Generation (RAG), Low Rank Adaptation (LoRA) fine-tuning, and quantization. Experimental results demonstrate that these enhancements yield significant improvements in reasoning accuracy by up to 10.45%, control adaptability by as much as 52.2%, and up to 10.5x increase in computational efficiency (tokens/s), validating the proposed framework's practicality for real-time deployment even on down-scaled robotic platforms. This work bridges high-level decision-making with low-level control adaptability, offering a synergistic framework for knowledge-driven and adaptive Autonomous Driving Systems (ADS).

Problem

• Improving edge-case handling in autonomous driving using LLMs
• Combining MPC and LLMs for adaptive control and safety
• Enabling efficient on-board LLM deployment via RAG and LoRA

Innovation

• Hybrid MPC and LLM architecture for driving
• On-board RAG, LoRA, quantization for efficiency
• LLM enhances decision-making and control adaptability