AdaDrive: Self-Adaptive Slow-Fast System for Language-Grounded Autonomous Driving

2025-11-09

AI Summary

To address the challenge of balancing high-level reasoning capability and real-time performance when integrating large language models (LLMs) into autonomous driving systems, this paper proposes AdaDrive—a novel adaptive slow-fast collaborative framework. Methodologically, AdaDrive introduces an adaptive activation loss function and a continuous scaling fusion strategy to dynamically determine whether and to what extent an LLM should be invoked, triggering LLM inference only in semantically complex or safety-critical scenarios—thereby avoiding fixed-frequency or excessive invocation. The framework fuses multimodal sensory inputs with conventional planners and employs contrastive learning to enable semantic-guided decision-making. Evaluated on a language-driven autonomous driving benchmark, AdaDrive achieves state-of-the-art performance with significantly reduced computational overhead. It is the first framework to realize context-aware, controllable LLM-planner collaboration while guaranteeing real-time execution.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Effectively integrating Large Language Models (LLMs) into autonomous driving requires a balance between leveraging high-level reasoning and maintaining real-time efficiency. Existing approaches either activate LLMs too frequently, causing excessive computational overhead, or use fixed schedules, failing to adapt to dynamic driving conditions. To address these challenges, we propose AdaDrive, an adaptively collaborative slow-fast framework that optimally determines when and how LLMs contribute to decision-making. (1) When to activate the LLM: AdaDrive employs a novel adaptive activation loss that dynamically determines LLM invocation based on a comparative learning mechanism, ensuring activation only in complex or critical scenarios. (2) How to integrate LLM assistance: Instead of rigid binary activation, AdaDrive introduces an adaptive fusion strategy that modulates a continuous, scaled LLM influence based on scene complexity and prediction confidence, ensuring seamless collaboration with conventional planners. Through these strategies, AdaDrive provides a flexible, context-aware framework that maximizes decision accuracy without compromising real-time performance. Extensive experiments on language-grounded autonomous driving benchmarks demonstrate that AdaDrive state-of-the-art performance in terms of both driving accuracy and computational efficiency. Code is available at https://github.com/ReaFly/AdaDrive.

Problem

• Balancing LLM reasoning with real-time efficiency in autonomous driving
• Adapting LLM activation to dynamic driving conditions instead of fixed schedules
• Modulating continuous LLM influence based on scene complexity and confidence

Innovation

• Adaptive activation loss for dynamic LLM invocation
• Continuous scaled LLM influence via adaptive fusion
• Slow-fast framework balancing reasoning and efficiency