Large Multimodal Models for Embodied Intelligent Driving: The Next Frontier in Self-Driving?

2026-01-13

AI Summary

This work addresses the limitations of traditional modular autonomous driving systems in achieving continuous environmental understanding, logical reasoning, and joint decision-making within closed-loop settings in open-world scenarios. To overcome these challenges, the paper proposes the first hybrid decision-making framework that synergistically integrates large multimodal models (LMMs) with deep reinforcement learning (DRL), featuring dual semantic-strategy driving mechanisms. In this architecture, LMMs provide high-level semantic comprehension and cognitive representations, while DRL enables real-time policy optimization and continual learning. By moving beyond sole reliance on LMMs, the framework significantly enhances decision intelligence and environmental adaptability in lane-change planning tasks, thereby advancing the development of embodied intelligent driving systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Multiagent Systems: Multiagent Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

The advent of Large Multimodal Models (LMMs) offers a promising technology to tackle the limitations of modular design in autonomous driving, which often falters in open-world scenarios requiring sustained environmental understanding and logical reasoning. Besides, embodied artificial intelligence facilitates policy optimization through closed-loop interactions to achieve the continuous learning capability, thereby advancing autonomous driving toward embodied intelligent (El) driving. However, such capability will be constrained by relying solely on LMMs to enhance EI driving without joint decision-making. This article introduces a novel semantics and policy dual-driven hybrid decision framework to tackle this challenge, ensuring continuous learning and joint decision. The framework merges LMMs for semantic understanding and cognitive representation, and deep reinforcement learning (DRL) for real-time policy optimization. We start by introducing the foundational principles of EI driving and LMMs. Moreover, we examine the emerging opportunities this framework enables, encompassing potential benefits and representative use cases. A case study is conducted experimentally to validate the performance superiority of our framework in completing lane-change planning task. Finally, several future research directions to empower EI driving are identified to guide subsequent work.

Problem

• Embodied Intelligence
• Large Multimodal Models
• Autonomous Driving
• Joint Decision-Making
• Continuous Learning

Innovation

• Large Multimodal Models
• Embodied Intelligence
• Hybrid Decision Framework
• Deep Reinforcement Learning
• Autonomous Driving