LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving

2026-04-09

AI Summary

This work addresses the limited generalization of autonomous driving systems in long-tail and open-world scenarios by proposing LMGenDrive, a unified framework that integrates multimodal large language models with generative world models. For the first time, it enables end-to-end generation of future driving videos and control signals directly from multi-view images and natural language instructions. The approach incorporates semantic priors and spatiotemporal imagination capabilities, and employs a three-stage progressive training strategy to enhance stability. Experimental results demonstrate that LMGenDrive significantly outperforms existing methods on challenging closed-loop driving benchmarks, achieving substantial improvements in instruction following, spatiotemporal understanding, and robustness in rare and complex driving scenarios.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation
• Computer Vision: Large Vision Models

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recent years have seen remarkable progress in autonomous driving, yet generalization to long-tail and open-world scenarios remains a major bottleneck for large-scale deployment. To address this challenge, some works use LLMs and VLMs for vision-language understanding and reasoning, enabling vehicles to interpret rare and safety-critical situations when generating actions. Others study generative world models to capture the spatio-temporal evolution of driving scenes, allowing agents to imagine possible futures before acting. Inspired by human intelligence, which unifies understanding and imagination, we explore a unified model for autonomous driving. We present LMGenDrive, the first framework that combines LLM-based multimodal understanding with generative world models for end-to-end closed-loop driving. Given multi-view camera inputs and natural-language instructions, LMGenDrive generates both future driving videos and control signals. This design provides complementary benefits: video prediction improves spatio-temporal scene modeling, while the LLM contributes strong semantic priors and instruction grounding from large-scale pretraining. We further propose a progressive three-stage training strategy, from vision pretraining to multi-step long-horizon driving, to improve stability and performance. LMGenDrive supports both low-latency online planning and autoregressive offline video generation. Experiments show that it significantly outperforms prior methods on challenging closed-loop benchmarks, with clear gains in instruction following, spatio-temporal understanding, and robustness to rare scenarios. These results suggest that unifying multimodal understanding and generation is a promising direction for more generalizable and robust embodied decision-making systems.

Problem

• autonomous driving
• generalization
• open-world scenarios
• long-tail scenarios
• embodied decision-making

Innovation

• multimodal understanding
• generative world model
• end-to-end driving
• large language model
• closed-loop autonomy