Generative Planning with 3D-vision Language Pre-training for End-to-End Autonomous Driving

2025-01-15

AI Summary

To address three core challenges in end-to-end autonomous driving—weak visual understanding, difficult decision reasoning, and poor scene generalization—this paper proposes GPVL, a unified vision-language framework. GPVL introduces the first 3D bird’s-eye-view (BEV) vision-language joint pretraining paradigm: multi-view images are encoded into BEV features and explicitly aligned with language representations. A cross-modal language model is then designed to autoregressively generate both high-level driving instructions and low-level fine-grained trajectories. By unifying perception, comprehension, and planning, GPVL enables full-stack autonomous driving conditioned on natural language commands. Evaluated on nuScenes, GPVL significantly outperforms state-of-the-art methods across key metrics, demonstrates strong generalization to unseen scenarios, and exhibits promising potential for real-time deployment.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Autonomous driving is a challenging task that requires perceiving and understanding the surrounding environment for safe trajectory planning. While existing vision-based end-to-end models have achieved promising results, these methods are still facing the challenges of vision understanding, decision reasoning and scene generalization. To solve these issues, a generative planning with 3D-vision language pre-training model named GPVL is proposed for end-to-end autonomous driving. The proposed paradigm has two significant aspects. On one hand, a 3D-vision language pre-training module is designed to bridge the gap between visual perception and linguistic understanding in the bird's eye view. On the other hand, a cross-modal language model is introduced to generate holistic driving decisions and fine-grained trajectories with perception and navigation information in an auto-regressive manner. Experiments on the challenging nuScenes dataset demonstrate that the proposed scheme achieves excellent performances compared with state-of-the-art methods. Besides, the proposed GPVL presents strong generalization ability and real-time potential when handling high-level commands in various scenarios. It is believed that the effective, robust and efficient performance of GPVL is crucial for the practical application of future autonomous driving systems. Code is available at https://github.com/ltp1995/GPVL

Problem

• Autonomous Driving
• Environmental Understanding
• Adaptive Decision-making

Innovation

• GPVL Model
• 3D Vision and Language Pre-training
• Advanced Driving Decision-making