UniUGP: Unifying Understanding, Generation, and Planing For End-to-end Autonomous Driving

2025-12-10

AI Summary

To address poor generalization of autonomous driving systems in long-tail scenarios—stemming from insufficient world knowledge and visual causal reasoning capabilities—this paper proposes an integrated understanding-generation-planning framework. It jointly processes multi-frame visual inputs and natural language instructions to simultaneously generate interpretable reasoning chains, physically consistent trajectories, and coherent future video predictions. Methodologically, we introduce a novel unified three-in-one collaborative architecture and a four-stage progressive training paradigm. For the first time, our approach integrates the semantic reasoning capacity of pre-trained vision-language models (VLMs) with the visual dynamic modeling capability of video diffusion models, while augmenting the planning module with chain-of-thought (CoT) reasoning. Leveraging a mixture-of-experts structure and joint fine-tuning on diverse autonomous driving datasets, our method achieves state-of-the-art performance across perception, reasoning, and decision-making tasks, significantly improving generalization and robustness in rare traffic scenarios.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Computer Vision: Visual Reasoning & Symbolic Representations
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Autonomous driving (AD) systems struggle in long-tail scenarios due to limited world knowledge and weak visual dynamic modeling. Existing vision-language-action (VLA)-based methods cannot leverage unlabeled videos for visual causal learning, while world model-based methods lack reasoning capabilities from large language models. In this paper, we construct multiple specialized datasets providing reasoning and planning annotations for complex scenarios. Then, a unified Understanding-Generation-Planning framework, named UniUGP, is proposed to synergize scene reasoning, future video generation, and trajectory planning through a hybrid expert architecture. By integrating pre-trained VLMs and video generation models, UniUGP leverages visual dynamics and semantic reasoning to enhance planning performance. Taking multi-frame observations and language instructions as input, it produces interpretable chain-of-thought reasoning, physically consistent trajectories, and coherent future videos. We introduce a four-stage training strategy that progressively builds these capabilities across multiple existing AD datasets, along with the proposed specialized datasets. Experiments demonstrate state-of-the-art performance in perception, reasoning, and decision-making, with superior generalization to challenging long-tail situations.

Problem

• Addresses autonomous driving limitations in long-tail scenarios
• Unifies reasoning, generation, and planning for end-to-end driving
• Enhances planning with visual dynamics and semantic reasoning

Innovation

• Unified framework synergizes reasoning, generation, planning
• Integrates pre-trained VLMs and video models for dynamics
• Uses hybrid expert architecture with multi-stage training