GaussianDWM++: Language-Grounded 3D Gaussian Driving World Model for Unified Scene Understanding, Editing, and Multi-Modal Generation

2026-08-17

AI Summary

This study addresses the limitations of existing driving world models regarding explicit 3D understanding, linguistic reasoning, and controllable 4D editing by proposing the Foundation Feature Gaussian Driving World Model. By introducing a novel foundation feature Gaussian tokenizer to construct open-vocabulary semantic fields, alongside a geometry-aware adapter and KL distribution alignment objective, this framework unifies scene understanding, instruction-based editing, and dynamic manipulation. The approach integrates 3D Gaussian primitives with multimodal feature distillation to effectively resolve fine-grained text-to-3D scene alignment. Extensive experiments demonstrate that the proposed method achieves state-of-the-art performance across multiple benchmarks, including scene understanding, planning reasoning, and controllable 4D generation, thereby significantly advancing the capabilities of autonomous driving world models.

Abstract

Driving World Models (DWMs) have recently advanced rapidly with generative models, yet most existing methods mainly focus on conditional scene generation and lack explicit 3D scene understanding, language-grounded reasoning, and controllable 4D editing capabilities. Moreover, commonly used point cloud, occupancy, or BEV representations make it difficult to achieve fine-grained alignment between textual information and the underlying 3D scene structure. To address these limitations, we propose a foundation-feature Gaussian driving world model that unifies scene understanding, language-grounded reasoning, controllable 4D editing, and multi-modal generation within a single framework. Specifically, we introduce a foundation-feature Gaussian tokenizer that directly distills Qwen/SigLIP visual-language features into 3D Gaussian primitives, building a compact open-vocabulary Gaussian semantic field. We further design a geometry-aware Gaussian adapter that combines importance-aware hierarchical selection with text-conditioned Perceiver-style cross-attention to aggregate dense Gaussian primitives into compact world tokens. To improve representation compatibility, we introduce a KL-based Gaussian--image distribution alignment objective that aligns Gaussian world tokens with foundation image tokens. Based on the aligned Gaussian representation, our framework further supports instruction-controllable scene editing, including weather-conditioned generation and dynamic vehicle manipulation. Extensive experiments on broader driving benchmarks demonstrate that our method achieves state-of-the-art performance across scene understanding, visual grounding, planning-oriented reasoning, and controllable 4D generation tasks. We will release the code and datasets publicly on Github.

Problem

• Driving World Models
• 3D Scene Understanding
• Language-Grounded Reasoning
• Controllable 4D Editing
• Text-3D Alignment

Innovation

• 3D Gaussian Splatting
• Driving World Model
• Vision-Language Features Distillation
• Controllable 4D Editing
• Open-Vocabulary Semantic Field