M$^\text{4}$World: A Multi-view Multimodal Driving World Model for Interactive Object Manipulation and Minute-long Streaming

2026-07-15

AI Summary

Existing methods for generating driving worlds suffer from limited object-level controllability and long-term temporal consistency. This work proposes a multi-view, multi-modal generative driving world model that enables interactive control over object layout and appearance through fine-grained conditional interfaces. The model employs a four-step denoising causal diffusion framework capable of generating minute-long, high-fidelity video streams. By integrating multi-stage training, few-shot visual reference customization, and vision-language model (VLM)-driven automated evaluation, the approach significantly enhances controllability, photorealism, and temporal stability in generating and editing long-tail driving scenarios.

Abstract

Driving-world generation has emerged as a core capability for scalable autonomous-driving simulation, yet existing methods remain limited in object-level controllability and long-horizon stability. We present M$^\text{4}$World, a Multi-view and Multimodal generative driving world model that synthesizes future surround-view video streams and synchronized LiDAR scans while supporting interactive object Manipulation and stable Minute-long streaming. Fine-grained object manipulation is realized through a flexible conditioning interface that supports explicit control over both the spatial layout and visual appearance of individual objects. Stable minute-long streaming, on the other hand, is achieved through a multi-stage training framework that enables online causal generation in only four denoising steps while maintaining coherent world dynamics throughout extended rollouts. Building on these components, we introduce an efficient few-clip post-training as well as a suite of visual reference-conditioned generation models, preserving general generation ability while allowing rare-case customization for long-tail controllability. To assess controllability beyond realism, we further introduce an automated VLM-based judging pipeline that evaluates scene-level condition adherence, view-wise object controllability, and cross-view object consistency. Comprehensive experiments show that M$^\text{4}$World consistently delivers high generation quality, precise controllability, and stable minute-long streaming. Together with downstream long-tail augmentation and scene editing, these results demonstrate the potential of M$^\text{4}$World for controllable, scalable driving simulation.

Problem

• driving-world generation
• object-level controllability
• long-horizon stability
• autonomous-driving simulation

Innovation

• interactive object manipulation
• minute-long streaming
• multi-view multimodal generation
• causal world modeling
• controllable driving simulation