OmniDrive: An LLM-Choreographed Multi-Agent World Model with Unified Latent Co-Compression for Multi-View Driving Video Generation

2026-06-16

AI Summary

This work addresses the challenges of fusing heterogeneous control signals and ensuring global 3D geometric consistency across multi-view videos in generative world models for autonomous driving. The authors propose a multi-agent collaborative generation framework grounded in a shared symbolic intermediate language. By reframing controllable multi-view video synthesis as a “choreography” task in latent space, three multimodal agents jointly produce position-aware token sequences, which are co-compressed with multi-view videos under a unified 3D VAE architecture that embeds cross-view geometric constraints. The method introduces an LLM-driven multi-agent collaboration mechanism and a token-level aligned joint representation unifying language, geometry, and pixels. Evaluated on nuScenes, it achieves a state-of-the-art BEV mAP of 21.6 and an FVD of 45.7; notably, a detector trained solely on synthetic data improves NDS by 2.4 on the real-world validation set.

Abstract

Generative world models for autonomous driving face two unresolved tensions: heterogeneous control injection, where free-form language, HD-maps, trajectories, and camera poses reside in incompatible representational spaces, and post-hoc cross-view fusion, where per-camera latents fail to encode global 3-D geometry. We trace both to a single root cause: the absence of a shared symbolic interlingua aligning language, geometry, and pixels at the latent-token level. We present DRIVE-CHOREO, an LLM-choreographed multi-agent world model that recasts controllable multi-view video generation as latent choreography. Three Qwen2.5-VL agents - a Director parsing user intent into a structured WorldScript, a Cartographer grounding it into spatially-anchored layout tokens, and an Auditor feeding cross-view critiques back as auxiliary supervision - jointly author a single position-aware token sequence. This sequence is co-compressed with the multi-view video via a view-time permutation that enforces inter-camera geometry within the convolutional receptive field of a 3-D VAE. On nuScenes, DRIVE-CHOREO sets new state-of-the-art multi-view consistency and BEV mAP (21.6) with competitive FVD (45.7); a detector trained purely on our synthetic data gains +2.4 NDS on the real validation split, validating downstream utility.

Problem

• heterogeneous control injection
• cross-view fusion
• world model
• multi-view driving video
• 3D geometry

Innovation

• latent co-compression
• LLM-choreographed multi-agent
• unified interlingua
• multi-view video generation
• 3D-aware world model