UNIVERSE: Unified Video Action Models for Autonomous Driving with Flexible Mask-Modulated Modality Generation

2026-07-06

AI Summary

Existing autonomous driving motion models struggle to effectively transfer scene dynamics knowledge from video prediction to trajectory generation, limiting their cross-domain generalization. This work proposes UNIVERSE, the first unified architecture based on a single Diffusion Transformer (DiT), which jointly trains future video latent representations and ego-vehicle trajectory tokens under shared parameters. It leverages mask-modulated diffusion and modality-decoupled visibility masks to enable direct video-supervised guidance for trajectory denoising. The framework supports three inference modes: trajectory-only, video-only, or joint. Evaluated on NAVSIM, UNIVERSE achieves a PDMS score of 91.0, substantially outperforming dual-DiT baselines. It also demonstrates strong zero-shot transfer to nuScenes and Bench2Drive without fine-tuning, while trajectory-only inference runs 4.3× faster.

Abstract

World Action Models (WAMs) have shown strong potential for improving action generalization in autonomous driving by using future video prediction as dense supervision for scene dynamics and temporal causality. However, it remains unclear which architecture better transfers video-modeling benefits to trajectory generation. Existing cascaded or dual-DiT designs separate video imagination from action prediction, weakening the transfer of video-learned world dynamics to the trajectory branch: the action model may still overfit dataset-specific driving priors, while the video model only indirectly regularizes planning. We propose UNIVERSE, a unified video-action model built upon a single mask-modulated Diffusion Transformer. By co-training future video latents and ego-trajectory tokens within shared generative parameters, UNIVERSE allows dense video supervision to directly shape trajectory denoising, leading to stronger cross-domain action generalization. To ensure causal validity and efficient deployment, we introduce a Modality-Decoupling Visibility Mask, which shares historical context across modalities while blocking mutual attention between future video and trajectory tokens. This prevents future-target leakage and enables trajectory-only inference by removing future-video denoising at test time, achieving a $4.3\times$ speedup over joint video-action rollout while maintaining comparable planning accuracy. The same model also supports video-only and joint video-action rollouts. Experiments show that UNIVERSE achieves 91.0 PDMS on NAVSIM (vs. 89.6 for the Two-DiT variant), and demonstrates strong zero-shot transfer to nuScenes and Bench2Drive without fine-tuning, while ablations confirm the importance of single-DiT unification, video co-training, and mask-based modality decoupling.

Problem

• autonomous driving
• video prediction
• trajectory generation
• action generalization
• world models

Innovation

• Unified Video-Action Model
• Mask-Modulated Diffusion Transformer
• Modality-Decoupling Visibility Mask
• Cross-Domain Action Generalization
• Dense Video Supervision