CausalDrive: Real-time Causal World Models for Autonomous Driving

2026-06-13

AI Summary

Existing world models for autonomous driving lack interactivity and real-time capability, hindering closed-loop policy learning and controllable causal reasoning. This work proposes the first text-driven, real-time controllable driving world renderer that, given only an initial frame, ego-vehicle trajectory, and textual prompts, enforces endogenous prediction of social interactions by excluding future non-player character (NPC) layouts and enables counterfactual interventions. The core innovation is a Context-Forced Diffusion-Matching Distillation (DMD) architecture that integrates continuous flow matching with self-correcting distillation to effectively mitigate autoregressive covariate shift and enhance computational efficiency. The system achieves real-time performance at 12 FPS and significantly reduces collision artifacts in generative closed-loop evaluation, reinforcement learning fine-tuning, and human-in-the-loop simulation, thereby strengthening policy robustness in realistic interactive scenarios.

Abstract

World models have emerged as a promising paradigm for scaling autonomous driving (AD) data, yet existing video generative models fall short as interactive simulators. Layout-conditioned renderers rely on "oracle" future trajectories of all background agents, rendering them strictly non-reactive. Conversely, pure action-conditioned predictors lack semantic control over complex interactions and suffer from prohibitive diffusion latencies, hindering closed-loop policy learning. To bridge this gap, we present CausalDrive, a controllable, real-time foundation driving world renderer. CausalDrive operates solely on the initial front-view frame, the ego-vehicle's trajectory, and a macroscopic text prompt. By excluding future NPC layouts, we compel the model to intrinsically predict causal interactions, enabling text-driven control over Driving Sociology, allowing users to dynamically orchestrate diverse counterfactual reactions to identical ego-actions. To overcome the efficiency bottleneck and address the covariate shift in autoregressive generation, we propose a novel Context-Forced DMD architecture. This combines continuous flow-matching with a self-correcting distillation objective, achieving interactive speeds of 12 FPS. This breakthrough transforms the passive video generator into a playable neural simulator. We demonstrate its versatility across three downstream applications: (1) generative closed-loop evaluation with significantly mitigated collision artifacts, (2) large-scale Reinforcement Learning (RL) post-training driven by a Video2Reward module, and (3) real-time human-in-the-loop simulation. Extensive experiments validate that policies trained within CausalDrive's reactive scenarios exhibit superior interaction capabilities in the real world.

Problem

• world models
• autonomous driving
• interactive simulation
• causal reasoning
• real-time generation

Innovation

• Causal World Models
• Real-time Simulation
• Text-driven Control
• Context-Forced DMD
• Autonomous Driving