How Can Driving World Models Do Counterfactual Prediction?

2026-08-11

AI Summary

This work addresses a fundamental mismatch in existing driving world models that employ direct action-conditioned generation for counterfactual prediction, which often fails to preserve the true subsequent states from original logs, leading to predictions that deviate from genuine counterfactuals. Building upon the causal inference framework of abduction–intervention–prediction, this study formally characterizes this discrepancy and introduces the first controllable simulation benchmark containing paired real and aligned counterfactual trajectories. Leveraging this benchmark, the authors propose a training-free post-hoc correction procedure that refines predictions from frozen world models by conditioning on observational evidence. Experiments demonstrate that this approach significantly improves counterfactual recovery rates and effectively reduces perceptual distance to ground-truth counterfactuals across two representative world models.

Abstract

Driving world models are often interpreted as counterfactual simulators for observed driving episodes: given a factual driving log, they are asked what would have happened under an alternative ego action. In this paper, we identify a fundamental mismatch between this goal and direct action-conditioned prediction. The direct prediction uses the shared history and the alternative action but not the factual continuation observed after that history. It can therefore generate a plausible future without preserving what actually happened in this episode. We formalize this gap using the causal recipe of abduction, action, and prediction and study it in a setting with a short time horizon, where the alternative ego action does not alter how surrounding agents evolve. To make the gap measurable, we construct a controlled simulation benchmark with factual outcomes and matched counterfactual outcomes. Across two representative world models, direct predictions fail to match the counterfactual ground truth, supporting our analysis. As a constructive check of this analysis, we introduce a deliberately simple, training-free pipeline that moves observed evidence into the counterfactual view and lets the frozen model complete what remains unknown. Even this simple construction raises the overall recovered fraction substantially and reduces perceptual distance to the matched counterfactual on both models. We hope this work draws attention to this gap and motivates better counterfactual prediction methods for driving world models.

Problem

• counterfactual prediction
• driving world models
• action-conditioned prediction
• causal reasoning
• simulation benchmark

Innovation

• counterfactual prediction
• driving world models
• causal reasoning
• abduction-action-prediction
• simulation benchmark