🤖 AI Summary
Existing world models for autonomous driving struggle to balance robustness and interpretability between pixel-level prediction and latent representations. This work proposes a hybrid world modeling framework that unifies pixel supervision and latent representation learning through a two-stage training strategy: in the pretraining stage, it jointly optimizes video latent feature prediction and pixel-level frame reconstruction; in the fine-tuning stage, it relies solely on latent features to drive an action expert. This approach achieves concurrent improvements in fine-grained spatiotemporal reasoning and noise robustness. Evaluated on NAVSIM v1/v2, the method significantly outperforms baselines based exclusively on pixels or latent representations and establishes a new benchmark for evaluating noise robustness in autonomous driving world models.
📝 Abstract
Vision-Language-Action (VLA) models augmented with world modeling represent a promising paradigm for end-to-end autonomous driving. While pixel-level future prediction enables fine-grained spatiotemporal reasoning, it compromises robustness in noisy driving scenarios. Conversely, latent-based world models alleviate this sensitivity but often incur limited interpretability and representational degradation due to absent pixel-level grounding. To reconcile this trade-off, we propose HyWorldVLA, a hybrid world-VLA framework that unifies pixel-level supervision and latent representation learning. In the pre-training stage, HyWorldVLA predicts video latents encoded by a pre-trained video VAE, while simultaneously reconstructing video frames to provide precise pixel-level grounding. During the subsequent co-fine-tuning phase, the model exclusively predicts latent features, which are fed into an action expert to generate trajectories. Extensive experiments on NAVSIM v1 and v2 benchmarks demonstrate that HyWorldVLA significantly outperforms both pixel-based and latent-based world model baselines. Notably, we present the first comprehensive qualitative and quantitative analysis of world model noise robustness in autonomous driving, establishing a new benchmark for evaluating future architectures.