OmniV2X: A Generative Foundation Planner for Efficient End-to-End Cooperative Driving

2026-06-19

AI Summary

This work addresses key challenges in existing cooperative driving approaches—namely high computational costs from dense 3D perception, sensitivity to scarce collaboration data, and incompatibility with standard communication protocols—by introducing generative foundation models into V2X cooperative driving for the first time. The proposed method employs an end-to-end trained multimodal multi-agent sequential planner that implicitly fuses heterogeneous inputs and generates driving decisions through trajectory-generation loss. By integrating cross-attention mechanisms with large-scale single-agent pretraining, it efficiently adapts to cooperative scenarios using only lightweight, protocol-compliant V2X tokens. Evaluated on DAIR-V2X-Seq, the approach surpasses current end-to-end methods using merely 10% of fine-tuning data and 1% of communication bandwidth, achieving significant gains in performance, computational efficiency, and robustness for real-world deployment.

Abstract

We present OmniV2X, a generative foundation model for vehicle-to-everything (V2X) cooperative driving. The model directly interprets independent context sequences comprising multi-modal and multi-agent observations. The new design mitigates the computational cost of dense 3D perception, the vulnerability to data scarcity in cooperative scenarios, and the poor compliance with standardized messaging in existing methods that fuse multi-modal inputs into a shared representation. For training, we present an end-to-end supervised pipeline using a downstream trajectory generation loss, in which a high-capacity generative sequence planner implicitly learns to steer the model and leverage multi-modal inputs via cross-attention injection. As a foundation model, we demonstrate that OmniV2X pre-trained on large-scale single-agent planning datasets can efficiently adapt to cooperative environments by integrating the conditioning context with lightweight, standard-compliant V2X tokens. Evaluated on the DAIR-V2X-Seq dataset, OmniV2X outperforms existing end-to-end cooperative driving baselines, achieving state-of-the-art performance with less than 10% of the fine-tune V2X dataset and less than 1% of the communication bandwidth. We conduct comprehensive evaluations to demonstrate its computational efficiency and robustness under real-world constraints.

Problem

• cooperative driving
• V2X
• multi-modal perception
• data scarcity
• standardized messaging

Innovation

• generative foundation model
• V2X cooperative driving
• cross-attention injection
• standard-compliant communication
• end-to-end trajectory planning