V2XCrafter: Learning to Generate Driving Scene Across Agents

2026-05-28

AI Summary

Existing cooperative driving systems are hindered by the scarcity of real-world V2X data and limited cross-scenario generalization, while conventional image generation methods struggle to simultaneously achieve high fidelity and cross-view physical consistency in multi-agent settings. To address these challenges, this work proposes the first controllable image generation framework tailored for multi-agent collaborative driving. It introduces a progressive multi-agent diffusion model augmented with a novel cross-agent attention mechanism—leveraging a collaborative view graph and learnable joint object representations—to effectively resolve attribute inconsistencies under dynamic viewpoints. The proposed approach enables high-fidelity, cross-view consistent street scene synthesis and significantly enhances downstream collaborative 3D object detection performance.

Abstract

Collaborative driving systems leverage vehicle-to-everything (V2X) communication for multi-agent collaborative perception to enhance driving safety, yet they remain constrained by scarce annotated real-world V2X driving datasets and limited generalization across diverse driving conditions. While image generation technology offers a feasible solution for data augmentation, existing methods tailored for single-vehicle multi-view scenarios face two fundamental challenges in multi-agent driving settings: (1) the expansion of the learning objective degrades generation quality, and (2) the highly dynamic variations across agents hinder the modeling of consistency for physical attributes (e.g., color, category) in jointly observed objects. To bridge this gap, we propose V2XCrafter, the first framework for generating controllable and realistic collaborative driving scene across agents' camera views. For effective learning, we develop a progressive multi-agent diffusion model based on a single-agent backbone, using neighboring agents' latent states as reference signals to progressively guide the single-to-multi diffusion. To address cross-vehicle inconsistency, we propose a cross-agent attention module that leverages a collaboration view graph and learnable jointly observed object representation to model the dynamic cross-agent camera view relationships. Experiments have shown that V2XCrafter can generate high-fidelity and controllable street views with consistency across agents, thereby effectively enhancing the downstream collaborative 3D object detection tasks.

Problem

• V2X
• multi-agent
• image generation
• consistency
• collaborative driving

Innovation

• multi-agent diffusion
• cross-agent attention
• V2X scene generation
• collaborative perception
• controllable image synthesis