ViCo3D: Empowering LiDAR-based Collaborative 3D Object Detection with Vision Foundation Models

2026-07-14

AI Summary

This work addresses the limitation of existing LiDAR-based cooperative 3D perception methods, which lack generic semantic priors due to training bird’s-eye-view (BEV) features from scratch, thereby hindering effective cross-agent collaboration. To overcome this, the study presents the first successful adaptation of large-scale pretrained vision foundation models—such as DINOv2—to LiDAR-based cooperative 3D object detection. The approach projects LiDAR point clouds into three-channel BEV images to extract rich semantic features, introduces a multi-scale BEV fusion module, and devises an ego-centric cross-agent feature aggregation mechanism. Evaluated on DAIR-V2X and V2XSet benchmarks, the method achieves state-of-the-art performance, with collaborative gains on DAIR-V2X reaching up to 1.8 times those of current approaches.

Abstract

LiDAR-based collaborative 3D perception in Vehicle-to-Everything (V2X) systems typically relies on fusing bird's-eye-view (BEV) features across agents. However, current BEV representations, typically extracted by LiDAR backbones trained from scratch, are geometry-dominated and lack general semantic priors, inherently limiting the efficacy of feature-level collaboration. Meanwhile, vision foundation models (VFMs) pretrained on large-scale image data have demonstrated strong capability in learning general-purpose and informative visual representations for 2D tasks, and have the potential to enhance agent-wise LiDAR BEV representations for collaboration. Despite this potential, adapting VFMs to LiDAR-based 3D detection remains challenging due to the substantial image-point cloud modality gap. To bridge this gap, we propose ViCo3D, a collaborative 3D object detection framework powered by VFMs. Specifically, ViCo3D adapts VFMs to LiDAR-based collaborative perception from three aspects: First, ViCo3D projects point clouds onto the BEV plane as three-channel images, enabling DINOv2 to extract BEV-space visual features from LiDAR inputs. Besides, to effectively integrate these DINOv2-derived features with LiDAR geometric features, ViCo3D introduces a multi-scale BEV fusion module within the single-agent encoder. In addition, ViCo3D adopts an ego-centric cross-agent fusion strategy to aggregate complementary information from multiple agents. Experiments on DAIR-V2X and V2XSet demonstrate that ViCo3D achieves state-of-the-art 3D detection performance. Remarkably, it delivers up to 1.8x greater collaborative gains than prior methods on DAIR-V2X. The code will be made public available for future investigation.

Problem

• LiDAR-based 3D object detection
• collaborative perception
• bird's-eye-view (BEV) representation
• semantic priors
• V2X systems

Innovation

• Vision Foundation Models
• LiDAR-based 3D Object Detection
• Collaborative Perception
• Bird's-Eye-View Fusion
• V2X