Self-Supervised Pre-training with Combined Datasets for 3D Perception in Autonomous Driving

2025-04-17

AI Summary

To address the reliance of 3D perception models on large-scale annotated data and their poor cross-dataset generalization in autonomous driving, this paper proposes the first cross-dataset self-supervised pretraining framework tailored for 3D perception. The framework leverages heterogeneous, unlabeled multi-source data—including images, point clouds, and BEV sequences—and introduces a prompt adapter-based domain bias correction mechanism to enable efficient domain adaptation alongside backbone network co-optimization. It supports unified pretraining across multiple downstream tasks, including 3D detection, BEV segmentation, 3D tracking, and occupancy prediction. Extensive experiments demonstrate consistent performance gains with increasing scale of unlabeled data, achieving significant improvements over state-of-the-art self-supervised and cross-domain methods on benchmarks such as nuScenes. The code will be made publicly available.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Unsupervised & Self-Supervised Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

The significant achievements of pre-trained models leveraging large volumes of data in the field of NLP and 2D vision inspire us to explore the potential of extensive data pre-training for 3D perception in autonomous driving. Toward this goal, this paper proposes to utilize massive unlabeled data from heterogeneous datasets to pre-train 3D perception models. We introduce a self-supervised pre-training framework that learns effective 3D representations from scratch on unlabeled data, combined with a prompt adapter based domain adaptation strategy to reduce dataset bias. The approach significantly improves model performance on downstream tasks such as 3D object detection, BEV segmentation, 3D object tracking, and occupancy prediction, and shows steady performance increase as the training data volume scales up, demonstrating the potential of continually benefit 3D perception models for autonomous driving. We will release the source code to inspire further investigations in the community.

Problem

• Explores large-scale pre-training for 3D autonomous driving perception
• Proposes self-supervised learning to reduce dataset bias in 3D models
• Improves downstream tasks like 3D detection and BEV segmentation

Innovation

• Self-supervised pre-training for 3D perception
• Combined heterogeneous datasets for training
• Prompt adapter for domain adaptation