🤖 AI Summary
This work addresses the triple challenge of scarce training scenes, limited point-level annotations, and absent corresponding 2D images in 3D point cloud semantic segmentation by proposing the PLOVIS framework. PLOVIS is the first method to jointly tackle these issues without relying on real 2D images; instead, it leverages open-vocabulary image segmentation models to generate high-quality 2D pseudo-labels from 3D point clouds. To enhance training efficiency, the framework incorporates a two-stage pseudo-label filtering mechanism and a class-balanced memory bank. Evaluated on four standard benchmarks—ScanNet, S3DIS, Toronto3D, and Semantic3D—PLOVIS achieves state-of-the-art performance using only dozens of scenes with fewer than 100 annotated points per scene, significantly outperforming existing approaches.
📝 Abstract
Semantic segmentation of 3D point cloud scenes is a crucial task for various applications. In real-world scenarios, training segmentation models often faces three concurrent forms of data insufficiency: scarcity of training scenes, scarcity of point-level annotations, and absence of 2D image sequences from which point clouds were reconstructed. Existing data-efficient algorithms typically address only one or two of these challenges, leaving the joint treatment of all three unexplored. This paper proposes a data-efficient training framework specifically designed to address the three forms of data insufficiency. Our proposed algorithm, called Point pseudo-Labeling via Open-Vocabulary Image Segmentation (PLOVIS), leverages an Open-Vocabulary Image Segmentation (OVIS) model as a pseudo label generator to compensate for the lack of training data. PLOVIS creates 2D images for pseudo-labeling directly from training 3D point clouds, eliminating the need for 2D image sequences. To mitigate the inherent noise and class imbalance in pseudo labels, we introduce a two-stage filtering of pseudo labels combined with a class-balanced memory bank for effective training. The two-stage filtering mechanism first removes low-confidence pseudo labels, then discards likely incorrect pseudo labels, thereby enhancing the quality of pseudo labels. Experiments on four benchmark datasets, i.e., ScanNet, S3DIS, Toronto3D, and Semantic3D, under realistic data-scarce conditions (a few tens of training 3D scenes, each annotated with only <100 3D points) demonstrate that PLOVIS consistently outperforms existing methods including standard fine-tuning strategies and state-of-the-art weakly supervised learning algorithms. Code will be made publicly available.