SAM-guided Pseudo Label Enhancement for Multi-modal 3D Semantic Segmentation

2025-02-02

AI Summary

To address the sparsity and unreliability of pseudo-labels in cross-domain multimodal 3D semantic segmentation, this paper pioneers the adaptation of the Segment Anything Model (SAM) to 3D domain adaptation. We propose a Geometry-Aware Progressive Propagation (GAPP) mechanism that leverages SAM’s robust 2D image priors to generate and expand high-quality pseudo-labels under 2D–3D alignment uncertainty, via cross-modal point cloud–image projection and geometric constraints. Furthermore, we enhance label consistency through multimodal feature alignment and majority-voting classification. Evaluated on challenging cross-domain benchmarks—including SemanticKITTI→nuScenes—our method significantly outperforms state-of-the-art approaches, achieving a +4.2% average mIoU improvement. This demonstrates the effective transferability of 2D visual priors to weakly supervised 3D learning, establishing a novel paradigm for leveraging foundation models in 3D domain adaptation.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: 3D Computer Vision
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Multi-modal 3D semantic segmentation is vital for applications such as autonomous driving and virtual reality (VR). To effectively deploy these models in real-world scenarios, it is essential to employ cross-domain adaptation techniques that bridge the gap between training data and real-world data. Recently, self-training with pseudo-labels has emerged as a predominant method for cross-domain adaptation in multi-modal 3D semantic segmentation. However, generating reliable pseudo-labels necessitates stringent constraints, which often result in sparse pseudo-labels after pruning. This sparsity can potentially hinder performance improvement during the adaptation process. We propose an image-guided pseudo-label enhancement approach that leverages the complementary 2D prior knowledge from the Segment Anything Model (SAM) to introduce more reliable pseudo-labels, thereby boosting domain adaptation performance. Specifically, given a 3D point cloud and the SAM masks from its paired image data, we collect all 3D points covered by each SAM mask that potentially belong to the same object. Then our method refines the pseudo-labels within each SAM mask in two steps. First, we determine the class label for each mask using majority voting and employ various constraints to filter out unreliable mask labels. Next, we introduce Geometry-Aware Progressive Propagation (GAPP) which propagates the mask label to all 3D points within the SAM mask while avoiding outliers caused by 2D-3D misalignment. Experiments conducted across multiple datasets and domain adaptation scenarios demonstrate that our proposed method significantly increases the quantity of high-quality pseudo-labels and enhances the adaptation performance over baseline methods.

Problem

• Pseudo-label Quality Improvement
• 3D Semantic Segmentation
• Adaptability for Autonomous Driving and VR

Innovation

• Image-guided Pseudo Label Refinement
• 3D Point Cloud Enhancement
• Semantic Segmentation Accuracy