SAMPro3D: Locating SAM Prompts in 3D for Zero-Shot Instance Segmentation

2023-11-29

AI Summary

This paper addresses zero-shot 3D instance segmentation without training. Leveraging the pre-trained Segment Anything Model (SAM), it generates 2D masks from multi-view RGB-D images and introduces a novel 3D prompting localization mechanism to back-project these masks into point cloud space, ensuring cross-frame view consistency. The method comprises: (1) geometry-guided cross-view prompt filtering; (2) automatic merging of surface parts belonging to the same instance; and (3) 3D mask reconstruction via projection alignment and multi-view fusion. Entirely training-free, it enables fine-grained segmentation. Experiments demonstrate competitive or superior performance against fully supervised methods across multiple benchmarks, with accuracy in certain scenes exceeding human annotation. To support fine-grained evaluation, the authors release ScanNet200-Fine50—a new benchmark dataset with refined category annotations.

Abstract

We introduce SAMPro3D for zero-shot instance segmentation of 3D scenes. Given the 3D point cloud and multiple posed RGB-D frames of 3D scenes, our approach segments 3D instances by applying the pretrained Segment Anything Model (SAM) to 2D frames. Our key idea involves locating SAM prompts in 3D to align their projected pixel prompts across frames, ensuring the view consistency of SAM-predicted masks. Moreover, we suggest selecting prompts from the initial set guided by the information of SAM-predicted masks across all views, which enhances the overall performance. We further propose to consolidate different prompts if they are segmenting different surface parts of the same 3D instance, bringing a more comprehensive segmentation. Notably, our method does not require any additional training. Extensive experiments on diverse benchmarks show that our method achieves comparable or better performance compared to previous zero-shot or fully supervised approaches, and in many cases surpasses human annotations. Furthermore, since our fine-grained predictions often lack annotations in available datasets, we present ScanNet200-Fine50 test data which provides fine-grained annotations on 50 scenes from ScanNet200 dataset. The project page can be accessed at https://mutianxu.github.io/sampro3d/.

Problem

• Zero-shot 3D instance segmentation
• Aligning SAM prompts in 3D
• Enhancing segmentation without additional training

Innovation

• SAM prompts located in 3D
• SAM applied to 2D frames
• No additional training required