Few-Click-Driven Interactive 3D Segmentation with Semantic Embedding

2026-05-09

AI Summary

Existing 3D interactive segmentation methods are often limited to single-object processing, rely on 2D priors, or require multiple iterative refinements, resulting in restricted generalization. This work proposes the first framework that directly supports multi-click joint reasoning on sparse point clouds. By integrating a point Transformer encoder with a hierarchical mask decoder, along with learnable semantic embeddings and a multi-level crop-and-fuse mechanism, the model simultaneously optimizes spatial masks and semantic predictions in a single forward pass. Eliminating sequential refinement and 2D dependencies, the approach significantly enhances both efficiency and generalization: under a single-click setting, it improves mIoU by over 20% compared to strong baselines, achieves 8–10% higher performance across datasets, and accurately segments most instances with just one click.

Abstract

Interactive segmentation allows efficient label generation by leveraging user-provided clicks to progressively refine predictions, which is critical when fully supervised labels are costly or generalization to unseen classes is needed. Existing 3D interactive methods are limited: most operate sequentially, predicting only one object per iteration with binary masks, while several recent approaches depend on 2D foundation models and camera alignment to bridge the 2D-3D gap. To address these limitations, we propose a novel interactive segmentation framework that operates directly on sparse, randomly downsampled 3D points and processes multiple object clicks in a single forward pass. Our framework consists of a point Transformer-based encoder and a hierarchical mask decoder, which integrates multi-level crop-and-merge operations conditioned on learnable semantic embeddings. Unlike prior interactive approaches that require repeated model updates after each manually corrective click, our method jointly reasons over all click queries, modeling inter-instance relationships and refining both spatial masks and semantic predictions through spatial and semantic embeddings. Extensive experiments demonstrate that our model improves the mIoU metric by over 20 percent compared to strong baselines and achieves 8-10 percent gains under cross-dataset evaluation for a one-click per instance setting, often requiring only a single click per object. Our approach provides a generalizable and efficient solution for interactive 3D instance segmentation, particularly suitable for real-time applications such as robotic manipulation, navigation, and rapid 3D semantic annotation.

Problem

• interactive segmentation
• 3D instance segmentation
• few-click
• semantic embedding
• point cloud

Innovation

• interactive 3D segmentation
• semantic embedding
• point Transformer
• few-click learning
• multi-instance reasoning