Probabilistic Interactive 3D Segmentation with Hierarchical Neural Processes

2025-05-03

AI Summary

Addressing the challenges of poor generalization from sparse clicks and unquantifiable uncertainty in 3D point cloud interactive segmentation, this paper proposes a hierarchical probabilistic segmentation framework based on neural processes. Methodologically, it introduces a dual-level latent variable structure—operating at both scene- and object-level—integrated with point cloud feature encoding and hierarchical variational inference; a probabilistic prototype modulator is further designed to enable object-aware global contextual modeling. Notably, it is the first approach in 3D interactive segmentation to support pixel-level calibrated uncertainty quantification. Evaluated on four mainstream benchmarks, the method achieves state-of-the-art segmentation accuracy while reducing average click count by 37%; it also improves uncertainty map error localization accuracy by 21.6%. These advances significantly enhance human-AI collaboration efficiency and model interpretability.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Segmentation
• Reasoning under Uncertainty: Probabilistic Programming

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Interactive 3D segmentation has emerged as a promising solution for generating accurate object masks in complex 3D scenes by incorporating user-provided clicks. However, two critical challenges remain underexplored: (1) effectively generalizing from sparse user clicks to produce accurate segmentation, and (2) quantifying predictive uncertainty to help users identify unreliable regions. In this work, we propose NPISeg3D, a novel probabilistic framework that builds upon Neural Processes (NPs) to address these challenges. Specifically, NPISeg3D introduces a hierarchical latent variable structure with scene-specific and object-specific latent variables to enhance few-shot generalization by capturing both global context and object-specific characteristics. Additionally, we design a probabilistic prototype modulator that adaptively modulates click prototypes with object-specific latent variables, improving the model's ability to capture object-aware context and quantify predictive uncertainty. Experiments on four 3D point cloud datasets demonstrate that NPISeg3D achieves superior segmentation performance with fewer clicks while providing reliable uncertainty estimations.

Problem

• Generalizing from sparse user clicks for accurate 3D segmentation
• Quantifying predictive uncertainty in interactive 3D segmentation
• Enhancing few-shot generalization with hierarchical latent variables

Innovation

• Hierarchical Neural Processes for 3D segmentation
• Probabilistic prototype modulator for uncertainty
• Scene and object-specific latent variables