NegROI: Click-Centric Uncertainty-Guided Refinement with Scene-Conditioned Negative Prompts for Robust Interactive 3D Segmentation

2026-07-07

AI Summary

This work addresses the challenges of interactive 3D segmentation under limited user clicks, where ambiguous boundaries and complex backgrounds often degrade performance and hinder cross-dataset generalization. The authors propose a Transformer-based interactive segmentation framework that leverages click-guided multi-resolution local refinement and models challenging background regions through scene-conditioned negative prompts. A key innovation is an uncertainty-driven selective optimization mechanism that adaptively focuses on ambiguous areas. The approach further introduces boundary-aware hard negative mining, diversity regularization for negative prompts, and context-aware prompts generated via cross-attention. Evaluated on ScanNet, S3DIS, and KITTI, the method significantly improves click efficiency, reduces false positives, and demonstrates superior generalization across diverse datasets.

Abstract

Interactive 3D segmentation aims to extract object masks in point clouds with minimal user clicks. Despite recent progress, most existing approaches still struggle with (i) coarse voxel resolution that blurs fine boundaries under limited clicks and (ii) hard false positives caused by confusing background structures. These issues are exacerbated by density and scale shifts across datasets (e.g., dense RGB-D reconstructions vs. sparse LiDAR scans), where fixed refinement heuristics and purely click-driven decoding generalize poorly. To address them, we propose NegROI -- a novel transformer-based interactive framework that couples click-centric multi-resolution refinement with scene-conditioned negative prompts. Given a coarse voxel prediction, it refines only a local Region Of Interest (ROI) around the current click on a finer grid and fuses refined logits back to the coarse mask. To improve robustness and efficiency, we introduce uncertainty-driven selective refinement that prioritizes ambiguous regions. Meanwhile, we model hard background patterns via a set of scene-conditioned negative prompts obtained by cross-attention over scene tokens. We further stabilize these prompts with a diversity regularizer. Finally, we propose boundary-aware hard negative mining to supervise negative-prompt attention toward boundary-proximal, high-confidence false positives. Our experiments on common benchmark datasets (i.e., ScanNet, S3DIS, and KITTI) demonstrate improved click efficiency and reduced false positives, with stronger cross-dataset robustness than the state-of-the-art baselines.

Problem

• interactive 3D segmentation
• false positives
• voxel resolution
• cross-dataset generalization
• point cloud segmentation

Innovation

• uncertainty-guided refinement
• negative prompts
• click-centric ROI
• boundary-aware hard negative mining
• scene-conditioned attention