SAPNet++: Evolving Point-Prompted Instance Segmentation with Semantic and Spatial Awareness

2026-02-25

AI Summary

This work addresses the challenges of instance segmentation from single-point prompts, which suffers from granularity ambiguity and boundary uncertainty, often failing to distinguish between holistic and local structures. To overcome these limitations, the paper proposes SAPNet++, introducing a spatial granularity-aware completeness scoring mechanism (S-MIL) in proposal selection for the first time. This mechanism is complemented by a point-distance-guided strategy and box mining to alleviate granularity ambiguity. Furthermore, a multi-level affinity refinement module is designed to integrate pixel-level and semantic-level cues during mask optimization, thereby enhancing boundary precision. Extensive experiments on four challenging benchmarks demonstrate that the proposed method significantly outperforms existing approaches, effectively resolving the granularity and boundary issues inherent in single-point-prompted instance segmentation.

Technology Category

• Computer Vision: Segmentation
• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Single-point annotation is increasingly prominent in visual tasks for labeling cost reduction. However, it challenges tasks requiring high precision, such as the point-prompted instance segmentation (PPIS) task, which aims to estimate precise masks using single-point prompts to train a segmentation network. Due to the constraints of point annotations, granularity ambiguity and boundary uncertainty arise the difficulty distinguishing between different levels of detail (eg. whole object vs. parts) and the challenge of precisely delineating object boundaries. Previous works have usually inherited the paradigm of mask generation along with proposal selection to achieve PPIS. However, proposal selection relies solely on category information, failing to resolve the ambiguity of different granularity. Furthermore, mask generators offer only finite discrete solutions that often deviate from actual masks, particularly at boundaries. To address these issues, we propose the Semantic-Aware Point-Prompted Instance Segmentation Network (SAPNet). It integrates Point Distance Guidance and Box Mining Strategy to tackle group and local issues caused by the point's granularity ambiguity. Additionally, we incorporate completeness scores within proposals to add spatial granularity awareness, enhancing multiple instance learning (MIL) in proposal selection termed S-MIL. The Multi-level Affinity Refinement conveys pixel and semantic clues, narrowing boundary uncertainty during mask refinement. These modules culminate in SAPNet++, mitigating point prompt's granularity ambiguity and boundary uncertainty and significantly improving segmentation performance. Extensive experiments on four challenging datasets validate the effectiveness of our methods, highlighting the potential to advance PPIS.

Problem

• point-prompted instance segmentation
• granularity ambiguity
• boundary uncertainty
• single-point annotation
• instance segmentation

Innovation

• point-prompted instance segmentation
• granularity ambiguity
• boundary uncertainty
• semantic-aware MIL
• multi-level affinity refinement