SAMRefiner: Taming Segment Anything Model for Universal Mask Refinement

2025-02-10

AI Summary

To address the low quality of coarse-grain segmentation masks and high annotation costs, this paper proposes SAMRefiner++, a general-purpose and efficient mask refinement framework that is plug-and-play compatible with any pre-trained segmentation model (e.g., SAM) without requiring additional annotations. Methodologically: (i) it introduces a novel noise-robust prompting mechanism integrating distance-guided points, context-aware elastic bounding boxes, and Gaussian-style masks; (ii) it designs a “Split–Tackle–Merge” (STM) pipeline to handle multi-object segmentation challenges; and (iii) it incorporates an unsupervised IoU-adaptive optimization module combining iterative calibration with adaptive prompt engineering. Extensive experiments on multiple benchmarks demonstrate significant improvements over state-of-the-art methods in both accuracy and inference efficiency, achieving new SOTA performance. The source code is publicly available.

Technology Category

• Computer Vision: Segmentation
• Search and Optimization: Sampling/Simulation-based Search
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

In this paper, we explore a principal way to enhance the quality of widely pre-existing coarse masks, enabling them to serve as reliable training data for segmentation models to reduce the annotation cost. In contrast to prior refinement techniques that are tailored to specific models or tasks in a close-world manner, we propose SAMRefiner, a universal and efficient approach by adapting SAM to the mask refinement task. The core technique of our model is the noise-tolerant prompting scheme. Specifically, we introduce a multi-prompt excavation strategy to mine diverse input prompts for SAM (i.e., distance-guided points, context-aware elastic bounding boxes, and Gaussian-style masks) from initial coarse masks. These prompts can collaborate with each other to mitigate the effect of defects in coarse masks. In particular, considering the difficulty of SAM to handle the multi-object case in semantic segmentation, we introduce a split-then-merge (STM) pipeline. Additionally, we extend our method to SAMRefiner++ by introducing an additional IoU adaption step to further boost the performance of the generic SAMRefiner on the target dataset. This step is self-boosted and requires no additional annotation. The proposed framework is versatile and can flexibly cooperate with existing segmentation methods. We evaluate our mask framework on a wide range of benchmarks under different settings, demonstrating better accuracy and efficiency. SAMRefiner holds significant potential to expedite the evolution of refinement tools. Our code is available at https://github.com/linyq2117/SAMRefiner.

Problem

• Enhance quality of coarse masks
• Universal mask refinement approach
• Reduce segmentation annotation cost

Innovation

• Universal mask refinement approach
• Noise-tolerant prompting scheme
• Self-boosted IoU adaption step