S^4M: Boosting Semi-Supervised Instance Segmentation with SAM

2025-04-07

AI Summary

Semi-supervised instance segmentation suffers from low-quality pseudo-labels, inaccurate instance localization, and category-agnostic bias and over-segmentation when directly applying the Segment Anything Model (SAM). Method: This paper proposes a novel SAM-driven teacher-student distillation paradigm. Specifically: (1) a boundary-aware consistency distillation mechanism preserves SAM’s fine-grained localization capability; (2) dynamic pseudo-label confidence calibration and instance-level matching enhancement mitigate category-agnostic bias and over-segmentation; (3) SAM’s prior knowledge is integrated with semi-supervised learning objectives. Results: The method achieves state-of-the-art performance on COCO and PASCAL VOC under semi-supervised settings—particularly with only 1%–10% labeled data—where it significantly improves mAP and mask accuracy. These results demonstrate its effectiveness and generalizability in low-data regimes.

Technology Category

• Machine Learning: Semi-Supervised Learning
• Computer Vision: Segmentation
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Semi-supervised instance segmentation poses challenges due to limited labeled data, causing difficulties in accurately localizing distinct object instances. Current teacher-student frameworks still suffer from performance constraints due to unreliable pseudo-label quality stemming from limited labeled data. While the Segment Anything Model (SAM) offers robust segmentation capabilities at various granularities, directly applying SAM to this task introduces challenges such as class-agnostic predictions and potential over-segmentation. To address these complexities, we carefully integrate SAM into the semi-supervised instance segmentation framework, developing a novel distillation method that effectively captures the precise localization capabilities of SAM without compromising semantic recognition. Furthermore, we incorporate pseudo-label refinement as well as a specialized data augmentation with the refined pseudo-labels, resulting in superior performance. We establish state-of-the-art performance, and provide comprehensive experiments and ablation studies to validate the effectiveness of our proposed approach.

Problem

• Limited labeled data hinders semi-supervised instance segmentation accuracy
• Unreliable pseudo-labels from teacher-student frameworks limit performance
• SAM's class-agnostic predictions cause over-segmentation in instance segmentation

Innovation

• Integrates SAM for precise localization
• Novel distillation method enhances semantic recognition
• Pseudo-label refinement boosts segmentation accuracy