Do Instance Priors Help Weakly Supervised Semantic Segmentation?

2026-04-13

AI Summary

This work addresses the high cost of dense pixel-wise annotations in semantic segmentation by proposing SeSAM, a framework that systematically analyzes and mitigates the limitations of the Segment Anything Model (SAM) under weak supervision. SeSAM effectively transfers SAM’s instance-level priors to semantic segmentation through a pipeline comprising category-aware mask decomposition, skeleton-point-based prompt sampling, weak-label-guided mask selection, and iterative pseudo-label refinement. By leveraging coarse masks, scribbles, or point annotations as weak supervisory signals, the method achieves substantial performance gains across diverse weakly supervised settings. Experimental results demonstrate that SeSAM significantly outperforms existing approaches while approaching the accuracy of fully supervised models, thereby offering a practical pathway to drastically reduce annotation costs without compromising segmentation quality.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Semi-Supervised Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Semantic segmentation requires dense pixel-level annotations, which are costly and time-consuming to acquire. To address this, we present SeSAM, a framework that uses a foundational segmentation model, i.e. Segment Anything Model (SAM), with weak labels, including coarse masks, scribbles, and points. SAM, originally designed for instance-based segmentation, cannot be directly used for semantic segmentation tasks. In this work, we identify specific challenges faced by SAM and determine appropriate components to adapt it for class-based segmentation using weak labels. Specifically, SeSAM decomposes class masks into connected components, samples point prompts along object skeletons, selects SAM masks using weak-label coverage, and iteratively refines labels using pseudo-labels, enabling SAM-generated masks to be effectively used for semantic segmentation. Integrated with a semi-supervised learning framework, SeSAM balances ground-truth labels, SAM-based pseudo-labels, and high-confidence pseudo-labels, significantly improving segmentation quality. Extensive experiments across multiple benchmarks and weak annotation types show that SeSAM consistently outperforms weakly supervised baselines while substantially reducing annotation cost relative to fine supervision.

Problem

• weakly supervised semantic segmentation
• instance priors
• Segment Anything Model
• annotation cost
• class-based segmentation

Innovation

• weakly supervised semantic segmentation
• Segment Anything Model (SAM)
• pseudo-labeling
• point prompting
• semi-supervised learning