SAM-MI: A Mask-Injected Framework for Enhancing Open-Vocabulary Semantic Segmentation with SAM

2025-11-25

AI Summary

Open-vocabulary semantic segmentation (OVSS) faces two key challenges with Segment Anything Model (SAM): over-segmentation and rigid mask-label binding. To address these, we propose SAM-MI, a novel framework introducing three technical innovations: (1) a text-guided sparse point prompter that enables semantic-aware region focusing; (2) shallow-mask aggregation to suppress redundant segmentations; and (3) a low-frequency–high-frequency decoupled mask injection mechanism that disentangles visual representation learning from textual alignment, thereby enhancing label-matching flexibility. Evaluated on the MESS benchmark, SAM-MI achieves a 16.7% absolute improvement in mean IoU over Grounded-SAM while accelerating inference by 1.6×. It consistently outperforms existing methods, establishing a new paradigm for efficient and accurate open-vocabulary universal segmentation.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Open-vocabulary semantic segmentation (OVSS) aims to segment and recognize objects universally. Trained on extensive high-quality segmentation data, the segment anything model (SAM) has demonstrated remarkable universal segmentation capabilities, offering valuable support for OVSS. Although previous methods have made progress in leveraging SAM for OVSS, there are still some challenges: (1) SAM's tendency to over-segment and (2) hard combinations between fixed masks and labels. This paper introduces a novel mask-injected framework, SAM-MI, which effectively integrates SAM with OVSS models to address these challenges. Initially, SAM-MI employs a Text-guided Sparse Point Prompter to sample sparse prompts for SAM instead of previous dense grid-like prompts, thus significantly accelerating the mask generation process. The framework then introduces Shallow Mask Aggregation (SMAgg) to merge partial masks to mitigate the SAM's over-segmentation issue. Finally, Decoupled Mask Injection (DMI) incorporates SAM-generated masks for guidance at low-frequency and high-frequency separately, rather than directly combining them with labels. Extensive experiments on multiple benchmarks validate the superiority of SAM-MI. Notably, the proposed method achieves a 16.7% relative improvement in mIoU over Grounded-SAM on the MESS benchmark, along with a 1.6$ imes$ speedup. We hope SAM-MI can serve as an alternative methodology to effectively equip the OVSS model with SAM.

Problem

• Addresses SAM's over-segmentation issue in semantic segmentation
• Resolves hard combinations between fixed masks and labels
• Enhances open-vocabulary segmentation with efficient mask integration

Innovation

• Uses sparse prompts to accelerate mask generation
• Aggregates partial masks to reduce over-segmentation
• Injects masks at different frequencies for guidance