Segment Anyword: Mask Prompt Inversion for Open-Set Grounded Segmentation

2025-05-23

AI Summary

Existing open-vocabulary image segmentation methods rely heavily on extensive training or fine-tuning, struggling to maintain segmentation coherence and consistency across diverse textual expressions. This paper introduces the first zero-training, language-guided open-vocabulary segmentation framework. It leverages frozen diffusion models (e.g., Stable Diffusion) to generate initial mask prompts from word-level cross-attention maps; incorporates dependency parsing to regularize visual prompts; and proposes a mask prompt inversion mechanism alongside linguistic-structure-driven visual embedding clustering to significantly enhance mask consistency and robustness. Evaluated on Pascal Context 59, gRefCOCO, and GranDf, our method achieves 52.5 mIoU (+6.8), 67.73 cIoU (+25.73), and 67.4 mIoU, respectively—outperforming state-of-the-art fine-tuning approaches and establishing the first high-accuracy, strongly generalizable zero-training grounded segmentation solution.

Technology Category

• Computer Vision: Segmentation
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Open-set image segmentation poses a significant challenge because existing methods often demand extensive training or fine-tuning and generally struggle to segment unified objects consistently across diverse text reference expressions. Motivated by this, we propose Segment Anyword, a novel training-free visual concept prompt learning approach for open-set language grounded segmentation that relies on token-level cross-attention maps from a frozen diffusion model to produce segmentation surrogates or mask prompts, which are then refined into targeted object masks. Initial prompts typically lack coherence and consistency as the complexity of the image-text increases, resulting in suboptimal mask fragments. To tackle this issue, we further introduce a novel linguistic-guided visual prompt regularization that binds and clusters visual prompts based on sentence dependency and syntactic structural information, enabling the extraction of robust, noise-tolerant mask prompts, and significant improvements in segmentation accuracy. The proposed approach is effective, generalizes across different open-set segmentation tasks, and achieves state-of-the-art results of 52.5 (+6.8 relative) mIoU on Pascal Context 59, 67.73 (+25.73 relative) cIoU on gRefCOCO, and 67.4 (+1.1 relative to fine-tuned methods) mIoU on GranDf, which is the most complex open-set grounded segmentation task in the field.

Problem

• Open-set image segmentation lacks consistency across text expressions
• Initial mask prompts are incoherent with complex image-text inputs
• Existing methods require extensive training or fine-tuning for segmentation

Innovation

• Training-free visual concept prompt learning
• Linguistic-guided visual prompt regularization
• Token-level cross-attention maps for segmentation