Diffuse2Seg: Diffusion Models Can Segment Anything Without Supervision

2026-09-06

AI Summary

本文提出Diffuse2Seg方法，利用文本到图像扩散模型自动生成多粒度实例掩码，无需监督即可实现开放世界实体分割。

Abstract

Open-world entity segmentation aims to predict masks for arbitrary objects across domains and at multiple granularities, from parts to whole objects. In this setting, SAM sets a strong standard: trained on SA-1B, comprising 11M images and over 1B carefully annotated masks, it achieves remarkable zero-shot performance. Collecting such labels is expensive and time-consuming, however, which limits how far this recipe can scale. Text-to-image diffusion models offer a way around this. Their intermediate features transfer well across perception tasks, and since object structure emerges as the model denoises a noise sample into an image conditioned on a text prompt, that structure is already encoded in these representations. They can therefore be exploited for open-world entity segmentation without retraining or supervision. Building on this observation, we introduce Diffuse2Seg, which repurposes generative diffusion models for automatic mask generation by propagating a grid of point prompts through their self-attention representations in an edge-preserving manner. Diffuse2Seg produces multi-granular instance masks and outperforms prior state-of-the-art label generators by 4.3-7.1 p.p. in AR_1000 across five domains. Training an instance segmentation model on these generated masks advances detector-free open-world segmentation by 7.4 and 7.7 p.p. on"things"and"stuff+things"datasets and surpasses the detector-based UnSAM on"stuff+things"by 2.1 p.p. in AR_1000. Finally, we show that a model trained on Diffuse2Seg labels provides a strong initialization for semi-supervised learning, outperforming its fully supervised counterpart with already 5k labeled images.

Problem

• Open-world entity segmentation
• unsupervised
• mask generation

Innovation

• Diffusion Models
• Unsupervised Segmentation
• Self-Attention Representations
• Edge-Preserving
• Open-World Entity Segmentation