Diffusion Model as a Generalist Segmentation Learner

2026-04-27

AI Summary

This work proposes DiGSeg, the first framework to leverage a pretrained diffusion model as a universal segmentation learner capable of supporting semantic, open-vocabulary, and cross-domain image segmentation without architectural modifications. By fusing image and mask representations in the latent space of the diffusion U-Net and incorporating multi-scale CLIP-aligned text features, DiGSeg enables structured segmentation driven jointly by visual appearance and arbitrary textual prompts. The approach eliminates the need for domain-specific architecture design, achieving state-of-the-art performance on standard semantic segmentation benchmarks while demonstrating exceptional open-vocabulary generalization across diverse cross-domain scenarios, including medical imaging, remote sensing, and agricultural applications.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Diffusion models are primarily trained for image synthesis, yet their denoising trajectories encode rich, spatially aligned visual priors. In this paper, we demonstrate that these priors can be utilized for text-conditioned semantic and open-vocabulary segmentation, and this approach can be generalized to various downstream tasks to make a general-purpose diffusion segmentation framework. Concretely, we introduce DiGSeg (Diffusion Models as a Generalist Segmentation Learner), which repurposes a pretrained diffusion model into a unified segmentation framework. Our approach encodes the input image and ground-truth mask into the latent space and concatenates them as conditioning signals for the diffusion U-Net. A parallel CLIP-aligned text pathway injects language features across multiple scales, enabling the model to align textual queries with evolving visual representations. This design transforms an off-the-shelf diffusion backbone into a universal interface that produces structured segmentation masks conditioned on both appearance and arbitrary text prompts. Extensive experiments demonstrate state-of-the-art performance on standard semantic segmentation benchmarks, as well as strong open-vocabulary generalization and cross-domain transfer to medical, remote sensing, and agricultural scenarios-without domain-specific architectural customization. These results indicate that modern diffusion backbones can serve as generalist segmentation learners rather than pure generators, narrowing the gap between visual generation and visual understanding.

Problem

• diffusion model
• semantic segmentation
• open-vocabulary segmentation
• generalist learner
• text-conditioned segmentation

Innovation

• diffusion model
• generalist segmentation
• text-conditioned segmentation
• open-vocabulary
• cross-domain transfer