Prompting Diffusion Models for Zero-Shot Instance Segmentation

2026-06-21

AI Summary

This work addresses the limitations of existing promptable segmentation models, which often suffer from false positives and oversegmentation in zero-shot settings and struggle to align with user intent. The authors propose Prompt2Seg, a novel framework that, for the first time, explicitly incorporates spatial prompts—such as 2D Gaussians or confidence maps—as conditional signals into the forward process of a frozen diffusion-based segmentation model. By introducing a spatial conditioning mechanism, the method effectively fuses generative priors with user-provided inputs. Remarkably, Prompt2Seg achieves strong cross-domain zero-shot generalization after fine-tuning on only a limited set of categories. Extensive experiments across seven diverse datasets—including sketches, first-person views, and X-ray images—demonstrate that the approach significantly outperforms baseline models, exhibiting superior zero-shot segmentation accuracy and broad applicability.

Abstract

Several disruptive research directions have recently emerged in computer vision, including foundation models achieving previously unseen zero-shot performance in scene understanding, even interactively, and generative models that synthesize extremely realistic images. The latter have also been shown to be highly effective in scene understanding tasks thanks to their rich priors. However, for promptable segmentation, foundation models struggle with accurately segmenting an object's region, leading to false positives and over-segmentation. Notably, early attempts that leverage generative priors use prompts only during post-processing, yielding suboptimal segments because the process is agnostic to the user input. In this paper, we target these limitations with Prompt2Seg, a spatial conditioning framework for diffusion-based segmentation. Prompt2Seg augments a frozen diffusion segmentation model with a conditioning branch. Our approach takes spatial prompts, represented as 2D Gaussians or confidence maps, as explicit input signals, training the model to respond directly to user intent. Fine-tuned on a deliberately constrained set of object categories drawn from Hypersim and Virtual KITTI 2, Prompt2Seg generalizes zero-shot to a wide range of unseen object types and visual domains. We evaluate on seven datasets ranging from standard benchmarks to more challenging domains, including paintings, egocentric views, and X-ray data. Furthermore, we demonstrate that Prompt2Seg consistently outperforms the underlying diffusion segmentation backbone across all benchmarks. Our results suggest that the rich priors encoded in generative pretraining, combined with principled spatial conditioning, offer a compelling path toward broadly generalizing interactive segmentation without large-scale mask supervision.

Problem

• zero-shot instance segmentation
• promptable segmentation
• diffusion models
• spatial conditioning
• generative priors

Innovation

• diffusion models
• zero-shot segmentation
• spatial prompting
• interactive segmentation
• foundation models