Early Semantic Grounding in Image Editing Models for Zero-Shot Referring Image Segmentation

2026-05-13

AI Summary

This work proposes a training-free, zero-shot referring image segmentation (RIS) method that leverages pre-trained instruction-driven image editing models to achieve language-guided, pixel-accurate object segmentation. The key insight is that intermediate representations generated during the earliest denoising steps already exhibit strong foreground-background separability and semantic localization capabilities. Building on this observation, the authors introduce a two-stage strategy that integrates spatial attention priors with feature-level semantic discrimination to produce precise segmentation masks within a single denoising step. Evaluated on RefCOCO, RefCOCO+, and RefCOCOg benchmarks, the method substantially outperforms existing zero-shot baselines, marking the first demonstration of the untapped potential for semantic segmentation latent in the early denoising stages of image editing diffusion models.

Abstract

Instruction-based image editing (IIE) models have recently demonstrated strong capability in modifying specific image regions according to natural language instructions, which implicitly requires identifying where an edit should be applied. This indicates that such models inherently perform language-conditioned visual semantic grounding. In this work, we investigate whether this implicit grounding can be leveraged for zero-shot referring image segmentation (RIS), a task that requires pixel-level localization of objects described by natural language expressions. Through systematic analysis, we reveal that strong foreground-background separability emerges in the internal representations of these models at the earliest denoising timestep, well before any visible image transformation occurs. Building on this insight, we propose a training-free framework that repurposes pretrained image editing models for RIS by exploiting their intermediate representations. Our approach decomposes localization into two complementary components: attention-based spatial priors that estimate where to focus, and feature-based semantic discrimination that determines what to segment. By leveraging feature-space separability, the framework produces accurate segmentation masks using only a single denoising step, without requiring full image synthesis. Extensive experiments on RefCOCO, RefCOCO+, and RefCOCOg demonstrate that our method achieves superior performance over existing zero-shot baselines.

Problem

• zero-shot referring image segmentation
• semantic grounding
• instruction-based image editing
• language-conditioned visual localization

Innovation

• zero-shot referring image segmentation
• instruction-based image editing
• semantic grounding
• feature-space separability
• training-free framework