S$^2$Edit: Text-Guided Image Editing with Precise Semantic and Spatial Control

2025-07-06

AI Summary

Existing text-guided image editing methods based on diffusion models suffer from coarse semantic control, inaccurate spatial localization, loss of subject identity and high-frequency details, and erroneous modifications to irrelevant regions due to semantic entanglement. To address these issues, we propose a Semantic-Spatial Dual-Decoupling Editing framework. Methodologically: (1) we introduce learnable text token embeddings to explicitly represent subject identity and impose feature-space orthogonality constraints to decouple identity from attribute semantics; (2) we incorporate object-mask-guided cross-attention to achieve spatially focused editing. Our approach requires only lightweight adaptation—no full-model fine-tuning—on pretrained text-to-image diffusion models. Extensive qualitative and quantitative evaluations demonstrate significant improvements over state-of-the-art methods in editing accuracy, identity preservation, and detail fidelity. Moreover, our framework successfully supports complex multi-attribute editing tasks, such as makeup transfer.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Recent advances in diffusion models have enabled high-quality generation and manipulation of images guided by texts, as well as concept learning from images. However, naive applications of existing methods to editing tasks that require fine-grained control, e.g., face editing, often lead to suboptimal solutions with identity information and high-frequency details lost during the editing process, or irrelevant image regions altered due to entangled concepts. In this work, we propose S$^2$Edit, a novel method based on a pre-trained text-to-image diffusion model that enables personalized editing with precise semantic and spatial control. We first fine-tune our model to embed the identity information into a learnable text token. During fine-tuning, we disentangle the learned identity token from attributes to be edited by enforcing an orthogonality constraint in the textual feature space. To ensure that the identity token only affects regions of interest, we apply object masks to guide the cross-attention maps. At inference time, our method performs localized editing while faithfully preserving the original identity with semantically disentangled and spatially focused identity token learned. Extensive experiments demonstrate the superiority of S$^2$Edit over state-of-the-art methods both quantitatively and qualitatively. Additionally, we showcase several compositional image editing applications of S$^2$Edit such as makeup transfer.

Problem

• Enables precise semantic and spatial image editing
• Preserves identity information during fine-grained editing
• Disentangles concepts to avoid irrelevant region alterations

Innovation

• Fine-tunes model with learnable identity text token
• Disentangles identity using orthogonality in text features
• Guides editing with object masks on attention maps