Style-Editor: Text-driven object-centric style editing

2024-08-16

AI Summary

This paper addresses text-driven object-level style editing—enabling precise, mask-free manipulation of target objects while preserving background consistency. The proposed method introduces three key innovations: (1) a patch-wise co-directional loss that enforces local semantic alignment between image patches and text embeddings; (2) text-matched patch selection, which dynamically identifies text-relevant image regions using CLIP-based similarity; and (3) an adaptive background preservation mechanism that jointly maintains structural integrity and stylistic coherence of the background. Evaluated on multiple benchmarks, the approach achieves significant improvements in text-image alignment and visual coherence, outperforming prior methods both quantitatively and qualitatively. To our knowledge, it is the first method to realize fine-grained, text-guided style editing that is mask-free, object-aware, and background-adaptive.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We present Text-driven object-centric style editing model named Style-Editor, a novel method that guides style editing at an object-centric level using textual inputs. The core of Style-Editor is our Patch-wise Co-Directional (PCD) loss, meticulously designed for precise object-centric editing that are closely aligned with the input text. This loss combines a patch directional loss for text-guided style direction and a patch distribution consistency loss for even CLIP embedding distribution across object regions. It ensures a seamless and harmonious style editing across object regions. Key to our method are the Text-Matched Patch Selection (TMPS) and Pre-fixed Region Selection (PRS) modules for identifying object locations via text, eliminating the need for segmentation masks. Lastly, we introduce an Adaptive Background Preservation (ABP) loss to maintain the original style and structural essence of the image's background. This loss is applied to dynamically identified background areas. Extensive experiments underline the effectiveness of our approach in creating visually coherent and textually aligned style editing.

Problem

• Text-driven object-centric style editing without masks
• Ensuring text-aligned style consistency across objects
• Preserving background style during object editing

Innovation

• Patch-wise Co-Directional loss for precise editing
• Text-Matched Patch Selection for object location
• Adaptive Background Preservation loss for original style