DreamMix: Decoupling Object Attributes for Enhanced Editability in Customized Image Inpainting

2024-11-26

arXiv.org

AI Summary

Existing text-guided image inpainting methods suffer from identity overfitting due to entanglement between textual prompts and intrinsic object attributes, hindering simultaneous preservation of object identity and flexible attribute editing. To address this, we propose a decoupled local-global diffusion inpainting framework featuring an Attribute Decoupling Mechanism (ADM) and a Text-Attribute Swapping (TAS) module—enabling, for the first time in customized inpainting, explicit disentanglement and independent control of object identity versus attributes such as color, material, and pose. By jointly learning implicit attribute representations and dynamically injecting text conditions, our method supports precise object insertion at specified locations and arbitrary fine-grained, text-driven edits. Extensive experiments demonstrate state-of-the-art performance across object insertion, attribute editing, and small-object restoration tasks, both quantitatively and qualitatively. Our code is publicly available.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Subject-driven image inpainting has emerged as a popular task in image editing alongside recent advancements in diffusion models. Previous methods primarily focus on identity preservation but struggle to maintain the editability of inserted objects. In response, this paper introduces DreamMix, a diffusion-based generative model adept at inserting target objects into given scenes at user-specified locations while concurrently enabling arbitrary text-driven modifications to their attributes. In particular, we leverage advanced foundational inpainting models and introduce a disentangled local-global inpainting framework to balance precise local object insertion with effective global visual coherence. Additionally, we propose an Attribute Decoupling Mechanism (ADM) and a Textual Attribute Substitution (TAS) module to improve the diversity and discriminative capability of the text-based attribute guidance, respectively. Extensive experiments demonstrate that DreamMix effectively balances identity preservation and attribute editability across various application scenarios, including object insertion, attribute editing, and small object inpainting. Our code is publicly available at https://github.com/mycfhs/DreamMix.

Problem

• Overcoming identity overfitting in subject-driven image inpainting
• Decoupling entangled object attributes from target text instructions
• Enabling identity preservation with arbitrary attribute modifications

Innovation

• Attribute Decoupling Mechanism synthesizes diverse image-text pairs
• Textual Attribute Substitution isolates attributes via orthogonal decomposition
• Disentangled Inpainting Framework separates local generation from global harmonization