Crafting Parts for Expressive Object Composition

2024-06-14

arXiv.org

AI Summary

Existing text-to-image diffusion models (e.g., Stable Diffusion) struggle to precisely localize and synthesize fine-grained object parts—such as “a panda’s bamboo cane” or “a robot’s glowing joints”—in a zero-shot setting, often resulting in part omission, misplacement, or semantic inconsistency. This work introduces PartCraft, the first fine-grained part-level diffusion localization and synthesis framework that operates without model fine-tuning. Leveraging only a pre-trained diffusion model, PartCraft enables spatially accurate part localization, binary mask generation, localized inpainting, and seamless multi-region compositing—all within the standard denoising process and without introducing auxiliary parameters or training overhead. Extensive qualitative and quantitative evaluations demonstrate that PartCraft significantly improves part fidelity, spatial consistency, and compositional novelty compared to prior part-control methods. It establishes an efficient, general-purpose, zero-shot paradigm for controllable image generation via part-aware editing.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Text-to-image generation from large generative models like Stable Diffusion, DALLE-2, etc., have become a common base for various tasks due to their superior quality and extensive knowledge bases. As image composition and generation are creative processes the artists need control over various parts of the images being generated. We find that just adding details about parts in the base text prompt either leads to an entirely different image (e.g., missing/incorrect identity) or the extra part details simply being ignored. To mitigate these issues, we introduce PartCraft, which enables image generation based on fine-grained part-level details specified for objects in the base text prompt. This allows more control for artists and enables novel object compositions by combining distinctive object parts. PartCraft first localizes object parts by denoising the object region from a specific diffusion process. This enables each part token to be localized to the right object region. After obtaining part masks, we run a localized diffusion process in each of the part regions based on fine-grained part descriptions and combine them to produce the final image. All the stages of PartCraft are based on repurposing a pre-trained diffusion model, which enables it to generalize across various domains without training. We demonstrate the effectiveness of part-level control provided by PartCraft qualitatively through visual examples and quantitatively in comparison to the contemporary baselines.

Problem

• Enables fine-grained part-level control in image generation
• Addresses attribute detail ignorance in text-to-image models
• Combines localized part attributes for novel object compositions

Innovation

• Training-free method for part-level image generation
• Localizes object parts via denoising diffusion process
• Combines localized part attributes for final image