Streamlining Image Editing with Layered Diffusion Brushes

2024-05-01

arXiv.org

AI Summary

Existing diffusion-based image editing methods suffer from limited localization control and interactivity. To address this, we propose a training-free, layer-based real-time editing framework. Its core innovation is the layered diffusion brush mechanism: fine-grained spatially masked intervention applied at intermediate denoising layers, decoupling region masks, visibility, and editing order to enable arbitrary, independent, and parallel layer editing. The method integrates prompt guidance with spatial constraints within a lightweight layer editor architecture, achieving efficient GPU-accelerated inference (140 ms per 512×512 image). User studies demonstrate significant improvements over InstructPix2Pix and Stable Diffusion Inpainting across attribute adjustment, error correction, and multi-step object placement tasks. To our knowledge, this is the first approach enabling high-fidelity, contextually consistent, interactive creative editing with real-time responsiveness and precise spatial control.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Prompt Engineering / Prompting
• Humans and AI: Intelligent User Interfaces

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Denoising diffusion models have recently gained prominence as powerful tools for a variety of image generation and manipulation tasks. Building on this, we propose a novel tool for real-time editing of images that provides users with fine-grained region-targeted supervision in addition to existing prompt-based controls. Our novel editing technique, termed Layered Diffusion Brushes, leverages prompt-guided and region-targeted alteration of intermediate denoising steps, enabling precise modifications while maintaining the integrity and context of the input image. We provide an editor based on Layered Diffusion Brushes modifications, which incorporates well-known image editing concepts such as layer masks, visibility toggles, and independent manipulation of layers; regardless of their order. Our system renders a single edit on a 512x512 image within 140 ms using a high-end consumer GPU, enabling real-time feedback and rapid exploration of candidate edits. We validated our method and editing system through a user study involving both natural images (using inversion) and generated images, showcasing its usability and effectiveness compared to existing techniques such as InstructPix2Pix and Stable Diffusion Inpainting for refining images. Our approach demonstrates efficacy across a range of tasks, including object attribute adjustments, error correction, and sequential prompt-based object placement and manipulation, demonstrating its versatility and potential for enhancing creative workflows.

Problem

• Enabling interactive localized editing with diffusion models
• Achieving non-destructive fine-grained edits in overlapping regions
• Reducing computational latency for real-time image manipulation

Innovation

• Training-free framework for layer-based image editing
• Self-contained layers enable independent non-destructive edits
• Intermediate latent caching achieves 140ms per edit speed