Don't Forget your Inverse DDIM for Image Editing

2025-05-14

AI Summary

To address the high computational cost and low reconstruction fidelity in real-image editing, this paper proposes SAGE: a method leveraging pretrained diffusion models to extract U-Net self-attention maps from the inverse DDIM sampling process, thereby establishing an attention-driven local reconstruction guidance mechanism that avoids full-image reconstruction. SAGE enables precise text-guided editing while preserving high fidelity in unedited regions. Its core innovation lies in the first explicit modeling of inverse-DDIM self-attention maps as reconstruction priors, coupled with a novel attention-weighted reconstruction loss. Quantitative evaluation across ten metrics shows SAGE achieves state-of-the-art performance on seven. A user study (N=47) further demonstrates its superior editing quality and consistency over existing approaches.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

The field of text-to-image generation has undergone significant advancements with the introduction of diffusion models. Nevertheless, the challenge of editing real images persists, as most methods are either computationally intensive or produce poor reconstructions. This paper introduces SAGE (Self-Attention Guidance for image Editing) - a novel technique leveraging pre-trained diffusion models for image editing. SAGE builds upon the DDIM algorithm and incorporates a novel guidance mechanism utilizing the self-attention layers of the diffusion U-Net. This mechanism computes a reconstruction objective based on attention maps generated during the inverse DDIM process, enabling efficient reconstruction of unedited regions without the need to precisely reconstruct the entire input image. Thus, SAGE directly addresses the key challenges in image editing. The superiority of SAGE over other methods is demonstrated through quantitative and qualitative evaluations and confirmed by a statistically validated comprehensive user study, in which all 47 surveyed users preferred SAGE over competing methods. Additionally, SAGE ranks as the top-performing method in seven out of 10 quantitative analyses and secures second and third places in the remaining three.

Problem

• Efficient real image editing using diffusion models
• Improving reconstruction quality in image editing
• Reducing computational intensity in diffusion-based editing

Innovation

• Leverages pre-trained diffusion models for editing
• Uses self-attention guidance in diffusion U-Net
• Inverse DDIM process for efficient reconstruction