Lightning-Fast Image Inversion and Editing for Text-to-Image Diffusion Models

2023-12-19

AI Summary

To address the slow inversion speed and low reconstruction fidelity in text-to-image diffusion models, this paper proposes Guided Newton–Raphson (GNR) inversion. GNR operates in the latent space by integrating diffusion-prior-guided constraints, thereby overcoming key limitations of conventional numerical root-finding methods—namely, slow convergence and susceptibility to out-of-distribution solutions. The method is compatible with major open-source models including Stable Diffusion, SDXL-Turbo, and Flux, as well as deterministic schedulers. It achieves high-fidelity inversion for SDXL-Turbo and Flux.1 in just 0.4 seconds. Experiments demonstrate substantial improvements in reconstruction quality, cross-image interpolation smoothness, and rare-object generation capability. Notably, GNR enables the first millisecond-scale, interactive high-quality image editing within diffusion-based frameworks.

Abstract

Diffusion inversion is the problem of taking an image and a text prompt that describes it and finding a noise latent that would generate the exact same image. Most current deterministic inversion techniques operate by approximately solving an implicit equation and may converge slowly or yield poor reconstructed images. We formulate the problem by finding the roots of an implicit equation and devlop a method to solve it efficiently. Our solution is based on Newton-Raphson (NR), a well-known technique in numerical analysis. We show that a vanilla application of NR is computationally infeasible while naively transforming it to a computationally tractable alternative tends to converge to out-of-distribution solutions, resulting in poor reconstruction and editing. We therefore derive an efficient guided formulation that fastly converges and provides high-quality reconstructions and editing. We showcase our method on real image editing with three popular open-sourced diffusion models: Stable Diffusion, SDXL-Turbo, and Flux with different deterministic schedulers. Our solution, Guided Newton-Raphson Inversion, inverts an image within 0.4 sec (on an A100 GPU) for few-step models (SDXL-Turbo and Flux.1), opening the door for interactive image editing. We further show improved results in image interpolation and generation of rare objects.

Problem

• Efficient image inversion and editing
• Fast convergence using Guided Newton-Raphson
• High-quality reconstruction in diffusion models

Innovation

• Newton-Raphson for inversion
• Guided formulation fast convergence
• Efficient image reconstruction editing