Decoupled Data Consistency with Diffusion Purification for Image Restoration

2024-03-10

arXiv.org

AI Summary

Existing diffusion-based image inpainting methods couple data-consistency gradients into the reverse sampling process, resulting in slow inference, limited acceleration potential, and a hard constraint that consistency updates cannot exceed the total number of sampling steps. This work proposes a decoupled paradigm: a two-stage alternating mechanism—reconstruction (data-consistency solving) and refinement (diffusion-based denoising)—enabling efficient optimization in latent space and seamless integration with consistency models to drastically reduce sampling steps. The method requires no modification to the diffusion backbone and is plug-and-play with diverse accelerated samplers. It achieves state-of-the-art performance across four fundamental restoration tasks—denoising, deblurring, inpainting, and super-resolution—while significantly reducing inference latency and improving speedup by 2–5×, all without compromising reconstruction fidelity.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Diffusion models have recently gained traction as a powerful class of deep generative priors, excelling in a wide range of image restoration tasks due to their exceptional ability to model data distributions. To solve image restoration problems, many existing techniques achieve data consistency by incorporating additional likelihood gradient steps into the reverse sampling process of diffusion models. However, the additional gradient steps pose a challenge for real-world practical applications as they incur a large computational overhead, thereby increasing inference time. They also present additional difficulties when using accelerated diffusion model samplers, as the number of data consistency steps is limited by the number of reverse sampling steps. In this work, we propose a novel diffusion-based image restoration solver that addresses these issues by decoupling the reverse process from the data consistency steps. Our method involves alternating between a reconstruction phase to maintain data consistency and a refinement phase that enforces the prior via diffusion purification. Our approach demonstrates versatility, making it highly adaptable for efficient problem-solving in latent space. Additionally, it reduces the necessity for numerous sampling steps through the integration of consistency models. The efficacy of our approach is validated through comprehensive experiments across various image restoration tasks, including image denoising, deblurring, inpainting, and super-resolution.

Problem

• Reducing computational overhead in diffusion-based image restoration
• Decoupling reverse process from data consistency steps
• Enhancing efficiency in latent space problem-solving

Innovation

• Decouples reverse process from data consistency
• Uses diffusion purification for prior enforcement
• Integrates consistency models to reduce steps