LD-RPS: Zero-Shot Unified Image Restoration via Latent Diffusion Recurrent Posterior Sampling

2025-07-01

AI Summary

Unified image restoration faces two major challenges: poor generalization and heavy reliance on paired training data—existing methods are either task-specific or require paired supervision, limiting their applicability to open-domain degradations. This paper proposes the first zero-shot, task-agnostic unified restoration framework, eliminating the need for paired data by leveraging a pre-trained latent diffusion model (LDM). Our approach innovatively integrates cyclic posterior sampling, multimodal semantic prior guidance, and a lightweight input alignment module. By modeling degradation-invariant semantic priors directly in the latent space, the method achieves robust restoration under unseen degradations. Extensive experiments demonstrate state-of-the-art performance across diverse tasks—including deblurring, denoising, super-resolution, and deraining—while significantly improving cross-degradation generalization. The framework enables practical zero-shot deployment in real-world scenarios without task-specific fine-tuning or ground-truth supervision.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Unified image restoration is a significantly challenging task in low-level vision. Existing methods either make tailored designs for specific tasks, limiting their generalizability across various types of degradation, or rely on training with paired datasets, thereby suffering from closed-set constraints. To address these issues, we propose a novel, dataset-free, and unified approach through recurrent posterior sampling utilizing a pretrained latent diffusion model. Our method incorporates the multimodal understanding model to provide sematic priors for the generative model under a task-blind condition. Furthermore, it utilizes a lightweight module to align the degraded input with the generated preference of the diffusion model, and employs recurrent refinement for posterior sampling. Extensive experiments demonstrate that our method outperforms state-of-the-art methods, validating its effectiveness and robustness. Our code and data will be available at https://github.com/AMAP-ML/LD-RPS.

Problem

• Unified image restoration across diverse degradations
• Eliminating need for task-specific designs or paired datasets
• Leveraging latent diffusion for zero-shot semantic priors

Innovation

• Uses pretrained latent diffusion model
• Incorporates multimodal understanding model
• Employs lightweight module for alignment