SSP-IR: Semantic and Structure Priors for Diffusion-based Realistic Image Restoration

2024-07-04

IEEE transactions on circuits and systems for video technology (Print)

AI Summary

To address semantic distortion, structural inaccuracy, and insufficient perceptual quality in severely degraded image restoration, this paper proposes a dual-prior collaborative diffusion restoration framework. First, it explicitly models semantic priors by integrating a multimodal large language model (MLLM) with implicit representations from the degraded input. Second, it designs an RGB+FFT joint-constrained structural processor to extract degradation-invariant structural priors. Third, it introduces a multi-level cross-attention mechanism to adaptively fuse the semantic and structural priors. Extensive experiments on both synthetic and real-world datasets demonstrate state-of-the-art performance: our method achieves superior PSNR, SSIM, and LPIPS scores, as well as the highest human perceptual ratings. It significantly improves semantic fidelity, structural accuracy, and visual realism—outperforming all existing approaches across quantitative and qualitative metrics.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Realistic image restoration is a crucial task in computer vision, and diffusion-based models for image restoration have garnered significant attention due to their ability to produce realistic results. Restoration can be seen as a controllable generation conditioning on priors. However, due to the severity of image degradation, existing diffusion-based restoration methods cannot fully exploit priors from low-quality images and still have many challenges in perceptual quality, semantic fidelity, and structure accuracy. Based on the challenges, we introduce a novel image restoration method, SSP-IR. Our approach aims to fully exploit semantic and structure priors from low-quality images to guide the diffusion model in generating semantically faithful and structurally accurate natural restoration results. Specifically, we integrate the visual comprehension capabilities of Multimodal Large Language Models (explicit) and the visual representations of the original image (implicit) to acquire accurate semantic prior. To extract degradation-independent structure prior, we introduce a Processor with RGB and FFT constraints to extract structure prior from the low-quality images, guiding the diffusion model and preventing the generation of unreasonable artifacts. Lastly, we employ a multi-level attention mechanism to integrate the acquired semantic and structure priors. The qualitative and quantitative results demonstrate that our method outperforms other state-of-the-art methods overall on both synthetic and real-world datasets. Our project page is https://zyhrainbow.github.io/projects/SSP-IR.

Problem

• Exploit semantic and structure priors
• Guide diffusion model for image restoration
• Enhance perceptual quality and semantic fidelity

Innovation

• Uses semantic and structure priors
• Integrates Multimodal Large Language Models
• Employs multi-level attention mechanism