Perceive-IR: Learning to Perceive Degradation Better for All-in-One Image Restoration

2024-08-28

arXiv.org

AI Summary

Existing all-in-one image restoration methods struggle to perceive degradation types and severity at a fine-grained level and rely on customized backbone networks, limiting generalizability and modular integration. To address this, we propose Perceive-IR, the first quality-driven multi-level prompting framework that aligns restored images with hierarchical quality prompts in the CLIP embedding space. We introduce a difficulty-adaptive perception loss that jointly suppresses interference from low- and medium-quality samples, enabling precise, quality-aware restoration. The framework adopts a backbone-agnostic, modular architecture supporting plug-and-play enhancement. Extensive experiments demonstrate state-of-the-art performance across blur, noise, JPEG compression, and mixed degradation tasks, achieving significant PSNR and SSIM improvements. Our approach validates the effectiveness and universality of fine-grained, quality-controllable image restoration.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Existing All-in-One image restoration methods often fail to perceive degradation types and severity levels simultaneously, overlooking the importance of fine-grained quality perception. Moreover, these methods often utilize highly customized backbones, which hinder their adaptability and integration into more advanced restoration networks. To address these limitations, we propose Perceive-IR, a novel backbone-agnostic All-in-One image restoration framework designed for fine-grained quality control across various degradation types and severity levels. Its modular structure allows core components to function independently of specific backbones, enabling seamless integration into advanced restoration models without significant modifications. Specifically, Perceive-IR operates in two key stages: 1) multi-level quality-driven prompt learning stage, where a fine-grained quality perceiver is meticulously trained to discern three tier quality levels by optimizing the alignment between prompts and images within the CLIP perception space. This stage ensures a nuanced understanding of image quality, laying the groundwork for subsequent restoration; 2) restoration stage, where the quality perceiver is seamlessly integrated with a difficulty-adaptive perceptual loss, forming a quality-aware learning strategy. This strategy not only dynamically differentiates sample learning difficulty but also achieves fine-grained quality control by driving the restored image toward the ground truth while pulling it away from both low- and medium-quality samples.

Problem

• Perceiving degradation types and severity levels simultaneously
• Overcoming backbone customization hindering adaptability
• Achieving fine-grained quality control in image restoration

Innovation

• Backbone-agnostic framework for diverse image restoration
• Multi-level quality-driven prompt learning in CLIP space
• Difficulty-adaptive perceptual loss for quality control