Learning Dual Transformers for All-In-One Image Restoration from a Frequency Perspective

2024-06-30

AI Summary

This work addresses unified image restoration across multiple degradation types (e.g., denoising, deraining, dehazing) with a novel frequency-domain modeling framework—the first single-model, all-in-one solution. Methodologically, it introduces a dual-Transformer architecture: a frequency-aware Degradation Estimator (Dformer) and a degradation-adaptive Restorer (Rformer), operating in synergy. Crucially, it explicitly decouples degradation modeling into the frequency domain (via DCT or wavelet transforms) and leverages this decomposition to condition self-attention mechanisms for selective band-wise restoration. This enables joint learning of degradation representations and frequency-band-adaptive reconstruction. The proposed method achieves state-of-the-art performance across five major restoration tasks. Moreover, it demonstrates significantly improved generalization to real-world degradations, spatially variant degradations, and unseen degradation intensities—marking a substantial advance in unified, frequency-guided image restoration.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

This work aims to tackle the all-in-one image restoration task, which seeks to handle multiple types of degradation with a single model. The primary challenge is to extract degradation representations from the input degraded images and use them to guide the model's adaptation to specific degradation types. Building on the insight that various degradations affect image content differently across frequency bands, we propose a new dual-transformer approach comprising two components: a frequency-aware Degradation estimation transformer (Dformer) and a degradation-adaptive Restoration transformer (Rformer). The Dformer captures the essential characteristics of various degradations by decomposing the input into different frequency components. By understanding how degradations affect these frequency components, the Dformer learns robust priors that effectively guide the restoration process. The Rformer then employs a degradation-adaptive self-attention module to selectively focus on the most affected frequency components, guided by the learned degradation representations. Extensive experimental results demonstrate that our approach outperforms existing methods in five representative restoration tasks, including denoising, deraining, dehazing, deblurring, and low-light enhancement. Additionally, our method offers benefits for handling, real-world degradations, spatially variant degradations, and unseen degradation levels.

Problem

• Handles multiple image degradations
• Extracts degradation representations
• Uses frequency-aware transformers

Innovation

• Dual-transformer for image restoration
• Frequency-aware degradation estimation
• Degradation-adaptive self-attention module