FAPE-IR: Frequency-Aware Planning and Execution Framework for All-in-One Image Restoration

2025-11-17

AI Summary

To address the weak generalization and poor interpretability of All-in-One Image Restoration (AIO-IR) models under complex, mixed degradation scenarios, this paper proposes FAPE-IR—a unified framework integrating semantic planning and frequency-domain restoration. Our method freezes a multimodal large language model as a semantic planner to generate frequency-aware restoration strategies; employs a LoRA-finetuned Mixture-of-Experts (MoE) architecture for dynamic routing between high- and low-frequency experts; and incorporates a diffusion-based executor, adversarial training, and frequency-domain regularization loss to enable frequency-adaptive, fine-grained reconstruction. Evaluated on seven benchmark tasks, FAPE-IR achieves state-of-the-art performance, significantly improving zero-shot generalization to unseen mixed degradations, effectively suppressing artifacts, and demonstrating superior robustness and interpretability.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Diffusion Models for Vision
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

All-in-One Image Restoration (AIO-IR) aims to develop a unified model that can handle multiple degradations under complex conditions. However, existing methods often rely on task-specific designs or latent routing strategies, making it hard to adapt to real-world scenarios with various degradations. We propose FAPE-IR, a Frequency-Aware Planning and Execution framework for image restoration. It uses a frozen Multimodal Large Language Model (MLLM) as a planner to analyze degraded images and generate concise, frequency-aware restoration plans. These plans guide a LoRA-based Mixture-of-Experts (LoRA-MoE) module within a diffusion-based executor, which dynamically selects high- or low-frequency experts, complemented by frequency features of the input image. To further improve restoration quality and reduce artifacts, we introduce adversarial training and a frequency regularization loss. By coupling semantic planning with frequency-based restoration, FAPE-IR offers a unified and interpretable solution for all-in-one image restoration. Extensive experiments show that FAPE-IR achieves state-of-the-art performance across seven restoration tasks and exhibits strong zero-shot generalization under mixed degradations.

Problem

• Developing unified models for multiple image degradations under complex conditions
• Overcoming limitations of task-specific designs in real-world restoration scenarios
• Providing interpretable solutions for all-in-one image restoration tasks

Innovation

• Uses frozen MLLM as planner for restoration plans
• Employs LoRA-MoE module with frequency-aware expert selection
• Introduces adversarial training and frequency regularization loss