Fractal-IR: A Unified Framework for Efficient and Scalable Image Restoration

2025-03-22

AI Summary

To address the high computational cost and poor generalization of vision transformers in multi-degradation and multi-resolution image restoration, this paper proposes a unified restoration framework based on fractal architecture. The framework enables progressive reconstruction—from fine details to global structure—via a local information recursive expansion mechanism, thereby circumventing the quadratic complexity of long-range self-attention. It introduces the first fractal neural network, integrating lightweight context aggregation with task-adaptive scaling, and establishes systematic scaling principles for large models in image restoration. Evaluated across seven mainstream restoration tasks—including 2× super-resolution and grayscale denoising (σ=50)—the method achieves state-of-the-art performance: +0.21 dB PSNR gain in 2× SR and +0.20 dB in denoising, while significantly improving inference efficiency and scalability.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Deep Neural Architectures and Foundation Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

While vision transformers achieve significant breakthroughs in various image restoration (IR) tasks, it is still challenging to efficiently scale them across multiple types of degradations and resolutions. In this paper, we propose Fractal-IR, a fractal-based design that progressively refines degraded images by repeatedly expanding local information into broader regions. This fractal architecture naturally captures local details at early stages and seamlessly transitions toward global context in deeper fractal stages, removing the need for computationally heavy long-range self-attention mechanisms. Moveover, we observe the challenge in scaling up vision transformers for IR tasks. Through a series of analyses, we identify a holistic set of strategies to effectively guide model scaling. Extensive experimental results show that Fractal-IR achieves state-of-the-art performance in seven common image restoration tasks, including super-resolution, denoising, JPEG artifact removal, IR in adverse weather conditions, motion deblurring, defocus deblurring, and demosaicking. For $2 imes$ SR on Manga109, Fractal-IR achieves a 0.21 dB PSNR gain. For grayscale image denoising on Urban100, Fractal-IR surpasses the previous method by 0.2 dB for $sigma=50$.

Problem

• Efficiently scaling vision transformers for diverse image restoration tasks
• Reducing computational costs of long-range self-attention mechanisms
• Achieving state-of-the-art performance across multiple IR tasks

Innovation

• Fractal-based design for progressive image refinement
• Eliminates heavy self-attention with fractal architecture
• Holistic scaling strategies for vision transformers