Efficient Visual State Space Model for Image Deblurring

2024-05-23

arXiv.org

AI Summary

To address the efficiency bottleneck imposed by the quadratic computational complexity of Vision Transformers (ViTs) in high-resolution image deblurring, this paper proposes the Efficient Visual State Space Model (EVSSM). Methodologically, EVSSM introduces a geometry-transform-driven dynamic visual scanning module that replaces conventional multi-directional fixed scanning, enabling non-local structural modeling while preserving linear computational complexity. Additionally, it designs a lightweight visual state space module integrating geometry-aware feature enhancement and end-to-end supervised training. Extensive experiments on multiple benchmarks and real-world blurred images demonstrate that EVSSM achieves state-of-the-art (SOTA) performance, with a 2.3× speedup in inference latency and a 37% reduction in parameter count—effectively balancing long-range dependency modeling capability and practical deployment efficiency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: State Estimation

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Convolutional neural networks (CNNs) and Vision Transformers (ViTs) have achieved excellent performance in image restoration. ViTs typically yield superior results in image restoration compared to CNNs due to their ability to capture long-range dependencies and input-dependent characteristics. However, the computational complexity of Transformer-based models grows quadratically with the image resolution, limiting their practical appeal in high-resolution image restoration tasks. In this paper, we propose a simple yet effective visual state space model (EVSSM) for image deblurring, leveraging the benefits of state space models (SSMs) to visual data. In contrast to existing methods that employ several fixed-direction scanning for feature extraction, which significantly increases the computational cost, we develop an efficient visual scan block that applies various geometric transformations before each SSM-based module, capturing useful non-local information and maintaining high efficiency. Extensive experimental results show that the proposed EVSSM performs favorably against state-of-the-art image deblurring methods on benchmark datasets and real-captured images.

Problem

• Overcome ViTs' high computational cost in image deblurring
• Improve non-local feature extraction efficiency for restoration
• Enhance frequency domain-based local information representation

Innovation

• Visual state space model for deblurring
• Efficient geometric transformations in scan block
• Frequency domain-based feedforward network