MDeRainNet: An Efficient Neural Network for Rain Streak Removal from Macro-pixel Images

2024-06-15

arXiv.org

AI Summary

Rain degradation severely compromises light field (LF) image quality; existing deraining methods fail to adequately model the spatio-temporal-angular global correlations inherent in 4D LF data, resulting in inconsistent sub-aperture view restoration. To address this, we propose MDeRainNet—the first LF deraining network operating directly on microlens image (MPI) representations. Our method introduces two key innovations: (i) an Extended Spatio-Temporal-Angular Interaction (ESAI) module and (ii) a Transformer-driven Spatio-Angular-Intensity Attention (SAIA) block, both explicitly modeling long-range cross-dimensional dependencies. Furthermore, we design a semi-supervised learning framework integrating multi-level KL-divergence constraints and color-residual-guided contrastive regularization to enhance generalization and inter-view consistency. Extensive experiments on synthetic and real-world LF datasets demonstrate that MDeRainNet significantly outperforms state-of-the-art methods, achieving substantial PSNR/SSIM improvements while ensuring highly balanced deraining quality across all sub-aperture views and more natural visual results.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Since rainy weather always degrades image quality and poses significant challenges to most computer vision-based intelligent systems, image de-raining has been a hot research topic. Fortunately, in a rainy light field (LF) image, background obscured by rain streaks in one sub-view may be visible in the other sub-views, and implicit depth information and recorded 4D structural information may benefit rain streak detection and removal. However, existing LF image rain removal methods either do not fully exploit the global correlations of 4D LF data or only utilize partial sub-views, resulting in sub-optimal rain removal performance and no-equally good quality for all de-rained sub-views. In this paper, we propose an efficient network, called MDeRainNet, for rain streak removal from LF images. The proposed network adopts a multi-scale encoder-decoder architecture, which directly works on Macro-pixel images (MPIs) to improve the rain removal performance. To fully model the global correlation between the spatial and the angular information, we propose an Extended Spatial-Angular Interaction (ESAI) module to merge them, in which a simple and effective Transformer-based Spatial-Angular Interaction Attention (SAIA) block is also proposed for modeling long-range geometric correlations and making full use of the angular information. Furthermore, to improve the generalization performance of our network on real-world rainy scenes, we propose a novel semi-supervised learning framework for our MDeRainNet, which utilizes multi-level KL loss to bridge the domain gap between features of synthetic and real-world rain streaks and introduces colored-residue image guided contrastive regularization to reconstruct rain-free images. Extensive experiments conducted on synthetic and real-world LFIs demonstrate that our method outperforms the state-of-the-art methods both quantitatively and qualitatively.

Problem

• Removes rain streaks from light field images effectively
• Exploits global 4D LF data correlations for better de-raining
• Improves generalization on real-world rain using semi-supervised learning

Innovation

• Multi-scale encoder-decoder on Macro-pixel images
• Transformer-based Spatial-Angular Interaction Attention
• Semi-supervised learning with KL loss