Semantics and Content Matter: Towards Multi-Prior Hierarchical Mamba for Image Deraining

2025-11-17

AI Summary

Rain streaks severely degrade the performance of vision systems, and existing deraining methods often compromise semantic and spatial detail fidelity. To address this, we propose the Multi-Prior Hierarchical Mamba network (MPHM), the first framework to jointly leverage heterogeneous visual priors—CLIP for global semantic guidance and DINOv2 for local structural modeling. We design a progressive prior fusion mechanism and hierarchical Mamba modules to mitigate prior conflicts. Furthermore, we introduce a Fourier-enhanced dual-path architecture that synergistically improves global contextual modeling and local detail recovery. Quantitatively, MPHM achieves a 0.57 dB PSNR gain over the previous state-of-the-art on Rain200H. Qualitatively, it demonstrates superior generalization to real-world rainy scenes, preserving both high-level semantics and fine-grained textures without artifacts.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Rain significantly degrades the performance of computer vision systems, particularly in applications like autonomous driving and video surveillance. While existing deraining methods have made considerable progress, they often struggle with fidelity of semantic and spatial details. To address these limitations, we propose the Multi-Prior Hierarchical Mamba (MPHM) network for image deraining. This novel architecture synergistically integrates macro-semantic textual priors (CLIP) for task-level semantic guidance and micro-structural visual priors (DINOv2) for scene-aware structural information. To alleviate potential conflicts between heterogeneous priors, we devise a progressive Priors Fusion Injection (PFI) that strategically injects complementary cues at different decoder levels. Meanwhile, we equip the backbone network with an elaborate Hierarchical Mamba Module (HMM) to facilitate robust feature representation, featuring a Fourier-enhanced dual-path design that concurrently addresses global context modeling and local detail recovery. Comprehensive experiments demonstrate MPHM's state-of-the-art performance, achieving a 0.57 dB PSNR gain on the Rain200H dataset while delivering superior generalization on real-world rainy scenarios.

Problem

• Addressing semantic and spatial detail loss in image deraining
• Integrating macro-semantic and micro-structural priors for rain removal
• Improving global context modeling and local detail recovery

Innovation

• Integrates CLIP and DINOv2 priors for semantic and structural guidance
• Uses progressive fusion injection to harmonize complementary prior cues
• Employs hierarchical Mamba with Fourier design for global-local modeling