SpikeDerain: Unveiling Clear Videos from Rainy Sequences Using Color Spike Streams

2025-03-26

AI Summary

Rain streak removal in rainy videos faces challenges including difficulty modeling high-speed raindrop motion with frame-based sensors and limitations of multimodal approaches due to hardware asynchrony and computational redundancy. This paper proposes the first color spike stream deraining paradigm specifically designed for rain streak interference, leveraging neuromorphic cameras to capture high-temporal-resolution spike events that precisely characterize dynamic rain streaks. We introduce a physically interpretable continuous rainfall spike synthesis model to mitigate the scarcity of real-world rainy spike data, and design an end-to-end cross-modal deraining network operating natively in the spike domain to enable efficient RGB–event co-modeling. Experiments demonstrate superior robustness under extreme rainfall conditions, significantly outperforming existing frame-based and event-based methods, and establishing new state-of-the-art performance across multiple benchmarks.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Restoring clear frames from rainy videos presents a significant challenge due to the rapid motion of rain streaks. Traditional frame-based visual sensors, which capture scene content synchronously, struggle to capture the fast-moving details of rain accurately. In recent years, neuromorphic sensors have introduced a new paradigm for dynamic scene perception, offering microsecond temporal resolution and high dynamic range. However, existing multimodal methods that fuse event streams with RGB images face difficulties in handling the complex spatiotemporal interference of raindrops in real scenes, primarily due to hardware synchronization errors and computational redundancy. In this paper, we propose a Color Spike Stream Deraining Network (SpikeDerain), capable of reconstructing spike streams of dynamic scenes and accurately removing rain streaks. To address the challenges of data scarcity in real continuous rainfall scenes, we design a physically interpretable rain streak synthesis model that generates parameterized continuous rain patterns based on arbitrary background images. Experimental results demonstrate that the network, trained with this synthetic data, remains highly robust even under extreme rainfall conditions. These findings highlight the effectiveness and robustness of our method across varying rainfall levels and datasets, setting new standards for video deraining tasks. The code will be released soon.

Problem

• Restoring clear videos from rainy sequences with fast-moving rain streaks
• Overcoming hardware and computational challenges in rain streak removal
• Generating synthetic rain data for robust deraining model training

Innovation

• Color Spike Stream Deraining Network for rain removal
• Physically interpretable rain streak synthesis model
• High robustness under extreme rainfall conditions