RainDancer: RGB-Event Video Deraining with Rain-Oriented Spiking Dynamics

2026-07-15

AI Summary

Existing video deraining methods rely solely on RGB sequences, making it challenging to disentangle rain streaks from scene textures, motion, and occlusions in dynamic rainy scenes, often resulting in blurry or distorted reconstructions. To address this, this work proposes RainDancer, a novel framework adopting a “decompose-then-interact” paradigm: it first decouples rain and background components within both RGB and event modalities separately, then fuses them through semantic alignment. The method innovatively introduces a rain-guided spiking neural network to model the sparse, bursty dynamics in event streams and, for the first time, enforces structural consistency and gradient-direction supervision directly in the event domain to effectively suppress cross-modal interference. Experiments demonstrate that RainDancer achieves state-of-the-art quantitative metrics, superior visual quality, and enhanced robustness on downstream perception tasks across both synthetic and real-world RGB-event deraining benchmarks.

Abstract

Video deraining aims to recover clean visual content from rainy videos for reliable perception under adverse weather. Existing methods mainly rely on RGB sequences and temporal redundancy, but RGB-only restoration remains ambiguous in dynamic rainy scenes, where rain streaks, textures, boundaries, motion, and occlusions may share similar visual patterns. Event cameras provide complementary motion-sensitive cues with high temporal resolution, but event streams also contain sensor noise and background-triggered responses, so direct RGB-Event fusion may introduce cross-modal interference. To address this issue, we propose RainDancer, a progressive RGB-Event video deraining framework based on a decompose-before-interact paradigm. The core idea is to separate rain and background components within each modality before cross-modal interaction. In the RGB branch, frame features are progressively decomposed into rain and background representations. In the event branch, a rain-oriented spiking neural network module captures sparse and bursty event dynamics associated with rain motion. Component-level fusion is then performed between semantically aligned representations for structure preservation and rain suppression. We further introduce event-domain supervision to regularize sparse event reconstruction, structural consistency, and gradient orientation. Experiments on synthetic and real RGB-Event video deraining datasets demonstrate superior quantitative performance, visual quality, and downstream perception robustness. Code is available at https://github.com/AE86-plus/RainDancer.

Problem

• video deraining
• RGB-Event fusion
• rain streaks
• event cameras
• cross-modal interference

Innovation

• RGB-Event fusion
• decompose-before-interact
• spiking neural network
• rain-oriented dynamics
• component-level fusion