Weaving Light and Time: Unified Harmonic-Geometric Representation Learning for Dense RGB-Event Parsing

2026-07-10

AI Summary

Existing RGB-event fusion methods are often limited by computational redundancy or difficulties in handling geometric disparity and cross-spectral aliasing. This work proposes Evita, the first unified backbone network designed for dense RGB-event parsing, which embeds co-learning modules within each encoder layer to achieve deep modality fusion. Its core innovations include geometric disparity correction, harmonic spectral resonance, and a transient global routing mechanism, complemented by spatial alignment, complex-frequency-domain texture transfer, and event-driven asymmetric attention. Leveraging hybrid pretraining on N-ImageNetV2 and stochastic event representations, Evita supports arbitrary event formats and achieves new state-of-the-art results on DELIVER, DDD17, and DSEC, striking a superior balance between accuracy and latency.

Abstract

Fusing standard RGB frames with asynchronous event streams has emerged as a definitive paradigm for robust perception in degraded environments. Although unified backbones have recently gained traction in multi-modal vision, adapting them to the RGB-Event domain remains fundamentally challenging. Existing architectures either resort to decoupled dual encoders that double computational overhead, or adopt generic unified designs that fail to resolve implicit geometric parallax and cross-spectral aliasing under the extreme representational divide between dense intensity grids and sparse kinematic spikes. To transcend these bottlenecks, we present Evita, the first unified backbone specifically engineered for dedicated dense RGB-Event parsing. To achieve profound modal synergy, Evita explicitly embeds a suite of intrinsic co-learning modules directly into every encoder layer. Specifically, it features Geometric Parallax Rectification for adaptive spatial alignment, Harmonic Spectral Resonance for texture transfer exclusively in the complex frequency domain, and Transient Global Routing for event-driven asymmetric attention. To guarantee robust feature extraction against spatial misalignments and decouple representations from specific event encodings, we construct N-ImageNetV2 alongside a stochastic event representation mixing pretraining protocol, empowering the network to seamlessly accommodate arbitrary event formats in downstream tasks. Extensive evaluations across the DELIVER, DDD17, and DSEC benchmarks confirm that Evita establishes new state-of-the-art metrics while delivering a superior accuracy-latency trade-off for real-time multimodal perception.The code are publicly available at: https://github.com/chaineypung/Evita.

Problem

• RGB-Event fusion
• unified representation learning
• geometric parallax
• cross-spectral aliasing
• dense parsing

Innovation

• unified backbone
• geometric parallax rectification
• harmonic spectral resonance
• transient global routing
• event-based vision