CMTFormer: Marrying Transformer with Hierarchical Information Interaction for RGB-Event Object Detection

2026-06-27

AI Summary

This work addresses the challenges of fusing RGB images and event streams—namely, modality heterogeneity, noise amplification, and redundant features—by proposing CMTFormer, a Transformer-based, stage-wise cross-modal interaction architecture. The method mitigates modality discrepancies through shallow-layer alignment, enhances feature representation in intermediate layers by leveraging texture and edge cues, and adaptively aggregates high-level semantics in deep layers via learnable weights. A spatial prior module is further integrated to improve localization accuracy. Evaluated on the DSEC-Detection and PKU-DAVIS-SOD benchmarks, CMTFormer substantially outperforms existing unimodal and multimodal approaches, demonstrating its effectiveness and state-of-the-art performance in RGB-event collaborative detection.

Abstract

Event cameras capture sparse brightness changes with high temporal resolution and high dynamic range, compensating for the deficiencies of the conventional RGB frames. However, previous multi-modal fusion techniques typically fail to handle the inherent heterogeneity between RGB frames and event streams, thus easily leading to noise amplification or redundant feature integration during cross-modal fusion. In this paper, we propose a Cross-Modal information inTeraction transFormer, coined as CMTFormer, which hierarchically integrates RGB and event information to achieve efficient and stable multimodal collaboration. Specifically, we design a shallow-to-deep information interaction scheme. In the shallow stage, we present the Shallow Alignment Module (SAM) to achieve an efficient fusion of RGB and event low-level features, which mitigates attribute disparities and prevents noisy information. In the middle stage, we devise the Cross-modal Enhancement Module (CEM) that utilizes texture and edge information to produce mutually reinforced middle-level features. In the deep stage, we present the Learnable Deep Fusion Module (LDFM) which performs high-level information aggregation through learnable weights, thus enabling the network to adaptively fuse RGB and event clues. A Spatial Prior Module is further designed to utilize global spatial information to enhance localization accuracy. Extensive experiments are conducted on two prevalent event-based object detection benchmarks, i.e., DSEC-Detection and PKU-DAVIS-SOD. Our CMTFormer consistently surpasses the detection counterparts in both uni-modal and multi-modal settings, strongly demonstrating the effectiveness of our paradigm. Codes will be available upon publication.

Problem

• RGB-Event Fusion
• Object Detection
• Cross-Modal Heterogeneity
• Event Camera
• Multimodal Integration

Innovation

• CMTFormer
• hierarchical information interaction
• RGB-event fusion
• cross-modal enhancement
• event-based object detection