Efficient RGB-T Object Detection via Sparse Cross-Modality Fusion

2026-06-29

AI Summary

This work addresses the high computational cost and deployment challenges of existing RGB-T object detection methods, which typically rely on dual-backbone architectures and dense cross-modal fusion across the entire image. To mitigate these limitations, the authors propose a sparse cross-modal fusion mechanism that first employs a lightweight single-modality model to generate high-recall region proposals. Multi-modal feature fusion and refinement are then performed exclusively within these sparse candidate regions. This two-stage framework substantially reduces redundant computation, achieving competitive detection accuracy while significantly lowering both model parameters and computational overhead. Moreover, the approach demonstrates strong scalability to high-resolution images, offering an efficient and practical solution for real-world RGB-T detection tasks.

Abstract

RGB-T detectors leverage the complementary strengths of visible and thermal infrared modalities, achieving robust performance under challenging conditions. Many of them resort to heavy dual backbones and exhaustive cross-modality fusion across the entire image, leading to impractically high computational costs. We observe that most image regions are smooth backgrounds (e.g., sky, ground) that can be easily handled by lightweight single-modality models. In light of this observation, we propose a sparse fusion mechanism for efficient RGB-T detection: first rapidly scanning the image to identify the proposals and then carefully examining the remaining sparse proposals via feature fusion. We propose a two-stage framework to instantiate this mechanism, which performs detection in two stages: 1) a lightweight and modality-specific detection stage that produces high-recall RoIs, and 2) a fusion-driven examination and refinement stage that filters out the false positives and refines the bounding boxes. This design enables the detector to adaptively allocate more computational resources to the potential foregrounds, improving the efficiency while ensuring detection accuracy. Extensive experiments show that our method achieves competitive performance with substantially fewer parameters and lower cost, while maintaining strong scalability to high-resolution images.

Problem

• RGB-T object detection
• cross-modality fusion
• computational efficiency
• dual backbones
• sparse fusion

Innovation

• sparse fusion
• RGB-T object detection
• cross-modality fusion
• two-stage detection
• efficient detection