Learning A Robust RGB-Thermal Detector for Extreme Modality Imbalance

2025-05-28

AI Summary

RGB-T object detection faces severe out-of-distribution (OOD) issues and training instability under extreme modality imbalance—common in real-world scenarios due to environmental or hardware constraints. To address this, we propose a baseline-assisted dual-detector architecture. Our method features: (1) a novel baseline-assisted collaborative training framework; (2) a modality-quality-aware dynamic weighting mechanism for adaptive cross-modal feature interaction; and (3) a modality pseudo-degradation data augmentation strategy that realistically emulates imbalanced modality distributions. Leveraging dual-branch design, interactive attention modules, and consistency-constrained learning, our approach reduces miss detection rates by 55% under extreme degradation. It significantly enhances robustness and generalization across multiple state-of-the-art detectors, demonstrating superior performance on benchmark RGB-T datasets without requiring architectural modifications to the base detectors.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

RGB-Thermal (RGB-T) object detection utilizes thermal infrared (TIR) images to complement RGB data, improving robustness in challenging conditions. Traditional RGB-T detectors assume balanced training data, where both modalities contribute equally. However, in real-world scenarios, modality degradation-due to environmental factors or technical issues-can lead to extreme modality imbalance, causing out-of-distribution (OOD) issues during testing and disrupting model convergence during training. This paper addresses these challenges by proposing a novel base-and-auxiliary detector architecture. We introduce a modality interaction module to adaptively weigh modalities based on their quality and handle imbalanced samples effectively. Additionally, we leverage modality pseudo-degradation to simulate real-world imbalances in training data. The base detector, trained on high-quality pairs, provides a consistency constraint for the auxiliary detector, which receives degraded samples. This framework enhances model robustness, ensuring reliable performance even under severe modality degradation. Experimental results demonstrate the effectiveness of our method in handling extreme modality imbalances~(decreasing the Missing Rate by 55%) and improving performance across various baseline detectors.

Problem

• Addresses extreme modality imbalance in RGB-Thermal object detection
• Proposes adaptive modality weighting for imbalanced data handling
• Enhances detector robustness under severe modality degradation

Innovation

• Base-and-auxiliary detector architecture for imbalance
• Modality interaction module adaptively weighs modalities
• Modality pseudo-degradation simulates real-world imbalances