Modality-Decoupled RGB-Thermal Object Detector via Query Fusion

2026-01-13

AI Summary

This work addresses the challenge of RGB-T object detection under extreme lighting or adverse weather conditions, where performance often degrades due to reliance on a single modality. To harness both modality complementarity and decoupling, the authors propose a dual-branch DETR-style detection framework featuring a stage-wise cross-modal query fusion mechanism that preserves complementary strengths while enabling high-quality, query-guided dynamic modality decoupling. A novel query selection and adaptation module is introduced, allowing each branch to be independently optimized using unpaired RGB or thermal data, thereby significantly enhancing modality robustness and independence. Extensive experiments demonstrate that the proposed method outperforms existing RGB-T detectors across multiple benchmarks, with particularly notable gains in scenarios involving severe modality degradation.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The advantage of RGB-Thermal (RGB-T) detection lies in its ability to perform modality fusion and integrate cross-modality complementary information, enabling robust detection under diverse illumination and weather conditions. However, under extreme conditions where one modality exhibits poor quality and disturbs detection, modality separation is necessary to mitigate the impact of noise. To address this problem, we propose a Modality-Decoupled RGB-T detection framework with Query Fusion (MDQF) to balance modality complementation and separation. In this framework, DETR-like detectors are employed as separate branches for the RGB and TIR images, with query fusion interspersed between the two branches in each refinement stage. Herein, query fusion is performed by feeding the high-quality queries from one branch to the other one after query selection and adaptation. This design effectively excludes the degraded modality and corrects the predictions using high-quality queries. Moreover, the decoupled framework allows us to optimize each individual branch with unpaired RGB or TIR images, eliminating the need for paired RGB-T data. Extensive experiments demonstrate that our approach delivers superior performance to existing RGB-T detectors and achieves better modality independence.

Problem

• RGB-Thermal object detection
• modality fusion
• modality decoupling
• cross-modality noise
• unpaired data

Innovation

• modality decoupling
• query fusion
• RGB-Thermal object detection
• unpaired training
• DETR-based detector