Dual-Correlation Hypergraph Network for Unaligned RGBT Video Object Detection and A Large-scale Benchmark

2026-07-09

AI Summary

This work addresses the performance degradation in RGB-thermal video object detection caused by spatial misalignment between modalities. To this end, the authors propose a Dual-Correlation Hypergraph Network (DHNet) that jointly models inter-frame temporal dependencies and cross-modal spatial correspondences through a Patch-level Spatial Alignment Module (PSAM) and a Dual Hypergraph Fusion Module (DHFM). Additionally, they introduce DVT-VOD1000, the first large-scale, multi-scenario benchmark dataset for RGBT video object detection, comprising 1,000 sequences. Extensive experiments on both VT-VOD50 and the newly curated DVT-VOD1000 demonstrate that DHNet significantly outperforms existing methods, achieving state-of-the-art performance and advancing the applicability of RGBT video object detection in complex real-world environments.

Abstract

RGB-Thermal (RGBT) Video Object Detection (VOD) has gained significant traction due to its ability to overcome the limitations of conventional RGB-based VOD under challenging conditions. However, spatial misalignment commonly exists between RGBT image pairs. To address this, we propose a Dual-Correlation Hypergraph Network (DHNet) that captures high-dimensional complementary information by explicitly modeling two types of correlations: temporal correlation across consecutive frames and spatial correlation from cross-modal features. Specifically, we first design a Patch-based Spatial Alignment Module (PSAM) to sequentially align the multimodal features at the local region level. Subsequently, we introduce a Dual Hypergraph Fusion Module (DHFM), which constructs separate temporal and multimodal hypergraphs to enhance object discriminability through dual-correlation learning. Furthermore, the field currently lacks a large-scale, scene-diverse benchmark dataset for comprehensive evaluation. To address this gap, we construct DVT-VOD1000, a large-scale RGBT VOD dataset containing 1,000 video sequences with 103,464 RGBT image pairs. The dataset covers diverse scenarios, including campuses, parks, transportation, rural areas, night scenes, rain, and snow. Comprehensive experiments on VT-VOD50 and our DVT-VOD1000 demonstrate that DHNet achieves state-of-the-art detection accuracy. The dataset and source code will be made publicly available on https://github.com/tzz-ahu/ to support academic research.

Problem

• RGBT Video Object Detection
• Spatial Misalignment
• Benchmark Dataset
• Unaligned Multimodal Data

Innovation

• Dual-Correlation Hypergraph Network
• Spatial Alignment
• Multimodal Fusion
• RGBT Video Object Detection
• Large-scale Benchmark