Modality-missing RGBT Tracking: Invertible Prompt Learning and High-quality Benchmarks

2023-12-25

International Journal of Computer Vision

AI Summary

To address the robustness challenge in RGB-T tracking—where performance drastically degrades under single-modality (infrared or visible-light) absence—this paper proposes the Reversible Prompt Learning (RPL) framework, the first to enable cross-modal feature complementarity and implicit reconstruction of missing modalities. Methodologically, RPL integrates a bidirectional prompt encoder based on invertible neural networks (INNs), cross-modal contrastive distillation, uncertainty-aware fusion, and synthetic-real joint augmentation. We further introduce MMRB, the first large-scale, real-scenario-driven RGB-T benchmark explicitly designed for modality-missing evaluation, featuring multi-level modality absence and dynamic occlusion. On MMRB, RPL achieves an AUC gain of 12.6% over state-of-the-art methods. Notably, under complete modality absence, it retains 78.3% of the original tracking accuracy while incurring only a 9-ms inference latency overhead.

Problem

• RGBT Tracking
• Modal Dropout
• Robustness

Innovation

• Reversible Prompt Learning
• RGBT Tracking
• Modal Dropout