FEMOT: Multi-Object Tracking using Frame and Event Cameras

2026-06-12

AI Summary

This work addresses the significant performance degradation of multi-object tracking with conventional RGB cameras under challenging conditions such as motion blur, low illumination, and overexposure. While event cameras offer high temporal resolution and high dynamic range, their application is hindered by the scarcity of large-scale annotated datasets. To bridge this gap, we introduce FEMOT, the first large-scale RGB-event multi-object tracking dataset, encompassing diverse real-world scenarios and 14 distinct challenge attributes. We further propose FEMOTR, a novel multimodal tracking framework that leverages frequency-domain decoupled fusion to effectively exploit the complementary information between RGB and event data. Extensive experiments on FEMOT and DSEC-MOT demonstrate the superiority of our approach, and we establish a comprehensive benchmark comprising over ten state-of-the-art trackers. Both the dataset and code are publicly released.

Abstract

Conventional RGB cameras have been widely used in multi-object tracking due to their ability to capture rich appearance and semantic information. However, their performance is often degraded under complex real-world challenges, such as motion blur, low illumination, and overexposure. Bio-inspired event cameras offer high temporal resolution and high dynamic range, providing complementary cues under extreme scenarios. Nevertheless, RGB-event multi-object tracking remains underexplored due to the lack of large-scale and well-annotated datasets. To address this issue, we propose FEMOT, a large-scale RGB-event multi-object tracking dataset that covers diverse real-world scenarios and 14 challenging attributes. With both RGB and event data as well as high-quality annotations, FEMOT provides a reliable platform for systematically evaluating RGB-event multi-object tracking methods. Based on FEMOT, we retrain and evaluate over ten strong trackers, thereby establishing a comprehensive benchmark for future research. Furthermore, we propose FEMOTR, a multimodal tracking framework that decouples RGB and event features and fuses them in the frequency domain, thereby effectively exploiting their complementary characteristics for robust object localization and identity association. Extensive experiments on FEMOT and DSEC-MOT datasets demonstrate the effectiveness of the proposed method. The source code and benchmark dataset have been released on https://github.com/Event-AHU/FEMOT.

Problem

• multi-object tracking
• event cameras
• RGB-event fusion
• dataset scarcity
• real-world challenges

Innovation

• event camera
• multi-object tracking
• multimodal fusion
• frequency domain fusion
• benchmark dataset