FlexEvent: Towards Flexible Event-Frame Object Detection at Varying Operational Frequencies

2024-12-09

AI Summary

Existing event detectors are constrained by fixed sampling frequencies, compromising simultaneous high temporal resolution and adaptability to dynamic scenes. To address this, we propose the first frequency-variable detection framework, breaking away from conventional synchronous paradigms. Our method introduces (i) the FlexFuse module for adaptive spatiotemporal fusion of event streams and RGB frames; (ii) the FlexTune mechanism for frequency-aware label refinement; and (iii) a joint learning paradigm integrating asynchronous event modeling with multimodal representation learning. Evaluated on large-scale event datasets, our approach significantly outperforms state-of-the-art methods, maintaining robust performance across 20–90 Hz while enabling real-time detection at up to 180 Hz. It establishes new benchmarks in generalization across static and dynamic scenes, as well as in robustness to varying motion patterns and illumination conditions.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications

Abstract

Event cameras offer unparalleled advantages for real-time perception in dynamic environments, thanks to the microsecond-level temporal resolution and asynchronous operation. Existing event detectors, however, are limited by fixed-frequency paradigms and fail to fully exploit the high-temporal resolution and adaptability of event data. To address these limitations, we propose FlexEvent, a novel framework that enables detection at varying frequencies. Our approach consists of two key components: FlexFuse, an adaptive event-frame fusion module that integrates high-frequency event data with rich semantic information from RGB frames, and FlexTune, a frequency-adaptive fine-tuning mechanism that generates frequency-adjusted labels to enhance model generalization across varying operational frequencies. This combination allows our method to detect objects with high accuracy in both fast-moving and static scenarios, while adapting to dynamic environments. Extensive experiments on large-scale event camera datasets demonstrate that our approach surpasses state-of-the-art methods, achieving significant improvements in both standard and high-frequency settings. Notably, our method maintains robust performance when scaling from 20 Hz to 90 Hz and delivers accurate detection up to 180 Hz, proving its effectiveness in extreme conditions. Our framework sets a new benchmark for event-based object detection and paves the way for more adaptable, real-time vision systems. Code is publicly available.

Problem

• Enables object detection at varying operational frequencies
• Adapts to dynamic environments with high-temporal resolution
• Integrates event data and RGB frames for improved accuracy

Innovation

• FlexEvent enables varying-frequency event detection
• FlexFuse integrates event and RGB frame data
• FlexTune adapts labels for different frequencies