🤖 AI Summary
This work addresses the challenge of adapting sparse, asynchronous event streams from neuromorphic cameras to mainstream deep learning models, which is hindered by existing approaches that rely on fixed temporal sub-binning and thereby disrupt fine-grained temporal structures, limiting high-frequency inference performance. To overcome this, the authors propose the FATE framework, which abandons conventional sub-binning and instead employs Pillar Encoding combined with continuous orthogonal polynomial bases within a macro accumulation window to approximate event dynamics, yielding dense pseudo-images that preserve rich temporal information. Furthermore, they introduce a frequency-aware training strategy augmented with a soft mean-teacher mechanism to generate high-density pseudo-labels, effectively decoupling supervision frequency during training from inference frequency. The method significantly outperforms strong baselines across multiple architectures, enabling robust object detection at up to 200 Hz with negligible increases in model parameters or inference latency.
📝 Abstract
Event cameras are bio-inspired sensors that asynchronously capture logarithmic intensity changes, offering inherent advantages in high-speed and high-dynamic-range scenarios. However, the sparse and asynchronous nature of event streams poses a fundamental challenge for modern deep learning architectures. To enable compatibility with standard models, most existing approaches partition the accumulation window into fixed temporal sub-bins. While effective for spatial processing, this internal discretization discards fine-grained temporal structure and constrains inference to the low temporal frequencies imposed by training supervision. To address this limitation, we propose FATE, a unified framework built upon a novel Pillar Encoding (PE). While operating over discrete macro-accumulation windows dictated by the target frequency, PE avoids internal temporal sub-binning. It organizes events into spatial pillars and approximates their intra-window evolution via projection onto a continuous-time orthogonal polynomial basis. This formulation yields an L2-optimal representation that retains rich temporal dynamics in a dense pseudo-image, mitigating information loss under sparse event conditions. To fully leverage this representation, we introduce Frequency-Aware Training (FAT), a soft mean-teacher curriculum that generates temporally dense pseudo-labels, effectively bridging the mismatch between low-frequency supervision and high-frequency inference. Extensive experiments demonstrate that FATE generalizes across architectural paradigms and consistently outperforms strong baselines. It enables robust object detection at high temporal resolutions up to 200 Hz, while incurring minimal overhead in parameter count and inference latency