FLEET: Token-Based Feature Extraction for Event Camera-based Reinforcement Learning

2026-08-17

AI Summary

This study addresses the high computational costs and temporal ambiguity inherent in reinforcement learning with event cameras by proposing FLEET, a novel feature extractor. Integrating random Fourier features, cross-attention, and tokenization, this method compresses variable-length event streams into fixed-size latent representations, thereby decoupling inference cost from resolution and enabling end-to-end learning without auxiliary losses. Experimental results demonstrate that FLEET outperforms state-of-the-art approaches on high-throughput benchmarks while exhibiting superior robustness to variations in observation frequency. Consequently, this work establishes a new paradigm for efficient event-driven control, effectively mitigating critical bottlenecks in processing asynchronous visual data for robotic applications.

Abstract

Event cameras generate asynchronous, high-frequency data streams offering spatially sparse information at lower latency than traditional cameras.In principle, these properties should be ideal for the design of control policies.However, reinforcement learning research in this field remains limited as existing approaches fail to fully exploit the sensor's properties.CNN-based methods negate the sensors benefits by aggregating events into sparse grids. This couples compute cost to sensor resolution and blurs the temporal information. Meanwhile, existing generative baselines rely on the availability of trajectory data to pretrain the model. We propose FLEET (Feature Learning from Events via Efficient Tokenization), a feature extractor that processes event sequences directly. Leveraging random Fourier features and cross-attention, our architecture compresses variable streams into fixed-size latent representations. This decouples inference cost of the feature extractor's backbone from the sensor's resolution, enabling end-to-end learning without auxiliary losses. We validate FLEET on a new, high-throughput benchmark. The results demonstrate that our sequence-based approach surpasses SOTA performance and exhibits superior robustness to variations in observation frequencies.

Problem

• Event Camera
• Reinforcement Learning
• Feature Extraction
• Asynchronous Data

Innovation

• Event Camera
• Tokenization
• Random Fourier Features
• Cross-Attention
• Reinforcement Learning