Perturbed State Space Feature Encoders for Optical Flow with Event Cameras

2025-04-14

AI Summary

To address the limited spatiotemporal modeling capability of existing neural networks for event-camera optical flow estimation, this paper proposes the Perturbed State-Space Feature Encoder (P-SSE). P-SSE introduces, for the first time, an adaptive perturbation mechanism applied to the state transition matrix, enhancing both the numerical stability and representation robustness of state-space models (SSMs). It further integrates bidirectional flow prediction with recurrent feature propagation to expand temporal context and supports multi-frame event voxel inputs. The architecture achieves a large receptive field while maintaining linear computational complexity. Evaluated end-to-end on DSEC-Flow and MVSEC, P-SSE reduces endpoint error (EPE) by 8.48% and 11.86%, respectively, establishing new state-of-the-art performance in event-based optical flow estimation.

Technology Category

• Intelligent Robots: State Estimation
• Computer Vision: Motion & Tracking
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

With their motion-responsive nature, event-based cameras offer significant advantages over traditional cameras for optical flow estimation. While deep learning has improved upon traditional methods, current neural networks adopted for event-based optical flow still face temporal and spatial reasoning limitations. We propose Perturbed State Space Feature Encoders (P-SSE) for multi-frame optical flow with event cameras to address these challenges. P-SSE adaptively processes spatiotemporal features with a large receptive field akin to Transformer-based methods, while maintaining the linear computational complexity characteristic of SSMs. However, the key innovation that enables the state-of-the-art performance of our model lies in our perturbation technique applied to the state dynamics matrix governing the SSM system. This approach significantly improves the stability and performance of our model. We integrate P-SSE into a framework that leverages bi-directional flows and recurrent connections, expanding the temporal context of flow prediction. Evaluations on DSEC-Flow and MVSEC datasets showcase P-SSE's superiority, with 8.48% and 11.86% improvements in EPE performance, respectively.

Problem

• Enhancing optical flow estimation with event cameras
• Overcoming temporal and spatial reasoning limitations in neural networks
• Improving stability and performance via perturbed state dynamics

Innovation

• Perturbed State Space Feature Encoders (P-SSE)
• Adaptive spatiotemporal feature processing
• Perturbation technique for SSM stability