Neuromorphic spatiotemporal optical flow: Enabling ultrafast visual perception beyond human capabilities

2024-09-10

AI Summary

Existing optical flow methods suffer from high latency (~0.6 s—four times human reaction time), hindering real-time robotic navigation in dynamic environments. This work proposes the first spatiotemporal joint neuromorphic optical flow architecture, implemented on a 2D van der Waals heterostructure floating-gate synaptic transistor array that natively encodes temporal dynamics in hardware, enabling millisecond-scale (1–2 ms) motion region localization. By embedding temporal priors directly into synaptic devices—bypassing conventional frame-based spatial modeling—it overcomes latency bottlenecks and achieves spatiotemporally consistent motion representation. The hardware delivers ultrafast response (<100 μs), non-volatility (>10⁴ s), and endurance (>8×10³ cycles). On the software side, inference latency drops to ~150 ms—4× lower than state-of-the-art—achieving 400% speedup while maintaining or improving accuracy.

Technology Category

• Machine Learning: Hardware-aware ML
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

Optical flow, inspired by the mechanisms of biological visual systems, calculates spatial motion vectors within visual scenes that are necessary for enabling robotics to excel in complex and dynamic working environments. However, current optical flow algorithms, despite human-competitive task performance on benchmark datasets, remain constrained by unacceptable time delays (~0.6 seconds per inference, 4X human processing speed) in practical deployment. Here, we introduce a neuromorphic optical flow approach that addresses delay bottlenecks by encoding temporal information directly in a synaptic transistor array to assist spatial motion analysis. Compared to conventional spatial-only optical flow methods, our spatiotemporal neuromorphic optical flow offers the spatial-temporal consistency of motion information, rapidly identifying regions of interest in as little as 1-2 ms using the temporal motion cues derived from the embedded temporal information in the two-dimensional floating gate synaptic transistors. Thus, the visual input can be selectively filtered to achieve faster velocity calculations and various task execution. At the hardware level, due to the atomically sharp interfaces between distinct functional layers in two-dimensional van der Waals heterostructures, the synaptic transistor offers high-frequency response (~100 {mu}s), robust non-volatility (>10000 s), and excellent endurance (>8000 cycles), enabling robust visual processing. In software benchmarks, our system outperforms state-of-the-art algorithms with a 400% speedup, frequently surpassing human-level performance while maintaining or enhancing accuracy by utilizing the temporal priors provided by the embedded temporal information.

Problem

• Optical Flow
• Real-time Processing
• Robotics Navigation

Innovation

• Neuromorphic Temporal-Spatial Optical Flow
• Integrated Time Information
• High-Speed Visual Processing