TDE-3: An improved prior for optical flow computation in spiking neural networks

2024-02-18

arXiv.org

AI Summary

To address the degraded direction selectivity of conventional two-node time-difference encoders (TDE-2) in textured environments—leading to reduced accuracy and energy efficiency in motion detection—this paper proposes TDE-3, a three-node spiking neural encoder with inhibitory input for event-camera-driven optical flow perception. Methodologically, we introduce, for the first time, a dual encoding scheme combining inter-spike interval (ISI) and spike count, trained via backpropagation through time (BPTT) with surrogate gradients. We also conduct the first quantitative joint evaluation of accuracy and energy efficiency for TDE-based optical flow encoding. Experiments show that TDE-3 achieves 20° angular error and 88% ground-truth correlation on synthetic event streams—matching state-of-the-art models in accuracy—while significantly enhancing directional selectivity per TDE unit and reducing overall spike rate, thereby outperforming TDE-2 in energy efficiency. Moreover, ISI encoding demonstrates superior robustness to spatial frequency variations.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Motion & Tracking
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Motion detection is a primary task required for robotic systems to perceive and navigate in their environment. Proposed in the literature bioinspired neuromorphic Time-Difference Encoder (TDE-2) combines event-based sensors and processors with spiking neural networks to provide real-time and energy-efficient motion detection through extracting temporal correlations between two points in space. However, on the algorithmic level, this design leads to loss of direction-selectivity of individual TDEs in textured environments. Here we propose an augmented 3-point TDE (TDE-3) with additional inhibitory input that makes TDE-3 direction-selectivity robust in textured environments. We developed a procedure to train the new TDE-3 using backpropagation through time and surrogate gradients to linearly map input velocities into an output spike count or an Inter-Spike Interval (ISI). Our work is the first instance of training a spiking neuron to have a specific ISI. Using synthetic data we compared training and inference with spike count and ISI with respect to changes in stimuli dynamic range, spatial frequency, and level of noise. ISI turns out to be more robust towards variation in spatial frequency, whereas the spike count is a more reliable training signal in the presence of noise. We performed the first in-depth quantitative investigation of optical flow coding with TDE and compared TDE-2 vs TDE-3 in terms of energy-efficiency and coding precision. Results show that on the network level both detectors show similar precision (20 degree angular error, 88% correlation with ground truth). Yet, due to the more robust direction-selectivity of individual TDEs, TDE-3 based network spike less and hence is more energy-efficient. Reported precision is on par with model-based methods but the spike-based processing of the TDEs provides allows more energy-efficient inference with neuromorphic hardware.

Problem

• Enhancing direction-selectivity in spiking neural networks for motion detection
• Training spiking neurons to output specific Inter-Spike Intervals (ISI)
• Comparing energy-efficiency and precision of TDE-2 and TDE-3 networks

Innovation

• Augmented 3-point TDE with inhibitory input
• Training using backpropagation through time
• Robust direction-selectivity in textured environments