Neuromorphic visual attention for Sign-language recognition on SpiNNaker

2026-05-07

AI Summary

This work addresses the challenge of achieving both real-time performance and energy efficiency in sign language recognition by proposing the first end-to-end neuromorphic system dedicated to this task. The system introduces a novel task-driven spiking visual attention mechanism integrated with a compact spiking neural network, enabling efficient edge deployment on the SpiNNaker platform. By synergistically combining neuromorphic sensing, event-driven computation, and specialized hardware, the approach achieves a simulation accuracy of 92.27% and a hardware-measured accuracy of 83.1%, while consuming only 0.565 mW of power and exhibiting an ultra-low latency of 3 ms. To the best of our knowledge, this represents the most energy-efficient solution currently available for sign language fingerspelling recognition.

Abstract

Sign-language recognition has achieved substantial gains in classification accuracy in recent years; however, the latency and power requirements of most existing methods limit their suitability for real-time deployment. Neuromorphic sensing and processing offer an alternative paradigm based on sparse, event-driven computation that supports low-latency and energy-efficient perception. In this work, we introduce an end-to-end neuromorphic architecture for American Sign Language (ASL) fingerspelling recognition that integrates a spiking visual attention mechanism for online region-of-interest extraction with a compact spiking neural network deployed on the SpiNNaker neuromorphic platform. We benchmark the proposed system against two datasets: a synthetically generated event-based version of the Sign Language MNIST dataset and a natively recorded ASL-DVS dataset, whilst providing a comprehensive overview of Sign-language recognition and related work. This work yields competitive performance in simulation (92.27%) and comparable performance on neuromorphic hardware deployment (83.1%), while achieving the most energy-efficient architecture (0.565 mW) and low latency (3 ms) across all benchmarked approaches. Despite its compact design, the system demonstrates the suitability of task-dependent visual attention applications for edge deployment.

Problem

• sign-language recognition
• real-time deployment
• latency
• power consumption
• neuromorphic computing

Innovation

• neuromorphic computing
• spiking neural networks
• visual attention
• event-based vision
• SpiNNaker