Generalizable, real-time neural decoding with hybrid state-space models

2025-06-05

AI Summary

Real-time neural decoding demands both ultra-low latency and strong generalization—yet conventional RNNs suffer from poor generalization, while Transformers incur prohibitive computational overhead. To address this, we propose POSSM, a hybrid architecture integrating cross-attentional spike tokenization with a causal recurrent state space model (SSM), establishing the first causal online SSM paradigm tailored for neural decoding. Our work is the first to empirically validate cross-species transfer gains—leveraging pretraining on primate motor cortex data to enhance human handwriting decoding. Through multi-dataset pretraining and an optimized real-time inference framework, POSSM achieves Transformer-level accuracy across monkey motor cortical recording, human handwriting, and speech decoding tasks, while accelerating GPU inference by up to 9×. This advances closed-loop brain–computer interfaces by simultaneously satisfying stringent real-time and generalization requirements.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Neuro-Symbolic Learning
• Humans and AI: Brain-Sensing and Analysis

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Real-time decoding of neural activity is central to neuroscience and neurotechnology applications, from closed-loop experiments to brain-computer interfaces, where models are subject to strict latency constraints. Traditional methods, including simple recurrent neural networks, are fast and lightweight but often struggle to generalize to unseen data. In contrast, recent Transformer-based approaches leverage large-scale pretraining for strong generalization performance, but typically have much larger computational requirements and are not always suitable for low-resource or real-time settings. To address these shortcomings, we present POSSM, a novel hybrid architecture that combines individual spike tokenization via a cross-attention module with a recurrent state-space model (SSM) backbone to enable (1) fast and causal online prediction on neural activity and (2) efficient generalization to new sessions, individuals, and tasks through multi-dataset pretraining. We evaluate POSSM's decoding performance and inference speed on intracortical decoding of monkey motor tasks, and show that it extends to clinical applications, namely handwriting and speech decoding in human subjects. Notably, we demonstrate that pretraining on monkey motor-cortical recordings improves decoding performance on the human handwriting task, highlighting the exciting potential for cross-species transfer. In all of these tasks, we find that POSSM achieves decoding accuracy comparable to state-of-the-art Transformers, at a fraction of the inference cost (up to 9x faster on GPU). These results suggest that hybrid SSMs are a promising approach to bridging the gap between accuracy, inference speed, and generalization when training neural decoders for real-time, closed-loop applications.

Problem

• Bridging accuracy-speed gap in real-time neural decoding
• Enhancing generalization across sessions and species
• Reducing computational costs for low-resource settings

Innovation

• Hybrid state-space model with cross-attention tokenization
• Multi-dataset pretraining for efficient generalization
• Fast inference speed with high decoding accuracy