DelRec: learning delays in recurrent spiking neural networks

2025-09-29

AI Summary

This work addresses the limitation in spiking neural networks (SNNs) where synaptic and axonal delays are fixed, thereby constraining temporal modeling capability. We propose DelRec—the first trainable delay framework compatible with arbitrary spiking neuron models. Its core innovations are: (1) end-to-end optimization of non-integer delays via differentiable interpolation combined with surrogate gradient learning; and (2) the first integration of learnable delays into recurrent spiking layers, decoupling delay modulation from neuronal state dynamics while maintaining compatibility with standard models such as leaky integrate-and-fire (LIF). Evaluated on Spiking Speech Commands and psMNIST, DelRec achieves state-of-the-art accuracy; on Heidelberg Digits, it matches the best reported performance. Remarkably, even with basic neuron models, DelRec demonstrates significant energy efficiency gains, underscoring the critical role of recurrent delay adaptation in SNN temporal representation learning.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Natural Language Processing: Learning & Optimization for NLP
• Machine Learning: Deep Neural Architectures and Foundation Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Spiking neural networks (SNNs) are a bio-inspired alternative to conventional real-valued deep learning models, with the potential for substantially higher energy efficiency. Interest in SNNs has recently exploded due to a major breakthrough: surrogate gradient learning (SGL), which allows training SNNs with backpropagation, strongly outperforming other approaches. In SNNs, each synapse is characterized not only by a weight but also by a transmission delay. While theoretical works have long suggested that trainable delays significantly enhance expressivity, practical methods for learning them have only recently emerged. Here, we introduce ''DelRec'', the first SGL-based method to train axonal or synaptic delays in recurrent spiking layers, compatible with any spiking neuron model. DelRec leverages a differentiable interpolation technique to handle non-integer delays with well-defined gradients at training time. We show that trainable recurrent delays outperform feedforward ones, leading to new state-of-the-art (SOTA) on two challenging temporal datasets (Spiking Speech Command, an audio dataset, and Permuted Sequential MNIST, a vision one), and match the SOTA on the now saturated Spiking Heidelberg Digit dataset using only vanilla Leaky-Integrate-and-Fire neurons with stateless (instantaneous) synapses. Our results demonstrate that recurrent delays are critical for temporal processing in SNNs and can be effectively optimized with DelRec, paving the way for efficient deployment on neuromorphic hardware with programmable delays. Our code is available at : https://github.com/alexmaxad/DelRec.

Problem

• Training axonal and synaptic delays in recurrent spiking neural networks
• Overcoming gradient challenges for non-integer delays with interpolation
• Enhancing temporal processing efficiency for neuromorphic hardware deployment

Innovation

• Trains axonal delays in recurrent spiking networks
• Uses differentiable interpolation for non-integer delays
• Achieves state-of-the-art on temporal datasets