Scalable Training of Continuous-Time Spiking Neural Networks with Differentiable Spike-Time Discretization

2026-07-16

AI Summary

This work addresses the prohibitive memory overhead of deep continuous-time spiking neural networks (SNNs), which arises from the precise tracking of spike timings and hinders scalability. To overcome this limitation, the authors propose a Differentiable Spike Time Discretization (DSTD) framework that maps irregular presynaptic spikes onto fixed time steps as differentiable weighted events, accurately approximating continuous membrane potential dynamics while drastically reducing memory consumption. By integrating the leaky integrate-and-fire (LIF) neuron model, time-to-first-spike (TTFS) encoding, and a temporal regularization mechanism inspired by synchronous firing chains, the approach effectively mitigates neuronal death and enables pipeline-like training. Experiments demonstrate successful training of a 9-layer CIFAR-10 and a 20-layer Fashion-MNIST convolutional SNN on a single GPU, achieving approximately 100× lower peak memory usage and 20× faster training speed.

Abstract

Continuous-time spiking neural networks (SNNs) provide an event-driven framework for temporal computation, computational neuroscience, and neuromorphic hardware. However, training deep continuous-time SNNs is severely constrained by the memory required for exact spike-time computation, which evaluates and retains candidate firing times over intervals determined by presynaptic spike ordering. Here we introduce a memory-efficient training framework based on differentiable spike-time discretization (DSTD) for leaky integrate-and-fire neurons with general membrane and synaptic time constants. DSTD maps irregular presynaptic spikes onto differentiable weighted events at fixed time points, replacing the input-dependent candidate dimension with $M$ fixed time intervals while accurately approximating continuous-time membrane-potential dynamics. This reduces candidate-related activation memory from $O(N_{\mathrm{out}}N_{\mathrm{in}})$ to $O(N_{\mathrm{out}}M)$ in the case of time-to-first-spike (TTFS) coding, where $N_{\mathrm{in}}$ and $N_{\mathrm{out}}$ denote the numbers of presynaptic and postsynaptic neurons, respectively. We further introduce synfire-chain-inspired temporal regularization that organizes layer-wise firing windows, mitigates dead-neuron failures, and enables pipeline-like processing. In dense LIF layers, DSTD reduced peak memory consumption by up to approximately 100-fold and training time by up to approximately 20-fold compared with exact spike-time computation. Together, these methods allowed us to train 9-layer convolutional SNNs on CIFAR-10 and 20-layer convolutional SNNs on Fashion-MNIST on a single GPU.

Problem

• spiking neural networks
• continuous-time
• memory efficiency
• scalable training
• spike-time computation

Innovation

• Differentiable Spike-Time Discretization
• Continuous-time Spiking Neural Networks
• Memory-Efficient Training
• Temporal Regularization
• Leaky Integrate-and-Fire Neurons