MD-SNN: Membrane Potential-aware Distillation on Quantized Spiking Neural Network

2025-12-03

AI Summary

Quantizing spiking neural networks (SNNs) often causes membrane potential mismatch, leading to substantial accuracy degradation. To address this, we propose a membrane-potential-aware knowledge distillation framework—the first to leverage dynamic membrane potentials as the distillation supervision signal. Our method jointly integrates quantization-aware training, batch normalization (BN) fusion, and spatio-temporal decorrelation to align the membrane potential evolution between full-precision and quantized SNNs, thereby mitigating activation deviations induced by weight and BN-layer quantization. The approach uniformly supports both static image datasets (CIFAR-10/100, TinyImageNet) and event-based neuromorphic data (N-Caltech101), achieving significant accuracy improvements across multiple benchmarks. Hardware evaluation demonstrates a 14.85× reduction in energy-delay-area product (EDAP), a 2.64× improvement in energy efficiency, and a 6.19× gain in area efficiency.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Hardware-aware ML
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Spiking Neural Networks (SNNs) offer a promising and energy-efficient alternative to conventional neural networks, thanks to their sparse binary activation. However, they face challenges regarding memory and computation overhead due to complex spatio-temporal dynamics and the necessity for multiple backpropagation computations across timesteps during training. To mitigate this overhead, compression techniques such as quantization are applied to SNNs. Yet, naively applying quantization to SNNs introduces a mismatch in membrane potential, a crucial factor for the firing of spikes, resulting in accuracy degradation. In this paper, we introduce Membrane-aware Distillation on quantized Spiking Neural Network (MD-SNN), which leverages membrane potential to mitigate discrepancies after weight, membrane potential, and batch normalization quantization. To our knowledge, this study represents the first application of membrane potential knowledge distillation in SNNs. We validate our approach on various datasets, including CIFAR10, CIFAR100, N-Caltech101, and TinyImageNet, demonstrating its effectiveness for both static and dynamic data scenarios. Furthermore, for hardware efficiency, we evaluate the MD-SNN with SpikeSim platform, finding that MD-SNNs achieve 14.85X lower energy-delay-area product (EDAP), 2.64X higher TOPS/W, and 6.19X higher TOPS/mm2 compared to floating point SNNs at iso-accuracy on N-Caltech101 dataset.

Problem

• Addresses accuracy loss from quantization in Spiking Neural Networks.
• Mitigates membrane potential mismatch in quantized SNN training.
• Improves hardware efficiency and energy consumption in SNNs.

Innovation

• Membrane potential-aware distillation for quantized SNNs
• Mitigates quantization mismatch in weight and membrane potential
• Enhances hardware efficiency with lower energy-delay-area product