Towards Accurate Binary Spiking Neural Networks: Learning with Adaptive Gradient Modulation Mechanism

2025-02-20

AI Summary

Binary spiking neural networks (BSNNs) suffer from slow convergence and low accuracy due to frequent weight sign flips—caused by binary weight constraints and non-differentiable spiking dynamics—disrupting optimization stability. Method: We propose the Adaptive Gradient Modulation Mechanism (AGMM), the first approach to theoretically characterize the intrinsic dynamical origin of sign flips in BSNNs. AGMM jointly regulates gradient scaling and adaptive learning rates to dynamically suppress gradient shocks, enhancing optimization robustness. It integrates spiking neural dynamics modeling with binary-constrained optimization. Results: On both static and neuromorphic benchmarks, AGMM achieves state-of-the-art accuracy for BSNNs, accelerates convergence by over 30%, reduces model size to 1/32 that of full-precision SNNs, and cuts inference energy consumption by more than 80%.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Neuro-Symbolic Learning
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Binary Spiking Neural Networks (BSNNs) inherit the eventdriven paradigm of SNNs, while also adopting the reduced storage burden of binarization techniques. These distinct advantages grant BSNNs lightweight and energy-efficient characteristics, rendering them ideal for deployment on resource-constrained edge devices. However, due to the binary synaptic weights and non-differentiable spike function, effectively training BSNNs remains an open question. In this paper, we conduct an in-depth analysis of the challenge for BSNN learning, namely the frequent weight sign flipping problem. To mitigate this issue, we propose an Adaptive Gradient Modulation Mechanism (AGMM), which is designed to reduce the frequency of weight sign flipping by adaptively adjusting the gradients during the learning process. The proposed AGMM can enable BSNNs to achieve faster convergence speed and higher accuracy, effectively narrowing the gap between BSNNs and their full-precision equivalents. We validate AGMM on both static and neuromorphic datasets, and results indicate that it achieves state-of-the-art results among BSNNs. This work substantially reduces storage demands and enhances SNNs' inherent energy efficiency, making them highly feasible for resource-constrained environments.

Problem

• BSNNs struggle with frequent weight sign flipping
• Training BSNNs effectively remains unresolved
• AGMM reduces weight sign flipping frequency

Innovation

• Adaptive Gradient Modulation Mechanism
• Binary Spiking Neural Networks
• Reduced weight sign flipping