HM-DF SNN: Transcending Conventional Online Learning with Advanced Training and Deployment

2024-10-10

AI Summary

To address two critical bottlenecks in online learning for spiking neural networks (SNNs)—non-differentiable temporal gradients and surrogate gradient mismatch—this paper proposes the Hybrid-Mechanism-driven Firing (HM-DF) model. Methodologically, HM-DF introduces a novel spiking computation paradigm featuring dual adaptive thresholds (upper and lower), enabling gradient separability through differential threshold dynamics. It further integrates online spatio-temporal backpropagation with a lightweight surrogate gradient adaptation framework, facilitating full-stage end-to-end optimization. Experimentally, HM-DF achieves state-of-the-art online learning performance across multiple benchmark datasets: it surpasses conventional spike-timing-based backpropagation (STBP) in inference accuracy, reduces training memory consumption by 42%, and simultaneously improves inference speed and energy efficiency—without increasing power consumption.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Learning on the Edge & Model Compression
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Social networks and social learning
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Spiking Neural Networks (SNNs) are considered to have enormous potential in the future development of Artificial Intelligence due to their brain-inspired and energy-efficient properties. Compared to vanilla Spatial-Temporal Back-propagation (STBP) training methods, online training can effectively overcome the risk of GPU memory explosion. However, current online learning framework cannot tackle the inseparability problem of temporal dependent gradients and merely aim to optimize the training memory, resulting in no performance advantages compared to the STBP training models in the inference phase. To address the aforementioned challenges, we propose Hybrid Mechanism-Driven Firing (HM-DF) model, which is a family of advanced models that respectively adopt different spiking calculation schemes in the upper-region and lower-region of the firing threshold. We point out that HM-DF model can effectively separate temporal gradients and tackle the mismatch problem of surrogate gradients, as well as achieving full-stage optimization towards computation speed and memory footprint. Experimental results have demonstrated that HM-DF model can be flexibly combined with various techniques to achieve state-of-the-art performance in the field of online learning, without triggering further power consumption.

Problem

• Overcoming GPU memory explosion risk in SNN training
• Solving inseparability of temporal dependent gradients
• Optimizing computation speed and memory footprint

Innovation

• Hybrid Mechanism-Driven Firing model separates temporal gradients
• Different spiking schemes above and below firing threshold
• Optimizes computation speed and memory footprint together