CLIF: Complementary Leaky Integrate-and-Fire Neuron for Spiking Neural Networks

2024-02-07

International Conference on Machine Learning

AI Summary

Spiking neural networks (SNNs) suffer from training difficulties due to the non-differentiability of leaky integrate-and-fire (LIF) neurons, resulting in gradient vanishing across time steps and inferior accuracy compared to artificial neural networks (ANNs). To address this, we propose the complementary LIF (CLIF) neuron—a novel spiking unit featuring a hyperparameter-free dual-path integration-and-firing mechanism. CLIF preserves strictly binary spike outputs while introducing an auxiliary differentiable pathway to mitigate temporal gradient decay. The design is theoretically sound and plug-and-play compatible with existing SNN architectures. Extensive experiments using surrogate gradient learning demonstrate that CLIF consistently outperforms state-of-the-art spiking neuron models across multiple benchmark datasets. Notably, under identical network topologies and training protocols, CLIF-based SNNs achieve accuracy on par with—or slightly exceeding—that of their ANN counterparts. The implementation is publicly available.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Natural Language Processing: Learning & Optimization for NLP
• Machine Learning: Deep Neural Architectures and Foundation Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Spiking neural networks (SNNs) are promising brain-inspired energy-efficient models. Compared to conventional deep Artificial Neural Networks (ANNs), SNNs exhibit superior efficiency and capability to process temporal information. However, it remains a challenge to train SNNs due to their undifferentiable spiking mechanism. The surrogate gradients method is commonly used to train SNNs, but often comes with an accuracy disadvantage over ANNs counterpart. We link the degraded accuracy to the vanishing of gradient on the temporal dimension through the analytical and experimental study of the training process of Leaky Integrate-and-Fire (LIF) Neuron-based SNNs. Moreover, we propose the Complementary Leaky Integrate-and-Fire (CLIF) Neuron. CLIF creates extra paths to facilitate the backpropagation in computing temporal gradient while keeping binary output. CLIF is hyperparameter-free and features broad applicability. Extensive experiments on a variety of datasets demonstrate CLIF's clear performance advantage over other neuron models. Furthermore, the CLIF's performance even slightly surpasses superior ANNs with identical network structure and training conditions. The code is available at https://github.com/HuuYuLong/Complementary-LIF.

Problem

• Improves training of spiking neural networks
• Addresses vanishing gradients in temporal dimension
• Enhances accuracy over traditional neuron models

Innovation

• CLIF Neuron enhances temporal gradient backpropagation
• CLIF maintains binary output without hyperparameters
• CLIF outperforms ANNs in identical conditions