Intrinsically Stable Spiking Neural Networks: Overcoming the Performance Barrier in the Absence of Batch Normalization

2026-06-30

AI Summary

This work addresses the severe performance degradation in deep spiking neural networks (SNNs) caused by firing rate decay when batch normalization (BN) is removed, a common issue that existing dynamic normalization methods alleviate at the cost of hardware efficiency. To overcome this limitation, the authors propose Intrinsically Stable SNNs (IS-SNN), which integrate topology-aware weight standardization and enhanced residual connections to offline fuse normalization operations during training. This design completely eliminates the need for normalization layers during inference, enabling purely accumulative and highly efficient computation. IS-SNN achieves high-performance training of deep SNNs without any runtime normalization for the first time, attaining 68.05% top-1 accuracy on ImageNet—surpassing prior BN-free approaches—and matching or exceeding dynamic BN methods across VGG, ResNet, and Transformer architectures, while reducing FPGA lookup table usage by 96.4%.

Abstract

The performance of deep spiking neural networks (SNNs) often relies on batch normalization (BN). However, the advanced dynamic BN variants used in state-of-the-art models introduce runtime multiplications, which weaken the hardware-efficiency motivation of SNNs. To address this tension, we identify catastrophic firing-rate decay as a primary cause of severe performance degradation in normalization-free SNNs. Guided by this insight, this work proposes the Intrinsically Stable SNN (IS-SNN) architecture, which removes activation-normalization layers by enforcing signal homeostasis through topology-aware weight standardization and modified residual connections. By folding the standardization operations into static weights offline, IS-SNN removes the runtime statistics tracking and multiplications introduced by activation normalization, restoring an accumulation-oriented inference datapath. Comprehensive experiments show that IS-SNN achieves performance competitive with or superior to computationally expensive dynamic BN techniques across VGG, ResNet, and Transformer-based models. Notably, it achieves a competitive accuracy of 68.05\% on ImageNet and overcomes the severe depth limitations of prior BN-free attempts. Together with a 96.4\% reduction in FPGA lookup table resource consumption for neuron implementations, these results support IS-SNN as a practical framework for building accurate and hardware-friendly deep neuromorphic systems.

Problem

• Spiking Neural Networks
• Batch Normalization
• Hardware Efficiency
• Performance Degradation
• Normalization-Free

Innovation

• Spiking Neural Networks
• Batch Normalization
• Weight Standardization
• Hardware Efficiency
• Signal Homeostasis