Reconsidering the energy efficiency of spiking neural networks

2024-08-29

arXiv.org

AI Summary

Existing energy-efficiency evaluations of spiking neural networks (SNNs) often neglect memory access and data movement overheads, leading to misleading conclusions. This work addresses the problem from a hardware-centric perspective by establishing a fine-grained energy model that comprehensively accounts for multi-level memory hierarchies and neuromorphic/spatial dataflow architectures—enabling, for the first time, systematic quantification of SNNs’ true energy efficiency. We propose a dual-regularized, sparsity-aware training strategy that jointly constrains both weights and activations to enhance neuronal sparsity. Furthermore, we derive an energy-efficiency constraint that explicitly balances temporal window length and sparsity. On CIFAR-10, our VGG16-SNN (T=6) achieves 94.18% accuracy while consuming only 69% of the energy required by a quantized ANN with comparable accuracy—demonstrating the hardware energy advantage of highly sparse SNNs (>93% sparsity). The code is publicly available.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Hardware-aware ML
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Energy management for devices in mobile Web and WoT environments

Abstract

Spiking neural networks (SNNs) are generally regarded as more energy-efficient because they do not use multiplications. However, most SNN works only consider the counting of additions to evaluate energy consumption, neglecting other overheads such as memory accesses and data movement operations. This oversight can lead to a misleading perception of efficiency, especially when state-of-the-art SNN accelerators operate with very small time window sizes. In this paper, we present a detailed comparison of the energy consumption of artificial neural networks (ANNs) and SNNs from a hardware perspective. We provide accurate formulas for energy consumption based on classical multi-level memory hierarchy architectures, commonly used neuromorphic dataflow architectures, and our proposed improved spatial-dataflow architecture. Our research demonstrates that to achieve comparable accuracy and greater energy efficiency than ANNs, SNNs require strict limitations on both time window size T and sparsity s. For instance, with the VGG16 model and a fixed T of 6, the neuron sparsity rate must exceed 93% to ensure energy efficiency across most architectures. Inspired by our findings, we explore strategies to enhance energy efficiency by increasing sparsity. We introduce two regularization terms during training that constrain weights and activations, effectively boosting the sparsity rate. Our experiments on the CIFAR-10 dataset, using T of 6, show that our SNNs consume 69% of the energy used by optimized ANNs on spatial-dataflow architectures, while maintaining an SNN accuracy of 94.18%. This framework, developed using PyTorch, is publicly available for use and further research.

Problem

• Evaluate true energy efficiency of SNNs vs QNNs
• Analyze overheads like data movement and memory access
• Identify operational regimes favoring SNNs' energy efficiency

Innovation

• Mapping rate-encoded SNNs to equivalent QNNs
• Detailed energy model for computation and data
• Identifying SNN efficiency regimes with specific parameters