SpikingMamba: Towards Energy-Efficient Large Language Models via Knowledge Distillation from Mamba

2025-10-06

AI Summary

To address the challenge of simultaneously improving energy efficiency and preserving performance in spiking neural network (SNN)-based large language models (LLMs), this paper proposes SpikingMamba—a highly energy-efficient sparsified LLM. Methodologically, we introduce TI-LIF, a ternary integer-valued leaky integrate-and-fire neuron, coupled with a smooth gradient compensation pathway to preserve semantic polarity and mitigate quantization-induced degradation. Furthermore, we employ single-stage knowledge distillation augmented with reinforcement learning to transfer zero-shot capabilities from Mamba without full pretraining, while incorporating sparse spike activations. Experiments demonstrate that SpikingMamba-1.3B achieves a 4.76× energy efficiency gain over its non-spiking counterpart while retaining strong zero-shot generalization. Its initial zero-shot accuracy drops by only 4.78%; after RL-based fine-tuning, it improves by an additional 2.55%, substantially outperforming existing spiking language models.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) have achieved remarkable performance across tasks but remain energy-intensive due to dense matrix operations. Spiking neural networks (SNNs) improve energy efficiency by replacing dense matrix multiplications with sparse accumulations. Their sparse spike activity enables efficient LLMs deployment on edge devices. However, prior SNN-based LLMs often sacrifice performance for efficiency, and recovering accuracy typically requires full pretraining, which is costly and impractical. To address this, we propose SpikingMamba, an energy-efficient SNN-based LLMs distilled from Mamba that improves energy efficiency with minimal accuracy sacrifice. SpikingMamba integrates two key components: (a) TI-LIF, a ternary-integer spiking neuron that preserves semantic polarity through signed multi-level spike representations. (b) A training-exclusive Smoothed Gradient Compensation (SGC) path mitigating quantization loss while preserving spike-driven efficiency. We employ a single-stage distillation strategy to transfer the zero-shot ability of pretrained Mamba and further enhance it via reinforcement learning (RL). Experiments show that SpikingMamba-1.3B achieves a 4.76$ imes$ energy benefit, with only a 4.78% zero-shot accuracy gap compared to the original Mamba, and achieves a further 2.55% accuracy improvement after RL.

Problem

• Developing energy-efficient spiking neural network large language models
• Reducing accuracy loss in SNN-based LLMs without full pretraining
• Enabling efficient deployment of LLMs on edge devices

Innovation

• Ternary-integer spiking neuron preserves semantic polarity
• Smoothed gradient compensation mitigates quantization loss
• Single-stage distillation transfers Mamba's zero-shot ability