Spike-driven Large Language Model

2026-04-11

AI Summary

Current spiking large language models are hindered by limited representational capacity, insufficient sparsity, and high computational overhead. This work proposes SDLLM—the first billion-parameter, fully sparse-addition-based, spiking-driven large language model. Its key innovations include a plug-and-play gamma-SQP two-stage spiking encoding scheme that aligns semantic representations, and a synergistic mechanism combining bidirectional symmetric quantization with membrane potential clipping, which substantially reduces both spike firing rates and required timesteps. Under the spiking paradigm, SDLLM achieves state-of-the-art performance: it lowers inference energy consumption by 7× while improving accuracy by 4.2%, offering an efficient and viable pathway for event-driven neuromorphic hardware.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Current Large Language Models (LLMs) are primarily based on large-scale dense matrix multiplications. Inspired by the brain's information processing mechanism, we explore the fundamental question: how to effectively integrate the brain's spiking-driven characteristics into LLM inference. Spiking Neural Networks (SNNs) possess spike-driven characteristics, and some works have attempted to combine SNNs with Transformers. However, achieving spike-driven LLMs with billions of parameters, relying solely on sparse additions, remains a challenge in the SNN field. To address the issues of limited representational capacity and sparsity in existing spike encoding schemes at the LLM level, we propose SDLLM, a spike-driven large language model that eliminates dense matrix multiplications through sparse addition operations. Specifically, we use the plug-and-play gamma-SQP two-step spike encoding method to ensure that the quantization process aligns with the model's semantic space, mitigating representation degradation caused by binary spikes. Furthermore, we introduce bidirectional encoding under symmetric quantization and membrane potential clipping mechanisms, leading to spike trains with no or low firing counts dominating, significantly reducing the model's spike firing rate, while halving the number of time steps. Experimental results show that SDLLM not only significantly reduces inference costs but also achieves state-of-the-art task performance under the spike-based paradigm. For example, compared to previous spike-based LLMs, SDLLM reduces energy consumption by 7x and improves accuracy by 4.2%. Our model provides inspiration for the architecture design of the next generation of event-driven neuromorphic chips.

Problem

• spike-driven
• large language models
• spiking neural networks
• sparse representation
• neuromorphic computing

Innovation

• spike-driven LLM
• sparse addition
• gamma-SQP encoding
• symmetric quantization
• neuromorphic computing