SpikeMLLM: Spike-based Multimodal Large Language Models via Modality-Specific Temporal Scales and Temporal Compression

2026-04-13

AI Summary

This work addresses the high computational cost of multimodal large language models (MLLMs), which hinders deployment in resource-constrained settings, and the challenges faced by spiking neural networks (SNNs) in handling heterogeneous modalities and high-resolution images—namely, inconsistent encoding schemes and redundant timesteps. To overcome these limitations, the authors propose the first spiking-based MLLM framework, introducing two key innovations: modality-specific timesteps (MSTS) guided by modality evolution disparities and a time-compressed leaky integrate-and-fire (TC-LIF) mechanism. These enable efficient, unified representation in the spiking domain with drastically reduced timesteps. Combined with ANN quantization and a custom RTL accelerator, the approach achieves only 0.72% and 1.19% performance degradation on InternVL2-8B and Qwen2VL-72B, respectively, while compressing inference to 1/4–1/3 of the original timesteps and delivering 9.06× higher throughput and 25.8× better energy efficiency compared to FP16 GPU execution.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Multimodal Large Language Models (MLLMs) have achieved remarkable progress but incur substantial computational overhead and energy consumption during inference, limiting deployment in resource-constrained environments. Spiking Neural Networks (SNNs), with their sparse event-driven computation, offer inherent energy efficiency advantages on neuromorphic hardware, yet extending them to MLLMs faces two key challenges: heterogeneous modalities make uniform spike encoding insufficient, and high-resolution image inputs amplify timestep unfolding overhead. We propose SpikeMLLM, the first spike-based framework for MLLMs, which unifies existing ANN quantization methods in the spiking representation space and incorporates Modality-Specific Temporal Scales (MSTS) guided by Modality Evolution Discrepancy (MED) and Temporally Compressed LIF (TC-LIF) for timestep compression from T=L-1 to T=log2(L)-1. Experiments on four representative MLLMs across diverse multimodal benchmarks show that SpikeMLLM maintains near-lossless performance under aggressive timestep compression (Tv/Tt=3/4), with average gaps of only 0.72% and 1.19% relative to the FP16 baseline on InternVL2-8B and Qwen2VL-72B. We further develop a dedicated RTL accelerator tailored to the spike-driven datapath, observing 9.06x higher throughput and 25.8x better power efficiency relative to an FP16 GPU baseline under a deployment-oriented co-design setting, suggesting the promise of algorithm-hardware co-design for efficient multimodal intelligence.

Problem

• Multimodal Large Language Models
• Spiking Neural Networks
• Computational Overhead
• Energy Efficiency
• Temporal Compression

Innovation

• Spiking Neural Networks
• Multimodal Large Language Models
• Temporal Compression
• Modality-Specific Temporal Scales
• Algorithm-Hardware Co-design