Multimodal Mamba: Decoder-only Multimodal State Space Model via Quadratic to Linear Distillation

2025-02-18

AI Summary

To address high computational complexity, large KV cache overhead, and reliance on separate visual encoders in multimodal large language model (MLLM) deployment, this paper introduces mmMamba—the first native linear-complexity multimodal state space model. We propose a three-stage progressive knowledge distillation framework to end-to-end transfer decoder-only MLLMs (e.g., HoVLE) into the Mamba architecture. We further introduce seeded Mamba token extraction and a configurable Transformer-Mamba hybrid layer design to flexibly balance accuracy and efficiency. Crucially, mmMamba eliminates the need for pretrained visual encoders or RNN components, significantly lowering deployment barriers. Experiments show that mmMamba-linear achieves 20.6× speedup and 75.8% memory reduction over HoVLE on 103K tokens; mmMamba-hybrid matches HoVLE’s performance while delivering 13.5× speedup and 60.2% memory savings. Code and models are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Recent Multimodal Large Language Models (MLLMs) have achieved remarkable performance but face deployment challenges due to their quadratic computational complexity, growing Key-Value cache requirements, and reliance on separate vision encoders. We propose mmMamba, a framework for developing linear-complexity native multimodal state space models through progressive distillation from existing MLLMs using moderate academic computational resources. Our approach enables the direct conversion of trained decoder-only MLLMs to linear-complexity architectures without requiring pre-trained RNN-based LLM or vision encoders. We propose an seeding strategy to carve Mamba from trained Transformer and a three-stage distillation recipe, which can effectively transfer the knowledge from Transformer to Mamba while preserving multimodal capabilities. Our method also supports flexible hybrid architectures that combine Transformer and Mamba layers for customizable efficiency-performance trade-offs. Distilled from the Transformer-based decoder-only HoVLE, mmMamba-linear achieves competitive performance against existing linear and quadratic-complexity VLMs, while mmMamba-hybrid further improves performance significantly, approaching HoVLE's capabilities. At 103K tokens, mmMamba-linear demonstrates 20.6$ imes$ speedup and 75.8% GPU memory reduction compared to HoVLE, while mmMamba-hybrid achieves 13.5$ imes$ speedup and 60.2% memory savings. Code and models are released at https://github.com/hustvl/mmMamba

Problem

• Reduces quadratic complexity in multimodal models.
• Eliminates need for separate vision encoders.
• Enhances speed and GPU memory efficiency.

Innovation

• Linear-complexity multimodal state space model
• Progressive distillation from MLLMs
• Hybrid Transformer-Mamba architectures