Query-based Cross-Modal Projector Bolstering Mamba Multimodal LLM

2026-06-03

AI Summary

This work addresses the quadratic computational complexity of Transformers in multimodal large language models when processing long sequences and the reliance of Mamba-based approaches on handcrafted 2D scanning orders for visual inputs. To overcome these limitations, the authors propose a query-driven cross-modal projector that, for the first time, integrates learnable query vectors with cross-attention mechanisms into the Mamba architecture. This enables dynamic, adaptive compression of visual tokens without requiring any predefined scanning order. Experimental results demonstrate that the proposed method significantly enhances both performance on multiple vision–language understanding benchmarks and inference throughput, offering an efficient and effective solution for multimodal sequence modeling.

Abstract

The Transformer's quadratic complexity with input length imposes an unsustainable computational load on large language models (LLMs). In contrast, the Selective Scan Structured State-Space Model, or Mamba, addresses this computational challenge effectively. This paper explores a query-based cross-modal projector designed to bolster Mamba's efficiency for vision-language modeling by compressing visual tokens based on input through the cross-attention mechanism. This innovative projector also removes the need for manually designing the 2D scan order of original image features when converting them into an input sequence for Mamba LLM. Experimental results across various vision-language understanding benchmarks show that the proposed cross-modal projector enhances Mamba-based multimodal LLMs, boosting both performance and throughput.

Problem

• Mamba
• multimodal LLM
• vision-language modeling
• cross-modal projector
• visual token compression

Innovation

• Mamba
• cross-modal projector
• query-based compression
• vision-language modeling
• structured state-space model