SAISA: Towards Multimodal Large Language Models with Both Training and Inference Efficiency

2025-02-04

AI Summary

Multimodal large language models (MLLMs) face a fundamental trade-off between training and inference efficiency: embedding alignment (e.g., LLaVA-1.5) incurs high inference overhead, while cross-attention alignment (e.g., Flamingo) suffers from prohibitive training costs. To address this, we propose SAISA—a self-attention architecture with input-space alignment. Our key contributions are: (1) the novel NAAViT mechanism, which empirically reveals substantial redundancy in attention among visual tokens; (2) elimination of intra-visual token attention, directly aligning visual features into the self-attention input space; and (3) co-designed lightweight self-attention and feed-forward networks, supporting flexible integration with diverse LLMs and vision encoders. Experiments show that SAISA reduces inference FLOPs by 66% and training budget by 26% compared to LLaVA-1.5, while achieving superior accuracy across multiple benchmarks. Ablation studies confirm strong generalization across architectures and tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Multimodal Large Language Models (MLLMs) mainly fall into two architectures, each involving a trade-off between training and inference efficiency: embedding space alignment (e.g., LLaVA-1.5) is inefficient during inference, while cross-attention space alignment (e.g., Flamingo) is inefficient in training. In this paper, we compare these two architectures and identify the key factors for building efficient MLLMs. A primary difference between them lies in how attention is applied to visual tokens, particularly in their interactions with each other. To investigate whether attention among visual tokens is necessary, we propose a new self-attention mechanism, NAAViT ( extbf{N}o extbf{A}ttention extbf{A}mong extbf{Vi}sual extbf{T}okens), which eliminates this type of attention. Our pilot experiment on LLaVA-1.5 shows that attention among visual tokens is highly redundant. Based on these insights, we introduce SAISA ( extbf{S}elf- extbf{A}ttention extbf{I}nput extbf{S}pace extbf{A}lignment), a novel architecture that enhance both training and inference efficiency. SAISA directly aligns visual features with the input spaces of NAAViT self-attention blocks, reducing computational overhead in both self-attention blocks and feed-forward networks (FFNs). Using the same configuration as LLaVA-1.5, SAISA reduces inference FLOPs by 66% and training budget by 26%, while achieving superior performance in terms of accuracy. Comprehensive ablation studies further validate the effectiveness of SAISA across various LLMs and visual encoders. The code and model will be publicly available at https://github.com/icip-cas/SAISA.

Problem

• Improves multimodal model efficiency
• Reduces computational overhead
• Enhances training and inference

Innovation

• NAAViT eliminates visual token attention
• SAISA aligns visual features efficiently
• SAISA reduces FLOPs and training budget