A-VL: Adaptive Attention for Large Vision-Language Models

2024-09-23

arXiv.org

AI Summary

To address the high memory and computational overhead in large vision-language models (LVLMs) caused by cross-modal attention redundancy, this paper proposes a Modality-Adaptive Attention (MAA) mechanism: for image tokens, it introduces dynamic critical-region pruning and perception-aware caching; for text tokens, it employs a local-context focusing strategy. MAA is the first to decouple and adaptively control long-range visual dependencies and linguistic locality. The method is plug-and-play, integrating modality-aware caching, hierarchical attention gating, and a lightweight inference framework. Evaluated across three vision-language task categories and five benchmark datasets, MAA reduces GPU memory consumption by up to 47%, decreases FLOPs by 39%, accelerates inference by 2.1×, and maintains zero accuracy degradation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The Large Vision-Language Model (LVLM) integrates computer vision and natural language processing techniques, offering substantial application potential. However, these models demand extensive resources during inference. Adaptive attention techniques can dynamically reduce computational redundancy and thus improve efficiency. Although current adaptive attention methods significantly reduce the memory requirements of Transformer-based language models, they are not tailored for LVLMs. We observe that LVLMs generate responses from both remote image tokens and local text tokens, and different modalities have different attention patterns. This observation inspires us to manage the attention for each modality separately. Specifically, for visual input, we store the cache of potentially useful information but only compute the most critical parts. For language input, we care more about local information. Based on our observation and analysis of vision-language attention patterns, we develop A-VL, a plug-and-play adaptive attention tailored for LVLM inference. Extensive evaluations on three vision-language tasks and five datasets show the effectiveness of our designs. Our approach A-VL outperforms existing adaptive attention methods in reducing memory usage and computational load without compromising performance.

Problem

• Reduces computational redundancy in LVLMs
• Tailors adaptive attention for vision-language tasks
• Improves efficiency without compromising performance

Innovation

• Adaptive attention for LVLMs
• Separate modality attention management
• Efficient critical parts computation