EMMA: Efficient Visual Alignment in Multi-Modal LLMs

2024-10-02

arXiv.org

AI Summary

To address inefficient vision-language fusion, reliance on complex adapter modules, and large-scale training data in multimodal large language models (MLLMs), this paper proposes EMMA—a lightweight cross-modal alignment module. Methodologically, EMMA introduces: (1) an efficient early-fusion mechanism with <0.2% parameter overhead, leveraging instruction-conditioned visual feature reweighting and a lightweight cross-attention adapter to generate instruction-aware visual representations; (2) an interpretable analytical framework that elucidates the intrinsic mechanisms of cross-modal alignment; and (3) comprehensive evaluation demonstrating an average 9.3% improvement across diverse domain-specific and general-purpose benchmarks. EMMA significantly mitigates hallucination and enhances robustness while preserving model simplicity and computational efficiency. The approach achieves substantial performance gains without architectural bloat or extensive retraining, offering a principled trade-off between effectiveness and parsimony in MLLM design.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Multi-modal Large Language Models (MLLMs) have recently exhibited impressive general-purpose capabilities by leveraging vision foundation models to encode the core concepts of images into representations. These are then combined with instructions and processed by the language model to generate high-quality responses. Despite significant progress in enhancing the language component, challenges persist in optimally fusing visual encodings within the language model for task-specific adaptability. Recent research has focused on improving this fusion through modality adaptation modules but at the cost of significantly increased model complexity and training data needs. In this paper, we propose EMMA (Efficient Multi-Modal Adaptation), a lightweight cross-modality module designed to efficiently fuse visual and textual encodings, generating instruction-aware visual representations for the language model. Our key contributions include: (1) an efficient early fusion mechanism that integrates vision and language representations with minimal added parameters (less than 0.2% increase in model size), (2) an in-depth interpretability analysis that sheds light on the internal mechanisms of the proposed method; (3) comprehensive experiments that demonstrate notable improvements on both specialized and general benchmarks for MLLMs. Empirical results show that EMMA boosts performance across multiple tasks by up to 9.3% while significantly improving robustness against hallucinations. Our code is available at https://github.com/SaraGhazanfari/EMMA

Problem

• Efficient fusion of visual and textual encodings in MLLMs
• Reducing model complexity and training data needs
• Improving task-specific adaptability and performance

Innovation

• Lightweight cross-modality fusion module
• Efficient early fusion with minimal parameters
• Interpretability analysis of fusion mechanisms