SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs

2024-08-21

arXiv.org

AI Summary

Current multimodal large language models (MLLMs) struggle to achieve token-level semantic alignment between vision and language under image-level alignment, limiting the visual understanding and reasoning capabilities of small-scale LLMs. To address this, we propose the first supervised token-level embedding alignment mechanism: leveraging vision–language priors from pre-trained models (e.g., CLIP), we distill cross-modal alignment knowledge via contrastive learning and design a lightweight adapter to precisely map visual tokens into the LLM’s embedding space. Our method requires no additional training data or inference overhead. Evaluated on multiple visual question answering and reasoning benchmarks, it improves performance of small-scale MLLMs by 3.2–5.7 points, while significantly enhancing model interpretability and generalization.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Multimodal Large Language Models (MLLMs) have recently demonstrated remarkable perceptual and reasoning abilities, typically comprising a Vision Encoder, an Adapter, and a Large Language Model (LLM). The adapter serves as the critical bridge between the visual and language components. However, training adapters with image-level supervision often results in significant misalignment, undermining the LLMs' capabilities and limiting the potential of Multimodal LLMs. To address this, we introduce Supervised Embedding Alignment (SEA), a token-level alignment method that leverages vision-language pre-trained models, such as CLIP, to align visual tokens with the LLM's embedding space through contrastive learning. This approach ensures a more coherent integration of visual and language representations, enhancing the performance and interpretability of multimodal LLMs while preserving their inherent capabilities. Extensive experiments show that SEA effectively improves MLLMs, particularly for smaller models, without adding extra data or inference computation. SEA also lays the groundwork for developing more general and adaptable solutions to enhance multimodal systems.

Problem

• Improving token-level visual-textual alignment in MLLMs
• Addressing suboptimal modality integration in multimodal systems
• Enhancing cross-modal understanding for smaller language models

Innovation

• Token-level supervision alignment method
• Minimal computational overhead preserves language
• Improves cross-modal understanding significantly