Parrot: Multilingual Visual Instruction Tuning

2024-06-04

arXiv.org

AI Summary

Multimodal large language models (MLLMs) suffer from degraded non-English visual instruction understanding due to English-dominant supervised fine-tuning data. Method: We propose a text-guided, language-level visual token alignment framework. Its core components include: (1) the first language-conditioned visual token generation mechanism; (2) a Mixture-of-Experts (MoE)-based architecture for multilingual visual representation alignment; and (3) the MMMB benchmark—the first large-scale multilingual multimodal evaluation suite covering 6 languages, 15 task categories, and 12K samples. Our approach integrates cross-modal cross-attention, MoE-driven multilingual joint fine-tuning, and synthetic data alignment. Results: Our method achieves state-of-the-art performance on both the multilingual subset of MMBench and MMMB. It significantly improves general multimodal capabilities across languages. Code and training data are fully open-sourced.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

The rapid development of Multimodal Large Language Models (MLLMs) like GPT-4V has marked a significant step towards artificial general intelligence. Existing methods mainly focus on aligning vision encoders with LLMs through supervised fine-tuning (SFT) to endow LLMs with multimodal abilities, making MLLMs' inherent ability to react to multiple languages progressively deteriorate as the training process evolves. We empirically find that the imbalanced SFT datasets, primarily composed of English-centric image-text pairs, lead to significantly reduced performance in non-English languages. This is due to the failure of aligning the vision encoder and LLM with multilingual tokens during the SFT process. In this paper, we introduce Parrot, a novel method that utilizes textual guidance to drive visual token alignment at the language level. Parrot makes the visual tokens condition on diverse language inputs and uses Mixture-of-Experts (MoE) to promote the alignment of multilingual tokens. Specifically, to enhance non-English visual tokens alignment, we compute the cross-attention using the initial visual features and textual embeddings, the result of which is then fed into the MoE router to select the most relevant experts. The selected experts subsequently convert the initial visual tokens into language-specific visual tokens. Moreover, considering the current lack of benchmarks for evaluating multilingual capabilities within the field, we collect and make available a Massive Multilingual Multimodal Benchmark which includes 6 languages, 15 categories, and 12,000 questions, named as MMMB. Our method not only demonstrates state-of-the-art performance on multilingual MMBench and MMMB, but also excels across a broad range of multimodal tasks. Both the source code and the training dataset of Parrot will be made publicly available. Code is available at: https://github.com/AIDC-AI/Parrot.

Problem

• Addresses multilingual token alignment degradation in MLLMs
• Proposes PARROT for language-specific visual token alignment
• Introduces MMMB benchmark for multilingual multimodal evaluation

Innovation

• Leverages textual guidance for visual token alignment
• Uses Mixture-of-Experts for multilingual token alignment
• Introduces Massive Multilingual Multimodal Benchmark