What Is Missing in Multilingual Visual Reasoning and How to Fix It

2024-03-03

arXiv.org

AI Summary

Current open-source multimodal large language models (MLLMs) significantly underperform proprietary systems (e.g., GPT-4V) on multilingual visual reasoning tasks, primarily constrained by three bottlenecks: limited multilingual capability, insufficient complex reasoning capacity, and suboptimal multimodal fusion. This paper presents the first systematic characterization of key dimensions governing multilingual visual reasoning and introduces a zero-shot, plug-and-play collaborative intervention framework comprising three orthogonal components: translate-test, visual programming, and caption-based multimodal grounding. The framework ensures both linguistic fairness and reasoning interpretability. Extensive experiments demonstrate substantial zero-shot improvements across multiple open-source MLLMs—+13.4% on LLaVA-v1.5-13B, +20.3% on LLaVA-v1.6-34B, and +16.7% on Qwen-VL—while yielding modest gains even for GPT-4V. Our approach provides a scalable, lightweight, and robust solution for open-source multilingual multimodal reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

NLP models today strive for supporting multiple languages and modalities, improving accessibility for diverse users. In this paper, we evaluate their multilingual, multimodal capabilities by testing on a visual reasoning task. We observe that proprietary systems like GPT-4V obtain the best performance on this task now, but open models lag in comparison. Surprisingly, GPT-4V exhibits similar performance between English and other languages, indicating the potential for equitable system development across languages. Our analysis on model failures reveals three key aspects that make this task challenging: multilinguality, complex reasoning, and multimodality. To address these challenges, we propose three targeted interventions including a translate-test approach to tackle multilinguality, a visual programming approach to break down complex reasoning, and a method that leverages image captioning to address multimodality. Our interventions achieve the best open performance on this task in a zero-shot setting, boosting open models LLaVA-v1.5-13B by 13.4%, LLaVA-v1.6-34B by 20.3%, and Qwen-VL by 16.7%, while also minorly improving GPT-4V's performance.

Problem

• Enhancing multilingual visual reasoning capabilities
• Improving open models' performance in multimodal tasks
• Addressing challenges in multilinguality, complex reasoning, and multimodality

Innovation

• Translate-test for multilinguality
• Visual programming for reasoning
• Image captioning for multimodality