MMA-ASIA: A Multilingual and Multimodal Alignment Framework for Culturally-Grounded Evaluation

2025-10-07

AI Summary

Current large language models (LLMs) exhibit significant deficiencies in multimodal understanding and reasoning within non-Western, resource-constrained cultural contexts—particularly across Asia—revealing critical gaps in cultural cognition and reliance on superficial, shortcut-based learning. Method: We introduce the first Asian-culture-focused, multilingual multimodal alignment evaluation framework, covering eight countries, ten languages, and 27,000 multiple-choice questions. It enables text-image-speech tri-modal input-level alignment and proposes a five-dimensional evaluation protocol with a dedicated cultural cognition verification module. Leveraging human-annotated data, cross-modal consistency testing, attention tracking, and Visual Prefix Replay (VPR)—a novel visual ablation technique—we systematically diagnose model limitations. Contribution/Results: Our framework establishes a reproducible, culturally grounded benchmark for multimodal LLMs and delivers actionable insights for developing culturally reliable models, directly addressing alignment failures in underrepresented sociocultural settings.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Large language models (LLMs) are now used worldwide, yet their multimodal understanding and reasoning often degrade outside Western, high-resource settings. We propose MMA-ASIA, a comprehensive framework to evaluate LLMs' cultural awareness with a focus on Asian contexts. MMA-ASIA centers on a human-curated, multilingual, and multimodally aligned multiple-choice benchmark covering 8 Asian countries and 10 languages, comprising 27,000 questions; over 79 percent require multi-step reasoning grounded in cultural context, moving beyond simple memorization. To our knowledge, this is the first dataset aligned at the input level across three modalities: text, image (visual question answering), and speech. This enables direct tests of cross-modal transfer. Building on this benchmark, we propose a five-dimensional evaluation protocol that measures: (i) cultural-awareness disparities across countries, (ii) cross-lingual consistency, (iii) cross-modal consistency, (iv) cultural knowledge generalization, and (v) grounding validity. To ensure rigorous assessment, a Cultural Awareness Grounding Validation Module detects "shortcut learning" by checking whether the requisite cultural knowledge supports correct answers. Finally, through comparative model analysis, attention tracing, and an innovative Vision-ablated Prefix Replay (VPR) method, we probe why models diverge across languages and modalities, offering actionable insights for building culturally reliable multimodal LLMs.

Problem

• Evaluating LLMs' cultural awareness degradation in non-Western contexts
• Assessing multimodal alignment across text, image, and speech modalities
• Measuring cultural knowledge generalization and cross-modal consistency

Innovation

• Multimodal aligned benchmark across text, image, speech
• Five-dimensional evaluation protocol for cultural awareness
• Vision-ablated Prefix Replay method probes model divergence