CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Concepts

2024-10-20

arXiv.org

AI Summary

Current vision-language models (VLMs) exhibit significant deficiencies in understanding culturally specific concepts and adapting to cultural context dynamically. Method: We introduce CROPE, the first cultural-concept-oriented visual question answering benchmark, which uniquely decouples the evaluation of cultural knowledge—assessing both parameter-embedded cultural priors and context-bound cultural reasoning. We propose a multimodal prompting framework, culture-aware VQA task design, and a controlled context injection and ablation analysis methodology. Contribution/Results: Experiments reveal that mainstream open-source VLMs underperform substantially on cultural concepts compared to general ones, and fail to establish robust cultural semantic alignment even with rich image-text context. Our work exposes a fundamental limitation in cross-cultural vision-language alignment and provides a reproducible benchmark and methodological foundation for developing more culturally inclusive multimodal models.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Question Answering

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

As Vision and Language models (VLMs) are reaching users across the globe, assessing their cultural understanding has become a critical challenge. In this paper, we introduce CROPE, a visual question answering benchmark designed to probe the knowledge of culture-specific concepts and evaluate the capacity for cultural adaptation through contextual information. This allows us to distinguish between parametric knowledge acquired during training and contextual knowledge provided during inference via visual and textual descriptions. Our evaluation of several state-of-the-art open VLMs shows large performance disparities between culture-specific and common concepts in the parametric setting. Moreover, experiments with contextual knowledge indicate that models struggle to effectively utilize multimodal information and bind culture-specific concepts to their depictions. Our findings reveal limitations in the cultural understanding and adaptability of current VLMs that need to be addressed toward more culturally inclusive models.

Problem

• Evaluating cultural understanding of Vision and Language Models
• Assessing models' adaptation to culture-specific concepts
• Identifying limitations in cultural adaptability of VLMs

Innovation

• CROPE benchmark for culture-specific concepts
• Evaluates VLM adaptation using contextual information
• Highlights VLM struggles with multimodal cultural integration