Seeing Culture: A Benchmark for Visual Reasoning and Grounding

2025-09-19

AI Summary

Existing vision-language models (VLMs) exhibit weak cultural reasoning capabilities on benchmark datasets and suffer from insufficient geographic representation—particularly lacking coverage of Southeast Asian cultures. Method: We introduce SCB, the first visual reasoning benchmark explicitly focused on Southeast Asian culture, comprising 1,065 culturally rich images and 3,178 multiple-choice visual question-answering (VQA) items. Each question requires not only selecting the correct cultural option but also generating instance-level segmentation masks to localize the corresponding cultural artifact—providing verifiable spatial evidence. Our two-stage evaluation framework jointly assesses cultural semantic reasoning and spatial grounding; distractors are structured by country-of-origin to heighten cultural discrimination difficulty. Results: Experiments reveal a significant performance gap between cultural reasoning and spatial evidence generation across state-of-the-art VLMs, exposing fundamental limitations in complex, culturally grounded multimodal understanding—and establishing SCB as a rigorous new benchmark with clear directions for advancement.

Technology Category

• Computer Vision: Language and Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Multimodal vision-language models (VLMs) have made substantial progress in various tasks that require a combined understanding of visual and textual content, particularly in cultural understanding tasks, with the emergence of new cultural datasets. However, these datasets frequently fall short of providing cultural reasoning while underrepresenting many cultures. In this paper, we introduce the Seeing Culture Benchmark (SCB), focusing on cultural reasoning with a novel approach that requires VLMs to reason on culturally rich images in two stages: i) selecting the correct visual option with multiple-choice visual question answering (VQA), and ii) segmenting the relevant cultural artifact as evidence of reasoning. Visual options in the first stage are systematically organized into three types: those originating from the same country, those from different countries, or a mixed group. Notably, all options are derived from a singular category for each type. Progression to the second stage occurs only after a correct visual option is chosen. The SCB benchmark comprises 1,065 images that capture 138 cultural artifacts across five categories from seven Southeast Asia countries, whose diverse cultures are often overlooked, accompanied by 3,178 questions, of which 1,093 are unique and meticulously curated by human annotators. Our evaluation of various VLMs reveals the complexities involved in cross-modal cultural reasoning and highlights the disparity between visual reasoning and spatial grounding in culturally nuanced scenarios. The SCB serves as a crucial benchmark for identifying these shortcomings, thereby guiding future developments in the field of cultural reasoning. https://github.com/buraksatar/SeeingCulture

Problem

• Existing cultural datasets lack cultural reasoning capabilities for VLMs
• Current datasets underrepresent many diverse cultures worldwide
• VLMs struggle with visual reasoning and spatial grounding in cultural contexts

Innovation

• Two-stage cultural reasoning with VQA
• Visual options organized by country origin
• Segmentation as evidence for artifact grounding