Evaluating Visual and Cultural Interpretation: The K-Viscuit Benchmark with Human-VLM Collaboration

2024-06-24

arXiv.org

AI Summary

Existing cultural evaluation benchmarks suffer from heavy reliance on manual curation, narrow cultural coverage, and low construction efficiency. To address these limitations, this paper introduces K-Viscuit—the first vision-language multiple-choice question benchmark explicitly designed for Korean culture. Methodologically, we propose a novel semi-automated human-in-the-loop framework: leveraging vision-language model (VLM) prompting and few-shot guided generation to produce culturally grounded questions, followed by native Korean speaker validation, external knowledge augmentation, and a multi-dimensional human evaluation protocol. Our contributions are threefold: (1) the open-sourced K-Viscuit dataset, publicly available on Hugging Face; (2) the first systematic empirical finding that mainstream open-source VLMs significantly underperform closed-source counterparts in East Asian cultural understanding; and (3) a flexible, transferable evaluation paradigm supporting both multiple-choice and open-ended cultural comprehension assessment.

Technology Category

• Computer Vision: Language and Vision
• Natural Language Processing: Question Answering
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

To create culturally inclusive vision-language models (VLMs), developing a benchmark that tests their ability to address culturally relevant questions is essential. Existing approaches typically rely on human annotators, making the process labor-intensive and creating a cognitive burden in generating diverse questions. To address this, we propose a semi-automated framework for constructing cultural VLM benchmarks, specifically targeting multiple-choice QA. This framework combines human-VLM collaboration, where VLMs generate questions based on guidelines, a small set of annotated examples, and relevant knowledge, followed by a verification process by native speakers. We demonstrate the effectiveness of this framework through the creation of exttt{K-Viscuit}, a dataset focused on Korean culture. Our experiments on this dataset reveal that open-source models lag behind proprietary ones in understanding Korean culture, highlighting key areas for improvement. We also present a series of further analyses, including human evaluation, augmenting VLMs with external knowledge, and the evaluation beyond multiple-choice QA. Our dataset is available at https://huggingface.co/datasets/ddehun/k-viscuit.

Problem

• Developing culturally inclusive vision-language models (VLMs)
• Reducing labor-intensive human annotation for diverse questions
• Assessing VLMs' understanding of Korean culture via K-Viscuit

Innovation

• Semi-automated framework for cultural VLM benchmarks
• Human-VLM collaboration for question generation
• Native speaker verification for cultural accuracy