VisTai: Benchmarking Vision-Language Models for Traditional Chinese in Taiwan

2025-03-13

AI Summary

Existing vision-language model (VLM) evaluation benchmarks critically lack support for Traditional Chinese—particularly the Taiwanese variant—hindering fair assessment and regional adaptation. Method: We introduce VisTai, the first comprehensive VLM benchmark tailored to Taiwanese Traditional Chinese, comprising two rigorously human-annotated datasets: VisTai-MCQ (multidisciplinary multiple-choice questions) and VisTai-Dialogue (culturally grounded image–text dialogues). Our dual-track evaluation framework jointly assesses academic knowledge reasoning and localized cultural contextualization, emphasizing character recognition, regional commonsense, and cultural modeling across education, history, and folklore domains. Contribution/Results: Experiments reveal substantial performance degradation of state-of-the-art VLMs on Traditional Chinese visual understanding tasks. VisTai fills a critical gap in VLM evaluation, providing an open-source, culturally informed benchmark that establishes a new paradigm for cross-regional language–vision alignment research.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

In this paper, we propose a comprehensive evaluation benchmark for Visual Language Models (VLM) in Traditional Chinese. Our evaluation suite, the first of its kind, contains two complementary components: (1) VisTai-MCQ, a collection of manually curated exam multi-choice questions from 21 academic subjects designed to test the broad knowledge and reasoning capabilities of VLMs; and (2) VisTai-Dialogue, an open dialogue benchmark comprising 131 image-question pairs manually created to evaluate VLMs' ability in free-form dialogue generation within Taiwanese cultural contexts. These benchmarks address a critical gap in the evaluation landscape, where existing benchmarks predominantly focus on English or Simplified Chinese, neglecting the unique linguistic and cultural aspects of Traditional Chinese used in regions like Taiwan and Hong Kong. Our analysis reveals significant performance differences across various VLMs and highlights specific challenges in processing Traditional Chinese visual content.

Problem

• Evaluate VLMs for Traditional Chinese in Taiwan.
• Address lack of benchmarks for Taiwanese cultural contexts.
• Highlight challenges in processing Traditional Chinese visuals.

Innovation

• First Traditional Chinese VLM evaluation benchmark
• Includes VisTai-MCQ and VisTai-Dialogue components
• Focuses on Taiwanese cultural and linguistic contexts