CulMind: Benchmarking Multimodal Understanding and Reasoning in Chinese Cultural Heritage

2026-06-19

AI Summary

Existing multimodal evaluation benchmarks for Chinese cultural heritage focus solely on answer accuracy, overlooking the completeness and plausibility of reasoning processes. To address this gap, this work introduces CulMind, a high-quality benchmark spanning over 100 museums and encompassing 50 tasks, along with its subset CulMind-R and an adaptive evaluation metric, ReaScore. This framework enables, for the first time, fine-grained assessment of multimodal large language models’ reasoning capabilities across visual, textual, stylistic, and historical dimensions. By dynamically weighting task-relevant dimensions, the proposed scoring mechanism aligns automatic evaluation more closely with expert judgments. Experiments on 14 state-of-the-art multimodal models reveal a significant discrepancy between answer correctness and reasoning quality, demonstrating the effectiveness and expert alignment of the proposed benchmark and metric.

Abstract

Evaluating Multimodal Large Language Models (MLLMs) in Chinese Cultural Heritage (CCH) requires fine-grained reasoning over visual, textual, stylistic, and historical clues. However, existing CCH benchmarks mainly emphasize final-answer accuracy, while the accuracy and completeness of reasoning processes remain underexplored. To address this gap, we introduce CulMind and CulMind-R: a high-quality benchmark for multimodal CCH covering 50 tasks from collections of more than 100 museums, and a 24-task reasoning subset that adaptively defines task-specific dimensions for reasoning process evaluation. To evaluate reasoning quality, we propose ReaScore, a task-adaptive metric that evaluates reasoning by automatically weighting task-relevant dimensions. Experiments on 14 leading MLLMs reveal a substantial gap between answers and reasoning, especially on challenging tasks. Further analysis shows that task-adaptive dimension selection and weighting better align evaluation results with expert judgments. Overall, our benchmark and metric support a more expert-aligned assessment of CCH understanding and offer a transferable reference for broader evaluations of cultural heritage. We publicly release the data, code, and evaluation scripts at https://github.com/ZevTsao/CulMind to facilitate reproducible research.

Problem

• Chinese Cultural Heritage
• Multimodal Large Language Models
• Reasoning Evaluation
• Benchmarking
• Reasoning Process

Innovation

• multimodal reasoning
• Chinese cultural heritage
• reasoning evaluation
• task-adaptive metric
• MLLM benchmark