CRaFT: An Explanation-Based Framework for Evaluating Cultural Reasoning in Multilingual Language Models
Current evaluations of multilingual large language models’ (LLMs’) cultural reasoning capabilities rely predominantly on answer accuracy, neglecting interpretability and cross-linguistic comparability. To address this, we propose CRaFT—the first explanation-based framework for cross-cultural reasoning assessment. CRaFT introduces a four-dimensional explanatory quality metric: cultural fluency, deviation, consistency, and linguistic adaptability. Leveraging the World Values Survey, we construct a culturally grounded, multilingual question–explanation dataset covering Arabic, Bengali, and Spanish (2,100+ instances). Empirical analysis reveals salient language-specific patterns: Arabic responses exhibit lower cultural fluency; Bengali reasoning achieves higher overall quality; GPT-4 demonstrates strong linguistic adaptability but weak consistency; conversely, FANAR shows high stability yet limited flexibility. CRaFT establishes a novel, interpretable, decomposable, and cross-linguistically comparable paradigm for evaluating culturally intelligent multilingual LLMs.