CRaFT: An Explanation-Based Framework for Evaluating Cultural Reasoning in Multilingual Language Models

2025-10-15

AI Summary

Current evaluations of multilingual large language models’ (LLMs’) cultural reasoning capabilities rely predominantly on answer accuracy, neglecting interpretability and cross-linguistic comparability. To address this, we propose CRaFT—the first explanation-based framework for cross-cultural reasoning assessment. CRaFT introduces a four-dimensional explanatory quality metric: cultural fluency, deviation, consistency, and linguistic adaptability. Leveraging the World Values Survey, we construct a culturally grounded, multilingual question–explanation dataset covering Arabic, Bengali, and Spanish (2,100+ instances). Empirical analysis reveals salient language-specific patterns: Arabic responses exhibit lower cultural fluency; Bengali reasoning achieves higher overall quality; GPT-4 demonstrates strong linguistic adaptability but weak consistency; conversely, FANAR shows high stability yet limited flexibility. CRaFT establishes a novel, interpretable, decomposable, and cross-linguistically comparable paradigm for evaluating culturally intelligent multilingual LLMs.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Correct answers do not necessarily reflect cultural understanding. We introduce CRaFT, an explanation-based multilingual evaluation framework designed to assess how large language models (LLMs) reason across cultural contexts. Rather than scoring outputs solely based on accuracy, CRaFT evaluates model explanations using four interpretable metrics: Cultural Fluency, Deviation, Consistency, and Linguistic Adaptation. We apply the framework to 50 culturally grounded questions from the World Values Survey, translated into Arabic, Bengali, and Spanish, and evaluate three models (GPT, DeepSeek, and FANAR) across over 2,100 answer-explanation pairs. Results reveal significant cross-lingual variation in reasoning: Arabic reduces fluency, Bengali enhances it, and Spanish remains largely stable. While GPT adapts more effectively across languages, it exhibits lower consistency; FANAR shows stable but rigid reasoning. These findings suggest that cultural awareness in LLMs is not intrinsic but emerges through linguistic framing. CRaFT offers a new lens for evaluating cross-cultural reasoning in multilingual settings, providing actionable insights for building culturally adaptive language models.

Problem

• Evaluating cultural reasoning in multilingual language models
• Assessing model explanations across cultural contexts
• Measuring cross-lingual variation in cultural understanding

Innovation

• Explanation-based framework for cultural reasoning evaluation
• Four interpretable metrics assess cultural fluency and adaptation
• Multilingual evaluation across 2100 answer-explanation pairs