MSQA: A Natively Sourced Multilingual and Multicultural SimpleQA Benchmark

2026-07-01

AI Summary

This study investigates whether multilingual large language models genuinely possess culture-bound understanding or merely exhibit an illusion of cultural alignment due to linguistic fluency. To this end, we introduce MSQA, the first natively crowdsourced multilingual and multicultural question-answering benchmark, spanning 11 language families, five cultural dimensions, and three difficulty levels. Evaluations of 18 prominent large language models reveal that their cultural comprehension strongly correlates with the degree of cultural exposure in their pretraining corpora, and they consistently display overconfidence and performance degradation on questions involving unfamiliar cultures. Our experiments further demonstrate that current inference-time techniques struggle to effectively compensate for gaps in cultural knowledge, underscoring the critical importance of native cultural data for achieving robust model localization.

Abstract

Multilingual fluency often invites a stronger assumption: a model that can speak a user's language must also understand the culture encoded by that language. We call this the Illusion of Cultural Alignment. To test this assumption directly, we introduce MSQA, a benchmark of 1,064 natively sourced questions across 11 language groups, five cultural dimensions, and three difficulty tiers. Unlike translated benchmarks, MSQA targets locally grounded knowledge and reduces shortcuts from English-centric cross-lingual transfer. Evaluating 18 LLMs, we find substantial cultural degradation and a pronounced Locality Effect: cultural competence tracks pre-training exposure more closely than general reasoning ability. We further show that common inference-time remedies do not dissolve the illusion. Models remain overconfident on unfamiliar cultural questions, repeated sampling yields unstable rather than reliable correctness, and retrieval augmentation helps unevenly on long-tail facts. These findings indicate that cultural alignment cannot be inferred from multilingual ability alone and requires deeper intervention than calibration, sampling, or retrieval at inference time

Problem

• cultural alignment
• multilingual models
• language models
• cultural knowledge
• cross-lingual transfer

Innovation

• cultural alignment
• natively sourced benchmark
• multilingual LLM evaluation
• locality effect
• cross-lingual transfer bias