Language over Content: Tracing Cultural Understanding in Multilingual Large Language Models

2025-10-18

AI Summary

This study investigates the independent contributions of linguistic and cultural factors to cultural understanding mechanisms in multilingual large language models (LLMs). To this end, we propose a circuit-analysis-based metric—activation path overlap—to quantify representational similarity across cultural contexts, using controlled contrastive experiments that decouple language and national variables. Our analysis focuses on semantic equivalence tasks under varying cultural backgrounds. Results demonstrate that language exerts a dominant influence on internal representations: activation paths exhibit significantly higher overlap within the same language across different countries than within the same country across different languages. However, the North Korea–South Korea case reveals substantial divergence in culture-related activation pathways despite near-identical linguistic forms, indicating that linguistic similarity does not guarantee cultural understanding consistency. This work is the first to isolate and quantitatively measure the distinct roles of language and culture in LLM reasoning, establishing an interpretable evaluation framework for developing culturally adaptive models.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Analogy

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs) are increasingly used across diverse cultural contexts, making accurate cultural understanding essential. Prior evaluations have mostly focused on output-level performance, obscuring the factors that drive differences in responses, while studies using circuit analysis have covered few languages and rarely focused on culture. In this work, we trace LLMs' internal cultural understanding mechanisms by measuring activation path overlaps when answering semantically equivalent questions under two conditions: varying the target country while fixing the question language, and varying the question language while fixing the country. We also use same-language country pairs to disentangle language from cultural aspects. Results show that internal paths overlap more for same-language, cross-country questions than for cross-language, same-country questions, indicating strong language-specific patterns. Notably, the South Korea-North Korea pair exhibits low overlap and high variability, showing that linguistic similarity does not guarantee aligned internal representation.

Problem

• Tracing cultural understanding mechanisms in multilingual large language models
• Disentangling language effects from cultural aspects in model responses
• Analyzing activation path overlaps across countries and languages

Innovation

• Measuring activation path overlaps for cultural analysis
• Varying country and language to isolate factors
• Using same-language pairs to disentangle cultural aspects