Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models

2025-04-05

AI Summary

Multilingual large language models (MLLMs) frequently generate inconsistent factual answers across languages, yet the underlying cause remains unclear. This paper identifies the root mechanism: early-layer knowledge representations are cross-lingually unified, while later layers progressively specialize per language; errors arise when the final layer fails to correctly map language-specific representations back to shared semantic space. We introduce the novel theory of *hierarchical language independence* and validate it via mechanistic interpretability techniques—including neuron activation tracing, representation-space projection, and inter-layer knowledge flow analysis. Building on this insight, we propose a lightweight linear shortcut intervention that bypasses the faulty final-layer computation, improving both accuracy and cross-lingual consistency without sacrificing task performance. Experiments show an average 38.7% reduction in cross-lingual factual inconsistency across diverse languages, with stable performance maintained on multilingual benchmarks including XNLI and XQA—demonstrating both efficacy and generalizability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Multilingual language models (MLMs) store factual knowledge across languages but often struggle to provide consistent responses to semantically equivalent prompts in different languages. While previous studies point out this cross-lingual inconsistency issue, the underlying causes remain unexplored. In this work, we use mechanistic interpretability methods to investigate cross-lingual inconsistencies in MLMs. We find that MLMs encode knowledge in a language-independent concept space through most layers, and only transition to language-specific spaces in the final layers. Failures during the language transition often result in incorrect predictions in the target language, even when the answers are correct in other languages. To mitigate this inconsistency issue, we propose a linear shortcut method that bypasses computations in the final layers, enhancing both prediction accuracy and cross-lingual consistency. Our findings shed light on the internal mechanisms of MLMs and provide a lightweight, effective strategy for producing more consistent factual outputs.

Problem

• Investigates cross-lingual inconsistency in multilingual language models
• Explores causes of inconsistent responses across languages
• Proposes method to improve accuracy and consistency

Innovation

• Mechanistic interpretability analyzes cross-lingual inconsistencies
• Linear shortcut bypasses final layer computations
• Enhances accuracy and cross-lingual consistency