Language Surgery in Multilingual Large Language Models

2025-06-14

AI Summary

This work addresses cross-lingual generation inconsistency—specifically, language interference—in multilingual large language models (MLLMs). We first discover an emergent cross-lingual representation alignment in hidden layers, which underlies this phenomenon. Leveraging this insight, we propose Inference-Time Language Control (ITLC), a fine-tuning-free method that injects language-specific signals into the latent space to precisely steer output language while preserving semantic fidelity. To formalize this, we introduce a cross-lingual disentanglement framework that explicitly separates language-invariant semantics from language-specific representations. Experiments demonstrate that ITLC significantly improves cross-lingual consistency across diverse multilingual generation tasks and effectively mitigates language interference in state-of-the-art MLLMs. Crucially, ITLC enables zero-shot, semantically lossless, language-controllable generation—a novel paradigm for multilingual LLM inference.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large Language Models (LLMs) have demonstrated remarkable generalization capabilities across tasks and languages, revolutionizing natural language processing. This paper investigates the naturally emerging representation alignment in LLMs, particularly in the middle layers, and its implications for disentangling language-specific and language-agnostic information. We empirically confirm the existence of this alignment, analyze its behavior in comparison to explicitly designed alignment models, and demonstrate its potential for language-specific manipulation without semantic degradation. Building on these findings, we propose Inference-Time Language Control (ITLC), a novel method that leverages latent injection to enable precise cross-lingual language control and mitigate language confusion in LLMs. Our experiments highlight ITLC's strong cross-lingual control capabilities while preserving semantic integrity in target languages. Furthermore, we demonstrate its effectiveness in alleviating the cross-lingual language confusion problem, which persists even in current large-scale LLMs, leading to inconsistent language generation. This work advances our understanding of representation alignment in LLMs and introduces a practical solution for enhancing their cross-lingual performance.

Problem

• Investigates representation alignment in multilingual LLM layers
• Proposes ITLC for precise cross-lingual language control
• Addresses language confusion in multilingual LLM generation

Innovation

• Leverages latent injection for language control
• Analyzes middle-layer representation alignment
• Mitigates language confusion in multilingual LLMs