Separating Syntax from Language: A Mechanistic Account of Translation in Multilingual LLMs

2026-09-01

AI Summary

研究通过构建多语言数据集和因果干预方法，揭示了多语言大模型在翻译过程中先构建目标语言的语序，再生成表面形式，分离出语法处理过程。

Abstract

Multilingual large language models (mLLMs) achieve strong performance in machine translation, yet our understanding of the mechanisms by which they transform representations from one language to another remains incomplete. Prior work suggests that translation decomposes into separable processes within an mLLM, where conceptual content is first represented independently, followed by a production into language-specific form. In this work, we show that translation is even more modular than previously assumed and that the output language production in translation processes is actually further separable into a syntax and a surface language process. We construct controlled multilingual datasets that isolate cross-linguistic differences in word-order and use causal interventions and probing to track how representations are transformed during translation. We find that models first construct target-side word-order before realizing the target language surface form. We identify individual attention heads that are selectively sensitive to syntactic transformations while remaining largely invariant to language identity. These results establish the commitment to a syntactic structure as an independent stage in translation, extending prior decompositions and showing how translation is implemented by functionally different components within mLLMs.

Problem

• multilingual large language models
• machine translation
• representation transformation
• syntax
• surface form

Innovation

• syntax
• surface language process
• attention heads
• word-order
• functional components