Exploring Translation Mechanism of Large Language Models

2025-02-17

AI Summary

Large language models (LLMs) achieve strong performance in multilingual translation, yet their internal translation mechanisms remain opaque. To address this, we conduct a component-level analysis of the translation process, integrating path patching, attention head attribution, and interpretable neuron modeling. Our study reveals that translation is predominantly governed by fewer than 5% of dedicated attention heads, which operate in distinct phases: extracting source-language semantics, indicators, and positional features. Furthermore, we identify a highly contributive sparse subnetwork—fine-tuning only 64 attention heads suffices to match full-parameter fine-tuning performance on translation tasks, while preserving the model’s general-purpose capabilities. These findings establish a fine-grained, interpretable, and intervenable framework for understanding LLM translation mechanisms, enabling targeted analysis and modification of translation-specific computational pathways.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large language models (LLMs) have succeeded remarkably in multilingual translation tasks. However, the inherent translation mechanisms of LLMs remain poorly understood, largely due to sophisticated architectures and vast parameter scales. In response to this issue, this study explores the translation mechanism of LLM from the perspective of computational components (e.g., attention heads and MLPs). Path patching is utilized to explore causal relationships between components, detecting those crucial for translation tasks and subsequently analyzing their behavioral patterns in human-interpretable terms. Comprehensive analysis reveals that translation is predominantly facilitated by a sparse subset of specialized attention heads (less than 5%), which extract source language, indicator, and positional features. MLPs subsequently integrate and process these features by transiting towards English-centric latent representations. Notably, building on the above findings, targeted fine-tuning of only 64 heads achieves translation improvement comparable to full-parameter tuning while preserving general capabilities.

Problem

• Understanding LLMs' translation mechanisms
• Identifying key components for translation
• Improving translation via targeted fine-tuning

Innovation

• Path patching explores causal relationships
• Sparse attention heads facilitate translation
• Targeted fine-tuning improves translation efficiency