One Model to Translate Them All? A Journey to Mount Doom for Multilingual Model Merging

2026-04-03

AI Summary

This study investigates the root cause of performance degradation in weight merging approaches for multilingual machine translation when target languages exhibit substantial divergence. By independently fine-tuning models on large-scale bilingual corpora and systematically evaluating standard merging strategies, the authors conduct representational analyses that reveal multilingual fine-tuning does not reinforce but rather redistributes language selectivity, leading to divergent high-level representations that violate merging assumptions. Leveraging span-conditioned neuron selectivity and layer-wise centered kernel alignment, they demonstrate that language-specific neurons are predominantly localized in the embedding and top Transformer layers, while intermediate layers remain highly shared. Fine-tuning diminishes the exclusivity of supervised-language neurons and concurrently isolates unsupervised-language neurons, thereby substantially impairing merged model translation performance.

Abstract

Weight-space model merging combines independently fine-tuned models without accessing original training data, offering a practical alternative to joint training. While merging succeeds in multitask settings, its behavior in multilingual contexts remains poorly understood. We systematically study weight-space merging for multilingual machine translation by fully fine-tuning language model on large-scale bilingual corpora and evaluating standard merging strategies. Our experiments reveal that merging degrades performance, especially when target languages differ. To explain this failure, we analyze internal representations using span-conditioned neuron selectivity and layer-wise centered kernel alignment. We find that language-specific neurons concentrate in embedding layers and upper transformer blocks, while intermediate layers remain largely shared across languages. Critically, fine-tuning redistributes rather than sharpens language selectivity: neurons for supervised and related languages become less exclusive, while those for unsupervised languages grow more isolated. This redistribution increases representational divergence in higher layers that govern generation. These findings suggest that multilingual fine-tuning may reshape geometry in ways that reduce compatibility with standard weight-space merging assumptions. Our work thus provides an explanation for why merging fails in multilingual translation scenarios.

Problem

• multilingual machine translation
• weight-space model merging
• language-specific neurons
• representational divergence
• fine-tuning

Innovation

• weight-space model merging
• multilingual machine translation
• neuron selectivity
• representational divergence
• fine-tuning dynamics