Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Models

2024-10-02

arXiv.org

AI Summary

To address the scarcity of annotated data for mathematical reasoning in non-English languages, this paper proposes a training-free, modular layer-swapping method. Leveraging fine-grained parameter interpretability analysis, it selectively exchanges Transformer layers between an English mathematical expert model and a general-purpose multilingual expert model, enabling zero-shot cross-lingual capability transfer. Unlike prior approaches, our method introduces interpretability-driven layer fusion to cross-lingual reasoning for the first time, supporting post-hoc capability reconfiguration. Evaluated on the MGSM multilingual mathematical reasoning benchmark, the approach achieves an average accuracy improvement of 10% across four low-resource languages—substantially outperforming both individual expert models and mainstream model merging techniques. This demonstrates that interpretable, layer-level parameter alignment can effectively bridge linguistic and task-specific capability gaps without additional training or supervision.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Model merging, such as model souping, is the practice of combining different models with the same architecture together without further training. In this work, we present a model merging methodology that addresses the difficulty of fine-tuning Large Language Models (LLMs) for target tasks in non-English languages, where task-specific data is often unavailable. We focus on mathematical reasoning and without in-language math data, facilitate cross-lingual transfer by composing language and math capabilities. Starting from the same pretrained model, we fine-tune separate"experts"on math instruction data in English and on generic instruction data in the target language. We then replace the top and bottom transformer layers of the math expert directly with layers from the language expert, which consequently enhances math performance in the target language. The resulting merged models outperform the individual experts and other merging methods on the math benchmark, MGSM, by 10% across four major languages where math instruction data is scarce. In addition, this layer swapping is simple, inexpensive, and intuitive, as it is based on an interpretative analysis of the most important parameter changes during the fine-tuning of each expert. The ability to successfully re-compose LLMs for cross-lingual transfer in this manner opens up future possibilities to combine model expertise, create modular solutions, and transfer reasoning capabilities across languages all post hoc.

Problem

• Addresses fine-tuning LLMs for non-English tasks without task-specific data.
• Enhances math performance in target languages via layer swapping.
• Improves cross-lingual transfer by combining language and math capabilities.

Innovation

• Layer swapping enhances cross-lingual math reasoning.
• Merging math and language experts improves performance.
• Interpretative analysis guides parameter changes effectively.