Multilingual Knowledge Transfer under Data Constraints via Lexical Interventions

2026-05-22

AI Summary

This work addresses the challenge of effectively transferring cross-lingual knowledge to improve multilingual language models’ performance on tasks requiring scientific reasoning and commonsense inference, particularly in low-resource target languages. The authors propose LINK, a method that operates during pretraining by performing word-level substitution on high-resource language corpora (e.g., English), replacing selected tokens with their counterparts from the target language using only a low-cost bilingual lexicon. Crucially, LINK requires no parallel corpora, additional models, or external translation systems. By injecting cross-lingual knowledge directly at the data level, the approach yields significant performance gains across eight languages and five model scales, achieving up to a 2× speedup in training time while matching or exceeding baseline performance.

Abstract

Cross-lingual knowledge transfer is critical for building high-performing multilingual language models for languages with insufficient training data. When target language data is scarce, the knowledge required for many downstream tasks involving scientific reasoning, commonsense inference, and world knowledge must be acquired primarily from the high-resource language, making effective knowledge transfer essential. Existing methods for improving such cross-lingual knowledge transfer require large amounts of parallel data, translation systems, auxiliary models, or additional training stages that are largely unavailable for many languages. We propose LINK - a data-level intervention method that improves knowledge transfer during model pretraining through lexical substitutions in high-resource part of pretraining data using bilingual vocabularies. For a given replacement ratio, randomly selected words in a portion of the high-resource (English) training corpus are swapped with their word-level translations, requiring no additional model training and only a bilingual vocabulary, which can be obtained at near-zero cost for virtually any language. Evaluation on eight languages across five model sizes shows notable improvements on downstream tasks in the target language, with up to a 2x speedup in training to reach equivalent performance.

Problem

• cross-lingual knowledge transfer
• data constraints
• multilingual language models
• low-resource languages
• lexical interventions

Innovation

• cross-lingual knowledge transfer
• lexical intervention
• multilingual language models
• data-level intervention
• bilingual vocabulary