What Causes Knowledge Loss in Multilingual Language Models?

2025-04-29

AI Summary

This work investigates catastrophic forgetting in cross-lingual transfer of multilingual large language models (MLLMs), emphasizing the impact of writing system differences—not merely parameter updates—on representation learning and knowledge retention. Using LoRA adapters, we conduct controlled experiments across 52 languages, systematically comparing non-shared, partially shared, and fully shared parameter configurations. We find, for the first time, that languages with non-Latin scripts exhibit significantly higher cross-lingual knowledge forgetting, establishing script type as a critical factor. Results demonstrate that carefully designed parameter sharing mechanisms substantially mitigate forgetting, while Latin-script languages show markedly greater cross-lingual transfer stability. Our study reveals the structural role of writing systems in multilingual modeling and provides novel, empirically grounded principles for training and adapting MLLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Cognitive Modeling & Cognitive Systems: Analogy

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Cross-lingual transfer in natural language processing (NLP) models enhances multilingual performance by leveraging shared linguistic knowledge. However, traditional methods that process all data simultaneously often fail to mimic real-world scenarios, leading to challenges like catastrophic forgetting, where fine-tuning on new tasks degrades performance on previously learned ones. Our study explores this issue in multilingual contexts, focusing on linguistic differences affecting representational learning rather than just model parameters. We experiment with 52 languages using LoRA adapters of varying ranks to evaluate non-shared, partially shared, and fully shared parameters. Our aim is to see if parameter sharing through adapters can mitigate forgetting while preserving prior knowledge. We find that languages using non-Latin scripts are more susceptible to catastrophic forgetting, whereas those written in Latin script facilitate more effective cross-lingual transfer.

Problem

• Investigates knowledge loss in multilingual language models
• Examines impact of linguistic differences on representational learning
• Assesses parameter sharing to mitigate catastrophic forgetting

Innovation

• Uses LoRA adapters for parameter sharing
• Evaluates non-shared and shared parameters
• Focuses on script impact on forgetting