Evaluating Cross-Lingual Unlearning in Multilingual Language Models

2026-01-10

arXiv.org

AI Summary

This work addresses the challenge of effectively unlearning specific factual knowledge in multilingual large language models, particularly in cross-lingual settings where existing unlearning methods exhibit significant performance degradation in non-training languages. To this end, the authors construct a multilingual TOFU benchmark spanning seven languages and scripts, enabling a systematic evaluation of mainstream unlearning algorithms across languages. Their analysis reveals, for the first time, the existence of a shared “interlingual space” within multilingual models. Leveraging this insight, they propose a subspace projection-based approach to achieve efficient and selective cross-lingual unlearning. Experimental results demonstrate that the proposed method substantially improves unlearning efficacy in non-training languages while preserving overall model utility, outperforming current state-of-the-art unlearning strategies.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We present the first comprehensive evaluation of cross-lingual unlearning in multilingual LLMs. Using translated TOFU benchmarks in seven language/script variants, we test major unlearning algorithms and show that most fail to remove facts outside the training language, even when utility remains high. However, subspace-projection consistently outperforms the other methods, achieving strong cross-lingual forgetting with minimal degradation. Analysis of learned task subspaces reveals a shared interlingua structure: removing this shared subspace harms all languages, while removing language-specific components selectively affects one. These results demonstrate that multilingual forgetting depends on geometry in weight space, motivating subspace-based approaches for future unlearning systems.

Problem

• cross-lingual unlearning
• multilingual language models
• machine unlearning
• language models
• knowledge forgetting

Innovation

• cross-lingual unlearning
• multilingual LLMs
• subspace projection
• interlingua
• weight space geometry