Knowledge Beyond Language: Bridging the Gap in Multilingual Machine Unlearning Evaluation

2026-05-14

AI Summary

Existing evaluation methods struggle to effectively measure the forgetting of cross-lingual information in multilingual large language models, thereby limiting privacy guarantees. This work proposes the first evaluation framework specifically designed for cross-lingual information distribution in machine unlearning, introducing two novel metrics—Knowledge Separability Score (KSS) and Knowledge Persistence Score (KPS)—to systematically characterize the consistency and completeness of forgetting across languages. Built upon a multilingual corpus, the framework compares the performance of various unlearning algorithms across diverse language pairs, uncovering unique phenomena in multilingual forgetting. The study thus offers an innovative evaluative perspective and empirical foundation for enhancing privacy protection in multilingual models.

Abstract

While LLMs are increasingly used in commercial services, they pose privacy risks such as leakage of sensitive personally identifiable information (PII). For LLMs trained on multilingual corpora, Multilingual Machine Unlearning (MMU) aims to remove information across multiple languages. However, prior MMU evaluations fail to capture such cross-linguistic distribution of information, being largely limited to direct extensions of per-language evaluation protocols. To this end, we propose two metrics to evaluate the information spread across languages: the Knowledge Separability Score (KSS) and the Knowledge Persistence Score (KPS). KSS measures the overall unlearning quality across multiple languages, while KPS more specifically aims to assess consistent removal of information among different language pairs. We evaluated various unlearning methods in the multilingual setting with these metrics and conducted comprehensive analyses. Through our investigation, we provide insights into unique phenomena exclusive to MMU and offer a new perspective on MMU evaluation.

Problem

• Multilingual Machine Unlearning
• cross-linguistic information
• evaluation metrics
• privacy leakage
• personally identifiable information

Innovation

• Multilingual Machine Unlearning
• Knowledge Separability Score
• Knowledge Persistence Score
• cross-linguistic evaluation
• LLM privacy