Probing Factual Knowledge Transfer with Training Data Interventions

2026-09-01

AI Summary

研究通过干预训练数据，探讨多语言模型在继续预训练时是否跨语言传递事实知识，发现事实转移非常有限。

Abstract

Do multilingual language models transfer factual knowledge across languages during continued pretraining, or do they mostly recall facts learned directly from the target-language data? To answer this question more reliably, we propose an intervention-based framework: starting from an English-pretrained model, we continue pretraining on Persian data from which specific facts have been systematically removed at varying levels of granularity. We construct SIFT, a resource of 500 triples across 20 relations, stratified by the cultural origin of each fact's subject into general (globally prominent) and Persian-related entities, designed for both systematic fact removal from training data and evaluation, with natively written Persian cloze templates. Our results show that fact transfer is very limited: under the strictest removal condition, a large majority of English-acquired facts fail to transfer into Persian. We further show that sentence-level co-occurrence removal is insufficient to eliminate fact signal, and that easier (randomly selected) negative candidate sets substantially inflate apparent transfer by rewarding shallow associative heuristics, while performance on a harder candidate set that allows for less reliance on heuristics is much lower. Finally, we show that source-language entity frequency has a large influence, with Persian-related facts, which are orders of magnitude rarer in the English corpus, hardly transferring.

Problem

• multilingual language models
• factual knowledge transfer
• continued pretraining
• cross-lingual transfer
• target-language data

Innovation

• intervention-based framework
• factual knowledge transfer
• multilingual language models
• cross-lingual pretraining
• SIFT resource