Toward Cross-Lingual Quality Classifiers for Multilingual Pretraining Data Selection

2026-04-22

AI Summary

This work addresses the challenge of training effective pretraining data quality classifiers for low-resource languages, which suffer from a scarcity of high-quality native data. The authors propose a quality classification method leveraging cross-lingual consistency in embedding spaces to transfer filtering capabilities from high-resource to low-resource languages. Their approach integrates third-quartile (Q3) sampling with a retention-rate tuning strategy to optimize decision boundaries. Experimental results demonstrate that, under a 1B-parameter model trained on 103B tokens, the proposed multilingual pooling method outperforms monolingual baselines in both ranking stability and overall accuracy. Specifically, it achieves a 1.2% improvement in French normalized accuracy and matches or exceeds monolingual performance on low-resource languages.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Multimodal Learning
• Knowledge Representation and Reasoning: Qualitative Reasoning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

As Large Language Models (LLMs) scale, data curation has shifted from maximizing volume to optimizing the signal-to-noise ratio by performing quality filtering. However, for many languages, native high quality data is insufficient to train robust quality classifiers. This work investigates the idea that quality markers in embedding space may show cross-lingual consistency, which would allow high-resource languages to subsidize the filtering of low-resource ones. We evaluate various filtering strategies, including cross-lingual transfer, third quartile sampling (Q3), and retention rate tuning. Our results demonstrate that massive multilingual pooling frequently outperforms monolingual baselines in both rank stability and aggregate accuracy for a 1B model trained on 103B tokens, delivering gains for high resource languages (1.2% increase in aggregate normalized accuracy for French) and matching or exceeding monolingual baselines for low-resource languages. However, we find that scale alone does not guarantee stability. Furthermore, for high-resource languages like French, we show that refining the decision boundary through third quartile sampling (Q3) or tuning the retention rate is necessary to fully leverage the multilingual signal.

Problem

• cross-lingual
• quality classification
• multilingual pretraining
• data filtering
• low-resource languages

Innovation

• cross-lingual transfer
• quality filtering
• multilingual pretraining
• third quartile sampling
• data curation