MultiSynt/MT: Trillion-Token Multi-Parallel Pre-Training Data Translated Across 36 Languages

2026-07-01

AI Summary

This work addresses the scarcity of non-English open-source pretraining corpora, which hinders the development of multilingual large language models. To overcome this limitation, the authors propose a machine translation–based synthesis approach that leverages the high-quality Nemotron-CC corpus and employs both Tower+ and OPUS-MT/HPLT-MT systems to generate a sentence-aligned parallel corpus spanning 36 European languages and approximately 4.8 trillion tokens—the first large-scale, open, multi-system-fused multilingual pretraining dataset of its kind. Experimental results demonstrate that, under a fixed budget of 100 billion tokens, models trained on this synthetic data achieve a ~15% performance gain over the native-data baseline HPLT 2.0; moreover, they reach equivalent final performance using only 72% of the token budget, substantially reducing reliance on scarce native multilingual data and exposing limitations in current evaluation benchmarks.

Abstract

Open web-scale pre-training corpora remain concentrated in English, limiting multilingual LLM development. We introduce MultiSynt/MT, an open synthetic parallel corpus with approximately 4.8 trillion target-language tokens across 36 European languages, produced by translating 100 billion high-quality Nemotron-CC tokens with Tower+ and OPUS-MT/HPLT-MT systems. For many medium- and lower-resource European languages, this is the largest openly available pre-training resource. On a broad multilingual benchmark suite, reference LLMs trained on MultiSynt/MT reach the final score of HPLT 2.0, a native-data baseline, using roughly 72% fewer pre-training tokens, and outperform it by approximately 15% relative at a matched 100B-token training budget. Our analyses also identify evaluation blind spots: standard multiple-choice benchmarks miss translation-quality differences that a fluency-sensitive LLM-as-judge evaluation cleanly recovers on the trained LLMs (with no fluency deficit in MultiSynt itself), and Norwegian idiomatic and culturally grounded tasks remain better served by native data. We release the corpus, including row-aligned translations from multiple systems, to support controlled research on multilingual pre-training data and evaluation.

Problem

• multilingual LLM
• pre-training data
• low-resource languages
• parallel corpus
• language coverage

Innovation

• synthetic parallel corpus
• multilingual pre-training
• translation-based data augmentation
• LLM-as-judge evaluation
• low-resource languages