HPLT~3.0: Very Large-Scale Multilingual Resources for LLM and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models

2025-11-02

AI Summary

To address the scarcity of high-quality, large-scale, fine-grained annotated data for multilingual large language models and machine translation research, this work introduces the first open-source, large-scale multilingual text data construction framework. It supports nearly 200 languages and scales to 30 trillion tokens, integrating end-to-end techniques including web page cleaning, noise-robust language identification, exact and fuzzy deduplication, PII detection, registry label annotation, text quality scoring, and synthetic parallel corpus generation. We propose the first native-task-based multilingual evaluation framework, releasing standardized benchmarks across nine languages and an automated assessment pipeline. The resulting dataset constitutes the largest publicly available multilingual pretraining corpus to date. Using it, we train 57 monolingual encoder-decoder models and multiple GPT-style monolingual models. All tooling—including data processing pipelines, evaluation benchmarks, and pretrained model families—is fully open-sourced.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

We present an ongoing initiative to provide open, very large, high-quality, and richly annotated textual datasets for almost 200 languages. At 30 trillion tokens, this is likely the largest generally available multilingual collection of LLM pre-training data. At 30 trillion tokens, this is likely the largest generally available multilingual collection of LLM pre-training data. These datasets are derived from web crawls from different sources and accompanied with a complete, open-source pipeline for document selection from web archives, text extraction from HTML, language identification for noisy texts, exact and near-deduplication, annotation with, among others, register labels, text quality estimates, and personally identifiable information; and final selection and filtering. We report on data quality probes through contrastive and analytical statistics, through manual inspection of samples for 24 languages, and through end-to-end evaluation of various language model architectures trained on this data. For multilingual LLM evaluation, we provide a comprehensive collection of benchmarks for nine European languages, with special emphasis on natively created tasks, mechanisms to mitigate prompt sensitivity, and refined normalization and aggregation of scores. Additionally, we train and evaluate a family of 57 monolingual encoder-decoder models, as well as a handful of monolingual GPT-like reference models. Besides the monolingual data and models, we also present a very large collection of parallel texts automatically mined from this data, together with a novel parallel corpus synthesized via machine translation.

Problem

• Creating massive open multilingual datasets for LLM training and machine translation
• Developing comprehensive evaluation benchmarks for multilingual language model assessment
• Providing high-quality monolingual and bilingual data with rich annotations and filtering

Innovation

• Largest multilingual LLM pre-training data collection
• Open-source pipeline for document selection and annotation
• Comprehensive multilingual evaluation benchmarks and models