Improving Romanian LLM Pretraining Data using Diversity and Quality Filtering

2025-11-02

AI Summary

Pretraining data for Romanian is scarce, heterogeneous in quality, and insufficiently diverse in topical coverage. Method: This paper proposes a lightweight multi-task filtering framework that jointly evaluates educational value, performs topic modeling, and analyzes syntactic and formatting diversity to conduct hierarchical quality screening of LLM-generated annotated texts. Unlike conventional unidimensional data cleaning, the framework systematically characterizes cross-lingual disparities between Romanian and English pretraining corpora across topic distribution, pedagogical relevance, and structural diversity. Contribution/Results: Empirical evaluation demonstrates that models trained on the filtered Romanian corpus achieve substantial performance gains on downstream tasks—including ROBUST and RONEC—validating the efficacy of structured, domain-aware data curation for low-resource language modeling. The work establishes a principled paradigm for small-language pretraining data construction, highlighting its critical role in enhancing model capabilities.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Large Language Models (LLMs) have recently exploded in popularity, often matching or outperforming human abilities on many tasks. One of the key factors in training LLMs is the availability and curation of high-quality data. Data quality is especially crucial for under-represented languages, where high-quality corpora are scarce. In this work we study the characteristics and coverage of Romanian pretraining corpora and we examine how they differ from English data. By training a lightweight multitask model on carefully LLM-annotated Romanian texts, we are able to analyze and perform multi-level filtering (e.g., educational value, topic, format) to generate high-quality pretraining datasets. Our experiments show noteworthy trends in the topics present in Romanian and English data, while also proving the effectiveness of filtering data through improved LLM pretraining performance across multiple benchmarks.

Problem

• Analyzing Romanian pretraining corpora characteristics and English comparisons
• Developing multi-level filtering methods for high-quality Romanian datasets
• Improving Romanian LLM performance through diversity and quality filtering

Innovation

• Multitask model analyzes Romanian text characteristics
• Performs multi-level filtering for dataset quality
• Uses LLM-annotated texts to improve pretraining performance