Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Data

2025-05-08

AI Summary

To address two key bottlenecks in LLM training data filtering—lagging quality assessment and subjectivity in seed sample selection—this paper proposes an efficient data filtering and validation pipeline. Methodologically, it (1) introduces a lightweight, real-time validation strategy that decouples data quality feedback from model training, and (2) pioneers a quality-hypothesis-driven seed sample self-optimization mechanism to iteratively enhance classifier generalization. The pipeline integrates fastText-based lightweight classification, model-guided filtering, bilingual (English–Chinese) consistency verification, and FineWeb/Chinese FineWeb augmentation. It yields Ultra-FineWeb, a high-quality corpus comprising 1 trillion English tokens and 120 billion Chinese tokens. Experiments demonstrate substantial improvements across multiple downstream LLM benchmarks, over 40% reduction in training cost, and strong robustness with minimal computational overhead.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Data quality has become a key factor in enhancing model performance with the rapid development of large language models (LLMs). Model-driven data filtering has increasingly become a primary approach for acquiring high-quality data. However, it still faces two main challenges: (1) the lack of an efficient data verification strategy makes it difficult to provide timely feedback on data quality; and (2) the selection of seed data for training classifiers lacks clear criteria and relies heavily on human expertise, introducing a degree of subjectivity. To address the first challenge, we introduce an efficient verification strategy that enables rapid evaluation of the impact of data on LLM training with minimal computational cost. To tackle the second challenge, we build upon the assumption that high-quality seed data is beneficial for LLM training, and by integrating the proposed verification strategy, we optimize the selection of positive and negative samples and propose an efficient data filtering pipeline. This pipeline not only improves filtering efficiency, classifier quality, and robustness, but also significantly reduces experimental and inference costs. In addition, to efficiently filter high-quality data, we employ a lightweight classifier based on fastText, and successfully apply the filtering pipeline to two widely-used pre-training corpora, FineWeb and Chinese FineWeb datasets, resulting in the creation of the higher-quality Ultra-FineWeb dataset. Ultra-FineWeb contains approximately 1 trillion English tokens and 120 billion Chinese tokens. Empirical results demonstrate that the LLMs trained on Ultra-FineWeb exhibit significant performance improvements across multiple benchmark tasks, validating the effectiveness of our pipeline in enhancing both data quality and training efficiency.

Problem

• Lack efficient data verification strategy for LLM training feedback
• No clear criteria for seed data selection in classifiers
• Need lightweight classifier to filter high-quality training data

Innovation

• Efficient verification strategy for LLM data quality
• Optimized seed data selection using verification feedback
• Lightweight fastText classifier for high-quality data filtering