GneissWeb: Preparing High Quality Data for LLMs at Scale

2025-02-19

AI Summary

To address the scarcity of high-quality, large-scale open-source data for large language model (LLM) pretraining, this work introduces GneissWeb—a 10-trillion-token, high-fidelity open-source dataset. We propose a synergistic data cleaning paradigm combining *shard-level exact substring deduplication* with *multi-stage quality filtering*, integrating ensemble-based quality scoring, heuristic rule-based filtering, and benchmark-driven evaluation. This approach preserves massive scale while substantially improving semantic purity and downstream generalization. Empirical evaluation shows that models pretrained on GneissWeb achieve an average +2.73 percentage points improvement over FineWeb-V1.1.0 across 11 diverse benchmarks; the gain remains robust at +1.75 points when extended to 20 benchmarks, demonstrating strong scalability and robustness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Data quantity and quality play a vital role in determining the performance of Large Language Models (LLMs). High-quality data, in particular, can significantly boost the LLM's ability to generalize on a wide range of downstream tasks. Large pre-training datasets for leading LLMs remain inaccessible to the public, whereas many open datasets are small in size (less than 5 trillion tokens), limiting their suitability for training large models. In this paper, we introduce GneissWeb, a large dataset yielding around 10 trillion tokens that caters to the data quality and quantity requirements of training LLMs. Our GneissWeb recipe that produced the dataset consists of sharded exact sub-string deduplication and a judiciously constructed ensemble of quality filters. GneissWeb achieves a favorable trade-off between data quality and quantity, producing models that outperform models trained on state-of-the-art open large datasets (5+ trillion tokens). We show that models trained using GneissWeb dataset outperform those trained on FineWeb-V1.1.0 by 2.73 percentage points in terms of average score computed on a set of 11 commonly used benchmarks (both zero-shot and few-shot) for pre-training dataset evaluation. When the evaluation set is extended to 20 benchmarks (both zero-shot and few-shot), models trained using GneissWeb still achieve a 1.75 percentage points advantage over those trained on FineWeb-V1.1.0.

Problem

• Enhancing LLM performance via quality data
• Addressing scarcity of large pre-training datasets
• Optimizing data quality and quantity balance

Innovation

• Sharded exact sub-string deduplication
• Judiciously constructed quality filters
• Yields 10 trillion tokens dataset