Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models

2025-04-19

AI Summary

Current large language model (LLM) pretraining datasets suffer from opacity in composition, and existing data filtering methods typically optimize only one dimension—e.g., quality, diversity, or redundancy—failing to holistically assess data quality. Method: We propose PRRC, a four-dimensional evaluation framework (Professionalism, Readability, Reasoning capability, Cleanliness), coupled with Meta-rater, a multidimensional weighted selection method. PRRC introduces the first collaborative assessment system; Meta-rater employs a dynamic weight learning mechanism based on surrogate models and validation loss regression to optimally fuse multiple quality objectives; and we perform 25-dimensional fine-grained annotation. Results: Experiments show that a 1.3B-parameter model trained on PRRC-filtered data achieves 2× faster convergence and +3.23 average points on downstream tasks; a 3.3B model demonstrates strong scalability on 100B tokens; and we open-source SlimPajama-627B, a high-quality subset.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The composition of pre-training datasets for large language models (LLMs) remains largely undisclosed, hindering transparency and efforts to optimize data quality, a critical driver of model performance. Current data selection methods, such as natural language quality assessments, diversity-based filters, and classifier-based approaches, are limited by single-dimensional evaluation or redundancy-focused strategies. To address these gaps, we propose PRRC to evaluate data quality across Professionalism, Readability, Reasoning, and Cleanliness. We further introduce Meta-rater, a multi-dimensional data selection method that integrates these dimensions with existing quality metrics through learned optimal weightings. Meta-rater employs proxy models to train a regression model that predicts validation loss, enabling the identification of optimal combinations of quality scores. Experiments demonstrate that Meta-rater doubles convergence speed for 1.3B parameter models and improves downstream task performance by 3.23, with scalable benefits observed in 3.3B models trained on 100B tokens. Additionally, we release the annotated SlimPajama-627B dataset, labeled across 25 quality metrics (including PRRC), to advance research in data-centric LLM development. Our work establishes that holistic, multi-dimensional quality integration significantly outperforms conventional single-dimension approaches, offering a scalable paradigm for enhancing pre-training efficiency and model capability.

Problem

• Evaluates data quality across multiple dimensions (Professionalism, Readability, Reasoning, Cleanliness).
• Improves pre-training efficiency and model performance via multi-dimensional data selection.
• Addresses limitations of single-dimensional evaluation in current data selection methods.

Innovation

• Multi-dimensional data selection with PRRC metrics
• Proxy models predict optimal quality score combinations
• Releases annotated SlimPajama-627B dataset for research