Datasheets Aren't Enough: DataRubrics for Automated Quality Metrics and Accountability

2025-06-02

AI Summary

Contemporary dataset papers frequently suffer from limited originality, insufficient diversity, inadequate quality control, and poor transparency regarding construction methodologies; existing datasheets are largely descriptive and lack quantifiable evaluation criteria or enforceable accountability mechanisms. Method: We propose DataRubrics, the first rubric-based framework for structured data quality assessment, integrating LLM-as-a-judge (e.g., GPT-4) with synthetic data techniques to enable automated, reproducible, and standardized quality scoring for both human- and model-generated datasets. Contribution/Results: The framework delivers an open-source evaluation toolkit (github.com/datarubrics/datarubrics), facilitating collaborative, measurable data review by reviewers and authors alike. It significantly enhances rigor, transparency, and trustworthiness in data-centric research through objective, interpretable, and auditable quality metrics.

Technology Category

• Knowledge Representation and Reasoning: Qualitative Reasoning
• Humans and AI: Crowd Sourcing and Human Computation
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content

Abstract

High-quality datasets are fundamental to training and evaluating machine learning models, yet their creation-especially with accurate human annotations-remains a significant challenge. Many dataset paper submissions lack originality, diversity, or rigorous quality control, and these shortcomings are often overlooked during peer review. Submissions also frequently omit essential details about dataset construction and properties. While existing tools such as datasheets aim to promote transparency, they are largely descriptive and do not provide standardized, measurable methods for evaluating data quality. Similarly, metadata requirements at conferences promote accountability but are inconsistently enforced. To address these limitations, this position paper advocates for the integration of systematic, rubric-based evaluation metrics into the dataset review process-particularly as submission volumes continue to grow. We also explore scalable, cost-effective methods for synthetic data generation, including dedicated tools and LLM-as-a-judge approaches, to support more efficient evaluation. As a call to action, we introduce DataRubrics, a structured framework for assessing the quality of both human- and model-generated datasets. Leveraging recent advances in LLM-based evaluation, DataRubrics offers a reproducible, scalable, and actionable solution for dataset quality assessment, enabling both authors and reviewers to uphold higher standards in data-centric research. We also release code to support reproducibility of LLM-based evaluations at https://github.com/datarubrics/datarubrics.

Problem

• Lack standardized metrics for dataset quality evaluation
• Insufficient transparency in dataset construction details
• Need scalable methods for synthetic data generation

Innovation

• Rubric-based evaluation metrics for datasets
• Synthetic data generation using LLM-as-a-judge
• DataRubrics framework for scalable quality assessment