Towards Best Practices for Open Datasets for LLM Training

2025-01-14

AI Summary

This study addresses systemic challenges in large language model (LLM) training—including high copyright risk, opaque data provenance, missing metadata, and insufficient cross-domain collaboration—by proposing the first open dataset construction framework specifically designed for LLM training. Methodologically, it integrates digital humanities principles, Schema.org metadata extensions, automated copyright status identification, and multi-source heterogeneous data cleaning and provenance tracking to unify legal compliance, technical traceability, and social collaboration. Key contributions include: (1) establishing practical standards and governance pathways for responsible open LLM training datasets; (2) releasing a reusable construction guideline, a legal compatibility assessment toolkit, and an interdisciplinary collaboration prototype; and (3) laying the foundation for the first large-scale, high-quality, fully openly licensed LLM training dataset—thereby significantly enhancing transparency, accountability, and sustainable innovation in AI development.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Many AI companies are training their large language models (LLMs) on data without the permission of the copyright owners. The permissibility of doing so varies by jurisdiction: in countries like the EU and Japan, this is allowed under certain restrictions, while in the United States, the legal landscape is more ambiguous. Regardless of the legal status, concerns from creative producers have led to several high-profile copyright lawsuits, and the threat of litigation is commonly cited as a reason for the recent trend towards minimizing the information shared about training datasets by both corporate and public interest actors. This trend in limiting data information causes harm by hindering transparency, accountability, and innovation in the broader ecosystem by denying researchers, auditors, and impacted individuals access to the information needed to understand AI models. While this could be mitigated by training language models on open access and public domain data, at the time of writing, there are no such models (trained at a meaningful scale) due to the substantial technical and sociological challenges in assembling the necessary corpus. These challenges include incomplete and unreliable metadata, the cost and complexity of digitizing physical records, and the diverse set of legal and technical skills required to ensure relevance and responsibility in a quickly changing landscape. Building towards a future where AI systems can be trained on openly licensed data that is responsibly curated and governed requires collaboration across legal, technical, and policy domains, along with investments in metadata standards, digitization, and fostering a culture of openness.

Problem

• Copyright Law
• Transparent AI Research
• High-Quality Dataset

Innovation

• Cross-domain Collaboration
• Data Standardization
• AI Transparency