No Language Data Left Behind: A Comparative Study of CJK Language Datasets in the Hugging Face Ecosystem

2025-07-06

AI Summary

East Asian languages (Chinese, Japanese, Korean—CJK) suffer from scarce, fragmented, and heterogeneous datasets in large language model (LLM) training, hindering equitable multilingual AI development. Method: This study conducts the first systematic evaluation of over 3,300 CJK datasets on Hugging Face, assessing scale, documentation completeness, licensing compliance, and provenance through integrated quantitative analysis and in-depth qualitative case studies. Contribution/Results: It reveals structural disparities across CJK data ecosystems: Chinese datasets are predominantly large-scale and institutionally driven (universities/corporations); Japanese datasets emphasize entertainment and subcultural domains; Korean datasets rely heavily on grassroots community curation. The work proposes a novel cross-lingual collaborative data stewardship framework, identifies how cultural norms and research governance shape data practices, and offers actionable recommendations—enhancing transparency, interoperability, and sustainable sharing—to bridge the non-English data gap with empirical grounding and methodological rigor.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Knowledge Representation and Reasoning: Knowledge Representation Languages
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Recent advances in Natural Language Processing (NLP) have underscored the crucial role of high-quality datasets in building large language models (LLMs). However, while extensive resources and analyses exist for English, the landscape for East Asian languages - particularly Chinese, Japanese, and Korean (CJK) - remains fragmented and underexplored, despite these languages together serving over 1.6 billion speakers. To address this gap, we investigate the HuggingFace ecosystem from a cross-linguistic perspective, focusing on how cultural norms, research environments, and institutional practices shape dataset availability and quality. Drawing on more than 3,300 datasets, we employ quantitative and qualitative methods to examine how these factors drive distinct creation and curation patterns across Chinese, Japanese, and Korean NLP communities. Our findings highlight the large-scale and often institution-driven nature of Chinese datasets, grassroots community-led development in Korean NLP, and an entertainment- and subculture-focused emphasis on Japanese collections. By uncovering these patterns, we reveal practical strategies for enhancing dataset documentation, licensing clarity, and cross-lingual resource sharing - ultimately guiding more effective and culturally attuned LLM development in East Asia. We conclude by discussing best practices for future dataset curation and collaboration, aiming to strengthen resource development across all three languages.

Problem

• Lack of high-quality CJK language datasets in NLP
• Fragmented resources for Chinese, Japanese, and Korean
• Cultural and institutional impacts on dataset availability

Innovation

• Analyze HuggingFace datasets cross-linguistically for CJK
• Quantitative and qualitative methods on 3,300 datasets
• Enhance documentation and licensing for CJK resources