SciLaD: A Large-Scale, Transparent, Reproducible Dataset for Natural Scientific Language Processing

2025-12-11

AI Summary

The scientific NLP community lacks large-scale, transparent, and reproducible multilingual academic datasets. Method: We introduce SciLaD—the first fully open-source, end-to-end reproducible scientific language dataset—comprising over 10 million English and 35 million multilingual scholarly articles in TEI XML format, sourced from authoritative repositories including PubMed Central and arXiv. We develop a standardized preprocessing pipeline built exclusively on open-source tools (e.g., GROBID, Apache Tika) to ensure cross-lingual, high-fidelity text extraction and structuring. Contribution/Results: SciLaD establishes a novel paradigm integrating open toolchains, publicly accessible data sources, TEI-based standardization, and verifiable evaluation—guaranteeing full reproducibility and community extensibility. Models pretrained on SciLaD achieve performance on par with state-of-the-art models of comparable scale across multiple scientific NLP benchmarks. The dataset, source code, and complete processing pipeline are publicly released under permissive open licenses.

Technology Category

• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Large pretrained models with web data
• Security and Privacy: Data transparency and provenance

Abstract

SciLaD is a novel, large-scale dataset of scientific language constructed entirely using open-source frameworks and publicly available data sources. It comprises a curated English split containing over 10 million scientific publications and a multilingual, unfiltered TEI XML split including more than 35 million publications. We also publish the extensible pipeline for generating SciLaD. The dataset construction and processing workflow demonstrates how open-source tools can enable large-scale, scientific data curation while maintaining high data quality. Finally, we pre-train a RoBERTa model on our dataset and evaluate it across a comprehensive set of benchmarks, achieving performance comparable to other scientific language models of similar size, validating the quality and utility of SciLaD. We publish the dataset and evaluation pipeline to promote reproducibility, transparency, and further research in natural scientific language processing and understanding including scholarly document processing.

Problem

• Creating a large-scale, open-source dataset for scientific language processing.
• Developing an extensible pipeline to ensure reproducibility and transparency.
• Pre-training and evaluating a model to validate dataset quality and utility.

Innovation

• Large-scale dataset built with open-source tools
• Extensible pipeline for generating scientific language data
• Pre-trained RoBERTa model validated on benchmarks