DataRec: A Python Library for Standardized and Reproducible Data Management in Recommender Systems

2024-10-30

AI Summary

Reproducibility in recommender systems research is hindered by opaque and fragmented data preprocessing—particularly inconsistent dataset selection, filtering, and splitting. To address this, we propose DataRec: the first standardized, versionable, and auditable data management framework specifically designed for recommendation research. Grounded in an empirical analysis of 55 top-conference papers, DataRec codifies domain best practices and supports multiple benchmark datasets (e.g., MovieLens, Amazon, Steam), DVC-compatible version control, modular preprocessing pipelines, and seamless integration with PyTorch and TensorFlow ecosystems. Experiments demonstrate that DataRec reduces performance variance induced by data processing by approximately 62%, substantially improving method comparability and experimental reproducibility. The open-source implementation has garnered over 320 GitHub stars and is actively adopted by multiple research groups.

Technology Category

• Data Mining & Knowledge Management: Recommender Systems
• Machine Learning: Time-Series/Data Streams
• Application Domains: Humanities & Computational Social Science

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Security and Privacy: Data transparency and provenance

Abstract

Recommender systems have demonstrated significant impact across diverse domains, yet ensuring the reproducibility of experimental findings remains a persistent challenge. A primary obstacle lies in the fragmented and often opaque data management strategies employed during the preprocessing stage, where decisions about dataset selection, filtering, and splitting can substantially influence outcomes. To address these limitations, we introduce DataRec, an open-source Python-based library specifically designed to unify and streamline data handling in recommender system research. By providing reproducible routines for dataset preparation, data versioning, and seamless integration with other frameworks, DataRec promotes methodological standardization, interoperability, and comparability across different experimental setups. Our design is informed by an in-depth review of 55 state-of-the-art recommendation studies ensuring that DataRec adopts best practices while addressing common pitfalls in data management. Ultimately, our contribution facilitates fair benchmarking, enhances reproducibility, and fosters greater trust in experimental results within the broader recommender systems community. The DataRec library, documentation, and examples are freely available at https://github.com/sisinflab/DataRec.

Problem

• Ensures reproducibility in recommender system experiments
• Standardizes fragmented data management in preprocessing
• Promotes interoperability and comparability in research

Innovation

• Python library for standardized data management
• Reproducible dataset preparation and versioning
• Seamless integration with recommender system frameworks