A General-Purpose Data Harmonization Framework: Supporting Reproducible and Scalable Data Integration in the RADx Data Hub

2025-03-03

AI Summary

To address the challenge of achieving FAIR interoperability and reusability for multi-source, heterogeneous scientific data—such as the RADx COVID-19 response data—in large-scale environments, this paper proposes a general-purpose, reproducible, and extensible data harmonization framework. Our approach introduces a novel harmonization paradigm based on parameterized primitive operations and automated execution tracing. It integrates a customizable data representation model, a configurable operation library, and mechanisms for transformation logging and dependency tracking—ensuring protocol reproducibility, process auditability, and transformation reusability. Evaluated in real-world deployment within the RADx Data Hub, the framework significantly lowers the barrier to entry for domain experts, improves harmonization efficiency and transparency, and enables high-quality cross-study analyses. This work provides a scalable, transferable technical pathway for FAIR-compliant data integration across diverse scientific domains.

Technology Category

• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust
• Application Domains: Humanities & Computational Social Science
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Security and Privacy: Data transparency and provenance
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Web Mining and Content Analysis: Web data integration and cleaning

Abstract

In the age of big data, it is important for primary research data to follow the FAIR principles of findability, accessibility, interoperability, and reusability. Data harmonization enhances interoperability and reusability by aligning heterogeneous data under standardized representations, benefiting both repository curators responsible for upholding data quality standards and consumers who require unified datasets. However, data harmonization is difficult in practice, requiring significant domain and technical expertise. We present a software framework to facilitate principled and reproducible harmonization protocols. Our framework implements a novel strategy of building harmonization transformations from parameterizable primitive operations and automated bookkeeping for executed transformations. We establish our data representation model and harmonization strategy and then present a proof-of-concept application in the context of the RADx Data Hub for COVID-19 pandemic response data. We believe that our framework offers a powerful solution for data scientists and curators who value transparency and reproducibility in data harmonization.

Problem

• Facilitates data harmonization for FAIR principles compliance
• Supports reproducible and scalable data integration processes
• Addresses challenges in aligning heterogeneous data sets

Innovation

• Parameterizable primitive operations for harmonization
• Automated bookkeeping of transformation executions
• Proof-of-concept in RADx Data Hub application