Problem
Research questions and friction points this paper is trying to address.
data readiness
scientific AI
automated transformation
provenance tracking
FAIR compliance
Automated Data Readiness for Scientific AI
🤖 AI Summary
Scientific data often require extensive manual curation before being usable for scientific AI, lacking a unified framework for automated conversion, readiness assessment, provenance tracking, and agent integration. This work proposes REDI, an open-source framework that automatically transforms raw scientific data into AI-ready formats through a five-stage, fully traceable pipeline—ingestion, preprocessing, transformation, structuring, and output—while exposing the resulting workflows as callable skills for AI agents. REDI is the first framework to unify these capabilities; its companion tool, SetGo, ensures FAIR compliance and enables automatic catalog publishing. Leveraging parallel distributed processing and I/O performance profiling, REDI demonstrates effectiveness across climate science, proteomics, materials science, and nuclear fusion, with the climate use case achieving near-ideal strong scaling up to 100 nodes on the Frontier supercomputer.
📝 Abstract
Leadership computing facilities steward large-scale scientific datasets that routinely require substantial transformation before serving as AI training data. However, no existing framework fully unifies automated transformation, readiness assessment, provenance tracking, and agent-native deployment. We present REDI, an open-source framework that addresses this gap through a unified five-stage pipeline (ingest, preprocess, transform, structure, and output) with per-stage instrumentation for reproducibility and deployment as an agent-callable skill; companion tool SetGo automates FAIR compliance and catalog publication. Evaluated across climate, proteomics, materials science, and nuclear fusion, REDI transforms all datasets from raw to AI-ready, with outputs validated against domain-expert references, and preliminary results show near-ideal parallel scaling to 100 nodes on Frontier for the climate case. Provenance-instrumented profiling reveals file I/O as the dominant pipeline cost, with format selection a first-order optimization lever. These results establish REDI as a cross-domain platform providing automated data readiness for scientific AI, transforming data preparation bottlenecks into reproducible, reusable community assets.
Innovation
Methods, ideas, or system contributions that make the work stand out.
automated data readiness
scientific AI
provenance tracking
FAIR compliance
agent-native deployment
🔎 Similar Papers
No similar papers found.
