MLPrE -- A tool for preprocessing and exploratory data analysis prior to machine learning model construction

2025-10-29

AI Summary

To address poor scalability, integration complexity, and inflexible configuration in preprocessing multi-source heterogeneous data for machine learning modeling and graph database construction, this paper proposes a lightweight, modular, JSON-driven automated data preprocessing framework. Built upon Spark DataFrames for efficient distributed processing, the framework defines 69 composable and parallelizable processing stages spanning input parsing, filtering, statistical analysis, feature engineering, and graph-structure transformation. Its declarative JSON-based configuration enables dynamic adaptation to varying data types and scales, significantly enhancing interoperability with workflow orchestration systems such as Apache Airflow. Experimental evaluation across six heterogeneous datasets demonstrates the framework’s generality and scalability: it successfully supports wine quality clustering analysis and end-to-end conversion of phosphosylation site–kinase interaction data into a graph database.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Web Mining and Content Analysis: Web data integration and cleaning

Abstract

With the recent growth of Deep Learning for AI, there is a need for tools to meet the demand of data flowing into those models. In some cases, source data may exist in multiple formats, and therefore the source data must be investigated and properly engineered for a Machine Learning model or graph database. Overhead and lack of scalability with existing workflows limit integration within a larger processing pipeline such as Apache Airflow, driving the need for a robust, extensible, and lightweight tool to preprocess arbitrary datasets that scales with data type and size. To address this, we present Machine Learning Preprocessing and Exploratory Data Analysis, MLPrE, in which SparkDataFrames were utilized to hold data during processing and ensure scalability. A generalizable JSON input file format was utilized to describe stepwise changes to that DataFrame. Stages were implemented for input and output, filtering, basic statistics, feature engineering, and exploratory data analysis. A total of 69 stages were implemented into MLPrE, of which we highlight and demonstrate key stages using six diverse datasets. We further highlight MLPrE's ability to independently process multiple fields in flat files and recombine them, otherwise requiring an additional pipeline, using a UniProt glossary term dataset. Building on this advantage, we demonstrated the clustering stage with available wine quality data. Lastly, we demonstrate the preparation of data for a graph database in the final stages of MLPrE using phosphosite kinase data. Overall, our MLPrE tool offers a generalizable and scalable tool for preprocessing and early data analysis, filling a critical need for such a tool given the ever expanding use of machine learning. This tool serves to accelerate and simplify early stage development in larger workflows.

Problem

• Preprocessing diverse data formats for machine learning models
• Addressing scalability limitations in existing data processing workflows
• Providing exploratory analysis capabilities for large-scale datasets

Innovation

• Uses SparkDataFrames for scalable data processing
• Implements JSON configuration for stepwise transformations
• Provides 69 modular stages for preprocessing workflows