data preprocessing and transformation

Existing ETL pipelines heavily rely on manual, context-sensitive design of transformation logic, resulting in poor generalizability and low reusability. To address this, we propose an example-driven autonomous ETL framework: given user-provided target data examples, it constructs a paired-sample-based planning engine that automatically infers and synthesizes high-fidelity, context-adapted data transformation programs. Integrated with modular ETL components and runtime monitoring, the framework enables end-to-end automation for multi-format, multi-structured, and multi-scale data processing. Experiments across 14 real-world, cross-domain datasets demonstrate that our approach substantially reduces human intervention while achieving high-precision transformations (average F1 score of 0.92), strong generalization across diverse schemas and formats, and practical engineering deployability.

This work addresses the significant limitations of spreadsheet-based analysis in reproducibility, auditability, version control, and automation. It proposes a migration pathway from Excel to research-grade analytical workflows by leveraging Python’s pandas library as a bridge. The study introduces an innovative set of Excel-to-pandas mapping rules, categorizes nine canonical workflow patterns, and compiles a catalog of common failure modes. Seven end-to-end real-world examples demonstrate the approach in practice. By retaining Excel as a familiar interface for input and output while integrating version control, automated refreshing, and seamless incorporation of statistical and machine learning methods, the proposed framework enables governed, reproducible, and auditable tabular data analysis.

Data standardization is critical in the data science lifecycle, yet existing tools (e.g., Pandas) require manual, error-prone coding, while LLM-based automation still demands expert prompt engineering and iterative interaction. To address this, we propose a declarative API-driven LLM-Agent framework that introduces *Dataprep.Clean*—a novel, column-type-aware standardization component enabling end-to-end automation via a single-line operation and one-shot natural language input. Our method integrates domain knowledge modeling with lightweight agent orchestration, eliminating programming prerequisites and enabling semantic cleaning of heterogeneous columns. Evaluated on real-world datasets, the approach achieves high accuracy and robustness across diverse standardization tasks. Deployed as an interactive web tool, it substantially lowers the barrier to entry for data practitioners. This work advances data preprocessing toward declarative, intelligent automation—bridging the gap between domain expertise and scalable, user-friendly tooling.

To address poor scalability, integration complexity, and inflexible configuration in preprocessing multi-source heterogeneous data for machine learning modeling and graph database construction, this paper proposes a lightweight, modular, JSON-driven automated data preprocessing framework. Built upon Spark DataFrames for efficient distributed processing, the framework defines 69 composable and parallelizable processing stages spanning input parsing, filtering, statistical analysis, feature engineering, and graph-structure transformation. Its declarative JSON-based configuration enables dynamic adaptation to varying data types and scales, significantly enhancing interoperability with workflow orchestration systems such as Apache Airflow. Experimental evaluation across six heterogeneous datasets demonstrates the framework’s generality and scalability: it successfully supports wine quality clustering analysis and end-to-end conversion of phosphosylation site–kinase interaction data into a graph database.

Data cleaning remains highly manual, inefficient, and error-prone. This paper proposes the first goal-driven LLM-based framework for automatic workflow generation: given a dirty table and a target query, it end-to-end generates a minimal viable clean table along with executable cleaning steps—including deduplication, missing-value imputation, and format standardization. Our contributions are threefold: (1) We introduce the first benchmark dataset comprising annotated quadruples of (goal, dirty table, cleaning workflow, cleaned answer); (2) We design a zero-shot, multi-stage prompting framework—requiring no fine-tuning—that decomposes the task into goal column identification, data quality diagnosis, and operation-parameter generation; (3) We empirically validate that off-the-shelf LLMs possess inherent reasoning capabilities sufficient to generate high-quality, executable cleaning workflows across three major LLM families, significantly reducing human intervention.

Prior work lacks systematic evaluation of how data processing frameworks impact end-to-end deep learning training and inference—particularly regarding performance-energy trade-offs across data loading, preprocessing, and batch feeding stages in conjunction with GPU computation. Method: We conduct the first comprehensive empirical study comparing Pandas, Polars, and Dask across diverse deep learning workloads—including CNNs and Transformers trained on ImageNet and WikiText—measuring runtime, memory footprint, disk I/O, and CPU/GPU power consumption under varying data scales and I/O characteristics. Contribution/Results: Polars achieves optimal latency–energy efficiency for medium-scale in-memory datasets; Dask scales effectively to ultra-large distributed workloads but exhibits lower energy efficiency; Pandas remains practical for small-batch, interactive tasks. Our findings bridge a critical gap in co-optimizing data engineering infrastructure with AI training pipelines, providing empirical guidance for green AI system design and framework selection in production ML systems.

This work addresses the challenge of reconciling high throughput and low query latency in traditional ETL pipelines when processing continuously arriving fresh data, where unpredictable preprocessing operations often create bottlenecks. The authors propose Fluid ETL Pipelines, which introduce, for the first time, an elastic and non-blocking preprocessing mechanism that decouples data ingestion from transformation. By dynamically scheduling preprocessing tasks based on resource availability and user interest—without blocking data ingestion—and leveraging preemptible computing resources such as Amazon Spot instances, the approach significantly reduces operational costs. Experimental results demonstrate that Fluid ETL Pipelines substantially improve the efficiency of exploring fresh data, offering a novel direction for accelerating real-time queries and enabling adaptive preprocessing management.

Existing automated data preparation tools lack robust semantic understanding and struggle with complex, context-dependent data quality issues. Method: This study investigates the efficacy of large language models (LLMs) in data profiling and cleaning on low-quality datasets. We propose a customized data quality assessment framework informed by a practitioner-focused user study, and systematically evaluate both general-purpose and fine-tuned table-centric LLMs—via prompt engineering—on tasks including anomaly detection, cleaning logic generation, and error repair, benchmarking against traditional tools (e.g., Trifacta, OpenRefine). Contribution/Results: LLMs significantly outperform conventional tools in contextual reasoning and generating interpretable, human-verifiable cleaning rules; however, their output precision and deterministic verifiability remain limited. This work establishes the first evaluation paradigm specifically designed for LLMs in data preparation and empirically validates their viability—and practical boundaries—as collaborative “data engineering partners.”

To address the lack of early, machine-readable descriptions of scientific data analysis workflows—hindering FAIR (Findable, Accessible, Interoperable, Reusable) compliance—this paper introduces dtreg, the first structured registration framework for statistical and machine learning pipelines targeting the pre-publication stage and supporting both Python and R. Its core contributions are: (1) a novel pre-analysis metadata registration mechanism; (2) a persistent, globally identifiable schema system covering mainstream statistical tests (e.g., t-tests) and ML methods; and (3) lightweight, automated RDF/Linked Data serialization to Turtle and JSON-LD. Leveraging object-oriented modeling, dynamic schema population, and export capabilities, dtreg enables end-to-end machine-readable workflow documentation. As an open-source infrastructure, it significantly enhances the findability, interoperability, and reusability of analytical methods in computational research.

This study systematically evaluates performance differences among Java, Python, and Scala for end-to-end ETL workloads on Apache Spark integrated with Apache Iceberg. We conduct controlled experiments across varying data scales (5 MB–1.6 GB) and operation complexities (basic transformations vs. complex merge operations), all within a uniform CSV→Spark→Iceberg pipeline. To our knowledge, this is the first standardized ETL benchmark enabling direct cross-language comparison under native Iceberg support. Results reveal nonlinear interactions among programming language, data volume, and operation type: Python exhibits superior throughput on small-scale data; performance converges across all three languages at medium scale (1.6 GB); and Scala significantly outperforms both Java and Python in high-complexity merge-intensive workloads. The findings provide empirical guidance for language selection in big-data ETL systems, along with a principled trade-off framework grounded in workload characteristics.