data engineering

Existing ETL pipelines heavily rely on manual, context-sensitive design of transformation logic, resulting in poor generalizability and low reusability. To address this, we propose an example-driven autonomous ETL framework: given user-provided target data examples, it constructs a paired-sample-based planning engine that automatically infers and synthesizes high-fidelity, context-adapted data transformation programs. Integrated with modular ETL components and runtime monitoring, the framework enables end-to-end automation for multi-format, multi-structured, and multi-scale data processing. Experiments across 14 real-world, cross-domain datasets demonstrate that our approach substantially reduces human intervention while achieving high-precision transformations (average F1 score of 0.92), strong generalization across diverse schemas and formats, and practical engineering deployability.

Large-scale text cleaning, filtering, and formatting for large language model (LLM) data engineering face high technical barriers and suffer from fragmented, non-integrated tooling. Method: This paper introduces the first low-code, block-based open-source ETL framework specifically designed for LLM data engineering. It features a modular, configuration-driven pipeline architecture with pluggable processor interfaces, enabling rapid integration of custom data processing logic; supports both CLI and programmatic API invocation to balance usability and flexibility. Contribution/Results: Experiments demonstrate efficient automated preprocessing of TB-scale corpora, significantly reducing LLM data preparation overhead. The framework is fully open-sourced, accompanied by tutorial videos and comprehensive documentation, thereby advancing standardization and community-driven development in LLM data engineering.

This work addresses the challenge of reconciling high throughput and low query latency in traditional ETL pipelines when processing continuously arriving fresh data, where unpredictable preprocessing operations often create bottlenecks. The authors propose Fluid ETL Pipelines, which introduce, for the first time, an elastic and non-blocking preprocessing mechanism that decouples data ingestion from transformation. By dynamically scheduling preprocessing tasks based on resource availability and user interest—without blocking data ingestion—and leveraging preemptible computing resources such as Amazon Spot instances, the approach significantly reduces operational costs. Experimental results demonstrate that Fluid ETL Pipelines substantially improve the efficiency of exploring fresh data, offering a novel direction for accelerating real-time queries and enabling adaptive preprocessing management.

Traditional ETL/ELT approaches struggle to simultaneously satisfy scalability, governance, and real-time processing requirements. While hybrid patterns such as ETLT and ELTL have emerged in practice, they lack formal definitions and systematic governance support. This paper formally establishes ETLT and ELTL as canonical data engineering design patterns and introduces their enhanced variants—ETLT++ and ELTL++—which integrate explicit data contracts, schema versioning, semantic metadata management, end-to-end lineage tracking, and continuous observability-driven monitoring. These mechanisms collectively ensure data quality, regulatory compliance, and trustworthiness. The proposed framework natively supports multi-cloud environments and unified stream-batch processing, significantly improving pipeline maintainability, auditability, and cost efficiency. By enabling standardized, verifiable design principles, it advances data architecture from empirically driven practices toward a rigorous, specification-based paradigm.

To address the joint optimization challenges of performance, maintainability, and collaborative efficiency in large-scale integrated machine learning within distributed data processing systems, this paper proposes Pipes—a declarative, modular data pipeline architecture. Pipes decomposes pipelines into logically encapsulated computation units, implemented atop Apache Spark with standardized interfaces and well-defined component boundaries—departing from conventional microservice paradigms to enable high-performance, maintainable ML pipeline development. In enterprise deployments, Pipes improves development efficiency by 50%, reduces collaborative debugging cycles from weeks to days, achieves 500× scalability, and delivers 10× higher throughput. Academic benchmarks show >5.7× throughput improvement and 99% CPU utilization. Its core contribution is the first deep integration of declarative abstractions with Spark’s native execution model, simultaneously advancing both development methodology and system performance.

This work addresses the unreliability of developer productivity dashboards, which often stems from ad hoc scripts that introduce undetected silent data gaps, eroding organizational trust. To resolve this, we propose a robust ELT pipeline grounded in DAG-based orchestration and the Medallion architecture, decoupling data extraction from transformation to preserve the immutability of raw data. Our approach introduces a state-driven dependency scheduling mechanism and, for the first time, treats metric pipelines as production-grade distributed systems. We emphasize the critical role of immutable raw history in enabling reliable metric redefinition. This methodology significantly enhances data reliability and freshness while effectively eliminating silent failures, thereby restoring organizational confidence in DevOps metrics.

The era of large language models (LLMs) faces critical challenges including insufficient high-quality data supply, fragmented data preparation pipelines, poor reproducibility, and lack of model-in-the-loop support. Method: We propose the first LLM-driven, unified data preparation framework for data-centric AI, featuring system-level abstractions and PyTorch-style APIs for modular design. We introduce DataFlow-Agent—the first agent that synthesizes executable data pipelines end-to-end from natural language specifications—and integrate LLM-powered operator synthesis, iterative validation, 200+ reusable operators, and six domain-agnostic pipeline templates. Results: Experiments on Text-to-SQL, code generation, and mathematical reasoning show our synthesized data significantly outperforms human-annotated and domain-specific synthetic data. Remarkably, just 10K samples surpass the performance of models trained on the million-scale Infinity-Instruct dataset, empirically validating the decisive impact of data quality on model performance.

To address low pipeline design efficiency, large semantic gaps, and high error rates in real-time data stream processing, this paper proposes the Hypergraph of Thoughts (HGoT) framework. HGoT leverages large language models (LLMs) to interpret high-level user intent and constructs a hypergraph-structured, multi-agent collaborative reasoning mechanism that enables end-to-end automated generation—from semantic understanding and logical modeling to cross-platform deployment and elastic optimization. Its core innovation lies in synergistically integrating LLM-based semantic comprehension with hypergraph-based symbolic reasoning to bridge the semantic gap between user intent and distributed system implementation. Additionally, HGoT introduces advanced query analysis and dynamic execution strategies to support automatic modeling and performance optimization of complex streaming logic. Experimental results demonstrate that, compared to conventional LLM-based code generation approaches, HGoT improves development efficiency by 6.3× and reduces error rates by 5.19×, significantly enhancing both generated code quality and system reliability.

This study addresses the critical data quality challenges in cloud-native ELT pipelines arising from heterogeneous data sources, schema evolution, and multi-backend environments. The authors propose a multi-layered testing framework that integrates orchestration-level validation, declarative dbt tests, LLM-generated semantic tests, and cross-storage consistency verification between DuckDB and Snowflake, all unified under Apache Airflow scheduling. Notably, this work pioneers the incorporation of LLM-driven semantic testing into production-grade ELT workflows. In anomaly injection experiments, the framework achieves a 128.57% improvement in detection rate, successfully identifying all 16 injected anomalies. Furthermore, three critical tables exhibit perfect consistency across both storage systems, with the end-to-end pipeline completing in just 106.58 seconds, substantially enhancing both detection coverage and engineering feasibility.

This study addresses the lack of systematic methodologies for selecting data architectures in modern organizations grappling with vast, heterogeneous data environments. To this end, it proposes the DATER conceptual framework, which establishes a unified taxonomy of technical requirements and systematically examines the historical evolution, core characteristics, and applicability boundaries of six prominent data architectures: data warehouses, data lakes, lakehouses, data fabrics, and data meshes. Through conceptual modeling and multidimensional comparative analysis, the framework clarifies overlaps and distinctions among these architectures, articulating their respective strengths and limitations. By offering a structured evaluation tool, DATER significantly enhances the strategic alignment and contextual appropriateness of data architecture design for both researchers and practitioners.