delta lake

To address the concurrent challenges of high throughput, low latency, and strong consistency in automotive data engineering, this paper systematically evaluates Delta Lake, Apache Iceberg, and Apache Hudi—the three leading lakehouse formats—on real-world vehicular time-series data. We assess their capabilities in schema modeling, change data capture (CDC), query performance, time travel, and ACID transaction compliance, integrating partitioning optimization, real-time ingestion, incremental processing, and cloud-native architecture, while coupling with end-to-end machine learning pipelines. Our key contribution is the first “use-case-driven” lakehouse format selection and hybrid composition framework tailored to automotive applications—including fleet management, predictive maintenance, and route optimization. Empirical results demonstrate that Delta Lake excels in data governance and ML-ready data preparation; Iceberg achieves superior batch query performance; and Hudi significantly outperforms the others in real-time upserts and sub-second analytical queries.

Traditional databases suffer from architectural bloat, strong external dependencies, and poor adaptability in rapidly evolving, nested-data-intensive, and research-oriented scenarios. To address these limitations, we propose ParquetDB—a lightweight, file-based, Python-native database built on PyArrow and Parquet’s columnar storage format. ParquetDB introduces a novel index-free predicate pushdown mechanism, supports schema-aware serialization, memory-mapped I/O, and vectorized query execution. Its design eliminates external dependencies while achieving high performance and cross-platform portability. Evaluated on the Alexandria 3D materials database (4.8 million deeply nested records), ParquetDB demonstrates significantly higher query throughput than SQLite and MongoDB, and achieves 3.2× faster serialization. This work establishes an efficient, minimalist, and reproducible paradigm for scientific data management.

Log-structured table formats (e.g., Delta Lake, Iceberg, Hudi) in data lakes suffer from excessive small files due to append-only writes and metadata-heavy operations, degrading query performance, increasing storage costs, and limiting system scalability. Existing compaction mechanisms lack flexibility, pursue narrow objectives, and fail to balance benefits against operational overhead. To address this, we propose Scalable Adaptive Compaction (SAC), an extensible, workload-aware, metadata-driven framework featuring dynamic threshold tuning, lightweight online evaluation, and a modular rule engine. SAC is production-deployed via the OpenHouse control plane. Evaluated on LinkedIn’s production workloads and synthetic benchmarks, SAC reduces file counts by up to 92%, improves typical query latency by 3.8×, and maintains bounded runtime overhead.

This work addresses the challenge of schema drift in data pipelines, which often manifests at runtime, causing errors and increasing maintenance overhead. The authors propose a lightweight framework built on Scala 3 that enforces structural contracts between producers and consumers at compile time. By leveraging the type system and compile-time metaprogramming, the framework automatically derives Spark DataFrame schemas from shared contracts and validates actual data structures prior to ingestion. It combines compile-time guarantees with policy-aware runtime comparators to support nested and optional fields as well as subset semantics, thereby ensuring both forward and backward compatibility. Empirical evaluation demonstrates the framework’s effectiveness in end-to-end workflows, with reproducible benchmarking conducted across two distinct environments.

This study addresses the challenge of selecting an optimal Data Lakehouse architecture based on data type and scale by presenting the first systematic evaluation of Apache Hudi, Apache Iceberg, and Delta Lake in terms of data ingestion efficiency and storage overhead for structured and semi-structured workloads. Conducted on the Apache Spark platform, the empirical comparison employs a four-stage ETL pipeline to assess the three frameworks under realistic conditions. Experimental results demonstrate that Delta Lake achieves the fastest data loading performance, while Iceberg excels in storage compression ratio and system stability. In contrast, Hudi exhibits comparatively lower efficiency in both batch ingestion and storage utilization. These findings provide critical empirical evidence and practical guidance for informed architectural decisions in Lakehouse deployments.

This work addresses the high commit latency incurred when directly executing transactions on low-latency object storage systems—such as S3 Express One Zone—due to their decentralized logging mechanisms. To overcome this limitation, the paper proposes an in-memory OLTP engine specifically optimized for millisecond-level object storage. The design introduces a novel commit protocol, request-level optimizations, and a streamlined object access strategy, collectively reducing commit latency while preserving strong consistency, high availability, and durability. Experimental evaluation demonstrates that the proposed system significantly lowers latency compared to baseline approaches on standard benchmarks, all while sustaining high throughput.

This work addresses the challenge of reconciling high throughput and low query latency in traditional ETL pipelines when processing continuously arriving fresh data, where unpredictable preprocessing operations often create bottlenecks. The authors propose Fluid ETL Pipelines, which introduce, for the first time, an elastic and non-blocking preprocessing mechanism that decouples data ingestion from transformation. By dynamically scheduling preprocessing tasks based on resource availability and user interest—without blocking data ingestion—and leveraging preemptible computing resources such as Amazon Spot instances, the approach significantly reduces operational costs. Experimental results demonstrate that Fluid ETL Pipelines substantially improve the efficiency of exploring fresh data, offering a novel direction for accelerating real-time queries and enabling adaptive preprocessing management.

This work addresses the susceptibility of Snowpark UDF execution to data skew induced by user-defined logic, which leads to task delays and inefficient resource utilization. To mitigate this issue, the authors propose a dynamic, fine-grained redistribution mechanism that integrates a state machine–based adaptive data distribution strategy, an eager redistribution policy, and a row-size prediction model to accurately detect and alleviate skew at runtime. Implemented within Snowflake’s general-purpose skew-handling framework, the approach leverages per-link state machines, dynamic row-level redistribution, and cost-aware scheduling to significantly reduce both execution time and resource consumption for large-scale UDF workloads. Experimental results demonstrate its superior performance over conventional static round-robin strategies.

This study addresses the lack of systematic methodologies for selecting data architectures in modern organizations grappling with vast, heterogeneous data environments. To this end, it proposes the DATER conceptual framework, which establishes a unified taxonomy of technical requirements and systematically examines the historical evolution, core characteristics, and applicability boundaries of six prominent data architectures: data warehouses, data lakes, lakehouses, data fabrics, and data meshes. Through conceptual modeling and multidimensional comparative analysis, the framework clarifies overlaps and distinctions among these architectures, articulating their respective strengths and limitations. By offering a structured evaluation tool, DATER significantly enhances the strategic alignment and contextual appropriateness of data architecture design for both researchers and practitioners.

Databases continuously evolve through operations such as schema changes, version updates, and data transformations; however, existing approaches typically address these functionalities in isolation, lacking a unified abstraction. This work proposes the first integrated model that unifies continuous schema evolution, version management, and data transformation within a single framework. Built upon general-purpose computational primitives, the model supports operation provenance, conditional update propagation, and change alerts, while employing a declarative mechanism to manage the co-evolution of dependent artifacts—including views and machine learning models. A prototype system implements this framework using an enhanced, parameterized Prolly Tree—a Merkle tree–inspired data structure—to construct a relational-like engine. Experimental evaluation demonstrates that the proposed approach is both feasible and offers tunable performance across diverse evolution scenarios.