data lake table formats

Open table formats (OTFs) such as Apache Iceberg and Delta Lake—widely adopted in lakehouse architectures—lack native support for cross-table ACID transactions and crash recovery, leading to non-repeatable reads, stale data, and high operational overhead. This paper presents the first non-intrusive solution that enables multi-table transactions, strong consistency isolation—including for long-running transactions—and fault tolerance on OTFs without modifying their storage formats or computation engines. Our approach introduces log-coordinated lightweight two-phase commit, snapshot version-chain management, atomic metadata writes, and incremental rollback. Evaluation shows minimal performance overhead: only 2.0% on YCSB write workloads and 2.5% on TPC-DS read workloads. The system supports high-concurrency mixed workloads while eliminating dirty reads and non-repeatable reads entirely, thereby ensuring production-grade consistency for cloud-scale analytical data.

This study addresses the challenge of selecting an optimal Data Lakehouse architecture based on data type and scale by presenting the first systematic evaluation of Apache Hudi, Apache Iceberg, and Delta Lake in terms of data ingestion efficiency and storage overhead for structured and semi-structured workloads. Conducted on the Apache Spark platform, the empirical comparison employs a four-stage ETL pipeline to assess the three frameworks under realistic conditions. Experimental results demonstrate that Delta Lake achieves the fastest data loading performance, while Iceberg excels in storage compression ratio and system stability. In contrast, Hudi exhibits comparatively lower efficiency in both batch ingestion and storage utilization. These findings provide critical empirical evidence and practical guidance for informed architectural decisions in Lakehouse deployments.

This work addresses the high computational cost and poor scalability of traditional data quality monitoring, which relies on full-table scans. The authors propose a zero-scan approach that leverages table-format metadata from Apache Iceberg, systematically utilizing column-level statistics—such as Theta sketches for distinct count estimation and KLL sketches for quantile approximation—generated at write time. Incremental, mergeable statistics are stored in Puffin sidecar files, enabling anomaly detection, distribution drift monitoring, and null-ratio tracking without accessing the underlying data. Deployed at LinkedIn across over 200,000 Iceberg tables (800+ PB), the method supports approximately 60% of user-defined rules at zero marginal compute cost, reduces resource consumption by 50%, and achieves near 90% coverage after extension.

Existing metadata catalogs (e.g., Hive Metastore, Iceberg, Delta Lake) face fundamental limitations in concurrent read/write throughput, strong consistency guarantees, and expressive query capabilities for hyperscale data systems. TreeCat addresses these challenges by introducing the first high-performance, purpose-built metadata catalog engine. It employs a hierarchical data model and a path-based query language; implements MVOCC—a multi-version optimistic concurrency control protocol ensuring serializable isolation; and pioneers an associative scan execution mechanism to accelerate metadata traversal and filtering. Its custom storage format is co-designed for efficient range queries and fine-grained version management. Experimental evaluation demonstrates that TreeCat delivers strict consistency under high concurrency while achieving up to 8.2× higher throughput for range queries compared to state-of-the-art systems, significantly outperforming existing solutions across key metadata management workloads.

Log-structured table formats (e.g., Delta Lake, Iceberg, Hudi) in data lakes suffer from excessive small files due to append-only writes and metadata-heavy operations, degrading query performance, increasing storage costs, and limiting system scalability. Existing compaction mechanisms lack flexibility, pursue narrow objectives, and fail to balance benefits against operational overhead. To address this, we propose Scalable Adaptive Compaction (SAC), an extensible, workload-aware, metadata-driven framework featuring dynamic threshold tuning, lightweight online evaluation, and a modular rule engine. SAC is production-deployed via the OpenHouse control plane. Evaluated on LinkedIn’s production workloads and synthetic benchmarks, SAC reduces file counts by up to 92%, improves typical query latency by 3.8×, and maintains bounded runtime overhead.

This work addresses the challenges of high startup latency and low query efficiency in graph analytics under lakehouse architectures. The authors propose a lakehouse-native graph processing engine that maps lakehouse tables to vertex and edge types in a property graph and enables efficient querying through GSQL. Key innovations include loading only graph topology to accelerate system initialization, designing a graph-aware caching mechanism, and developing two lakehouse-optimized parallel primitives for graph computation. Experimental evaluation demonstrates that the proposed system significantly outperforms PuppyGraph—the current state-of-the-art—in both startup time and query latency across a range of workloads.

This work addresses the high cost of maintaining consistency for materialized views in high-throughput and real-time scenarios, where existing systems offer limited SQL operator support or rely on manual tuning of refresh strategies. The paper proposes an end-to-end incremental materialized view maintenance engine that treats materialized views as first-class constructs within declarative data pipelines. Built on Apache Spark, the system features a modular and generalizable incremental refresh architecture capable of operating across heterogeneous data sources and query engines. It integrates a cost model, an automated refresh planner, and an execution optimizer to dynamically select optimal refresh strategies and enable cross-view batched co-optimization. Experimental evaluation demonstrates significant performance improvements, saving billions of CPU seconds daily across thousands of production pipelines.

Traditional ETL approaches rely on manually predefined schemas, making them inflexible for dynamically analyzing multi-source, heterogeneous tables in data lakes in response to natural language queries. This work proposes the first table integration framework tailored for natural language querying, which combines a lightweight graph model with Steiner tree search. Building upon a two-stage schema linking process, the framework leverages large language models to automatically generate structured views that support corresponding SQL queries. The proposed method reduces large model invocation costs to one-fifth of the original while achieving over a 30% improvement in query accuracy across four real-world datasets, significantly outperforming existing techniques.

Existing SQL systems struggle to efficiently query unstructured text data—such as agent trajectories and chat logs—and conventional data lake solutions cannot be lightweightly integrated into client-side JavaScript environments. This work proposes Squirreling, the first embedded query engine designed for AI-native client applications, which natively supports direct reads of Parquet and Apache Iceberg formats and interleaves asynchronous LLM-based user-defined functions with analytical SQL execution. Built atop three open-source JavaScript libraries totaling less than 70 KB, Squirreling achieves up to 300× speedup over DuckDB-WASM on filter-limited queries and 192× on sort-limited queries, while completing a ten-task agent analytics benchmark suite at two-thirds the computational cost, substantially reducing both latency and resource overhead.

This work addresses the operational complexity and architectural coupling introduced by conventional vector similarity search systems in compute-storage disaggregated environments, which typically rely on a separate indexing layer. To resolve this, the authors propose deeply integrating a distributed approximate nearest neighbor (ANN) index into the Apache Iceberg table format. The approach leverages Puffin sidecar files to store sharded Vamana graphs and employs Iceberg’s snapshot mechanism to atomically bind indexes with data, enabling versioning and time travel. A coordinator-executor protocol is introduced, featuring a hierarchical probing strategy where coordinators cache compact centroid indexes while executors store large graphs on SSDs. The design reuses Iceberg’s REST catalog optimistic concurrency control for index commits. This solution is the first to host billion-scale vector graph indexes within Puffin, achieving a favorable trade-off between recall and latency while preserving Iceberg’s native transactional capabilities and significantly reducing system complexity.