clickhouse

This work proposes a fully automated, end-to-end approach to synthesizing specialized OLAP engines tailored to specific analytical workloads. Unlike general-purpose OLAP systems, which incur structural overhead that impedes peak performance, the proposed method leverages large language model–driven code generation, iterative performance feedback, automated correctness verification, and co-optimization of storage and query execution architectures to generate complete, custom engines without human intervention. For the first time, this framework enables the automatic synthesis of high-performance OLAP engines within minutes to hours, achieving order-of-magnitude speedups over state-of-the-art general-purpose systems such as DuckDB. By eliminating the traditional trade-off between engineering effort and performance inherent in manual specialization, the approach breaks a key bottleneck in analytical data system design.

This work addresses key challenges in multimodal data analytics, including inefficient I/O, rigid query optimization, and performance degradation caused by resource decoupling. To overcome these limitations, the authors propose a cloud-native OLAP engine that integrates a unified table engine—combining two-layer logical abstraction with physically consistent layout—alongside CrossCache, a cluster-wide shared SSD cache, and NexusFS, a virtual file system enabling efficient local data access. Furthermore, the system incorporates a query optimizer that leverages historical execution traces and AI-driven insights to enhance plan selection. Experimental results demonstrate that the proposed system significantly outperforms existing solutions in multimodal query efficiency, resource utilization, and end-to-end latency, enabling highly effective coordination among analytical, batch, and incremental workloads.

Open table formats (OTFs) such as Apache Iceberg and Delta Lake—widely adopted in lakehouse architectures—lack native support for cross-table ACID transactions and crash recovery, leading to non-repeatable reads, stale data, and high operational overhead. This paper presents the first non-intrusive solution that enables multi-table transactions, strong consistency isolation—including for long-running transactions—and fault tolerance on OTFs without modifying their storage formats or computation engines. Our approach introduces log-coordinated lightweight two-phase commit, snapshot version-chain management, atomic metadata writes, and incremental rollback. Evaluation shows minimal performance overhead: only 2.0% on YCSB write workloads and 2.5% on TPC-DS read workloads. The system supports high-concurrency mixed workloads while eliminating dirty reads and non-repeatable reads entirely, thereby ensuring production-grade consistency for cloud-scale analytical data.

This work addresses the scarcity of real-world SQL workloads and limitations of existing synthetic query generation methods—such as rigid templates, schema hallucination, and simplistic join structures—that hinder effective training of query optimizers. The authors propose SynQL, a rule-based, deterministic SQL synthesis framework that constructs abstract syntax trees by traversing foreign key graphs in database schemas, eschewing probabilistic generation to guarantee syntactic and schema-level validity. SynQL introduces a configuration vector to explicitly control join topology, analytical complexity, and predicate selectivity, while modeling core query components including multi-table joins, projections, aggregations, and range predicates. Workloads generated on TPC-H and IMDb achieve a topological entropy of 1.53 bits; tree-based models trained on this data attain R² ≥ 0.79 on test sets with sub-millisecond inference latency.

Existing metadata catalogs (e.g., Hive Metastore, Iceberg, Delta Lake) face fundamental limitations in concurrent read/write throughput, strong consistency guarantees, and expressive query capabilities for hyperscale data systems. TreeCat addresses these challenges by introducing the first high-performance, purpose-built metadata catalog engine. It employs a hierarchical data model and a path-based query language; implements MVOCC—a multi-version optimistic concurrency control protocol ensuring serializable isolation; and pioneers an associative scan execution mechanism to accelerate metadata traversal and filtering. Its custom storage format is co-designed for efficient range queries and fine-grained version management. Experimental evaluation demonstrates that TreeCat delivers strict consistency under high concurrency while achieving up to 8.2× higher throughput for range queries compared to state-of-the-art systems, significantly outperforming existing solutions across key metadata management workloads.

Existing SQL systems struggle to efficiently query unstructured text data—such as agent trajectories and chat logs—and conventional data lake solutions cannot be lightweightly integrated into client-side JavaScript environments. This work proposes Squirreling, the first embedded query engine designed for AI-native client applications, which natively supports direct reads of Parquet and Apache Iceberg formats and interleaves asynchronous LLM-based user-defined functions with analytical SQL execution. Built atop three open-source JavaScript libraries totaling less than 70 KB, Squirreling achieves up to 300× speedup over DuckDB-WASM on filter-limited queries and 192× on sort-limited queries, while completing a ten-task agent analytics benchmark suite at two-thirds the computational cost, substantially reducing both latency and resource overhead.

This work addresses the challenges of implementing analytical logic in production-grade systems using imperative code, which often leads to code duplication, definition drift, and poor testability. To overcome these issues, the authors propose modeling analytical workflows as typed, declarative directed acyclic graphs (QDAGs), enabling, for the first time, a unified representation of complex analytical pipelines in large-scale production environments. The system efficiently orchestrates multi-step operations through on-demand execution, memoization, pruning, and parallel scheduling, supporting diverse primitives such as Apache Pinot queries, service invocations, SQLite joins, jq transformations, conditional logic, differentially private aggregations, and nested QDAG calls. Deployed across over 100 production use cases at LinkedIn on more than 500 hosts, the system achieves median orchestration overhead of approximately 10 milliseconds and 99th-percentile latency under 50 milliseconds, significantly enhancing reusability, consistency, and testability while meeting interactive latency requirements.

This study addresses the lack of systematic methodologies in modern enterprises for selecting databases across multiple paradigms, a process often driven by ad hoc experience. The authors propose a cross-paradigm database architecture evaluation and selection framework grounded in nine dimensions, employing a three-stage workflow—workload characterization, constraint filtering, and compatibility scoring—to enable scientifically informed decisions for AI-ready data platforms. The work identifies three emergent trends in database evolution: compute-storage disaggregation, workload-driven specialization, and convergence toward AI-ready platforms, and introduces the first unified reference architecture tailored for AI applications. Validated through a financial fraud detection case study, the approach demonstrates the advantages of a hybrid polyglot architecture, integrating lakehouse storage, feature engineering, and semantic retrieval layers to effectively support modern analytics, machine learning, and retrieval-augmented generation (RAG) applications.

This work addresses the resource waste and performance bottlenecks in concurrent analytical queries caused by redundant computation. It proposes a dynamic query folding mechanism that integrates newly arriving queries into ongoing executions through a state-centric execution model and query grafting technique. By safely sharing intermediate operator states among co-running queries and employing state lenses to provide each query with a dedicated view of the shared state, the approach ensures semantic correctness while enabling state sharing, dynamic fusion, and cooperative multi-query execution. Evaluated on TPC-H–derived dynamic concurrent workloads, the method achieves up to a 2.17× throughput improvement over the baseline and reduces the 95th-percentile response time to 0.17× of the baseline under overload conditions.

This work addresses the challenge of automatically selecting high-impact secondary indexes in SQL databases to efficiently utilize CPU, I/O, and storage resources under dynamic workloads. The authors propose AIM (Automatic Index Manager), an end-to-end, configurable index management system that features a workload-driven, wide composite index recommendation algorithm for rapid convergence. AIM incorporates a resource-aware evaluation mechanism that reduces reliance on the query optimizer and provides a strict “no performance regression” guarantee alongside metric-driven, interpretable recommendations. Extensive experiments across thousands of production databases demonstrate that AIM quickly generates near-optimal physical designs, substantially improving resource efficiency while ensuring no degradation in online query performance.