amazon redshift

Existing metadata catalogs (e.g., Hive Metastore, Iceberg, Delta Lake) face fundamental limitations in concurrent read/write throughput, strong consistency guarantees, and expressive query capabilities for hyperscale data systems. TreeCat addresses these challenges by introducing the first high-performance, purpose-built metadata catalog engine. It employs a hierarchical data model and a path-based query language; implements MVOCC—a multi-version optimistic concurrency control protocol ensuring serializable isolation; and pioneers an associative scan execution mechanism to accelerate metadata traversal and filtering. Its custom storage format is co-designed for efficient range queries and fine-grained version management. Experimental evaluation demonstrates that TreeCat delivers strict consistency under high concurrency while achieving up to 8.2× higher throughput for range queries compared to state-of-the-art systems, significantly outperforming existing solutions across key metadata management workloads.

Existing relational databases lack the persistent isolation and concurrent branching capabilities required by agent workloads, hindering support for speculative modifications and nonlinear state exploration. This work presents the first systematic definition and quantitative evaluation of database branching for intelligent agents, introducing BranchBench—a benchmark combining parameterized macrobenchmarks (modeling branch-mutate-evaluate cycles) and microbenchmarks (measuring branch lifecycle overhead). We evaluate systems including Neon, DoltgreSQL, TigerBeetle, Xata, and PostgreSQL, revealing that fast-branching systems suffer 5–4000× read performance degradation as branch depth increases, while fast data operating systems exhibit 25–1500× higher latency in branch creation and switching. Our experiments demonstrate that none of the current systems can effectively scale to support representative agent workloads such as software engineering tasks, fault reproduction, or Monte Carlo Tree Search (MCTS).

Existing TPC-style benchmarks (e.g., TPC-H/DS) fail to capture two critical characteristics of real-world cloud data warehouse workloads—query repetitiveness and string-intensive operations—leading to inaccurate system evaluation. To address this, we propose Redbench, a novel synthetic benchmark generation framework grounded in empirical cloud workload trace analysis. Rather than relying on superficial execution metrics, Redbench models intrinsic workload signals—including query pattern distributions, repetition cycles, and string-operation intensity—via integrated techniques for query pattern extraction, repetitiveness modeling, and string-aware enhancement. It enables end-to-end, reproducible, and customizable synthesis of realistic workloads from production traces. Experimental evaluation demonstrates that Redbench-generated workloads significantly improve fidelity and effectively expose performance disparities across four major commercial cloud data warehouses under diverse optimization strategies, thereby providing a high-fidelity, reproducible foundation for rigorous system assessment and optimization.

This work addresses the challenge of materialized view selection under increasingly complex database workloads by explicitly modeling incremental maintenance cost within the optimization objective and integrating it into an integer linear programming framework. The approach innovatively leverages subquery containment analysis to devise strategies for generating high-quality initial solutions and constructing effective neighborhoods, guided by metrics such as subquery frequency, utility, or utility per unit of storage. These strategies efficiently steer local search toward superior solutions. Experimental evaluation on the Redbench benchmark demonstrates that the proposed method significantly outperforms BIGSUBS in both optimization utility and the quality of selected views.

This work proposes OceanBase Mercury, a distributed near real-time analytical processing system built upon OceanBase, designed to deliver enterprise-grade analytical capabilities—including multi-tenancy, high availability, and elastic scalability—for petabyte-scale data. Traditional OLAP systems struggle to simultaneously support real-time transactions and efficient analytics at scale, often suffering from high data redundancy, complex cross-system synchronization, and poor timeliness. Mercury addresses these challenges through three key innovations: an adaptive columnar storage format with hybrid layout optimization, a materialized view differential refresh mechanism that ensures temporal consistency, and a polymorphic vectorized execution engine compatible with three distinct data formats. Experimental results on real-world workloads demonstrate that Mercury achieves 1.3–3.1× faster query latency than specialized OLAP engines while maintaining sub-second response times, effectively balancing analytical depth with operational agility.

Multi-cloud ETL faces significant challenges, including high cross-cloud data movement costs, incompatibility among heterogeneous SQL engines, complex orchestration, and fragmented security policies. This work presents the first systematic evaluation of predicate pushdown optimization in multi-cloud ETL scenarios and proposes a novel协同 strategy that integrates localized pushdown with data federation to offload transformation logic across heterogeneous query engines such as Amazon Redshift and Google BigQuery. By intelligently pushing computation closer to the data sources while leveraging federated querying capabilities, the approach substantially reduces cross-cloud data transfer, thereby lowering end-to-end execution latency and cost. The proposed paradigm offers an efficient and scalable solution for optimizing multi-cloud data integration workflows.

To address replication latency, race conditions, and high network overhead induced by pointer-based designs in cross-region, high-throughput NoSQL database storage, this paper proposes the “Chunked Object” pattern. It decomposes oversized objects into ordered data chunks and stores them directly within a multi-region replicated key-value store (e.g., DynamoDB Global Tables), collocating data and metadata within a single consistency domain. This eliminates asynchronous replication between the database and external object storage, thereby significantly reducing cross-region p99 consistency attainment time. Evaluated in production under heavy load (>200K TPH, 1 MB/transaction), the approach demonstrates robust effectiveness and generalizability. It is applicable to any key-value store supporting per-item size limits and cross-region strong synchronization, offering a practical, system-agnostic solution for scalable, low-latency global data management.

Databases continuously evolve through operations such as schema changes, version updates, and data transformations; however, existing approaches typically address these functionalities in isolation, lacking a unified abstraction. This work proposes the first integrated model that unifies continuous schema evolution, version management, and data transformation within a single framework. Built upon general-purpose computational primitives, the model supports operation provenance, conditional update propagation, and change alerts, while employing a declarative mechanism to manage the co-evolution of dependent artifacts—including views and machine learning models. A prototype system implements this framework using an enhanced, parameterized Prolly Tree—a Merkle tree–inspired data structure—to construct a relational-like engine. Experimental evaluation demonstrates that the proposed approach is both feasible and offers tunable performance across diverse evolution scenarios.

This paper addresses version control challenges for multidimensional structured data—namely, temporal evolution, spatial collaboration, and design iteration—by introducing Operational Differencing, a novel paradigm. Methodologically, it incorporates high-level semantic operations (e.g., schema changes, refactorings) into the version model; adopts an append-only branch history with a repository-free, lightweight “copy-as-branch” architecture; and enables operational query representation and future-tense execution. Contributions include: (1) the first systematic support for automatic schema-adaptive query rewriting under schema evolution; (2) precise, fine-grained diff/merge across structural transformations; (3) resolution of four out of eight canonical schema evolution challenges; and (4) a simplified versioning experience with no explicit repository and asymptotically zero branching overhead.

This work addresses the tendency of existing large language model (LLM)-driven Text-to-SQL agents to over-explore in fine-grained API environments, often incorporating irrelevant schema elements and thereby degrading SQL generation accuracy. To mitigate this issue, the paper introduces a novel mechanism that embeds guiding instructions directly within data system API responses to actively regulate the agent’s exploration behavior. This approach enhances exploration efficiency while preserving query precision. Notably, it represents the first method to leverage API response instructions for constraining excessive exploration. Experimental results demonstrate that the proposed guiding instructions reduce over-exploration by a factor of 4.6 and improve SQL generation accuracy by up to 12.4% (approximately 4 percentage points).