Intra-Query Runtime Elasticity for Cloud-Native Data Analysis

2025-02-25

AI Summary

To address the resource waste and unpredictable query latency caused by static degree of parallelism (DOP) configuration during query execution in cloud-native OLAP systems, this paper proposes Intra-Query Runtime Elasticity (IQRE). IQRE enables dynamic, zero-interruption DOP adjustment at any execution point without halting data processing. It introduces the first delay-constraint-driven “what-if” automated tuning service integrated with a user-friendly interactive interface. Built upon the Presto execution model, IQRE features a lightweight runtime scheduler, an elastic task redistribution protocol, and an online resource-aware optimization framework. Experimental evaluation under realistic workloads demonstrates that IQRE reduces average computational resource consumption by 37% while guaranteeing 100% compliance with end-to-end query latency constraints.

Technology Category

• Data Mining & Knowledge Management: Intelligent Query Processing
• Search and Optimization: Distributed Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications

Abstract

We propose the concept of Intra-Query Runtime Elasticity (IQRE) for cloud-native data analysis. IQRE enables a cloud-native OLAP engine to dynamically adjust a query's Degree of Parallelism (DOP) during execution. This capability allows users to utilize cloud computing resources more cost-effectively. We present Accordion, the first IQRE query engine. Accordion can adjust the parallelism of a query at any point during query execution without pausing data processing. It features a user-friendly interface and an auto-tuner backed by a"what-if"service to allow users to adjust the DOP according to their query latency constraints. The design of Accordion follows the execution model in Presto, an open-source distributed SQL query engine developed at Meta. We present the implementation of Accordion and demonstrate its ease of use, showcasing how it enables users to minimize compute resource consumption while meeting their query time constraints.

Problem

• Dynamic adjustment of query parallelism
• Cost-effective cloud resource utilization
• Minimizing compute resource consumption

Innovation

• Dynamic Degree of Parallelism
• Real-time Query Adjustment
• Cloud Resource Optimization