Stage-Level Executor Allocation in Apache Spark with Cost-Performance Trade-offs

2026-07-13

AI Summary

This work addresses the performance bottlenecks and resource inefficiencies caused by static global resource configuration in distributed processing systems by proposing a fine-grained, dynamic resource allocation method tailored to Apache Spark’s execution stages. The approach introduces, for the first time, a stage-level executor count recommendation mechanism guided by user preferences, overcoming the limitations of traditional application-level static configurations. It employs a tree ensemble model to predict stage-wise execution time and cost under varying resource allocations and integrates an optimization strategy to enable personalized trade-offs between performance and cost. Evaluated on TPC-DS and SQLStorm benchmarks, the method reduces average costs by 50% and 40.5% respectively compared to baseline approaches, with only 16% and 29% additional performance overhead, thereby significantly enhancing resource utilization efficiency and scheduling flexibility.

Abstract

Allocating executors (i.e. compute resources) to distributed processing systems must balance resource costs of scaling-out unnecessarily against artificial, performance-limiting bottlenecks. Naive approaches may allocate executors at the application level, which have predictable costs and performance but are almost guaranteed to be sub-optimal for each of the thousands of diverse, individual stages executed by the application. Users may also have explicit preferences, such as completing an application within a specific time budget while minimizing cost, that existing solutions usually fail to support. We propose a novel method for determining the number of executors per stage in a serverless Apache Spark environment, enabling users to specify their desired cost-performance tradeoff. Our approach trains tree-ensemble models to estimate the run times and costs of a stage as a function of allocated resources. These estimates are then used to recommend resources for each stage individually. We evaluate our approach on TPC-DS and SQLStorm benchmarks and compare it against two baselines. Depending on the user-defined trade-off parameter and setup, our approach achieves approx. 50% cost savings across 103 TPC-DS queries with only a approx. 16% slowdown, and approx. 40.5% on 96 SQLStorm queries at a approx. 29% slowdown.

Problem

• executor allocation
• cost-performance trade-off
• stage-level scheduling
• Apache Spark
• resource optimization

Innovation

• stage-level allocation
• cost-performance trade-off
• executor optimization
• tree-ensemble model
• serverless Spark