🤖 AI Summary
This work addresses the challenge of Spark SQL performance tuning, which is hindered by an enormous configuration space and the prohibitive cost of full-fidelity evaluation, making it difficult to identify high-quality configurations within practical time limits. To overcome this, the authors propose MFTune, a novel approach that introduces a query-level multi-fidelity partitioning mechanism to construct a representative, low-cost performance surrogate using a carefully selected subset of SQL queries with high correlation to overall workload behavior. MFTune further integrates density-based clustering for parameter space compression, tailored transfer learning, and a two-stage warm-start strategy to dramatically enhance tuning efficiency. Experimental results on TPC-H and TPC-DS benchmarks demonstrate that MFTune consistently outperforms five state-of-the-art methods under realistic time constraints, rapidly converging to superior configurations.
📝 Abstract
Apache Spark SQL is a cornerstone of modern big data analytics.However,optimizing Spark SQL performance is challenging due to its vast configuration space and the prohibitive cost of evaluating massive workloads. Existing tuning methods predominantly rely on full-fidelity evaluations, which are extremely time-consuming,often leading to suboptimal performance within practical budgets.While multi-fidelity optimization offers a potential solution, directly applying standard techniques-such as data volume reduction or early stopping-proves ineffective for Spark SQL as they fail to preserve performance correlations or represent true system bottlenecks. To address these challenges, we propose MFTune, an efficient multi-fidelity framework that introduces a query-based fidelity partitioning strategy, utilizing representative SQL subsets to provide accurate, low-cost proxies. To navigate the huge search space, MFTune incorporates a density-based optimization mechanism for automated knob and range compression, alongside an adapted transfer learning approach and a two-phase warm start to further accelerate the tuning process. Experimental results on TPC-H and TPC-DS benchmarks demonstrate that MFTune significantly outperforms five state-of-the-art tuning methods, identifying superior configurations within practical time constraints.