Problem
Research questions and friction points this paper is trying to address.
resource allocation
ad-hoc queries
over-provisioning
cost-performance trade-off
cloud-native data warehouse
ScaleSense: Cost-Intelligent Scaling Framework via Learned Resource Estimation in Alibaba AnalyticDB
🤖 AI Summary
This work addresses the performance-cost imbalance in cloud-native serverless data warehouses, where heterogeneous ad-hoc queries simultaneously cause resource over-provisioning and non-CPU bottlenecks such as I/O saturation. To tackle this challenge, we propose ScaleSense, a novel framework that introduces a multidimensional resource prediction mechanism integrating query plan topology with hardware specifications. ScaleSense employs a multifaceted query encoder and a quantile regression predictor to accurately model resource consumption, coupled with a lightweight autoscaling controller and a Pareto-front optimization strategy to dynamically adjust resource allocation according to business priorities. Evaluation on over 1.36 million production queries demonstrates that ScaleSense reduces costs by up to 5.22× under performance constraints, improves resource provisioning accuracy by 76.7%, and incurs minimal inference overhead while seamlessly adapting to diverse business objectives without retraining.
📝 Abstract
Cloud-native serverless data warehouses achieve fine-grained elasticity by decoupling storage from compute, yet determining the optimal resource allocation for highly heterogeneous ad-hoc queries remains a formidable industrial challenge. Our analysis of production workloads in Alibaba AnalyticDB exposes a costly ``provisioning trap'': the fear of catastrophic resource depletion drives users to blindly over-provision resources, wasting immense monetary budgets without alleviating non-CPU bottlenecks (e.g., I/O saturation). To break this impasse, we propose ScaleSense, a proactive, query-level resource scaling framework. Specifically, it features a multi-faceted query encoder that jointly models plan topologies and hardware specifications. Crucially, a quantile-based resource predictor estimates multi-dimensional physical footprints, acting as a reliable safety net for optimal resource scaling. An auto-scaling controller then navigates the performance-cost Pareto frontier, dynamically tailoring allocations to specific business priorities without requiring model retraining. Evaluations on over 1.36 million production queries show that ScaleSense achieves state-of-the-art prediction accuracy with good prediction interval coverage. By achieving a 76.7% relative improvement in optimal resource configuration selection over the best baseline, this approach addresses the critical performance-cost trade-off while maintaining low-overhead inference latency, confirming its practical performance in production deployments. Under the performance-optimization policy, ScaleSense satisfies user-defined performance requirements while reducing monetary cost by up to 5.22x.
Innovation
Methods, ideas, or system contributions that make the work stand out.
resource estimation
serverless data warehouse
quantile-based prediction
cost-performance trade-off
auto-scaling
🔎 Similar Papers
No similar papers found.
