Index-Assisted Stratified Sampling for Online Aggregation

2026-04-30

AI Summary

This work addresses the challenge of high-variance-induced sampling costs in index-assisted approximate query processing for ad-hoc aggregation over frequently updated flat data, where existing methods struggle to balance accuracy and low latency. To overcome this limitation, we propose a two-stage online stratified sampling framework that, for the first time, integrates stratified sampling into index-assisted online aggregation. In the first stage, samples are used simultaneously for real-time estimation and to refine the subsequent sampling strategy; the second stage performs efficient sampling based on an optimized stratification structure and Neyman allocation. We develop greedy and dynamic programming algorithms tailored to an index-aware sampling cost model to balance efficiency and accuracy. Experimental results demonstrate that our approach achieves up to 3× speedup over index-assisted uniform sampling and up to 98,708× speedup compared to traditional scan-based stratified sampling.

Abstract

Ad-hoc queries over frequently updated data in a flat schema are common in real-time data analysis applications and often require very low latency. Online aggregation can achieve so by providing approximate aggregation answers with confidence bound guarantees. It relies on the ability to draw samples online in a linear time to sample size rather than database size, which can be supported by index-assisted Sampling-based Approximate Query Processing (S-AQP) systems. However, the query latencies of approximate queries in these systems can still suffer from excessive sampling cost required to achieve a desired confidence bound, due to increased sample size for data with high variance in value distribution and selectivity. Classic stratified sampling methods with Neyman allocation can minimize sample size in theory, but several challenges prevent it from being applicable in index-assisted S-AQP systems, including requiring apriori statistics, high optimization cost, and inaccurate sampling cost model based on sample size. Towards that, we design index-assisted stratified sampling for online aggregation, which features a two-phase sampling framework. Samples drawn from first phase are used for both online aggregation and optimizing future sampling cost, while the second phase continues the online aggregation using the optimized strata. We prove optimal stratification and sample size allocation strategies for index-based sampling cost model, and design several greedy and dynamic programming based optimization methods to balance optimization cost and effectiveness in cost reduction. We evaluate our methods on several real-world and synthetic datasets and queries, and the results show ours consistently achieve good speedup and, in extreme cases, up to 3x speedup and 98708x speedup, when compared to index-assisted uniform sampling and classic scan-based stratified sampling respectively.

Problem

• online aggregation
• stratified sampling
• index-assisted sampling
• approximate query processing
• sampling cost

Innovation

• stratified sampling
• online aggregation
• index-assisted sampling
• approximate query processing
• Neyman allocation