🤖 AI Summary
This study addresses the high computational cost of full-dataset scans for data quality assessment—such as detecting missing values, duplicates, and outliers—which hinders near real-time monitoring. The authors systematically evaluate nine progressive sampling strategies across diverse real-world and synthetic datasets, comparing blind sampling methods (e.g., uniform random, clustering-based, Yamane) against proxy-guided approaches (e.g., MCMC, DAG-based, stratified weighting). Contrary to the prevailing assumption that incorporating prior knowledge improves accuracy, large-scale empirical results reveal that representative blind sampling consistently outperforms proxy-guided techniques, primarily due to mismatches between proxy metrics and actual data quality issues. Notably, under a 5% sampling budget, uniform random sampling achieves an average relative error of merely 0.49% with near-linear scalability, whereas DAG-guided methods incur 11–49× higher errors and run 28–47× slower, demonstrating that simple sampling strategies are better suited for production-grade data quality monitoring.
📝 Abstract
Data quality profiling -- computing missing-value rates, duplicate fractions, outlier densities, and functional-dependency violations -- is foundational for data-centric AI pipelines, yet exhaustive scans over millions of rows are prohibitively slow for near-real-time monitoring. Progressive sampling is the standard alternative; the open question is which strategy best preserves profile fidelity at scale. We benchmark nine sampling strategies -- blind (random uniform, geometric, Yamane, cluster) and proxy-guided (Metropolis-Hastings, DAG, stratified by column type or quality score, importance-weighted) -- on three real-world datasets (NYC 311, NYPD arrests, UCI Adult; up to 500K rows), an IoT sensor stream (2.3M rows), two ultra-large real datasets including Ultra-Marathon Running (up to 7.4M rows), and synthetic data scaled to 5x10^6 rows. Contrary to the assumption sharpens estimates, blind representative samplers dominate uniformly. At a 5% budget, random uniform achieves 0.49% mean relative error on NYC 311; DAG-guided MCMC yields 19.5% (approx. 40x worse), and across all real datasets DAG is 11-49x worse (Wilcoxon W=0, p=0.002, n=9 pairs). Cluster sampling matches random uniform (MRE 0.110 vs. 0.111); proxy-guided methods share DAG's failure mode (MRE 0.20-0.35). At scale, random uniform is near-linear (O(N^{0.964})) while DAG is super-linear (O(N^{1.272})), running 28--47x slower on ultra-large data with 6x worse accuracy. The root cause is an IQR proxy mismatch: proxy-guided samplers over-pursue numeric outliers, while quality defects concentrate in categorical columns invisible to the proxy. The actionable finding: representativeness, not domain knowledge, determines sampler quality -- schema-free random uniform or cluster sampling suffices for production-grade quality profiling at scale.