Adaptive Data Analysis for Growing Data

2024-05-22

arXiv.org

AI Summary

Existing differential privacy methods for adaptive data analysis assume static datasets, rendering them ill-suited for dynamically growing data—where maintaining both generalization and statistical validity remains challenging. Method: We establish, for the first time, a generalization bound for adaptive analysis under dynamic data growth; introduce a time-varying empirical accuracy bound theorem; and integrate a clipped Gaussian mechanism with a batched query framework to achieve progressively tightening error guarantees as data accumulates. Contribution/Results: Theoretically, our approach requires sample complexity scaling only with the square root of the number of queries—strictly improving upon static partitioning baselines. Empirically, it significantly enhances estimation accuracy and practical utility across statistical query tasks. Our framework provides a novel paradigm for streaming or incremental adaptive analysis, uniquely balancing rigorous theoretical guarantees with real-world feasibility.

Technology Category

• Machine Learning: Privacy
• Data Mining & Knowledge Management: Intelligent Query Processing
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Security and Privacy: Data transparency and provenance
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Reuse of data in adaptive workflows poses challenges regarding overfitting and the statistical validity of results. Previous work has demonstrated that interacting with data via differentially private algorithms can mitigate overfitting, achieving worst-case generalization guarantees with asymptotically optimal data requirements. However, such past work assumes data is static and cannot accommodate situations where data grows over time. In this paper we address this gap, presenting the first generalization bounds for adaptive analysis on dynamic data. We allow the analyst to adaptively schedule their queries conditioned on the current size of the data, in addition to previous queries and responses. We also incorporate time-varying empirical accuracy bounds and mechanisms, allowing for tighter guarantees as data accumulates. In a batched query setting, the asymptotic data requirements of our bound grows with the square-root of the number of adaptive queries, matching prior works'improvement over data splitting for the static setting. We instantiate our bound for statistical queries with the clipped Gaussian mechanism, where it empirically outperforms baselines composed from static bounds.

Problem

• Addresses overfitting in adaptive data reuse workflows
• Provides generalization bounds for dynamically growing datasets
• Enables adaptive query scheduling with time-varying accuracy guarantees

Innovation

• Dynamic data generalization bounds for adaptive analysis
• Time-varying accuracy bounds with accumulating data
• Adaptive query scheduling with clipped Gaussian mechanism