Is Randomness Necessary for Adaptive Data Analysis?

2026-07-08

AI Summary

This work investigates whether randomness is essential in adaptive data analysis (ADA), particularly for computationally unbounded analysts. Through information-theoretic arguments and adversarial constructions, it establishes the first rigorous lower bound in the random oracle model, proving that any deterministic mechanism can reliably answer at most $\tilde{O}(n)$ adaptive queries—significantly fewer than the $n^2$-order achievable by randomized mechanisms. This result demonstrates that randomness plays a fundamental role in overcoming the linear query barrier, thereby preventing overfitting and spurious discoveries. The findings reveal an inherent limitation of deterministic approaches in ADA, underscoring the necessity of randomization for sustaining statistical validity under adaptive querying.

Abstract

The Adaptive Data Analysis (ADA) problem formalizes the challenge of preventing false discovery and overfitting when a dataset is repeatedly reused. Formally, our input is a dataset containing $n$ i.i.d. samples from an unknown distribution $\mathcal{P}$ over a domain $\mathcal{X}$, and our goal is to answer a sequence of $k$ adaptively chosen statistical queries with respect to $\mathcal{P}$. The main question is how many queries we can support (i.e., how large $k$ can be), primarily as a function of the number of samples $n$. This question has been intensively studied and is relatively well-understood for randomized mechanisms: there are computationally efficient mechanisms that support $k \approx n^2$ queries, and no computationally efficient mechanism can answer $k \gg n^2$ queries. In this paper, we address a fundamental question: is randomness necessary for ADA? Despite a decade of work on ADA, this question remains open. A folklore observation dating back to the initial works on ADA is that randomness is not necessary when the analyst is computationally bounded. Yet, the necessity of randomness against computationally unbounded analysts has remained elusive. Our main contribution resolves this gap in the information-theoretic Random Oracle model. Perhaps surprisingly, we show that randomness is strictly necessary to answer a non-trivial number of adaptive queries: when the analyst is unbounded, any deterministic mechanism can be forced to fail after just $k = \tilde{O} (n)$ queries.

Problem

• Adaptive Data Analysis
• Randomness
• Overfitting
• Statistical Queries
• False Discovery

Innovation

• Adaptive Data Analysis
• Randomness
• Deterministic Mechanisms
• Statistical Queries
• Information-Theoretic Lower Bound