The Role of Randomness in Stability

2025-02-11

AI Summary

This paper investigates the minimal amount of randomness required for algorithmic stability in PAC learning, unifying the randomness complexity of replicability and differential privacy. Methodologically, it establishes the first necessary and sufficient condition for finite randomness complexity: an hypothesis class admits bounded randomness complexity if and only if its Littlestone dimension is finite; moreover, the complexity decays logarithmically in the excess error. A “weak-to-strong” boosting theorem is introduced, reducing randomness complexity to the global stability of deterministic algorithms and establishing a tight equivalence between them. Key contributions include: (1) a complete characterization of the intrinsic randomness requirements for PAC learning; (2) resolving the list-replicability problem posed at STOC 2024; and (3) providing the first general lower bound and constructive upper bound on randomness complexity for complex tasks such as high-dimensional classification.

Technology Category

• Machine Learning: Learning Theory
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Security and Privacy: Privacy-enhancing technologies
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Stability is a central property in learning and statistics promising the output of an algorithm $A$ does not change substantially when applied to similar datasets $S$ and $S'$. It is an elementary fact that any sufficiently stable algorithm (e.g. one returning the same result with high probability, satisfying privacy guarantees, etc.) must be randomized. This raises a natural question: can we quantify how much randomness is needed for algorithmic stability? We study the randomness complexity of two influential notions of stability in learning: replicability, which promises $A$ usually outputs the same result when run over samples from the same distribution (and shared random coins), and differential privacy, which promises the output distribution of $A$ remains similar under neighboring datasets. The randomness complexity of these notions was studied recently in (Dixon et al. ICML 2024) and (Cannone et al. ITCS 2024) for basic $d$-dimensional tasks (e.g. estimating the bias of $d$ coins), but little is known about the measures more generally or in complex settings like classification. Toward this end, we prove a `weak-to-strong' boosting theorem for stability: the randomness complexity of a task $M$ (either under replicability or DP) is tightly controlled by the best replication probability of any deterministic algorithm solving the task, a weak measure called `global stability' that is universally capped at $frac{1}{2}$ (Chase et al. FOCS 2023). Using this, we characterize the randomness complexity of PAC Learning: a class has bounded randomness complexity iff it has finite Littlestone dimension, and moreover scales at worst logarithmically in the excess error of the learner. This resolves a question of (Chase et al. STOC 2024) who asked for such a characterization in the equivalent language of (error-dependent) `list-replicability'.

Problem

• Quantify randomness for algorithmic stability
• Study randomness in replicability and differential privacy
• Characterize randomness complexity in PAC Learning

Innovation

• Quantifies randomness for algorithmic stability.
• Proves weak-to-strong boosting theorem.
• Characterizes randomness complexity in PAC Learning.