General-Purpose $f$-DP Estimation and Auditing in a Black-Box Setting

2025-02-10

AI Summary

This work addresses the challenge of verifying $f$-differential privacy ($f$-DP) in black-box settings, where neither algorithmic internals nor data distribution assumptions are available. We propose the first general, assumption-free auditing framework for $f$-DP that requires no prior knowledge of the mechanism or distributional assumptions. Our method integrates nonparametric density estimation with optimal classification boundary theory to reconstruct the full $f$-DP trade-off curve and detect violations with statistically rigorous guarantees. Theoretical analysis establishes uniform consistency of the estimator. Experiments across diverse DP mechanisms demonstrate rapid error convergence and high violation detection power, significantly outperforming existing empirical $f$-DP evaluation methods. Our core contribution is the first black-box, distribution-agnostic, and model-free estimator of the complete $f$-DP trade-off curve—overcoming prior reliance on white-box access or strong parametric modeling assumptions—and providing a trustworthy, empirically grounded auditing tool for practical $f$-DP deployment.

Technology Category

• Machine Learning: Privacy
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Non-convex Optimization

Application Category

• Security and Privacy: Data transparency and provenance
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

In this paper we propose new methods to statistically assess $f$-Differential Privacy ($f$-DP), a recent refinement of differential privacy (DP) that remedies certain weaknesses of standard DP (including tightness under algorithmic composition). A challenge when deploying differentially private mechanisms is that DP is hard to validate, especially in the black-box setting. This has led to numerous empirical methods for auditing standard DP, while $f$-DP remains less explored. We introduce new black-box methods for $f$-DP that, unlike existing approaches for this privacy notion, do not require prior knowledge of the investigated algorithm. Our procedure yields a complete estimate of the $f$-DP trade-off curve, with theoretical guarantees of convergence. Additionally, we propose an efficient auditing method that empirically detects $f$-DP violations with statistical certainty, merging techniques from non-parametric estimation and optimal classification theory. Through experiments on a range of DP mechanisms, we demonstrate the effectiveness of our estimation and auditing procedures.

Problem

• Statistical assessment of $f$-DP
• Black-box validation of differential privacy
• Efficient $f$-DP violation detection

Innovation

• Black-box $f$-DP estimation
• No prior algorithm knowledge
• Non-parametric estimation technique