Optimal Guarantees for Auditing Rényi Differentially Private Machine Learning

2026-05-20

AI Summary

This work addresses the problem of effectively auditing machine learning algorithms that claim to satisfy Rényi differential privacy (RDP) in a black-box setting. The authors propose a hypothesis testing framework that leverages the Donsker–Varadhan variational estimator to directly estimate the Rényi divergence between outputs from adjacent executions, thereby disentangling statistical estimation error from genuine privacy leakage. This approach establishes, for the first time, information-theoretically optimal sample complexity for RDP auditing based on the Donsker–Varadhan estimator and provides explicit non-asymptotic confidence intervals. Empirical evaluations on MNIST and CIFAR-10 demonstrate that the proposed black-box audit significantly outperforms existing methods for small to moderate Rényi orders, yielding substantially tighter lower bounds on the algorithm’s true RDP guarantee.

Abstract

We study black-box auditing for machine learning algorithms that claim R \ 'enyi differential privacy (RDP) guarantees. We introduce an auditing framework, based on hypothesis testing, that directly estimates Rényi divergence between neighboring executions using the Donsker-Varadhan (DV) variational estimator. Our analysis yields explicit and non-asymptotic confidence intervals for RDP auditing via class-restricted DV estimators, separating statistical estimation error from algorithmic privacy leakage. We prove matching minimax lower bounds showing that, up to logarithmic factors, our sample-complexity guarantees are information-theoretically optimal, thereby establishing the first optimal guarantees for auditing RDP via DV estimators. Empirically, we instantiate our framework for auditing DP-SGD in a fully black-box setting. Across MNIST and CIFAR-10, and over a wide range of privacy regimes, our auditors produce a strong overall improvement on empirical RDP lower bounds compared to prior state-of-the-art black-box methods especially at small and moderate Rényi orders where accurate auditing is most challenging.

Problem

• Rényi differential privacy
• privacy auditing
• black-box auditing
• machine learning
• Rényi divergence

Innovation

• Rényi differential privacy
• black-box auditing
• Donsker-Varadhan estimator
• minimax optimality
• privacy leakage