Tighter Privacy Auditing of DP-SGD in the Hidden State Threat Model

2024-05-23

International Conference on Learning Representations

AI Summary

Under the hidden-state threat model—where attackers observe only the final model and cannot access intermediate training updates—a significant gap exists between the empirical privacy lower bounds obtained via auditing of DP-SGD and its theoretical privacy upper bounds. Method: This paper introduces the first gradient-optimization-based privacy auditing framework, which constructs adaptive gradient sequences that maximize the final model’s privacy loss, enabling trajectory modeling and empirical boundary analysis. Contributions/Results: (1) Hiding intermediate states does not inherently improve privacy; inserting adversarial gradients at all steps yields zero privacy gain. (2) Tight matching between empirical lower and theoretical upper bounds is achieved under specific loss landscapes and large-batch regimes. (3) The proposed auditing method substantially surpasses existing empirical lower bounds. Collectively, these findings expose the looseness of current theoretical guarantees and establish a new, empirically grounded benchmark for real-world privacy assurance under hidden-state assumptions.

Technology Category

• Machine Learning: Privacy
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Machine learning models can be trained with formal privacy guarantees via differentially private optimizers such as DP-SGD. In this work, we focus on a threat model where the adversary has access only to the final model, with no visibility into intermediate updates. In the literature, this hidden state threat model exhibits a significant gap between the lower bound from empirical privacy auditing and the theoretical upper bound provided by privacy accounting. To challenge this gap, we propose to audit this threat model with adversaries that craft a gradient sequence designed to maximize the privacy loss of the final model without relying on intermediate updates. Our experiments show that this approach consistently outperforms previous attempts at auditing the hidden state model. Furthermore, our results advance the understanding of achievable privacy guarantees within this threat model. Specifically, when the crafted gradient is inserted at every optimization step, we show that concealing the intermediate model updates in DP-SGD does not enhance the privacy guarantees. The situation is more complex when the crafted gradient is not inserted at every step: our auditing lower bound matches the privacy upper bound only for an adversarially-chosen loss landscape and a sufficiently large batch size. This suggests that existing privacy upper bounds can be improved in certain regimes.

Problem

• Auditing privacy loss in DP-SGD hidden state threat model
• Bridging gap between empirical and theoretical privacy bounds
• Evaluating impact of crafted gradients on privacy guarantees

Innovation

• Auditing hidden state model with crafted gradients
• Concealing intermediate updates lacks privacy enhancement
• Improved privacy bounds for specific loss landscapes