DP-SGD with weight clipping

2023-10-27

arXiv.org

AI Summary

In differentially private stochastic gradient descent (DP-SGD), fixed gradient clipping introduces estimation bias and excessive noise injection, degrading model utility and privacy budget efficiency. Method: This paper proposes an adaptive privacy-preserving method based on a dynamically updated upper bound of the Lipschitz constant. It is the first to jointly model the current parameter location and the Lipschitz upper bound, enabling fine-grained, real-time estimation of gradient sensitivity—thereby enabling coordinated, adaptive adjustment of both the clipping threshold and the noise scale. Contribution/Results: Compared to conventional fixed-threshold clipping, our method significantly reduces estimation bias, improves ε–δ privacy budget utilization, and yields a tighter privacy loss bound. Extensive experiments on multiple benchmark datasets demonstrate that, under identical privacy budgets, our approach achieves substantially higher model accuracy than existing DP-SGD variants—validating both theoretical guarantees and practical performance gains.

Abstract

Recently, due to the popularity of deep neural networks and other methods whose training typically relies on the optimization of an objective function, and due to concerns for data privacy, there is a lot of interest in differentially private gradient descent methods. To achieve differential privacy guarantees with a minimum amount of noise, it is important to be able to bound precisely the sensitivity of the information which the participants will observe. In this study, we present a novel approach that mitigates the bias arising from traditional gradient clipping. By leveraging a public upper bound of the Lipschitz value of the current model and its current location within the search domain, we can achieve refined noise level adjustments. We present a new algorithm with improved differential privacy guarantees and a systematic empirical evaluation, showing that our new approach outperforms existing approaches also in practice.

Problem

• Enhance differential privacy in deep learning
• Reduce bias from gradient clipping methods
• Optimize noise levels for privacy guarantees

Innovation

• DP-SGD with weight clipping
• Public Lipschitz value utilization
• Refined noise level adjustments