GeoClip: Geometry-Aware Clipping for Differentially Private SGD

2025-06-06

AI Summary

This paper addresses the coarse-grained selection of gradient clipping thresholds in DP-SGD, which disrupts the privacy–utility trade-off. We propose a geometry-aware adaptive gradient clipping method. Its core innovation is the first introduction of a principal-component-analysis-inspired coordinate transformation: gradient clipping and noise injection are performed in a newly aligned geometric basis—learned from the gradient distribution—enabling estimation of optimal clipping directions without additional privacy cost. We theoretically establish convergence and derive a closed-form solution for the optimal clipping threshold. Experiments on tabular and image datasets demonstrate that, under identical privacy budgets (ε ≤ 8), our method significantly outperforms existing adaptive clipping approaches: it reduces noise variance by 12.7% on average and improves test accuracy by 1.9–4.3 percentage points.

Technology Category

• Machine Learning: Privacy
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Learning to Search

Application Category

• Security and Privacy: Large-scale security measurements
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

Differentially private stochastic gradient descent (DP-SGD) is the most widely used method for training machine learning models with provable privacy guarantees. A key challenge in DP-SGD is setting the per-sample gradient clipping threshold, which significantly affects the trade-off between privacy and utility. While recent adaptive methods improve performance by adjusting this threshold during training, they operate in the standard coordinate system and fail to account for correlations across the coordinates of the gradient. We propose GeoClip, a geometry-aware framework that clips and perturbs gradients in a transformed basis aligned with the geometry of the gradient distribution. GeoClip adaptively estimates this transformation using only previously released noisy gradients, incurring no additional privacy cost. We provide convergence guarantees for GeoClip and derive a closed-form solution for the optimal transformation that minimizes the amount of noise added while keeping the probability of gradient clipping under control. Experiments on both tabular and image datasets demonstrate that GeoClip consistently outperforms existing adaptive clipping methods under the same privacy budget.

Problem

• Optimizing gradient clipping threshold in DP-SGD for privacy-utility trade-off
• Addressing gradient coordinate correlations ignored by current adaptive methods
• Reducing noise addition while controlling gradient clipping probability

Innovation

• Geometry-aware gradient clipping framework
• Transformed basis aligned with gradient geometry
• Adaptive transformation without privacy cost