When Gradient Clipping Becomes a Control Mechanism for Differential Privacy in Deep Learning

2026-02-11

AI Summary

In differentially private deep learning, fixed gradient clipping thresholds often lead to optimization bias or noise-dominated updates, degrading model performance. This work proposes the first adaptive gradient clipping method grounded in control theory, integrating a feedback control mechanism into differentially private training. By leveraging lightweight spectral analysis of model weights—including spectral decomposition and estimation of heavy-tailed spectral indicators—the method dynamically adjusts the clipping threshold. A logarithmic-domain multiplicative feedback controller is employed to ensure stable optimization. Crucially, this approach incurs no additional privacy cost, effectively balancing gradient clipping and noise injection without increasing privacy loss, thereby significantly improving both training stability and accuracy of privacy-preserving models.

Technology Category

• Machine Learning: Privacy
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Security and Privacy: Privacy-enhancing technologies
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Privacy-preserving training on sensitive data commonly relies on differentially private stochastic optimization with gradient clipping and Gaussian noise. The clipping threshold is a critical control knob: if set too small, systematic over-clipping induces optimization bias; if too large, injected noise dominates updates and degrades accuracy. Existing adaptive clipping methods often depend on per-example gradient norm statistics, adding computational overhead and introducing sensitivity to datasets and architectures. We propose a control-driven clipping strategy that adapts the threshold using a lightweight, weight-only spectral diagnostic computed from model parameters. At periodic probe steps, the method analyzes a designated weight matrix via spectral decomposition and estimates a heavy-tailed spectral indicator associated with training stability. This indicator is smoothed over time and fed into a bounded feedback controller that updates the clipping threshold multiplicatively in the log domain. Because the controller uses only parameters produced during privacy-preserving training, the resulting threshold updates are post-processing and do not increase privacy loss beyond that of the underlying DP optimizer under standard composition accounting.

Problem

• differential privacy
• gradient clipping
• deep learning
• adaptive clipping
• privacy-preserving training

Innovation

• gradient clipping
• differential privacy
• spectral analysis
• feedback control
• privacy-preserving training