DP-$\lambda$CGD: Efficient Noise Correlation for Differentially Private Model Training

2026-01-29

AI Summary

This work proposes a novel inter-iteration noise correlation mechanism for differentially private training that overcomes the substantial memory overhead of existing approaches—such as matrix factorization–based methods—which require storing historical noise vectors. The proposed method correlates noise only with that of the immediately preceding iteration and leverages controlled cancellation combined with a pseudorandom generator to regenerate noise without any additional storage. This approach is the first to achieve noise correlation while preserving the memory efficiency of standard DP-SGD, incurring negligible computational overhead. Consequently, it significantly improves model accuracy without increasing memory consumption beyond that of conventional DP-SGD.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Generation
• Search and Optimization: Learning to Search

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Differentially private stochastic gradient descent (DP-SGD) is the gold standard for training machine learning models with formal differential privacy guarantees. Several recent extensions improve its accuracy by introducing correlated noise across training iterations. Matrix factorization mechanisms are a prominent example, but they correlate noise across many iterations and require storing previously added noise vectors, leading to substantial memory overhead in some settings. In this work, we propose a new noise correlation strategy that correlates noise only with the immediately preceding iteration and cancels a controlled portion of it. Our method relies on noise regeneration using a pseudorandom noise generator, eliminating the need to store past noise. As a result, it requires no additional memory beyond standard DP-SGD. We show that the computational overhead is minimal and empirically demonstrate improved accuracy over DP-SGD.

Problem

• differential privacy
• noise correlation
• memory overhead
• DP-SGD
• model training

Innovation

• differential privacy
• noise correlation
• DP-SGD
• memory efficiency
• pseudorandom noise generation