Near Exact Privacy Amplification for Matrix Mechanisms

2024-10-08

arXiv.org

AI Summary

This work addresses the challenge of privacy parameter computation in differentially private (DP) machine learning under the joint effects of stochastic minibatching and cross-iteration correlated noise—specifically, matrix mechanisms. We propose the first near-exact privacy amplification analysis framework applicable to arbitrary lower-triangular nonnegative correlation matrices. Methodologically, we integrate Monte Carlo privacy accounting with lower-triangular noise modeling, enabling the first tight privacy bound estimation for general correlated noise structures. Our framework supports joint optimization of the correlation matrix under privacy amplification constraints and introduces a practical “ball-and-bin” minibatching mechanism as a robust alternative to Poisson sampling. Experiments demonstrate that our approach achieves lower root-mean-square error (RMSE) than state-of-the-art methods on prefix-sum tasks and significantly improves the privacy–utility trade-off in deep learning tasks.

Technology Category

• Machine Learning: Privacy
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Learning to Search

Application Category

• Security and Privacy: Privacy-enhancing technologies
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

We study the problem of computing the privacy parameters for DP machine learning when using privacy amplification via random batching and noise correlated across rounds via a correlation matrix $ extbf{C}$ (i.e., the matrix mechanism). Past work on this problem either only applied to banded $ extbf{C}$, or gave loose privacy parameters. In this work, we give a framework for computing near-exact privacy parameters for any lower-triangular, non-negative $ extbf{C}$. Our framework allows us to optimize the correlation matrix $ extbf{C}$ while accounting for amplification, whereas past work could not. Empirically, we show this lets us achieve smaller RMSE on prefix sums than the previous state-of-the-art (SOTA). We also show that we can improve on the SOTA performance on deep learning tasks. Our two main technical tools are (i) using Monte Carlo accounting to bypass composition, which was the main technical challenge for past work, and (ii) a"balls-in-bins"batching scheme that enables easy privacy analysis and is closer to practical random batching than Poisson sampling.

Problem

• Computing precise DP parameters for matrix mechanisms
• Optimizing correlation matrix C with amplification
• Improving RMSE and deep learning performance

Innovation

• Monte Carlo accounting bypasses composition challenges
• Balls-in-bins batching simplifies privacy analysis
• Optimizes correlation matrix with amplification consideration