Improved Differentially Private Continual Observation Using Group Algebra

2024-12-03

arXiv.org

AI Summary

This work addresses additive error optimization for weighted prefix sums under differential privacy in continual observation settings, motivated by high-dimensional private learning for word prediction in Gboard’s billion-user deployment. Methodologically, it introduces group representation theory—specifically the Dedekind–Frobenius–Schur theory—to analyze differentially private mechanisms for the first time, revealing an intrinsic connection between weighted prefix sums and group matrices; it then integrates ℓₚ-norm error bounds with representation-theoretic tools within a factorization norm framework to design the first efficient constructive decomposition algorithm. Contributions include: (i) the first polynomial-time constructive mechanism achieving Mathias’ (1993) optimal non-constructive upper bound; (ii) substantial improvements over Fichtenberger et al. (ICML ’23) and Henzinger et al. (SODA ’23); and (iii) the first tight additive error bounds for generalized weight functions—including sliding windows—under continual observation.

Technology Category

• Machine Learning: Privacy
• Search and Optimization: Mixed Discrete/Continuous Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Differentially private weighted prefix sum under continual observation is a crucial component in the production-level deployment of private next-word prediction for Gboard, which, according to Google, has over a billion users. More specifically, Google uses a differentially private mechanism to sum weighted gradients in its emph{private follow-the-regularized leader} algorithm. Apart from efficiency, the additive error of the private mechanism is crucial as multiplied with the square root of the model's dimension $d$ (with $d$ ranging up to $10$ trillion, for example, Switch Transformers or M6-10T), it determines the accuracy of the learning system. So, any improvement in leading constant matters significantly in practice. In this paper, we show a novel connection between mechanisms for continual weighted prefix sum and a concept in representation theory known as the group matrix introduced in correspondence between Dedekind and Frobenius (1897) and generalized by Schur (1904). To the best of our knowledge, this is the first application of group algebra to analyze differentially private algorithms. Using this connection, we analyze a class of matrix norms known as {em factorization norms} that give upper and lower bounds for the additive error under general $ell_p$-norms of the matrix mechanism. This allows us to give the first efficient factorization that matches the best-known non-constructive upper bound on the factorization norm by Mathias (1993) for the matrix used in Google's deployment and also improves on the previous best-known constructive bound of Fichtenberger et al. (ICML 2023) and Henzinger et al. (SODA 2023) and the first upper bound on the additive error for a large class of weight functions for weighted prefix sum problems, including the sliding window matrix (Bolot et al. (ICDT 2013).

Problem

• Improving differentially private continual observation
• Enhancing additive error in private mechanisms
• Applying group algebra to private algorithms

Innovation

• Group algebra application
• Matrix norms analysis
• Improved error bounds