Back to Square Roots: An Optimal Bound on the Matrix Factorization Error for Multi-Epoch Differentially Private SGD

2025-05-17

AI Summary

Existing matrix decomposition approaches for differentially private stochastic gradient descent (DP-SGD) yield loose error bounds over multiple training epochs, hindering theoretical understanding and practical performance. Method: This paper proposes the Banded Inverse Square Root (BISR) explicit decomposition—a novel method that constructs an inverse correlation matrix with banded structure, enabling precise characterization of cumulative error across training cycles. Contribution/Results: BISR is the first decomposition to provably achieve tight, matching upper and lower bounds on the estimation error, establishing asymptotic optimality and resolving a long-standing theoretical gap. Compared to prior methods, BISR maintains state-of-the-art empirical performance while significantly improving computational efficiency, implementation simplicity, and analytical tractability. By unifying rigorous privacy guarantees with practical scalability, BISR provides a principled, theoretically grounded tool for privacy-preserving machine learning.

Technology Category

• Machine Learning: Privacy
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Non-convex Optimization

Application Category

• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

Matrix factorization mechanisms for differentially private training have emerged as a promising approach to improve model utility under privacy constraints. In practical settings, models are typically trained over multiple epochs, requiring matrix factorizations that account for repeated participation. Existing theoretical upper and lower bounds on multi-epoch factorization error leave a significant gap. In this work, we introduce a new explicit factorization method, Banded Inverse Square Root (BISR), which imposes a banded structure on the inverse correlation matrix. This factorization enables us to derive an explicit and tight characterization of the multi-epoch error. We further prove that BISR achieves asymptotically optimal error by matching the upper and lower bounds. Empirically, BISR performs on par with state-of-the-art factorization methods, while being simpler to implement, computationally efficient, and easier to analyze.

Problem

• Close gap in multi-epoch matrix factorization error bounds
• Develop optimal banded inverse square root factorization method
• Achieve tight and explicit multi-epoch error characterization

Innovation

• Introduces Banded Inverse Square Root (BISR) method
• Achieves asymptotically optimal error bounds
• Simpler, efficient, and easier to analyze