Improved Sample Complexity for Private Nonsmooth Nonconvex Optimization

2024-10-08

arXiv.org

AI Summary

This paper studies differentially private (DP) nonsmooth nonconvex stochastic and empirical risk minimization (ERM), aiming to efficiently compute Goldstein stationary points. To address the high sample complexity of existing DP algorithms, we propose both single-pass and multi-pass DP optimization methods. The single-pass algorithm achieves sample complexity $ ilde{O}(1/(alphaeta^3) + d/(varepsilonalphaeta^2) + d^{3/4}/(varepsilon^{1/2}alphaeta^{5/2}))$, while the multi-pass variant improves it to $ ilde{O}(d/eta^2 + d^{3/4}/(varepsilonalpha^{1/2}eta^{3/2}))$—both strictly outperforming Zhang et al. [2024] by a factor of $Omega(sqrt{d})$, constituting the first such improvement. We further establish the first generalization guarantee for Goldstein stationary points in DP-ERM, deriving the first nonconvex nonsmooth DP generalization error bound. Our core technical innovation lies in unifying differential privacy mechanisms, Goldstein subgradient analysis, and generalization error analysis—yielding simultaneous advances in sample efficiency and theoretical completeness.

Technology Category

• Search and Optimization: Non-convex Optimization
• Machine Learning: Privacy
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Security and Privacy: Data transparency and provenance
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

We study differentially private (DP) optimization algorithms for stochastic and empirical objectives which are neither smooth nor convex, and propose methods that return a Goldstein-stationary point with sample complexity bounds that improve on existing works. We start by providing a single-pass $(epsilon,delta)$-DP algorithm that returns an $(alpha,eta)$-stationary point as long as the dataset is of size $widetilde{Omega}left(1/alphaeta^{3}+d/epsilonalphaeta^{2}+d^{3/4}/epsilon^{1/2}alphaeta^{5/2} ight)$, which is $Omega(sqrt{d})$ times smaller than the algorithm of Zhang et al. [2024] for this task, where $d$ is the dimension. We then provide a multi-pass polynomial time algorithm which further improves the sample complexity to $widetilde{Omega}left(d/eta^2+d^{3/4}/epsilonalpha^{1/2}eta^{3/2} ight)$, by designing a sample efficient ERM algorithm, and proving that Goldstein-stationary points generalize from the empirical loss to the population loss.

Problem

• Improving sample complexity for private nonsmooth nonconvex optimization
• Designing DP algorithms for stochastic empirical objectives
• Achieving better bounds for Goldstein-stationary points

Innovation

• Single-pass DP algorithm for nonconvex optimization
• Improved sample complexity with polynomial time
• Goldstein-stationary points generalize empirical loss