Second-Order Convergence in Private Stochastic Non-Convex Optimization

2025-05-21

AI Summary

This work addresses the challenge of finding second-order stationary points (SOSPs) in stochastic nonconvex optimization under differential privacy (DP). Existing methods suffer from inaccurate convergence error bounds—neglecting gradient variance—and reliance on inefficient private model selection mechanisms. To overcome these limitations, we propose a perturbed SGD framework incorporating Gaussian noise injection and a generic gradient oracle. Crucially, we introduce *model drift distance* as a novel, privacy-preserving surrogate for second-order information to detect and escape saddle points—eliminating the need for additional private selection. Furthermore, we establish the first DP-SOSP convergence guarantee for *heterogeneous distributed settings*, correcting prior erroneous error bounds. Our theoretical analysis achieves the optimal second-order convergence rate. Empirical evaluation on real-world datasets demonstrates substantial improvements in the privacy–utility trade-off, particularly in high-dimensional distributed environments.

Technology Category

• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Privacy

Application Category

• Security and Privacy: Data transparency and provenance
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

We investigate the problem of finding second-order stationary points (SOSP) in differentially private (DP) stochastic non-convex optimization. Existing methods suffer from two key limitations: (i) inaccurate convergence error rate due to overlooking gradient variance in the saddle point escape analysis, and (ii) dependence on auxiliary private model selection procedures for identifying DP-SOSP, which can significantly impair utility, particularly in distributed settings. To address these issues, we propose a generic perturbed stochastic gradient descent (PSGD) framework built upon Gaussian noise injection and general gradient oracles. A core innovation of our framework is using model drift distance to determine whether PSGD escapes saddle points, ensuring convergence to approximate local minima without relying on second-order information or additional DP-SOSP identification. By leveraging the adaptive DP-SPIDER estimator as a specific gradient oracle, we develop a new DP algorithm that rectifies the convergence error rates reported in prior work. We further extend this algorithm to distributed learning with arbitrarily heterogeneous data, providing the first formal guarantees for finding DP-SOSP in such settings. Our analysis also highlights the detrimental impacts of private selection procedures in distributed learning under high-dimensional models, underscoring the practical benefits of our design. Numerical experiments on real-world datasets validate the efficacy of our approach.

Problem

• Finding second-order stationary points in private stochastic non-convex optimization
• Addressing inaccurate convergence error rates in saddle point escape analysis
• Eliminating dependence on auxiliary private model selection procedures

Innovation

• Uses perturbed stochastic gradient descent (PSGD) framework
• Employs Gaussian noise injection for differential privacy
• Leverages adaptive DP-SPIDER estimator for gradient oracles