Characterizing the Accuracy-Communication-Privacy Trade-off in Distributed Stochastic Convex Optimization

2025-01-06

AI Summary

This work investigates the fundamental trade-off among estimation accuracy, communication cost, and privacy guarantee in distributed differentially private stochastic convex optimization (DP-SCO). We consider a setting with $M$ clients, each holding $N$ i.i.d. local samples, aiming to collaboratively minimize a global convex loss function while satisfying strict local differential privacy constraints. First, we establish a tight trivariate trade-off characterizing accuracy, number of communication rounds, and privacy budget. Second, we propose a novel distributed algorithm based on Vaidya’s cutting-plane method, achieving optimal convergence rate under differential privacy. Third, we derive a matching information-theoretic lower bound, demonstrating that our upper and lower bounds are tight up to constant factors. Collectively, these results provide foundational theoretical guarantees and an algorithmic paradigm for privacy-aware distributed learning.

Technology Category

• Machine Learning: Privacy
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

We consider the problem of differentially private stochastic convex optimization (DP-SCO) in a distributed setting with $M$ clients, where each of them has a local dataset of $N$ i.i.d. data samples from an underlying data distribution. The objective is to design an algorithm to minimize a convex population loss using a collaborative effort across $M$ clients, while ensuring the privacy of the local datasets. In this work, we investigate the accuracy-communication-privacy trade-off for this problem. We establish matching converse and achievability results using a novel lower bound and a new algorithm for distributed DP-SCO based on Vaidya's plane cutting method. Thus, our results provide a complete characterization of the accuracy-communication-privacy trade-off for DP-SCO in the distributed setting.

Problem

• Distributed Computing
• Data Privacy
• Optimization

Innovation

• Distributed Optimization
• Privacy Protection
• Balanced Algorithm Design