DPZV: Resource Efficient ZO Optimization For Differentially Private VFL

2025-02-27

AI Summary

This paper addresses three critical challenges in vertical federated learning (VFL): gradient leakage risks, high computational and communication overheads for large-scale models, and memory bottlenecks inherent to zeroth-order methods. To this end, we propose the first backpropagation-free, lightweight collaborative training framework for VFL. Our approach introduces: (1) a two-point stochastic zeroth-order optimization mechanism that eliminates reliance on gradient backpropagation; (2) Gaussian noise injection integrated with rigorous (ε,δ)-differential privacy, enabling strong privacy guarantees without a trusted third party; and (3) an asynchronous communication protocol with theoretical convergence analysis for non-convex objectives. Experiments demonstrate a 90% reduction in client memory usage, substantial decreases in communication and computation costs, and consistent accuracy improvements over baselines on image and NLP benchmarks. Moreover, our method achieves optimal privacy–utility trade-offs under ε ≤ 10.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Learning & Optimization for CV
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Security and Privacy: Privacy-enhancing technologies
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Vertical Federated Learning (VFL) enables collaborative model training across feature-partitioned data, yet faces significant privacy risks and inefficiencies when scaling to large models. We propose DPZV, a memory-efficient Zeroth-Order(ZO) optimization framework that integrates differential privacy (DP) with vertical federated learning, addressing three critical challenges: (1) privacy vulnerabilities from gradient leakage, (2) high computation/communication costs of first-order methods, and (3) excessive memory footprint in conventional zeroth-order approaches. Our framework eliminates backpropagation through two-point gradient estimation, reducing client memory usage by 90% compared to first-order counterparts while enabling asynchronous communication. By strategically injecting Gaussian noise on the server, DPZV achieves rigorous $(epsilon, delta)$-DP guarantees without third-party trust assumptions. Theoretical analysis establishes a convergence rate matching centralized case under non-convex objectives. Extensive experiments on image and NLP benchmarks demonstrate that DPZV outperforms all baselines in accuracy while providing strong privacy assurances ($epsilon leq 10$) and requiring far fewer computation resources, establishing new state-of-the-art privacy-utility tradeoffs for resource-constrained VFL deployments.

Problem

• Addresses privacy risks in vertical federated learning.
• Reduces computation and communication costs significantly.
• Minimizes memory usage in zeroth-order optimization.

Innovation

• Memory-efficient ZO optimization for VFL
• Differential privacy with Gaussian noise injection
• Asynchronous communication reduces client memory usage