Deep Learning under Fractional-Order Differential Privacy

2026-05-10

AI Summary

This work addresses the suboptimal privacy-utility trade-off in traditional DP-SGD, which neglects historical gradient information. The authors propose FO-DP-SGD, the first method to integrate a fractional-order memory mechanism into the DP-SGD framework. Prior to noise addition, FO-DP-SGD employs a fractional-order recursive query to perform power-law weighted aggregation of the current clipped gradient and historical private outputs, thereby injecting long-memory effects. The approach combines ℓ₂ sensitivity analysis, Poisson subsampled Gaussian mechanism, and Rényi differential privacy composition, preserving standard privacy accounting while enabling fine-tuned optimization of the privacy-utility balance through the fractional-order parameter. Experiments demonstrate that FO-DP-SGD significantly outperforms DP-SGD and multiple private optimization baselines on SVHN, CIFAR-10, and CIFAR-100, achieving notably higher test accuracy and improved privacy-utility performance.

Abstract

Differentially private stochastic gradient descent (DP-SGD) is a standard approach to privacy-preserving learning based on per-example clipping, subsampling, Gaussian perturbation, and privacy accounting. Classical DP-SGD releases a noisy version of the current clipped subsampled gradient sum. We propose Fractional-Order Differentially Private Stochastic Gradient Descent (\textbf{FO-DP-SGD}), a mechanism-level extension that replaces this current-only query, before Gaussian noise is added, with a fractional recursive query combining the current clipped sum with a finite-window, power-law-weighted aggregation of previously released private sum-level outputs. This injects fractional memory into the release mechanism while preserving the standard \emph{sum-then-noise-then-divide} structure. Under add/remove adjacency with Poisson subsampling, the current-step sensitivity analysis shows that the only newly data-dependent term is the scaled current clipped sum. Hence, conditioned on the private history, the effective \(\ell_2\)-sensitivity is at most \(βC\), where \(C\) is the clipping threshold and \(β\in(0,1]\) controls the current-step contribution. Thus, FO-DP-SGD admits standard per-step Rényi differential privacy accounting via a Poisson-subsampled Gaussian mechanism with effective noise-to-sensitivity ratio \(σ/β\), and composes to yield overall \((\varepsilon,δ)\)-differential privacy guarantees. FO-DP-SGD provides a framework for studying long-memory effects in private optimization. The fractional order, memory window, and mixing coefficient govern the trade-off among current-step sensitivity, signal retention, and private-history influence. Experiments on SVHN, CIFAR-10, and CIFAR-100 show improved test accuracy and privacy--utility performance over DP-SGD and private baselines including DP-Adam, DP-IS, SA-DP-SGD, ADP-AdamW, DP-SAT, and DP-Adam-AC.

Problem

• Differential Privacy
• Deep Learning
• Stochastic Gradient Descent
• Privacy-Utility Trade-off
• Memory Mechanism

Innovation

• Fractional-Order DP
• FO-DP-SGD
• Differential Privacy
• Memory-Augmented Optimization
• Private Stochastic Gradient Descent