Optimal Rates for Pure {\varepsilon}-Differentially Private Stochastic Convex Optimization with Heavy Tails

2026-04-07

AI Summary

This work addresses heavy-tailed stochastic convex optimization under pure ε-differential privacy, where gradients satisfy only bounded k-th order moments rather than Lipschitz continuity. The authors propose a novel private optimization framework based on a Lipschitz extension of the empirical loss, which for the first time characterizes the minimax optimal excess risk rate for this setting. Remarkably, even when the worst-case Lipschitz constant is unbounded, the method achieves—within logarithmic factors—the optimal risk bound with high probability in polynomial time. The theoretical analysis accommodates structured domains such as Euclidean balls, ellipsoids, and polytopes, and the algorithm integrates robust estimation with differential privacy mechanisms to provide rigorous high-probability guarantees.

Abstract

We study stochastic convex optimization (SCO) with heavy-tailed gradients under pure epsilon-differential privacy (DP). Instead of assuming a bound on the worst-case Lipschitz parameter of the loss, we assume only a bounded k-th moment. This assumption allows for unbounded, heavy-tailed stochastic gradient distributions, and can yield sharper excess risk bounds. The minimax optimal rate for approximate (epsilon, delta)-DP SCO is known in this setting, but the pure epsilon-DP case has remained open. We characterize the minimax optimal excess-risk rate for pure epsilon-DP heavy-tailed SCO up to logarithmic factors. Our algorithm achieves this rate in polynomial time with high probability. Moreover, it runs in polynomial time with probability 1 when the worst-case Lipschitz parameter is polynomially bounded. For important structured problem classes - including hinge/ReLU-type and absolute-value losses on Euclidean balls, ellipsoids, and polytopes - we achieve the same excess-risk guarantee in polynomial time with probability 1 even when the worst-case Lipschitz parameter is infinite. Our approach is based on a novel framework for privately optimizing Lipschitz extensions of the empirical loss. We complement our excess risk upper bound with a novel high probability lower bound.

Problem

• differential privacy
• stochastic convex optimization
• heavy-tailed gradients
• minimax rate
• excess risk

Innovation

• pure differential privacy
• heavy-tailed gradients
• stochastic convex optimization
• minimax optimal rate
• Lipschitz extension