Revisiting the Provable-Auditable Privacy Gap of DP-SGD

2026-08-28

AI Summary

本文针对DP-SGD的理论隐私与实际隐私差距问题，提出了一种轻量级防御框架，通过优化方法提高实证隐私下界，同时保持理论隐私水平。

Abstract

Differential privacy (DP) has traditionally been used to provide theoretical upper bounds on an algorithm's stability to changing its training data. In modern private machine learning applications, achieving strong tradeoffs between utility and theoretical privacy is challenging, and thus one may optimistically hope that existing theoretical privacy analyses are loose. Recent work on privacy auditing has adopted a dual viewpoint, instead lower bounding the true privacy of an algorithm by constructing empirical distinguishing events. The auditing literature has thus far yielded a pessimistic outlook on the looseness of theoretical privacy bounds for DP-SGD, the de facto private training method in modern ML, as nearly-matching empirical lower bounds have been achieved under various threat models [NHSBTJCT23, AC24, CBP25]. In this work, we propose the empirical privacy lower bound of an algorithm as a concrete metric to optimize for, complementary to the theoretical upper bound. We give a lightweight defense framework that generically augments optimization methods in the ML pipeline to have significantly-improved empirical privacy on standard benchmarks. Moreover, we show that our framework comes at no theoretical privacy cost when augmenting DP-SGD, unlike previously-proposed defenses against membership inference attacks. We evaluate our defense against a broad range of audit constructions, models, and datasets to demonstrate its flexibility.

Problem

• Differential Privacy
• DP-SGD
• Privacy Auditing
• Theoretical Privacy Bounds
• Empirical Lower Bounds

Innovation

• Empirical Privacy Lower Bound
• Defense Framework
• DP-SGD Augmentation