Detectability in Diversity: Improved Canary Crafting for Privacy Auditing in One Run

2026-05-26

AI Summary

This work addresses the limitation of existing single-run privacy auditing methods, whose estimates of privacy leakage are often weakened due to interference among multiple canary samples. To mitigate this issue, the authors propose a novel two-level optimization strategy that integrates influence function–based greedy initialization with an embedding-space diversity objective. This approach simultaneously enhances the detectability of individual canaries and reduces their mutual interference. As a result, the method substantially strengthens the privacy leakage signal in a single training run, achieving more accurate and efficient privacy auditing than current techniques while requiring lower computational overhead.

Abstract

Privacy auditing aims to empirically assess privacy leakage in machine learning models using membership inference attacks (MIAs), and to derive lower bounds on differential privacy (DP) parameters. Recent one-run auditing methods address the high cost of standard approaches by relying on a single training run with multiple "canary" points whose inclusion or exclusion must be detected by the auditor. In this work, we study the problem of efficiently crafting canaries for one-run privacy auditing. Motivated by recent theoretical insights suggesting that interference between canaries contributes to weaker leakage estimates compared to multi-run methods, we propose to optimize canaries to be both highly detectable and minimally interfering. Our approach combines a greedy initialization based on influence functions with a bilevel optimization procedure that maximizes distinguishability while promoting diversity in embedding space, enabling the use of computationally efficient bilevel algorithms. Experiments show that our method achieves stronger privacy leakage estimates at a lower computational cost than existing canary crafting approaches.

Problem

• privacy auditing
• canary crafting
• membership inference attacks
• differential privacy
• one-run auditing

Innovation

• canary crafting
• privacy auditing
• membership inference attacks
• bilevel optimization
• embedding diversity