Membership Inference Attacks with False Discovery Rate Control

2025-08-09

AI Summary

Deep learning models are vulnerable to membership inference attacks (MIAs), yet existing methods fail to provide verifiable false discovery rate (FDR) control under unknown data distributions and non-member probability dependencies. This paper introduces, for the first time, FDR control into MIAs, proposing a general plug-in framework grounded in hypothesis testing and posterior calibration—requiring no distributional assumptions and supporting both black-box and continual learning settings. We theoretically establish that the framework guarantees FDR ≤ α for any underlying data distribution. Empirically, on benchmarks including CIFAR-10, CIFAR-100, and ImageNet, it achieves high attack success rates (>90%) while tightly controlling FDR within prespecified thresholds (e.g., 0.05 or 0.1), substantially outperforming prior approaches in both statistical rigor and practical efficacy.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent studies have shown that deep learning models are vulnerable to membership inference attacks (MIAs), which aim to infer whether a data record was used to train a target model or not. To analyze and study these vulnerabilities, various MIA methods have been proposed. Despite the significance and popularity of MIAs, existing works on MIAs are limited in providing guarantees on the false discovery rate (FDR), which refers to the expected proportion of false discoveries among the identified positive discoveries. However, it is very challenging to ensure the false discovery rate guarantees, because the underlying distribution is usually unknown, and the estimated non-member probabilities often exhibit interdependence. To tackle the above challenges, in this paper, we design a novel membership inference attack method, which can provide the guarantees on the false discovery rate. Additionally, we show that our method can also provide the marginal probability guarantee on labeling true non-member data as member data. Notably, our method can work as a wrapper that can be seamlessly integrated with existing MIA methods in a post-hoc manner, while also providing the FDR control. We perform the theoretical analysis for our method. Extensive experiments in various settings (e.g., the black-box setting and the lifelong learning setting) are also conducted to verify the desirable performance of our method.

Problem

• Control false discovery rate in membership inference attacks
• Provide guarantees for non-member probability estimation
• Integrate with existing MIA methods post-hoc

Innovation

• Novel MIA method with FDR control
• Wrapper integrating existing MIA methods
• Theoretical analysis and extensive experiments