Membership Inference Attacks as Privacy Tools: Reliability, Disparity and Ensemble

2025-06-16

AI Summary

Membership inference attacks (MIAs) suffer from low reliability in privacy assessment, exhibiting substantial variability across attack methods and individual instances. This work systematically quantifies MIA unreliability for the first time, revealing up to 35% AUC discrepancy among state-of-the-art attacks. To address this, we propose a dual-dimension evaluation framework—Coverage–Stability—that jointly measures attack coverage (i.e., applicability across models and datasets) and result stability (i.e., consistency across perturbations and random seeds). Furthermore, we design three ensemble MIA methods integrating multiple heterogeneous strategies, achieving an average AUC improvement of 8.2% across eight benchmark datasets. Our approach significantly enhances both robustness and interpretability of privacy risk assessment. This work establishes a more credible, reproducible, and principled benchmarking paradigm for evaluating model privacy leakage.

Technology Category

• Machine Learning: Privacy
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Computer Vision: Bias, Fairness & Privacy

Application Category

• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Membership inference attacks (MIAs) pose a significant threat to the privacy of machine learning models and are widely used as tools for privacy assessment, auditing, and machine unlearning. While prior MIA research has primarily focused on performance metrics such as AUC, accuracy, and TPR@low FPR - either by developing new methods to enhance these metrics or using them to evaluate privacy solutions - we found that it overlooks the disparities among different attacks. These disparities, both between distinct attack methods and between multiple instantiations of the same method, have crucial implications for the reliability and completeness of MIAs as privacy evaluation tools. In this paper, we systematically investigate these disparities through a novel framework based on coverage and stability analysis. Extensive experiments reveal significant disparities in MIAs, their potential causes, and their broader implications for privacy evaluation. To address these challenges, we propose an ensemble framework with three distinct strategies to harness the strengths of state-of-the-art MIAs while accounting for their disparities. This framework not only enables the construction of more powerful attacks but also provides a more robust and comprehensive methodology for privacy evaluation.

Problem

• Assess reliability and disparity in membership inference attacks
• Investigate disparities among different MIA methods and instantiations
• Propose ensemble framework to improve privacy evaluation robustness

Innovation

• Systematic investigation of MIA disparities
• Novel framework with coverage and stability analysis
• Ensemble framework with three distinct strategies