MIBench: A Comprehensive Benchmark for Model Inversion Attack and Defense

2024-10-07

arXiv.org

AI Summary

Existing research on model inversion attacks and defenses lacks standardized, reproducible evaluation benchmarks, leading to unfair method comparisons and unreliable defense assessments. Method: We introduce MIBench—the first comprehensive benchmark for model inversion, integrating 16 state-of-the-art attack and defense algorithms with nine standardized evaluation protocols. It supports modular deployment, multi-dimensional analysis (e.g., input resolution, model architecture, loss functions), and joint evaluation of cross-model/task transferability and robustness–accuracy trade-offs. Contribution/Results: MIBench establishes the first unified, extensible, and operationally grounded evaluation framework for model inversion. It significantly mitigates experimental inconsistency and assessment fragmentation, enabling rigorous, comparable, and reproducible privacy-security evaluations. As a result, it provides a trustworthy, standardized infrastructure for advancing privacy-preserving machine learning research.

Technology Category

• Machine Learning: Privacy
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Model Inversion (MI) attacks aim at leveraging the output information of target models to reconstruct privacy-sensitive training data, raising widespread concerns on privacy threats of Deep Neural Networks (DNNs). Unfortunately, in tandem with the rapid evolution of MI attacks, the lack of a comprehensive, aligned, and reliable benchmark has emerged as a formidable challenge. This deficiency leads to inadequate comparisons between different attack methods and inconsistent experimental setups. In this paper, we introduce the first practical benchmark for model inversion attacks and defenses to address this critical gap, which is named extit{MIBench}. This benchmark serves as an extensible and reproducible modular-based toolbox and currently integrates a total of 16 state-of-the-art attack and defense methods. Moreover, we furnish a suite of assessment tools encompassing 9 commonly used evaluation protocols to facilitate standardized and fair evaluation and analysis. Capitalizing on this foundation, we conduct extensive experiments from multiple perspectives to holistically compare and analyze the performance of various methods across different scenarios, which overcomes the misalignment issues and discrepancy prevalent in previous works. Based on the collected attack methods and defense strategies, we analyze the impact of target resolution, defense robustness, model predictive power, model architectures, transferability and loss function. Our hope is that this extit{MIBench} could provide a unified, practical and extensible toolbox and is widely utilized by researchers in the field to rigorously test and compare their novel methods, ensuring equitable evaluations and thereby propelling further advancements in the future development.

Problem

• Model Inversion attacks reconstruct sensitive data from model outputs.
• Lack of standardized benchmarks hinders attack and defense comparisons.
• MIBench provides a comprehensive framework for evaluating MI attacks and defenses.

Innovation

• Developed MIBench for model inversion benchmarking
• Integrated 19 attack and defense methods
• Established 9 standardized evaluation protocols