Hyperparameters in Score-Based Membership Inference Attacks

2025-02-10

AI Summary

This work investigates the feasibility of fractional membership inference attacks (MIAs) when target model hyperparameters are unknown—particularly in transfer learning and differential privacy (DP) settings. We propose an output-distribution-matching method to automatically select hyperparameters for shadow models without access to the target model’s hyperparameters, enabling construction of high-fidelity surrogate models. We provide the first theoretical proof—and empirically validate—that this approach achieves MIA performance nearly indistinguishable from baseline attacks assuming known hyperparameters. Furthermore, we find that hyperparameter optimization (HPO) using training data yields no significant improvement in MIA success under DP-aware transfer learning. Our core contribution is the relaxation of the hyperparameter-dependency assumption, establishing a new, more robust and practical paradigm for MIAs in realistic settings, supported by both theoretical analysis and empirical evidence.

Technology Category

• Machine Learning: Privacy
• Search and Optimization: Learning to Search
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Membership Inference Attacks (MIAs) have emerged as a valuable framework for evaluating privacy leakage by machine learning models. Score-based MIAs are distinguished, in particular, by their ability to exploit the confidence scores that the model generates for particular inputs. Existing score-based MIAs implicitly assume that the adversary has access to the target model's hyperparameters, which can be used to train the shadow models for the attack. In this work, we demonstrate that the knowledge of target hyperparameters is not a prerequisite for MIA in the transfer learning setting. Based on this, we propose a novel approach to select the hyperparameters for training the shadow models for MIA when the attacker has no prior knowledge about them by matching the output distributions of target and shadow models. We demonstrate that using the new approach yields hyperparameters that lead to an attack near indistinguishable in performance from an attack that uses target hyperparameters to train the shadow models. Furthermore, we study the empirical privacy risk of unaccounted use of training data for hyperparameter optimization (HPO) in differentially private (DP) transfer learning. We find no statistically significant evidence that performing HPO using training data would increase vulnerability to MIA.

Problem

• Hyperparameter selection for shadow models
• Privacy risk in transfer learning
• Membership Inference Attack effectiveness

Innovation

• Hyperparameter selection via output distribution matching
• MIA without target hyperparameters knowledge
• HPO impact on DP transfer learning privacy