Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Data

2024-09-29

arXiv.org

AI Summary

This paper addresses the legal evidentiary challenge of proving training data provenance for foundation models. We argue that membership inference attacks (MIAs) are fundamentally unsuitable for judicial settings due to their inability to construct a valid null hypothesis distribution and guarantee low false-positive rates—critical requirements for legal admissibility. Through rigorous theoretical analysis grounded in statistical hypothesis testing, we formally demonstrate that existing MIAs lack statistical reliability as legally admissible evidence. To overcome this limitation, we propose two verifiable alternatives: (1) a deterministic proof paradigm based on data extraction attacks, and (2) a statistically grounded verification paradigm integrating controlled canary data with enhanced MIAs. Both approaches provably achieve bounded false-positive rates, establishing the first evidence-generation framework for model training provenance that is both statistically rigorous and practically deployable in legal contexts.

Technology Category

• Machine Learning: Other Foundations of Machine Learning
• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Security and Privacy: Data transparency and provenance
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

We consider the problem of a training data proof, where a data creator or owner wants to demonstrate to a third party that some machine learning model was trained on their data. Training data proofs play a key role in recent lawsuits against foundation models trained on web-scale data. Many prior works suggest to instantiate training data proofs using membership inference attacks. We argue that this approach is fundamentally unsound: to provide convincing evidence, the data creator needs to demonstrate that their attack has a low false positive rate, i.e., that the attack's output is unlikely under the null hypothesis that the model was not trained on the target data. Yet, sampling from this null hypothesis is impossible, as we do not know the exact contents of the training set, nor can we (efficiently) retrain a large foundation model. We conclude by offering two paths forward, by showing that data extraction attacks and membership inference on special canary data can be used to create sound training data proofs.

Problem

• Membership inference attacks cannot reliably prove model training data.
• Demonstrating low false positive rates in attacks is fundamentally unsound.
• Data extraction and canary data attacks offer alternative proof methods.

Innovation

• Membership inference attacks for data proof
• Data extraction attacks for evidence creation
• Canary data for membership inference validation