CheckMIABench: Firm Foundations For Membership Inference Attacks on Language Models

2026-06-15

AI Summary

Current evaluations of membership inference attacks (MIAs) against language models suffer from statistical invalidity due to distributional shifts between member and non-member data, hindering fair comparisons. This work proposes an unbiased evaluation benchmark that leverages the temporal in-distribution property observed during model training: by utilizing intermediate checkpoints of open-source large language models (e.g., Pythia, OLMo), it constructs temporally adjacent member and non-member datasets that share the same underlying distribution. We introduce Pandora_LLM, a modular and open-source MIA attack library, and conduct systematic evaluations across models ranging from 70M to 7B parameters. Our experiments reveal the true performance of various MIA methods under this unbiased setting, thereby advancing standardized and reliable assessment of membership privacy risks in language models.

Abstract

Membership inference attacks (MIAs) are a canonical way to assess a machine learning model's privacy properties. Although several attempts have been made to evaluate MIAs on language models, the extant literature has suffered numerous difficulties in constructing clean evaluations to test new techniques. In particular, subtle distribution shifts between member and non-member sets can undermine the statistical validity of MIAs; recent work has underscored this by showing that "blind" methods with no access to the underlying model can perform far better than published methods on the same benchmarks. This paper constructs a benchmark for principled evaluation of MIAs against LLMs, by leveraging the insight that training data before and after a fixed point during training are drawn from the same distribution. Therefore, all open-source models with intermediate checkpoints and public training data can be converted into MIA testbeds. We apply our framework to a half-dozen published attacks on the Pythia and OLMo family of models, from 70M to 7B parameters. To facilitate further privacy research, we open-source a modular library for designing and implementing attacks in this setting: https://github.com/safr-ai-lab/pandora_llm.

Problem

• membership inference attacks
• language models
• evaluation benchmark
• distribution shift
• privacy evaluation

Innovation

• Membership Inference Attacks
• Language Models
• Benchmarking
• Distribution Shift
• Privacy Evaluation