Boundary-targeted Membership Inference Attacks on Safety Classifiers

2026-05-21

AI Summary

This work addresses the privacy risks of safety classifiers trained on sensitive data involving self-harm and mental health, which are vulnerable to membership inference attacks that can leak user information. The authors propose a boundary-targeting strategy that, for the first time, integrates low-confidence samples with membership inference attacks to expose how models rely on memorization rather than generalization when handling ambiguous inputs, thereby amplifying membership signals. Experimental results demonstrate that, at a 5% false positive rate, the method successfully recovers 19% of conversations labeled as indicating emotional distress—achieving 3.5 times the performance of the current state-of-the-art approach. The study further validates that adding noise effectively mitigates this privacy vulnerability.

Abstract

Safety classifiers are essential safeguards within generative AI systems, filtering harmful content or identifying at-risk users when interacting with large language models. Despite their necessity, these models are trained on sensitive datasets including discussions of self-harm and mental health, raising important, yet poorly understood, privacy concerns. Membership inference attacks (MIAs) allow adversaries to infer membership of examples used to train models. In this work, we hypothesize that identifying the examples on which the classifier is least confident are informative for an adversary to infer membership. This reflects a localized failure of generalization, where the model relies on memorization to resolve ambiguity in the training set. To investigate this, we introduce a new boundary-targeted selection strategy that identifies low confidence examples that amplify the signal of an examples membership within a training set. Our experimental results show that an adversary can recover 19\% of the conversations a safety classifier flagged as indicating user distress, at a 5\% false-positive rate, on a classifier fine-tuned for detecting a user who may require emotional support. This is $3.5$ times more than attacking using state-of-the-art MIA methods alone. Finally, we characterize the boundary laying examples and show that content-based filtering is ineffective for protection, and existing noise strategies can effectively mitigate susceptibility of these examples.

Problem

• Membership Inference Attacks
• Safety Classifiers
• Privacy
• Boundary-targeted Attacks
• Sensitive Data

Innovation

• membership inference attacks
• safety classifiers
• boundary-targeted selection
• low-confidence examples
• privacy leakage