Mask-based Membership Inference Attacks for Retrieval-Augmented Generation

2024-10-26

arXiv.org

AI Summary

In Retrieval-Augmented Generation (RAG) systems, knowledge base document membership is vulnerable to inference, yet existing Membership Inference Attacks (MIAs) suffer from low reliability and poor interpretability. Method: We propose the first fine-grained MIA framework grounded in text masking. It systematically masks candidate document segments and leverages the RAG system’s contextual completion capability to determine whether each masked segment originates from the internal knowledge base—enabling document-level, fine-grained membership inference. Crucially, the masking strategy is co-designed with RAG’s intrinsic retrieval-and-generation mechanism to suppress interference from large language model priors and irrelevant retrieved passages. Results: Extensive experiments across multiple datasets demonstrate that our method achieves an average 23.6% improvement in attack accuracy, significantly reduces false positive rate, exhibits strong robustness, and maintains high specificity and full interpretability.

Technology Category

• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Data Mining & Knowledge Management: Knowledge Acquisition from the Web
• Knowledge Representation and Reasoning: Argumentation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Retrieval-Augmented Generation (RAG) has been an effective approach to mitigate hallucinations in large language models (LLMs) by incorporating up-to-date and domain-specific knowledge. Recently, there has been a trend of storing up-to-date or copyrighted data in RAG knowledge databases instead of using it for LLM training. This practice has raised concerns about Membership Inference Attacks (MIAs), which aim to detect if a specific target document is stored in the RAG system's knowledge database so as to protect the rights of data producers. While research has focused on enhancing the trustworthiness of RAG systems, existing MIAs for RAG systems remain largely insufficient. Previous work either relies solely on the RAG system's judgment or is easily influenced by other documents or the LLM's internal knowledge, which is unreliable and lacks explainability. To address these limitations, we propose a Mask-Based Membership Inference Attacks (MBA) framework. Our framework first employs a masking algorithm that effectively masks a certain number of words in the target document. The masked text is then used to prompt the RAG system, and the RAG system is required to predict the mask values. If the target document appears in the knowledge database, the masked text will retrieve the complete target document as context, allowing for accurate mask prediction. Finally, we adopt a simple yet effective threshold-based method to infer the membership of target document by analyzing the accuracy of mask prediction. Our mask-based approach is more document-specific, making the RAG system's generation less susceptible to distractions from other documents or the LLM's internal knowledge. Extensive experiments demonstrate the effectiveness of our approach compared to existing baseline models.

Problem

• Detects specific document in RAG databases.
• Improves accuracy of membership inference attacks.
• Reduces influence from unrelated documents and LLMs.

Innovation

• Mask-Based Membership Inference Attacks
• Masking algorithm for document specificity
• Threshold-based method for membership inference