Who Stole Your Data? A Method for Detecting Unauthorized RAG Theft

2025-10-08

AI Summary

This work addresses the growing threat of unauthorized data exfiltration in Retrieval-Augmented Generation (RAG) systems by proposing the first comprehensive watermarking-based plagiarism detection framework tailored for RAG. Methodologically, it introduces (1) RPD—the first cross-domain benchmark dataset for RAG plagiarism detection; (2) a dual-layer watermarking scheme integrating semantic and lexical watermarks to ensure both robustness against tampering and imperceptibility; and (3) a question-answering–driven detection architecture grounded in statistical hypothesis testing, employing an evidence-accumulation “detective-style” inference process that significantly enhances resilience against adversarial evasion (e.g., prompt perturbation, retrieval parameter manipulation). Experiments demonstrate high detection accuracy and strong generalization across diverse RAG configurations and attack scenarios, establishing a practical, deployable solution for intellectual property protection in RAG systems.

Technology Category

• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Retrieval-augmented generation (RAG) enhances Large Language Models (LLMs) by mitigating hallucinations and outdated information issues, yet simultaneously facilitates unauthorized data appropriation at scale. This paper addresses this challenge through two key contributions. First, we introduce RPD, a novel dataset specifically designed for RAG plagiarism detection that encompasses diverse professional domains and writing styles, overcoming limitations in existing resources. Second, we develop a dual-layered watermarking system that embeds protection at both semantic and lexical levels, complemented by an interrogator-detective framework that employs statistical hypothesis testing on accumulated evidence. Extensive experimentation demonstrates our approach's effectiveness across varying query volumes, defense prompts, and retrieval parameters, while maintaining resilience against adversarial evasion techniques. This work establishes a foundational framework for intellectual property protection in retrieval-augmented AI systems.

Problem

• Detecting unauthorized data theft in retrieval-augmented generation systems
• Developing watermarking protection for semantic and lexical content
• Creating datasets and frameworks for RAG plagiarism detection

Innovation

• Introduces RPD dataset for RAG plagiarism detection
• Develops dual-layered watermarking at semantic and lexical levels
• Implements interrogator-detective framework with statistical testing