Ward: Provable RAG Dataset Inference via LLM Watermarks

2024-10-04

arXiv.org

AI Summary

This work addresses the challenge of detecting unauthorized use of external data in Retrieval-Augmented Generation (RAG) systems. We formally define the black-box RAG Dataset Inference (RAG-DI) problem—the first such formalization—and construct the first practical, application-oriented RAG-DI benchmark to bridge the gap in real-world evaluation. Methodologically, we propose Ward, a watermarking framework leveraging LLM-generated watermarks, integrating statistical hypothesis testing with reverse-engineering of RAG system behavior to provide provable detection guarantees. Ward achieves high accuracy and strong robustness against noise, model updates, and other perturbations, while maintaining low query overhead. Extensive experiments demonstrate that Ward consistently outperforms existing baselines across accuracy, efficiency, and robustness—establishing the first systematic methodology for data copyright protection in RAG environments.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

RAG enables LLMs to easily incorporate external data, raising concerns for data owners regarding unauthorized usage of their content. The challenge of detecting such unauthorized usage remains underexplored, with datasets and methods from adjacent fields being ill-suited for its study. We take several steps to bridge this gap. First, we formalize this problem as (black-box) RAG Dataset Inference (RAG-DI). We then introduce a novel dataset designed for realistic benchmarking of RAG-DI methods, alongside a set of baselines. Finally, we propose Ward, a method for RAG-DI based on LLM watermarks that equips data owners with rigorous statistical guarantees regarding their dataset's misuse in RAG corpora. Ward consistently outperforms all baselines, achieving higher accuracy, superior query efficiency and robustness. Our work provides a foundation for future studies of RAG-DI and highlights LLM watermarks as a promising approach to this problem.

Problem

• Detecting unauthorized RAG dataset usage
• Developing RAG-DI methods with watermarks
• Enhancing dataset misuse detection accuracy

Innovation

• Introduces RAG Dataset Inference
• Develops Ward using LLM watermarks
• Ensures statistical misuse guarantees