RAG-WM: An Efficient Black-Box Watermarking Approach for Retrieval-Augmented Generation of Large Language Models

2025-01-09

AI Summary

Existing RAG systems lack intellectual property (IP) protection mechanisms suitable for black-box settings; conventional watermarking techniques are vulnerable to semantic rewriting, fragment deletion, or knowledge expansion. This paper proposes the first black-box knowledge watermarking framework tailored for RAG systems. It encodes watermarks as injectable textual sequences derived from entity-relation tuples and implements watermark embedding and unauthorized usage detection via a multi-LLM collaborative architecture—comprising a watermark generator, a shadow RAG+LLM system, and a watermark discriminator—without requiring internal model access. By integrating entity-relation-driven encoding and adversarial robustness enhancement, the framework achieves >96% detection accuracy across four mainstream LLMs and five real-world tasks. It demonstrates strong robustness against semantic rewriting, fragment deletion, and knowledge injection/expansion attacks, while effectively evading existing watermark detection methods.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Adversarial Attacks & Robustness
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

In recent years, tremendous success has been witnessed in Retrieval-Augmented Generation (RAG), widely used to enhance Large Language Models (LLMs) in domain-specific, knowledge-intensive, and privacy-sensitive tasks. However, attackers may steal those valuable RAGs and deploy or commercialize them, making it essential to detect Intellectual Property (IP) infringement. Most existing ownership protection solutions, such as watermarks, are designed for relational databases and texts. They cannot be directly applied to RAGs because relational database watermarks require white-box access to detect IP infringement, which is unrealistic for the knowledge base in RAGs. Meanwhile, post-processing by the adversary's deployed LLMs typically destructs text watermark information. To address those problems, we propose a novel black-box"knowledge watermark"approach, named RAG-WM, to detect IP infringement of RAGs. RAG-WM uses a multi-LLM interaction framework, comprising a Watermark Generator, Shadow LLM&RAG, and Watermark Discriminator, to create watermark texts based on watermark entity-relationship tuples and inject them into the target RAG. We evaluate RAG-WM across three domain-specific and two privacy-sensitive tasks on four benchmark LLMs. Experimental results show that RAG-WM effectively detects the stolen RAGs in various deployed LLMs. Furthermore, RAG-WM is robust against paraphrasing, unrelated content removal, knowledge insertion, and knowledge expansion attacks. Lastly, RAG-WM can also evade watermark detection approaches, highlighting its promising application in detecting IP infringement of RAG systems.

Problem

• Intellectual Property Protection
• Watermarking Techniques
• Large Language Models

Innovation

• RAG-WM
• Knowledge Watermarking
• RAG Plagiarism Detection