WET: Overcoming Paraphrasing Vulnerabilities in Embeddings-as-a-Service with Linear Transformation Watermarks

2024-08-29

arXiv.org

AI Summary

Existing Embeddings-as-a-Service (EaaS) watermarking schemes are vulnerable to paraphrasing attacks, failing to protect service providers’ intellectual property. Method: This work first systematically characterizes the intrinsic mechanism by which paraphrasing erodes watermarks, and proposes a robust watermarking method based on invertible linear embedding-space transformations. It embeds watermarks into semantic-invariant subspaces, preserving downstream task performance (degradation < 0.5%) while ensuring paraphrasing resilience. Contribution/Results: We introduce a principled adversarial paraphrasing modeling and evaluation framework, validating our method across multiple mainstream LLM embedding services and paraphrasing models. Our approach achieves >99% watermark detection accuracy and withstands diverse paraphrasing strategies—including syntactic, lexical, and semantic variants—without watermark removal. This constitutes the first theoretically sound, empirically validated, and practically deployable copyright protection solution for EaaS.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Embeddings-as-a-Service (EaaS) is a service offered by large language model (LLM) developers to supply embeddings generated by LLMs. Previous research suggests that EaaS is prone to imitation attacks -- attacks that clone the underlying EaaS model by training another model on the queried embeddings. As a result, EaaS watermarks are introduced to protect the intellectual property of EaaS providers. In this paper, we first show that existing EaaS watermarks can be removed by paraphrasing when attackers clone the model. Subsequently, we propose a novel watermarking technique that involves linearly transforming the embeddings, and show that it is empirically and theoretically robust against paraphrasing.

Problem

• EaaS vulnerable to imitation attacks cloning models
• Existing EaaS watermarks removable by paraphrasing attacks
• Propose robust linear transformation watermark against paraphrasing

Innovation

• Linear transformation for watermarking embeddings
• Robust against paraphrasing attacks
• Protects EaaS intellectual property