An Efficient Watermarking Method for Latent Diffusion Models via Low-Rank Adaptation

2024-10-26

arXiv.org

AI Summary

To address the low embedding efficiency, degradation of generation quality, and insufficient robustness in watermarking latent diffusion models (LDMs), this paper introduces Low-Rank Adaptation (LoRA) to LDM watermarking for the first time, proposing a lightweight, backbone-weight-free watermarking method. Specifically, trainable low-rank matrices are injected into critical layers of the U-Net architecture, and a dynamically weighted multi-task loss function is designed to jointly optimize generation fidelity and watermark robustness. Experiments demonstrate that the method achieves rapid watermark embedding, preserves generation quality without degradation (FID increase ≤ 0.3), attains a bit error rate below 0.5%, and achieves zero false-negative detection (FNR = 0). It significantly outperforms existing fine-tuning-based watermarking approaches in efficiency, fidelity, and robustness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The rapid proliferation of deep neural networks (DNNs) is driving a surge in model watermarking technologies, as the trained deep models themselves serve as intellectual properties. The core of existing model watermarking techniques involves modifying or tuning the models' weights. However, with the emergence of increasingly complex models, ensuring the efficiency of watermarking process is essential to manage the growing computational demands. Prioritizing efficiency not only optimizes resource utilization, making the watermarking process more applicable, but also minimizes potential impacts on model performance. In this letter, we propose an efficient watermarking method for latent diffusion models (LDMs) which is based on Low-Rank Adaptation (LoRA). We specifically choose to add trainable low-rank matrices to the existing weight matrices of the models to embed watermark, while keeping the original weights frozen. Moreover, we also propose a dynamic loss weight tuning algorithm to balance the generative task with the watermark embedding task, ensuring that the model can be watermarked with a limited impact on the quality of the generated images. Experimental results show that the proposed method ensures fast watermark embedding and maintains a very low bit error rate of the watermark, a high-quality of the generated image, and a zero false negative rate (FNR) for verification.

Problem

• Efficiently embeds watermarks in large latent diffusion models
• Balances generative quality and watermark fidelity dynamically
• Preserves model integrity while maintaining robustness across datasets

Innovation

• Uses Low-Rank Adaptation for efficient watermark embedding
• Implements dynamic loss weighting to balance objectives
• Preserves original model weights while embedding watermark