Your Text Encoder Can Be An Object-Level Watermarking Controller

2025-03-15

AI Summary

To address the limitations of object-level watermarking for AI-generated images—namely, poor flexibility, weak robustness, and high model overhead—this paper proposes a lightweight, text-driven invisible watermarking method. Our approach fine-tunes only the token embedding layer of the text encoder within a latent diffusion model (LDM), enabling on-demand, spatially localized object-level watermark embedding. Innovatively, we repurpose the text encoder as a watermark controller, supporting plug-and-play cross-model deployment. By injecting watermark signals prior to diffusion denoising, we significantly enhance resilience against adversarial perturbations. Experiments demonstrate 99% bit accuracy for 48-bit watermarks, with merely ~0.001M additional parameters—reducing parameter overhead by five orders of magnitude compared to full fine-tuning. The method thus achieves a favorable trade-off among fine-grained controllability, strong robustness, and deployment efficiency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Invisible watermarking of AI-generated images can help with copyright protection, enabling detection and identification of AI-generated media. In this work, we present a novel approach to watermark images of T2I Latent Diffusion Models (LDMs). By only fine-tuning text token embeddings $W_*$, we enable watermarking in selected objects or parts of the image, offering greater flexibility compared to traditional full-image watermarking. Our method leverages the text encoder's compatibility across various LDMs, allowing plug-and-play integration for different LDMs. Moreover, introducing the watermark early in the encoding stage improves robustness to adversarial perturbations in later stages of the pipeline. Our approach achieves $99%$ bit accuracy ($48$ bits) with a $10^5 imes$ reduction in model parameters, enabling efficient watermarking.

Problem

• Enables object-level watermarking in AI-generated images
• Improves robustness against adversarial image manipulations
• Reduces model parameters significantly for efficient watermarking

Innovation

• Fine-tunes text token embeddings for watermarking
• Enables object-level watermarking in AI-generated images
• Ensures robust, plug-and-play LDM integration