PT-Mark: Invisible Watermarking for Text-to-image Diffusion Models via Semantic-aware Pivotal Tuning

2025-04-15

AI Summary

To address semantic distortion caused by watermark embedding in text-to-image diffusion models—a critical challenge in copyright protection—this paper proposes PT-Mark, a semantic-aware invisible watermarking method. Its core innovation is the first-of-its-kind “semantic-aware critical-path tuning” mechanism, which jointly achieves semantic alignment and watermark preservation throughout the full denoising process via spatial-saliency-guided dual-branch fine-tuning. By integrating latent-space trajectory analysis, dynamic text-embedding adaptation, and multi-step pivotal tuning, PT-Mark ensures watermark embedding without perturbing the generative distribution. Experiments demonstrate that PT-Mark improves semantic fidelity by 10% in SSIM, PSNR, and LPIPS metrics; achieves state-of-the-art robustness against real-world distortions (e.g., compression, resizing, cropping); and accelerates inference by 4× compared to baseline methods.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Watermarking for diffusion images has drawn considerable attention due to the widespread use of text-to-image diffusion models and the increasing need for their copyright protection. Recently, advanced watermarking techniques, such as Tree Ring, integrate watermarks by embedding traceable patterns (e.g., Rings) into the latent distribution during the diffusion process. Such methods disrupt the original semantics of the generated images due to the inevitable distribution shift caused by the watermarks, thereby limiting their practicality, particularly in digital art creation. In this work, we present Semantic-aware Pivotal Tuning Watermarks (PT-Mark), a novel invisible watermarking method that preserves both the semantics of diffusion images and the traceability of the watermark. PT-Mark preserves the original semantics of the watermarked image by gradually aligning the generation trajectory with the original (pivotal) trajectory while maintaining the traceable watermarks during whole diffusion denoising process. To achieve this, we first compute the salient regions of the watermark at each diffusion denoising step as a spatial prior to identify areas that can be aligned without disrupting the watermark pattern. Guided by the region, we then introduce an additional pivotal tuning branch that optimizes the text embedding to align the semantics while preserving the watermarks. Extensive evaluations demonstrate that PT-Mark can preserve the original semantics of the diffusion images while integrating robust watermarks. It achieves a 10% improvement in the performance of semantic preservation (i.e., SSIM, PSNR, and LPIPS) compared to state-of-the-art watermarking methods, while also showing comparable robustness against real-world perturbations and four times greater efficiency.

Problem

• Preserve image semantics while embedding watermarks
• Avoid distribution shift caused by watermark patterns
• Enhance watermark robustness and traceability

Innovation

• Semantic-aware Pivotal Tuning for watermarking
• Aligns generation trajectory with original semantics
• Optimizes text embedding to preserve watermarks