MOLM: Mixture of LoRA Markers

2025-09-30

AI Summary

Image provenance and attribution face challenges including poor watermark robustness, vulnerability to removal, and high overhead for key updates. This paper proposes a general verifiable watermarking framework based on parameter perturbation: a binary key dynamically activates lightweight LoRA adapters without retraining, enabling instantaneous key switching; a routing mechanism selects LoRA tokens within critical layers—residual blocks and attention modules—to embed key-dependent, imperceptible information. Evaluated on Stable Diffusion and FLUX, the method demonstrates strong robustness against compression, geometric distortions, regeneration, and black-box adversarial attacks, achieves high key recovery rates, and preserves image quality and generation fidelity with negligible degradation. Key innovations include a key-driven dynamic LoRA routing mechanism and a zero-retraining paradigm for key updates.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Data transparency and provenance

Abstract

Generative models can generate photorealistic images at scale. This raises urgent concerns about the ability to detect synthetically generated images and attribute these images to specific sources. While watermarking has emerged as a possible solution, existing methods remain fragile to realistic distortions, susceptible to adaptive removal, and expensive to update when the underlying watermarking key changes. We propose a general watermarking framework that formulates the encoding problem as key-dependent perturbation of the parameters of a generative model. Within this framework, we introduce Mixture of LoRA Markers (MOLM), a routing-based instantiation in which binary keys activate lightweight LoRA adapters inside residual and attention blocks. This design avoids key-specific re-training and achieves the desired properties such as imperceptibility, fidelity, verifiability, and robustness. Experiments on Stable Diffusion and FLUX show that MOLM preserves image quality while achieving robust key recovery against distortions, compression and regeneration, averaging attacks, and black-box adversarial attacks on the extractor.

Problem

• Detecting and attributing synthetically generated images to specific sources
• Addressing fragility of watermarking methods against realistic distortions
• Reducing expensive model updates when watermarking keys change

Innovation

• Key-dependent parameter perturbation for watermarking
• Routing-based LoRA adapters in model blocks
• Robust key recovery against various attacks