Filtering Memorization from Parameter-Space in Diffusion Models

2026-05-11

AI Summary

This work addresses the risk of training data memorization in diffusion models fine-tuned with Low-Rank Adaptation (LoRA), which poses significant copyright and privacy concerns—particularly when only LoRA weights are shared, rendering existing mitigation strategies ineffective. To tackle this challenge, the authors propose Base-Anchored Filtering (BAF), a post-processing framework that operates without access to original training data or additional retraining. BAF leverages spectral decomposition to map LoRA updates into channel space and selectively preserves generalizable features while suppressing memorized components based on their alignment with the principal subspace of the pre-trained backbone model. Notably, BAF achieves memory reduction using only the LoRA weights themselves. Extensive experiments across multiple datasets and diffusion architectures demonstrate that BAF substantially mitigates memorization risks while maintaining or even enhancing generation quality.

Abstract

Low-Rank Adaptation (LoRA) has become a widely used mechanism for customizing diffusion models, enabling users to inject new visual concepts or styles through lightweight parameter updates. However, LoRAs can memorize training images, causing generated outputs to reproduce copyrighted or sensitive content. This risk is particularly concerning in LoRA-sharing ecosystems, where users distribute trained LoRAs without releasing the underlying training data. Existing approaches for mitigating memorization rely on access to the training pipeline, training data, or control over the inference process, making them difficult to apply when only the released LoRA weights are available. We propose \textbf{Base-Anchored Filtering (BAF)}, a training-free and data-free framework for post-hoc memorization mitigation in diffusion LoRAs. BAF decomposes LoRA updates into spectral channels and measures their alignment with the principal subspace of the pretrained backbone. Channels strongly aligned with this subspace are retained as generalizable adaptations, while weakly aligned channels are suppressed as potential carriers of memorized content. Experiments on multiple datasets and diffusion backbones demonstrate that BAF consistently reduces memorization while preserving or even improving generation quality. Our code is available in the supplementary material.

Problem

• memorization
• diffusion models
• LoRA
• copyright
• privacy

Innovation

• Base-Anchored Filtering
• LoRA
• memorization mitigation
• diffusion models
• training-free