SolidMark: Evaluating Image Memorization in Generative Models

2025-03-01

AI Summary

Existing image memorization evaluation metrics suffer from dataset bias and lack the capability to perform per-image, pixel-level memorization assessment. To address this, we propose SolidMark—the first interpretable, fine-grained image memorization evaluation method. SolidMark integrates feature similarity comparison, multi-scale perceptual hashing, and statistical significance testing to generate pixel-level memorization scores for each synthesized image. Crucially, it operates without reliance on the original training dataset, thereby enhancing robustness and reproducibility. Leveraging SolidMark, we systematically re-evaluate state-of-the-art data-unlearning techniques and publicly release enhanced models and full implementation code. Experiments demonstrate that SolidMark achieves superior detection accuracy and discriminative power across cross-dataset and cross-model settings. As a result, it establishes a reliable, general-purpose benchmark for safety assessment of generative models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Recent works have shown that diffusion models are able to memorize training images and emit them at generation time. However, the metrics used to evaluate memorization and its mitigation techniques suffer from dataset-dependent biases and struggle to detect whether a given specific image has been memorized or not. This paper begins with a comprehensive exploration of issues surrounding memorization metrics in diffusion models. Then, to mitigate these issues, we introduce $ m style{font-variant: small-caps}{SolidMark}$, a novel evaluation method that provides a per-image memorization score. We then re-evaluate existing memorization mitigation techniques. We also show that $ m style{font-variant: small-caps}{SolidMark}$ is capable of evaluating fine-grained pixel-level memorization. Finally, we release a variety of models based on $ m style{font-variant: small-caps}{SolidMark}$ to facilitate further research for understanding memorization phenomena in generative models. All of our code is available at https://github.com/NickyDCFP/SolidMark.

Problem

• Evaluates image memorization in diffusion models.
• Introduces SolidMark for per-image memorization scoring.
• Assesses pixel-level memorization and mitigation techniques.

Innovation

• Introduces SolidMark for per-image memorization scoring
• Evaluates pixel-level memorization in diffusion models
• Releases models to study generative model memorization