DriftScope: Measuring The Hidden Effects of Diffusion Model Adaptation

2026-06-30

AI Summary

This work addresses a critical yet overlooked issue in diffusion models: when adapting to new tasks such as concept customization or unlearning, these models implicitly degrade semantically unrelated visual concepts—a phenomenon invisible to conventional metrics like FID or KID. The study systematically uncovers this latent concept drift and introduces DriftScope, a prompt-level diagnostic tool that requires neither ground-truth data nor internal model access. By integrating sparse autoencoders, zero-shot classification, and soft prompt optimization, DriftScope enables token-level attribution of drift and ranks concepts by their susceptibility to degradation. Experiments demonstrate substantial performance drops, with zero-shot accuracy for specific categories declining by up to 18.9 points, thereby validating both the existence of implicit concept drift and the efficacy and interpretability of the proposed diagnostic framework.

Abstract

Adapting pre-trained text-to-image diffusion models, whether to learn new visual concepts or erase unwanted ones, is routinely evaluated on its intended effects alone. We argue this framing is incomplete. Through sparse autoencoder analysis and zero-shot classification, we demonstrate that adaptation systematically damages semantically unrelated concepts in ways that aggregate metrics structurally cannot surface: when damage is severe enough for FID and KID to respond, the model is already nearly unusable; when the model remains functional, FID and KID stay flat while specific classes silently suffer worst-case zero-shot accuracy drops of up to 18.9 points and concept-level distributions shift dramatically. This pattern appears at both ends of the adaptation spectrum (concept customization and concept unlearning), suggesting it is a systematic consequence of weight-level modification rather than an artifact of any particular method. To surface this hidden drift before deployment, we introduce DriftScope, a prompt-level diagnostic tool that takes any two model checkpoints and returns a ranked list of tokens whose visual concepts have shifted most between them. DriftScope optimizes a soft prompt to attribute drift at the token level without requiring access to real data or model internals. The result is an interpretable, concept-level audit that aggregate evaluation cannot provide.

Problem

• diffusion model adaptation
• hidden drift
• concept damage
• model evaluation
• semantic shift

Innovation

• DriftScope
• diffusion model adaptation
• concept drift
• soft prompt optimization
• zero-shot classification