Look But Don't Touch with Sparse Autoencoders for Unlearning in Diffusion Models

2026-06-30

AI Summary

This work addresses the out-of-distribution activations and visual artifacts that arise when sparse autoencoders (SAEs) are used for concept intervention in diffusion models via direct manipulation of the latent space. To mitigate these issues, the authors propose a decoupled strategy: rather than modifying SAE activations directly, the SAE serves solely as a semantic detector to localize regions corresponding to the target concept. Concept erasure is then achieved by replacing the embeddings of the identified image patches. This approach avoids direct latent-space intervention, preserving activation statistics while substantially reducing visual artifacts and improving erasure fidelity. The study further highlights a fundamental gap between concept detection and intervention, underscoring the strengths of SAEs in interpretability analysis and their limitations when employed for direct latent manipulation.

Abstract

Sparse autoencoders (SAEs) have recently been proposed as interpretable tools for concept-level manipulation, under the assumption that isolated features can serve as controllable intervention points. In this work, we systematically evaluate this assumption in the context of object erasure and steering in diffusion models. We show that while SAEs reliably detect and localize semantic concepts within diffusion model activations, direct intervention in their latent space frequently induces out-of-distribution activations, resulting in severe visual artifacts. To disentangle detection from intervention, we use SAE activations purely as semantic detectors to identify image regions containing the target object, and replace those patch embeddings with the ones that do not contain it. This detection-based replacement preserves the diffusion model's activation statistics and produces significantly cleaner erasure results than latent steering. Our findings reveal a fundamental gap between concept detection and concept intervention in diffusion models: monosemantic or sparse features are not inherently suitable as control knobs for steering. These results position SAEs as powerful interpretability tools for analyzing generative models, but highlight important limitations when used for direct manipulation, such as unlearning.

Problem

• unlearning
• diffusion models
• sparse autoencoders
• concept intervention
• visual artifacts

Innovation

• sparse autoencoders
• diffusion models
• concept erasure
• semantic detection
• unlearning