Forget-It-All: Multi-Concept Machine Unlearning via Concept-Aware Neuron Masking

2026-01-07

arXiv.org

AI Summary

This work addresses the challenge of efficiently and effectively unlearning multiple concepts—such as copyrighted or sensitive content—from text-to-image diffusion models without retraining. The authors propose a plug-and-play, training-free framework that identifies neurons associated with target concepts through contrastive concept saliency analysis and constructs a unified multi-concept mask by integrating spatial and temporal information for precise neuron pruning. By employing a neuron mask fusion strategy, the method achieves low hyperparameter sensitivity and eliminates the need for additional training, significantly improving unlearning performance across three benchmark forgetting tasks while preserving the semantic fidelity and visual quality of generated images.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The widespread adoption of text-to-image (T2I) diffusion models has raised concerns about their potential to generate copyrighted, inappropriate, or sensitive imagery learned from massive training corpora. As a practical solution, machine unlearning aims to selectively erase unwanted concepts from a pre-trained model without retraining from scratch. While most existing methods are effective for single-concept unlearning, they often struggle in real-world scenarios that require removing multiple concepts, since extending them to this setting is both non-trivial and problematic, causing significant challenges in unlearning effectiveness, generation quality, and sensitivity to hyperparameters and datasets. In this paper, we take a unique perspective on multi-concept unlearning by leveraging model sparsity and propose the Forget It All (FIA) framework. FIA first introduces Contrastive Concept Saliency to quantify each weight connection's contribution to a target concept. It then identifies Concept-Sensitive Neurons by combining temporal and spatial information, ensuring that only neurons consistently responsive to the target concept are selected. Finally, FIA constructs masks from the identified neurons and fuses them into a unified multi-concept mask, where Concept-Agnostic Neurons that broadly support general content generation are preserved while concept-specific neurons are pruned to remove the targets. FIA is training-free and requires only minimal hyperparameter tuning for new tasks, thereby promoting a plug-and-play paradigm. Extensive experiments across three distinct unlearning tasks demonstrate that FIA achieves more reliable multi-concept unlearning, improving forgetting effectiveness while maintaining semantic fidelity and image quality.

Problem

• machine unlearning
• multi-concept forgetting
• text-to-image diffusion models
• concept removal
• model editing

Innovation

• machine unlearning
• concept-aware neuron masking
• multi-concept forgetting
• diffusion models
• model sparsity