DuMo: Dual Encoder Modulation Network for Precise Concept Erasure

2025-01-02

AI Summary

This work addresses the secure erasure of undesirable concepts—such as NSFW content and copyright-sensitive elements—in text-to-image diffusion models. We propose Eraser with PRior Knowledge (EPR), a method that performs high-frequency, detail-level concept removal within the skip-connection feature space while keeping the U-Net backbone frozen. To enable precise control, we introduce Time-Layer MOdulation (TLMO), a novel mechanism that adaptively modulates erasure strength across timesteps and network layers. EPR further employs a dual-encoder collaborative architecture to preserve non-target semantics and structural fidelity. Evaluated on three challenging tasks—explicit content removal, cartoon concept elimination, and artistic style suppression—EPR achieves state-of-the-art performance, significantly outperforming existing approaches. The implementation is publicly available.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: Generation

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The exceptional generative capability of text-to-image models has raised substantial safety concerns regarding the generation of Not-Safe-For-Work (NSFW) content and potential copyright infringement. To address these concerns, previous methods safeguard the models by eliminating inappropriate concepts. Nonetheless, these models alter the parameters of the backbone network and exert considerable influences on the structural (low-frequency) components of the image, which undermines the model's ability to retain non-target concepts. In this work, we propose our Dual encoder Modulation network (DuMo), which achieves precise erasure of inappropriate target concepts with minimum impairment to non-target concepts. In contrast to previous methods, DuMo employs the Eraser with PRior Knowledge (EPR) module which modifies the skip connection features of the U-NET and primarily achieves concept erasure on details (high-frequency) components of the image. To minimize the damage to non-target concepts during erasure, the parameters of the backbone U-NET are frozen and the prior knowledge from the original skip connection features is introduced to the erasure process. Meanwhile, the phenomenon is observed that distinct erasing preferences for the image structure and details are demonstrated by the EPR at different timesteps and layers. Therefore, we adopt a novel Time-Layer MOdulation process (TLMO) that adjusts the erasure scale of EPR module's outputs across different layers and timesteps, automatically balancing the erasure effects and model's generative ability. Our method achieves state-of-the-art performance on Explicit Content Erasure, Cartoon Concept Removal and Artistic Style Erasure, clearly outperforming alternative methods. Code is available at https://github.com/Maplebb/DuMo

Problem

• Text-to-Image Generation
• Content Moderation
• Image Quality Preservation

Innovation

• DuMo Network
• EPR Module
• TLMO Mechanism