OrthoEraser: Coupled-Neuron Orthogonal Projection for Concept Erasure

2026-03-11

AI Summary

This work addresses the challenge of removing sensitive concepts from text-to-image generative models without compromising benign semantic content, a problem exacerbated by the high entanglement of features in latent space. The authors propose a disentanglement approach based on sparse autoencoders that identifies vulnerable benign features through coupled neurons and reformulates concept erasure as an orthogonal projection of sensitive vectors onto their null space. To achieve precise disentanglement at the high-dimensional feature level, the method introduces an analytical gradient-based orthogonalization strategy. Experimental results demonstrate that the proposed technique effectively eliminates harmful content while significantly outperforming existing baselines and preserving the integrity of the model’s benign semantic structure.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Text-to-image (T2I) models face significant safety risks from adversarial induction, yet current concept erasure methods often cause collateral damage to benign attributes when suppressing selected neurons entirely. This occurs because sensitive and benign semantics exhibit non-orthogonal superposition, sharing activation subspaces where their respective vectors are inherently entangled. To address this issue, we propose OrthoEraser, which leverages sparse autoencoders (SAE) to achieve high-resolution feature disentanglement and subsequently redefines erasure as an analytical orthogonalization projection that preserves the benign manifold's invariance. OrthoEraser first employs SAE to decompose dense activations and segregate sensitive neurons. It then uses coupled neuron detection to identify non-sensitive features vulnerable to intervention. The key novelty lies in an analytical gradient orthogonalization strategy that projects erasure vectors onto the null space of the coupled neurons. This orthogonally decouples the sensitive concepts from the identified critical benign subspace, effectively preserving non-sensitive semantics. Experimental results on safety demonstrate that OrthoEraser achieves high erasure precision, effectively removing harmful content while preserving the integrity of the generative manifold, and significantly outperforming SOTA baselines. This paper contains results of unsafe models.

Problem

• concept erasure
• text-to-image models
• adversarial induction
• semantic entanglement
• collateral damage

Innovation

• orthogonal projection
• concept erasure
• sparse autoencoder
• coupled neuron
• feature disentanglement