MANCE: Manifold Aware Concept Erasure

2026-07-04

AI Summary

This work addresses the challenge of removing specific concepts from representations while preserving unrelated information. It introduces the manifold constraint hypothesis, positing that natural representations reside on low-dimensional structured manifolds and that concept erasure should operate within this manifold to minimize collateral information loss. Building on this insight, the authors propose MANCE—the first method to explicitly incorporate manifold constraints into concept erasure—along with enhanced variants MANCE+ and MANCE++. These approaches integrate manifold projection, iterative refinement, closed-form preprocessing, and classifier-guided intervention. Evaluated across 119 textual and visual benchmarks, the MANCE family consistently outperforms existing methods, with MANCE++ achieving state-of-the-art performance in nonlinear erasure tasks, thereby validating the efficacy of manifold-constrained concept removal.

Abstract

Concept erasure aims to remove a target concept from a representation while preserving the other information encoded in it. This is difficult because representations encode many concepts that are often correlated with the erasure target, so removing the target risks damaging them. We propose the Manifold Constraint Hypothesis (MCH): if natural representations concentrate on a structured, lower-dimensional manifold, then interventions should be constrained to that manifold and better preserve other information encoded in the representation during interventions. We instantiate MCH in a new concept erasure method: MANifold aware Concept Erasure (MANCE). MANCE performs iterative updates to the representations using signals from a classifier that predicts a target concept. We estimate the manifold using representations obtained from natural inputs, and then we project the concept removal update to the estimated manifold. We perform extensive evaluation on 119 settings spanning text and vision, including 13 language models, three NLP concepts, and 40 CelebA-CLIP attributes. Employing MANCE on top of previous methods shows consistent improved leakage results. We also introduce MANCE+ and MANCE++, which prepend a closed-form erasure algorithm before employing MANCE, achieving better leakage--surgicality tradeoffs relative to matched full-space updates. MANCE++, our best method, achieves state-of-the-art results on nonlinear concept erasure. These results support MCH in the erasure setting: interventions should be constrained to the natural representation manifold.

Problem

• concept erasure
• representation manifold
• information preservation
• target concept removal
• correlated concepts

Innovation

• Manifold Constraint Hypothesis
• Concept Erasure
• Representation Manifold
• MANCE
• Surgicality