RepSelect: Robust LLM Unlearning via Representation Selectivity

2026-06-15

AI Summary

This work addresses the challenge of unlearning specific knowledge from large language models without compromising general capabilities or robustness against attacks—a limitation of existing methods, which often yield shallow and reversible forgetting. The authors propose a representation-selectivity-based deep unlearning mechanism that identifies and isolates representations exclusively associated with the target knowledge to be forgotten. By leveraging the discrepancy between retain and forget sets, the method employs PCA-guided gradient clipping to suppress dominant weight gradient components prior to parameter updates. This approach achieves the first effective disentanglement between unlearned content and general knowledge in representation space, establishing a unified framework applicable across diverse architectures including Llama 3, Qwen, Gemma, and DeepSeek. Experiments on biohazard and misuse-prone tasks show that post-unlearning relearning accuracy drops by 4–50× compared to five baselines, while near-complete resistance to few-shot prompt attacks demonstrates significantly enhanced irreversibility and robustness of forgetting.

Abstract

Making large language models (LLMs) deeply forget specific knowledge and values without sacrificing general capabilities remains a central challenge in unlearning. However, current methods are easily reversed by fine-tuning or few-shot prompting, suggesting their forgetting is only shallow. We identify the root cause. Existing methods target representations shared with both the retain set and the subspace recovered by a fine-tuning attacker, making unlearning both disruptive to general capabilities and easy to reverse. We propose RepSelect (Representation Selectivity), isolates forget-set-specific representations by collapsing top principal components of weight gradients before each update, leaving general capabilities intact while limiting what fine-tuning can recover. We evaluate across two forget categories, biohazardous knowledge and abusive tendencies, and four model families spanning dense and Mixture-of-Experts architectures (Llama 3, Qwen 3.5, Gemma 4 E4B, DeepSeek V2 Lite). Compared to five popular baselines (GradDiff, NPO, SimNPO, RMU, UNDIAL), RepSelect achieves a 4-50x larger reduction in post-relearning answer accuracy than the strongest baseline, and is near-perfectly robust to few-shot prompting attacks. Targeting selective representations is thus an important step towards deep and robust LLM forgetting.

Problem

• LLM unlearning
• representation selectivity
• robust forgetting
• knowledge removal
• model safety

Innovation

• Representation Selectivity
• LLM Unlearning
• Robust Forgetting
• Gradient Principal Components
• Fine-tuning Resistance