ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models

2026-05-15

AI Summary

Large language models often retain sensitive information during training, posing significant privacy and security risks. This work reframes machine unlearning as a knowledge remapping problem and introduces a closed-form multiplicative parameter update mechanism that enforces representational orthogonality. By doing so, it effectively maps sensitive inputs to a neutral state and precisely erases their original representations using only a few samples. The approach admits a gradient-based variant that naturally extends to multiple samples, enabling efficient and targeted forgetting. Extensive evaluation demonstrates that the method substantially outperforms existing baselines while preserving the model’s overall performance and generalization capabilities.

Abstract

Large language models inevitably retain sensitive information, defined as inputs that may induce harmful generations, due to training on massive web corpora, raising concerns for privacy and safety. Existing machine unlearning methods primarily rely on retraining or aggressive fine-tuning, which are either computationally expensive or prone to degrading related knowledge and overall model utility. In this work, we reformulate machine unlearning as a precise knowledge re-mapping problem via model editing. We propose ZeroUnlearn, a few-shot unlearning framework. It overwrites sensitive inputs by mapping them to a neutral target state and removing their original representations. ZeroUnlearn enforces representational orthogonality through a multiplicative parameter update with a closed-form solution, enabling efficient and targeted unlearning. We further extend ZeroUnlearn to a gradient-based variant for multi-sample unlearning. Experiments demonstrate that our approach outperforms existing baselines while preserving general model utility. Our code is available at the github: https://github.com/XMUDeepLIT/ZeroUnlearn.

Problem

• knowledge unlearning
• large language models
• sensitive information
• privacy
• model utility

Innovation

• knowledge unlearning
• model editing
• representational orthogonality
• few-shot learning
• large language models