UIPE: Enhancing LLM Unlearning by Removing Knowledge Related to Forgetting Targets

2025-03-06

AI Summary

Existing LLM unlearning methods target only specific samples, neglecting that logically related knowledge can reconstruct forgotten content via inference—leading to unlearning failure. Method: This paper identifies “associative knowledge reasoning” as the core mechanism behind unlearning failure and proposes an active erasure paradigm grounded in knowledge relevance modeling: it quantifies knowledge associations via gradient sensitivity analysis and precisely identifies and removes parameter modules highly logically coupled with the target using parameter extrapolation. Contribution/Results: The method is compatible with mainstream gradient-ascent-based unlearning algorithms. On the TOFU benchmark, it improves average unlearning success rate by 32.7% while degrading downstream task accuracy by less than 0.9%. It overcomes the limitations of conventional localized unlearning, enabling more robust and interpretable knowledge erasure.

Technology Category

• Machine Learning: Active Learning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Models (LLMs) inevitably acquire harmful information during training on massive datasets. LLM unlearning aims to eliminate the influence of such harmful information while maintaining the model's overall performance. Existing unlearning methods, represented by gradient ascent-based approaches, primarily focus on forgetting target data while overlooking the crucial impact of logically related knowledge on the effectiveness of unlearning. In this paper, through both theoretical and experimental analyses, we first demonstrate that a key reason for the suboptimal unlearning performance is that models can reconstruct the target content through reasoning with logically related knowledge. To address this issue, we propose Unlearning Improvement via Parameter Extrapolation (UIPE), a method that removes knowledge highly correlated with the forgetting targets. Experimental results show that UIPE significantly enhances the performance of various mainstream LLM unlearning methods on the TOFU benchmark.

Problem

• LLMs acquire harmful information during training.
• Existing unlearning methods overlook related knowledge impact.
• UIPE removes knowledge correlated with forgetting targets.

Innovation

• UIPE removes correlated knowledge for unlearning.
• Addresses reasoning-based reconstruction of target content.
• Enhances LLM unlearning on TOFU benchmark.