Unveiling Entity-Level Unlearning for Large Language Models: A Comprehensive Analysis

2024-06-22

International Conference on Computational Linguistics

AI Summary

This paper addresses the underexplored problem of entity-level knowledge forgetting in large language models (LLMs)—the complete erasure of all associations related to a specific entity (e.g., a person or organization)—motivated by real-world requirements such as copyright compliance, distinct from prevalent instance-level forgetting. We propose the first systematic formal definition and empirical analysis framework for this task. Through controlled experiments and attribution analysis, we identify two key determinants of forgetting efficacy: the extent of knowledge coverage about the target entity in the model and the size of the forgetting set; notably, entities introduced via fine-tuning are more susceptible to forgetting. Benchmarking state-of-the-art unlearning algorithms reveals their consistent failure on entity-level forgetting. Our work fills a critical gap in the literature, providing both theoretical foundations and empirically grounded guidelines for developing efficient, controllable entity forgetting mechanisms.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language model unlearning has garnered increasing attention due to its potential to address security and privacy concerns, leading to extensive research in the field. However, much of this research has concentrated on instance-level unlearning, specifically targeting the removal of predefined instances containing sensitive content. This focus has left a significant gap in the exploration of full entity-level unlearning, which is critical in real-world scenarios such as copyright protection. To this end, we propose a novel task of Entity-level unlearning, which aims to erase entity-related knowledge from the target model completely. To thoroughly investigate this task, we systematically evaluate trending unlearning algorithms, revealing that current methods struggle to achieve effective entity-level unlearning. Then, we further explore the factors that influence the performance of the unlearning algorithms, identifying that knowledge coverage and the size of the forget set play pivotal roles. Notably, our analysis also uncovers that entities introduced through fine-tuning are more vulnerable to unlearning than pre-trained entities. These findings collectively offer valuable insights for advancing entity-level unlearning for LLMs.

Problem

• Addressing the gap in entity-level unlearning for LLMs
• Evaluating current unlearning methods' failure in entity removal
• Identifying key factors affecting entity unlearning performance

Innovation

• Proposes entity-level unlearning for LLMs
• Evaluates trending unlearning algorithms
• Identifies key factors influencing unlearning performance