PreUnlearn: Auditing Collateral Knowledge Damage Before Large Language Model Unlearning

2026-06-16

AI Summary

This work addresses the collateral damage incurred when large language models unlearn specific knowledge, which often impairs not only related but also distant-domain knowledge in ways that lack clear boundaries. Framing forgetting-set auditing as a pre-forgetting prediction task—a first from a data-centric perspective—the study quantifies the association between forgetting and evaluation sets by analyzing data interaction features and semantic distances. The findings reveal that collateral damage decays with semantic distance yet can propagate across domains, and that interaction features exhibit the strongest predictive power. This demonstrates that implicit signals embedded in the geometric structure of data enable effective estimation of potential damage even before unlearning occurs, thereby establishing an early-warning mechanism for reliable and controllable knowledge unlearning.

Abstract

Machine unlearning for large language models (LLMs) aims to remove specified knowledge while preserving the rest of the model's capabilities. However, the boundary between knowledge to forget and knowledge to retain is often unclear, since related and even distant information may be entangled in the model. In this paper, we study LLM unlearning from a data-centric perspective and measure how unlearning effects propagate from the forget set to same-domain and distant-domain knowledge. We find a consistent decay pattern: collateral damage is strongest near the forget set, weakens with semantic distance, but does not disappear at domain boundaries. We further ask whether such damage can be audited before unlearning is executed. We formulate forget-set auditing as a pre-unlearning prediction task and analyze which data features are most predictive of downstream damage. Our results show that interaction features between the forget set and evaluation set provide the strongest signals, suggesting that collateral damage is partly reflected in data geometry before model updates occur. These findings position forget-set auditing as an early warning tool for identifying risky unlearning runs and designing more reliable unlearning procedures.

Problem

• machine unlearning
• large language models
• collateral damage
• knowledge forgetting
• data auditing

Innovation

• machine unlearning
• collateral damage
• forget-set auditing
• data-centric analysis
• semantic distance