LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearning

2026-07-02

AI Summary

Large language models (LLMs) are prone to memorizing sensitive personally identifiable information (PII) from their training data, yet existing unlearning methods lack reliable evaluation of parameter-level knowledge erasure, often conflating true forgetting with output obfuscation. This work introduces the first LLM unlearning benchmark with ground-truth, parameter-level localization labels: by injecting synthetic PII into specific parameters of the OLMo model via masked continual pretraining, it enables precise assessment of whether unlearning algorithms target the actual weights storing the knowledge. Experiments reveal that current state-of-the-art methods, while effective at the output level, exhibit poor parameter localization accuracy and remain vulnerable to resurfacing attacks. In contrast, once accurate parameter localization is achieved, even simple gradient-based interventions yield robust and verifiable unlearning performance.

Abstract

LLMs memorize sensitive training data, including personally identifiable information (PII), creating a pressing need for reliable post hoc removal methods. Unlearning has emerged as a promising solution, with state-of-the-art(SOTA) methods often following a localize-first, unlearn-second paradigm that targets specific model parameters. However, existing benchmarks evaluate unlearning solely at the output level, leaving open the question of whether unlearning truly erases knowledge from a model's parameters or merely obfuscates it, a concern reinforced by the success of resurfacing attacks. To bridge this gap, we introduce LACUNA: the first unlearning testbed with ground-truth parameter-level localization. LACUNA injects PII of synthetic individuals into predefined parameters of 1B and 7B OLMo-based models via masked continual pretraining, enabling direct evaluation of whether unlearning targets the weights responsible for knowledge storage. We use LACUNA to benchmark current SOTA unlearning methods and find that, despite strong output-level performance, existing methods are highly imprecise and susceptible to resurfacing attacks. We further show that when localization is successful, even a simple gradient-based unlearning method achieves strong erasure and robustness to resurfacing attacks, highlighting the importance of precise unlearning. We release LACUNA to complement behavioral evaluations and drive further advances in robust, localization-based unlearning.

Problem

• LLM unlearning
• localization precision
• parameter-level evaluation
• PII removal
• resurfacing attacks

Innovation

• unlearning
• localization precision
• parameter-level evaluation
• resurfacing attacks
• LLM memorization