Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem

2026-07-10

AI Summary

This work addresses the unreliable evaluation of machine unlearning in large language models, where existing methods often suffer from under-unlearning or over-unlearning, failing to precisely remove target knowledge while fully preserving unrelated capabilities. The study formulates unlearning as an asymmetric generalization problem and proposes evaluating unlearning efficacy across diverse queries while systematically probing retained knowledge to verify model integrity. Its key contributions include the introduction of SUITE, a novel evaluation protocol and training corpus grounded in fine-grained annotations that clearly delineate the boundary between forgotten and retained knowledge, and JensUn++, an unlearning algorithm that achieves state-of-the-art trade-offs across multiple settings. Experiments on three mainstream large language models demonstrate significant improvements in both forgetting accuracy and capability retention, highlighting the critical role of training data structure in effective unlearning.

Abstract

Machine unlearning in LLMs is the targeted removal of specific knowledge while preserving all other capabilities, critical for privacy and safety. Yet existing benchmarks measure it unreliably. They miss knowledge that resurfaces under paraphrased or indirect queries, a failure we call under-forgetting, and lack the semantic, syntactic, and lexical probes needed to verify that unrelated knowledge is preserved, a failure we call over-forgetting. Both failures reflect an asymmetric generalization problem. Forget evaluation must cover diverse query formulations of the same target facts, testing whether forgetting holds beyond exact training prompts. Retain evaluation must probe a far larger and implicitly defined set, namely every fact disjoint from the forget target. The retain set thus defines the effective forget set, yet current datasets provide no fine-grained annotation of this forget-retain boundary. We address this with SUITE, an evaluation protocol and training corpus that captures forget-retain structure for real-world factual domains. Methods trained on SUITE improve substantially, showing that training data is as important as algorithmic design. Building on the obtained insights, we introduce JensUn++, an unlearning algorithm that achieves the best forget-retain utility trade-off across three LLMs, in both sequential and joint unlearning settings. Code and datasets are available at https://amitpeleg.github.io/forget-narrowly-retain-broadly

Problem

• machine unlearning
• asymmetric generalization
• under-forgetting
• over-forgetting
• forget-retain boundary

Innovation

• machine unlearning
• asymmetric generalization
• forget-retain boundary
• SUITE benchmark
• JensUn++