SAGE: Retain-Aware Post-Hoc Sanitization of Final Unlearning Vector

2026-06-16

AI Summary

This work addresses the fundamental trade-off in large language models between unlearning harmful knowledge and preserving useful capabilities. The authors propose SAGE, a source-agnostic post-hoc purification method that enhances retention without re-running the original unlearning procedure. SAGE leverages spectral activation geometry analysis on a small retention proxy to identify dominant activation directions and suppresses update components along high-energy retention directions through a closed-form, source-anchored optimization objective. The study introduces “retention activation bias” as a novel, general-purpose metric for quantifying retention damage. Extensive experiments demonstrate that SAGE consistently improves retention performance across diverse unlearning algorithms, model scales, and benchmarks while maintaining the original forgetting efficacy, thereby validating its effectiveness and broad applicability.

Abstract

Large Language Model (LLM) unlearning aims to remove undesirable knowledge or behaviors while preserving retained capabilities. Current unlearning methods all involve a trade-off between unlearning and retention. We have found that the retention activation bias can also be used to quantify the damage an unlearning method inflicts on retention, without considering the specific implementation of the unlearning process. This allows us to restore retention performance for any unlearning method using a post-hoc approach. Therefore, we propose a complementary post-hoc setting to sanitize the final update vector without rerunning the original unlearning pipeline. In this setting, we design SAGE, Spectral Activation-GEometry Sanitization, a source-agnostic correction for final unlearning updates. SAGE collects real module inputs from a small retain proxy, extracts their dominant activation geometry, and solves a source-anchored optimization objective in closed form, which suppresses update components aligned with high-energy retained directions while preserving the source method's forgetting carrier. Across multiple unlearning methods, model scales, and benchmarks, SAGE consistently relieves the retain-forget trade-off, identifying post-hoc sanitization of final vectors as a practical and underexplored axis for machine unlearning.

Problem

• machine unlearning
• retain-forget trade-off
• large language models
• post-hoc sanitization
• retention performance

Innovation

• post-hoc sanitization
• activation geometry
• retain-forget trade-off
• machine unlearning
• update vector correction