Sparsity-Aware Unlearning for Large Language Models

2026-01-31

AI Summary

Existing machine unlearning methods suffer significant performance degradation when applied to sparse large language models, struggling to effectively remove sensitive information. This work proposes a novel unlearning paradigm tailored for sparse large language models, which decouples the unlearning objective from the model’s sparsification goal for the first time. The approach employs gradient masking to steer parameter updates toward retained weights and incorporates an importance-aware redistribution strategy to compensate for the impact of pruned parameters. By integrating gradient modulation, parameter redistribution, and pruning structure, the method achieves efficient removal of sensitive data while substantially outperforming existing unlearning techniques and preserving overall model performance.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Security and Privacy: Large-scale security measurements

Abstract

Large Language Models (LLMs) inevitably memorize sensitive information during training, posing significant privacy risks. Machine unlearning has emerged as a promising solution to selectively remove such information without full retraining. However, existing methods are designed for dense models and overlook model sparsification-an essential technique for efficient LLM deployment. We find that unlearning effectiveness degrades substantially on sparse models. Through empirical analysis, we reveal that this degradation occurs because existing unlearning methods require updating all parameters, yet sparsification prunes substantial weights to zero, fundamentally limiting the model's forgetting capacity. To address this challenge, we propose Sparsity-Aware Unlearning (SAU), which decouples unlearning from sparsification objectives through gradient masking that redirects updates to surviving weights, combined with importance-aware redistribution to compensate for pruned parameters. Extensive experiments demonstrate that SAU significantly outperforms existing methods on sparse LLMs, achieving effective forgetting while preserving model utility.

Problem

• machine unlearning
• large language models
• model sparsification
• privacy
• parameter pruning

Innovation

• sparsity-aware unlearning
• large language models
• machine unlearning
• model sparsification
• gradient masking