Understanding Fine-tuning in Approximate Unlearning: A Theoretical Perspective

2024-10-04

AI Summary

In machine unlearning, fine-tuning (FT) preserves model performance but fails to fully erase the influence of target data—particularly in linear regression, where pretraining bias persists post-unlearning. This work presents the first theoretical analysis of FT-based unlearning within the linear regression framework, revealing that its failure stems from coupled parameter updates that do not disentangle dependencies between target and retained data. To address this, we propose Retained-data-based Weight Significance Masking (RBM): a novel masking strategy that constructs a data-aware significance map using only retained samples to dynamically suppress redundant parameters. We theoretically prove that RBM simultaneously improves both unlearning accuracy (UA) and retention accuracy (RA). Extensive experiments on synthetic and real-world datasets demonstrate that RBM consistently outperforms existing masking methods and significantly enhances model fairness.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Machine Unlearning has emerged as a significant area of research, focusing on `removing' specific subsets of data from a trained model. Fine-tuning (FT) methods have become one of the fundamental approaches for approximating unlearning, as they effectively retain model performance. However, it is consistently observed that naive FT methods struggle to forget the targeted data. In this paper, we present the first theoretical analysis of FT methods for machine unlearning within a linear regression framework, providing a deeper exploration of this phenomenon. Our analysis reveals that while FT models can achieve zero remaining loss, they fail to forget the forgetting data, as the pretrained model retains its influence and the fine-tuning process does not adequately mitigate it. To address this, we propose a novel Retention-Based Masking (RBM) strategy that constructs a weight saliency map based on the remaining dataset, unlike existing methods that focus on the forgetting dataset. Our theoretical analysis demonstrates that RBM not only significantly improves unlearning accuracy (UA) but also ensures higher retaining accuracy (RA) by preserving overlapping features shared between the forgetting and remaining datasets. Experiments on synthetic and real-world datasets validate our theoretical insights, showing that RBM outperforms existing masking approaches in balancing UA, RA, and disparity metrics.

Problem

• Analyzes Fine-tuning in Machine Unlearning
• Proposes Retention-Based Masking strategy
• Improves unlearning and retaining accuracy

Innovation

• Fine-tuning in unlearning
• Retention-Based Masking strategy
• Weight saliency map construction