Auditing Approximate Machine Unlearning for Differentially Private Models

2025-08-26

AI Summary

This work exposes an implicit privacy risk in approximate machine unlearning methods under differential privacy (DP): while existing approaches assume retained data remains unaffected by unlearning, empirical evidence shows their DP guarantees may be compromised. To address this, we propose a dual-dimension privacy evaluation framework—assessing both forgotten and retained samples—and introduce A-LiRA, a lightweight auditing method that integrates data augmentation to reduce shadow model overhead, enabling efficient and reproducible joint evaluation of unlearning efficacy and privacy leakage. Experiments reveal that mainstream approximate unlearning algorithms significantly degrade the DP guarantees for retained samples. Our findings advance the development of truly DP-compliant machine unlearning paradigms and support community validation through open-sourced code.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Safety and Robustness
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Security and Privacy: Security and privacy of machine learning and AI applications
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Approximate machine unlearning aims to remove the effect of specific data from trained models to ensure individuals' privacy. Existing methods focus on the removed records and assume the retained ones are unaffected. However, recent studies on the emph{privacy onion effect} indicate this assumption might be incorrect. Especially when the model is differentially private, no study has explored whether the retained ones still meet the differential privacy (DP) criterion under existing machine unlearning methods. This paper takes a holistic approach to auditing both unlearned and retained samples' privacy risks after applying approximate unlearning algorithms. We propose the privacy criteria for unlearned and retained samples, respectively, based on the perspectives of DP and membership inference attacks (MIAs). To make the auditing process more practical, we also develop an efficient MIA, A-LiRA, utilizing data augmentation to reduce the cost of shadow model training. Our experimental findings indicate that existing approximate machine unlearning algorithms may inadvertently compromise the privacy of retained samples for differentially private models, and we need differentially private unlearning algorithms. For reproducibility, we have pubished our code: https://anonymous.4open.science/r/Auditing-machine-unlearning-CB10/README.md

Problem

• Auditing privacy risks of retained samples after unlearning
• Verifying differential privacy compliance post-unlearning process
• Developing efficient membership inference attacks for practical auditing

Innovation

• Auditing privacy risks for unlearned and retained samples
• Proposing differential privacy and MIA criteria for evaluation
• Developing efficient A-LiRA attack with data augmentation