Rectifying Privacy and Efficacy Measurements in Machine Unlearning: A New Inference Attack Perspective

2025-06-16

AI Summary

Existing machine unlearning evaluation methods suffer from inaccurate privacy-risk and effectiveness measurements—relying solely on average-case analysis, random-sample testing, and incomplete retraining baselines. Method: We propose RULI, a fine-grained evaluation framework grounded in likelihood-ratio testing and dual-objective inference attacks. RULI introduces a gamified modeling paradigm, enabling, for the first time, simultaneous quantification of sample-level unlearning effectiveness and privacy leakage risk within a unified framework—thereby overcoming the limitations of average-case analysis. Contribution/Results: Through adversarial game-theoretic analysis and cross-modal (image/text) empirical evaluation, RULI exposes systematic underestimation of privacy risks by mainstream unlearning methods. Its inference attack achieves significantly higher success rates than state-of-the-art baselines, demonstrating superior sensitivity and discriminative power in assessing unlearning quality and privacy preservation.

Technology Category

• Machine Learning: Privacy
• Game Theory and Economic Paradigms: Adversarial Learning
• Natural Language Processing: Safety and Robustness

Application Category

• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Machine unlearning focuses on efficiently removing specific data from trained models, addressing privacy and compliance concerns with reasonable costs. Although exact unlearning ensures complete data removal equivalent to retraining, it is impractical for large-scale models, leading to growing interest in inexact unlearning methods. However, the lack of formal guarantees in these methods necessitates the need for robust evaluation frameworks to assess their privacy and effectiveness. In this work, we first identify several key pitfalls of the existing unlearning evaluation frameworks, e.g., focusing on average-case evaluation or targeting random samples for evaluation, incomplete comparisons with the retraining baseline. Then, we propose RULI (Rectified Unlearning Evaluation Framework via Likelihood Inference), a novel framework to address critical gaps in the evaluation of inexact unlearning methods. RULI introduces a dual-objective attack to measure both unlearning efficacy and privacy risks at a per-sample granularity. Our findings reveal significant vulnerabilities in state-of-the-art unlearning methods, where RULI achieves higher attack success rates, exposing privacy risks underestimated by existing methods. Built on a game-based foundation and validated through empirical evaluations on both image and text data (spanning tasks from classification to generation), RULI provides a rigorous, scalable, and fine-grained methodology for evaluating unlearning techniques.

Problem

• Evaluating privacy risks in inexact machine unlearning methods
• Identifying gaps in current unlearning evaluation frameworks
• Proposing a robust framework for fine-grained unlearning assessment

Innovation

• Introduces RULI framework for unlearning evaluation
• Uses dual-objective attack for efficacy and privacy
• Validated on image and text data tasks