Auditing of Unlearning Algorithms

2026-07-07

AI Summary

Existing machine unlearning algorithms lack effective means to rigorously verify whether the influence of training data has been genuinely removed. This work proposes a general auditing framework based on membership inference attacks, which leverages hypothesis testing to compute a lower bound (ε) on data-dependent unlearning parameters, thereby establishing, for the first time, empirical falsifiability of unlearning claims. The framework systematically evaluates diverse unlearning techniques—including model pruning, fine-tuning on retention sets, and rollback-based deletion—on CIFAR-100 and Shakespeare datasets. Experimental results reveal that theoretically grounded methods, such as model pruning, achieve markedly small ε bounds, whereas empirical approaches yield substantially larger ε values, highlighting a significant disparity in their practical unlearning efficacy.

Abstract

Evaluating whether unlearning algorithms truly remove training data influence remains an open challenge. We propose a practical auditor that computes data-dependent lower bounds on the unlearning parameter $\varepsilon$ using membership inference attacks. Evaluating multiple unlearning algorithms, we find a sharp separation: algorithms with rigorous guarantees, such as model clipping and rewind-to-delete, achieve very small $\varepsilon$ bounds that do not falsify their unlearning guarantees, whereas empirical methods such as Hessian-based unlearning, interleaved ascent-descent, ascent on the forget set, and fine-tuning on the retain set exhibit large bounds, indicating poor unlearning. Our auditor provides a practical tool for empirically falsifying unlearning claims through a hypothesis-testing framework, and we validate it on CIFAR-100 and Shakespeare text.

Problem

• unlearning algorithms
• membership inference attacks
• data influence removal
• empirical evaluation
• privacy auditing

Innovation

• unlearning algorithms
• membership inference attacks
• auditing
• epsilon bounds
• hypothesis testing