Unlearning with Asymmetric Sources: Improved Unlearning-Utility Trade-off with Public Data

2026-05-11

AI Summary

This work addresses the significant degradation in model utility caused by high noise in existing provable machine unlearning methods under large-scale deletion requests, as well as the unexplored role of public data in this context. The authors propose Asymmetric Langevin Unlearning (ALU), a novel framework that, for the first time, incorporates public data into machine unlearning. By leveraging asymmetric Langevin dynamics and variational Rényi divergence analysis, they theoretically establish that ALU reduces unlearning cost by a factor of $O(1/n_{\text{pub}}^2)$. The method remains effective under distribution shift, substantially improves model utility, provides strong resistance against membership inference attacks, and supports unlearning of a constant fraction of training data at scale—outperforming both full retraining and current symmetric unlearning approaches.

Abstract

Noise-based certified machine unlearning currently faces a hard ceiling: the noise magnitude required to certify unlearning typically destroys model utility, particularly for large-scale deletion requests. While leveraging public data is a standard technique in differential privacy to relax this tension, its role in unlearning remains unexplored. We address this gap by introducing Asymmetric Langevin Unlearning (ALU), a framework that uses public data to mitigate privacy costs. We prove that public data injection suppresses the unlearning cost by a factor of $O(1/n_{\mathrm{pub}}^2)$, guaranteeing a strict computational advantage over retraining. This establishes a new control mechanism: practitioners can mitigate the need for high noise-and the associated utility loss-by increasing the volume of public data. Crucially, we analyze the realistic setting of distribution mismatch, explicitly characterizing how shifts between public and private sources impact utility. We show that ALU enables mass unlearning of constant dataset fractions -- a regime where standard symmetric methods become impractical -- while maintaining high utility. Empirical evaluations using variational Rényi divergence and membership inference attacks confirm that ALU effectively thwarts privacy attacks while preserving utility under reasonable distribution shifts.

Problem

• machine unlearning
• utility trade-off
• public data
• noise-based certification
• distribution mismatch

Innovation

• machine unlearning
• public data
• asymmetric Langevin dynamics
• utility-privacy trade-off
• distribution mismatch