Towards Certified Unlearning for Deep Neural Networks

2024-08-01

International Conference on Machine Learning

AI Summary

Certified unlearning—guaranteeing rigorous mathematical bounds on model behavior after data removal—remains theoretically intractable for deep neural networks (DNNs) under non-convex optimization, especially when models are not fully converged. Method: This work pioneers the extension of certified unlearning to non-convex DNN training, enabling dynamic, time-sensitive unlearning requests during training. We propose an efficient gradient correction mechanism based on inverse-Hessian approximation, integrated with local stability analysis for non-convex optimization and a sequential unlearning strategy. Contribution/Results: Our approach achieves strict ε-certified unlearning while substantially reducing computational overhead compared to baselines. Experiments on three real-world datasets demonstrate significant improvements in unlearning efficiency—measured by both speed and fidelity—while provably satisfying the ε-certified unlearning guarantee. This establishes a new paradigm for practical, trustworthy machine learning systems.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Data transparency and provenance

Abstract

In the field of machine unlearning, certified unlearning has been extensively studied in convex machine learning models due to its high efficiency and strong theoretical guarantees. However, its application to deep neural networks (DNNs), known for their highly nonconvex nature, still poses challenges. To bridge the gap between certified unlearning and DNNs, we propose several simple techniques to extend certified unlearning methods to nonconvex objectives. To reduce the time complexity, we develop an efficient computation method by inverse Hessian approximation without compromising certification guarantees. In addition, we extend our discussion of certification to nonconvergence training and sequential unlearning, considering that real-world users can send unlearning requests at different time points. Extensive experiments on three real-world datasets demonstrate the efficacy of our method and the advantages of certified unlearning in DNNs.

Problem

• Extending certified unlearning to nonconvex deep neural networks
• Reducing time complexity via efficient Hessian approximation
• Enabling certified unlearning for nonconvergence and sequential requests

Innovation

• Extends certified unlearning to nonconvex DNNs
• Uses inverse Hessian approximation for efficiency
• Supports sequential unlearning at different times