The Forgetting-Retention Dilemma: Certified Unlearning Theory in Continual Learning

2026-06-29

AI Summary

This work establishes the first theoretical connection between continual learning and machine unlearning, formalizing post-unlearning performance as an excess risk minimization problem decomposed into a continual learning risk term and a forgetting loss term. Building on this framework, the study systematically adapts two theoretically grounded unlearning approaches—gradient-based and Hessian-based methods—and reveals that while the former exhibits weaker forgetting capability, it incurs nearly zero storage overhead. To balance effectiveness and efficiency, the authors propose a hybrid unlearning strategy. Theoretical analysis, supported by excess risk upper bounds for non-convex models, together with empirical validation, demonstrates that the proposed method significantly reduces storage costs while effectively preserving post-unlearning performance, thereby confirming the inherent trade-off between forgetting and knowledge retention and the efficacy of the introduced approach.

Abstract

Machine unlearning aims to eliminate the influence of specific data from trained models to safeguard privacy. However, this presents a significant challenge in the context of continual learning (CL), where models update sequentially on dynamic datasets. A major limitation is that current certified unlearning algorithms fail to account for the complex, cumulative model evolution inherent to CL framework. In this work, we establish the first theoretical foundation bridging CL and machine unlearning. We formulate the CL's unlearning objective as the minimization of post-unlearning excess risk, which decomposes into CL excess risk and unlearning loss, characterizing the fundamental trade-off between preserving historical knowledge and targeted forgetting. Under mild assumptions, we first establish an upper bound for the CL excess risk in non-convex models. We then adapt two certified unlearning approaches, gradient-based and Hessian-based, to the CL framework. Our analysis reveals that while the gradient-based approach is less effective than the Hessian-based method in minimizing unlearning loss, it offers the distinct advantage of nearly zero storage overhead for enabling unlearning. This insight motivates a hybrid strategy that reduces storage costs while maintaining post-unlearning performance. Experimental results further validate our theoretical findings.

Problem

• machine unlearning
• continual learning
• forgetting-retention dilemma
• excess risk
• certified unlearning

Innovation

• Certified Unlearning
• Continual Learning
• Excess Risk
• Gradient-based Unlearning
• Hessian-based Unlearning