Benchmarking Unlearning for Vision Transformers

2026-02-23

AI Summary

This work addresses the lack of a systematic benchmark for machine unlearning (MU) in vision transformers (VTs), as existing research has primarily focused on CNNs, large language models, and diffusion models. We present the first comprehensive MU evaluation framework tailored to mainstream VT architectures such as ViT and Swin-T, incorporating varying dataset scales and complexities, as well as both single-step and continual unlearning protocols. Our approach introduces an unlearning strategy grounded in the model’s training data memorization mechanisms and proposes unified metrics to jointly assess forgetting efficacy and retained model utility. Experiments reveal significant differences in data memorization behavior between VTs and CNNs, and demonstrate that leveraging memorization-aware strategies substantially improves unlearning performance. This study establishes a reproducible, fair, and holistic platform for evaluating VT unlearning methods and provides strong baselines for future research.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Security and Privacy: Data transparency and provenance

Abstract

Research in machine unlearning (MU) has gained strong momentum: MU is now widely regarded as a critical capability for building safe and fair AI. In parallel, research into transformer architectures for computer vision tasks has been highly successful: Increasingly, Vision Transformers (VTs) emerge as strong alternatives to CNNs. Yet, MU research for vision tasks has largely centered on CNNs, not VTs. While benchmarking MU efforts have addressed LLMs, diffusion models, and CNNs, none exist for VTs. This work is the first to attempt this, benchmarking MU algorithm performance in different VT families (ViT and Swin-T) and at different capacities. The work employs (i) different datasets, selected to assess the impacts of dataset scale and complexity; (ii) different MU algorithms, selected to represent fundamentally different approaches for MU; and (iii) both single-shot and continual unlearning protocols. Additionally, it focuses on benchmarking MU algorithms that leverage training data memorization, since leveraging memorization has been recently discovered to significantly improve the performance of previously SOTA algorithms. En route, the work characterizes how VTs memorize training data relative to CNNs, and assesses the impact of different memorization proxies on performance. The benchmark uses unified evaluation metrics that capture two complementary notions of forget quality along with accuracy on unseen (test) data and on retained data. Overall, this work offers a benchmarking basis, enabling reproducible, fair, and comprehensive comparisons of existing (and future) MU algorithms on VTs. And, for the first time, it sheds light on how well existing algorithms work in VT settings, establishing a promising reference performance baseline.

Problem

• machine unlearning
• Vision Transformers
• benchmarking
• unlearning evaluation
• model forgetting

Innovation

• machine unlearning
• Vision Transformers
• memorization
• benchmarking
• forgetting quality