PULSE: Practical Evaluation Scenarios for Large Multimodal Model Unlearning

2025-07-01

AI Summary

Existing large multimodal models (LMMs) lack a realistic, scenario-oriented evaluation framework for machine unlearning—particularly for quantifying pretrained knowledge erasure and long-term sustainability under sequential forgetting requests. Method: We propose the first practical unlearning evaluation framework specifically designed for large vision-language models. It introduces a dual-dimensional assessment protocol: (1) “pretrained knowledge forgetting” and (2) “sequential request sustainability,” integrated with controlled-variable experiments, knowledge provenance analysis, and longitudinal performance tracking. We conduct multi-stage empirical evaluations of mainstream unlearning algorithms under this framework. Results: Our findings reveal that current methods fail to effectively erase pretrained memorization, and exhibit significant performance degradation under repeated forgetting requests. This work establishes a reproducible benchmark and delivers critical empirical insights to guide robust unlearning mechanism design.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

In recent years, unlearning techniques, which are methods for inducing a model to "forget" previously learned information, have attracted attention as a way to address privacy and copyright concerns in large language models (LLMs) and large multimodal models (LMMs). While several unlearning benchmarks have been established for LLMs, a practical evaluation framework for unlearning in LMMs has been less explored. Specifically, existing unlearning benchmark for LMMs considers only scenarios in which the model is required to unlearn fine-tuned knowledge through a single unlearning operation. In this study, we introduce PULSE protocol for realistic unlearning scenarios for LMMs by introducing two critical perspectives: (i) Pre-trained knowledge Unlearning for analyzing the effect across different knowledge acquisition phases and (ii) Long-term Sustainability Evaluation to address sequential requests. We then evaluate existing unlearning methods along these dimensions. Our results reveal that, although some techniques can successfully unlearn knowledge acquired through fine-tuning, they struggle to eliminate information learned during pre-training. Moreover, methods that effectively unlearn a batch of target data in a single operation exhibit substantial performance degradation when the same data are split and unlearned sequentially.

Problem

• Lack of practical unlearning evaluation for large multimodal models
• Existing benchmarks ignore pre-trained knowledge unlearning scenarios
• Current methods fail in sequential unlearning sustainability

Innovation

• Introduces PULSE protocol for LMM unlearning scenarios
• Evaluates pre-trained knowledge unlearning across phases
• Assesses long-term sustainability of sequential unlearning