PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Models

2025-03-16

AI Summary

Privacy and security risks arising from multimodal large language models’ (MLLMs) reliance on internet-scale data remain unaddressed by existing machine unlearning (MU) methods, which lack systematic evaluation and well-defined task formulations. To bridge this gap, we introduce PEBench—the first MU benchmark tailored for MLLMs—featuring: (i) a formally defined MU task scope; (ii) a privacy-safe, reproducible synthetic multimodal dataset containing fabricated personal entities and events; and (iii) a three-dimensional evaluation framework assessing unlearning accuracy, retention of critical knowledge, and vision-language consistency. We conduct systematic experiments using cross-modal alignment prompting, counterfactual perturbation, and six state-of-the-art MU methods. Results reveal widespread failure of current MU techniques in erasing visual knowledge, pinpointing insufficient vision-language decoupling as a key bottleneck. PEBench establishes the foundation for verifiable, auditable unlearning standards in MLLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

In recent years, Multimodal Large Language Models (MLLMs) have demonstrated remarkable advancements in tasks such as visual question answering, visual understanding, and reasoning. However, this impressive progress relies on vast amounts of data collected from the internet, raising significant concerns about privacy and security. To address these issues, machine unlearning (MU) has emerged as a promising solution, enabling the removal of specific knowledge from an already trained model without requiring retraining from scratch. Although MU for MLLMs has gained attention, current evaluations of its efficacy remain incomplete, and the underlying problem is often poorly defined, which hinders the development of strategies for creating more secure and trustworthy systems. To bridge this gap, we introduce a benchmark, named PEBench, which includes a dataset of personal entities and corresponding general event scenes, designed to comprehensively assess the performance of MU for MLLMs. Through PEBench, we aim to provide a standardized and robust framework to advance research in secure and privacy-preserving multimodal models. We benchmarked 6 MU methods, revealing their strengths and limitations, and shedding light on key challenges and opportunities for MU in MLLMs.

Problem

• Assessing machine unlearning efficacy in Multimodal Large Language Models
• Addressing privacy and security concerns in MLLMs
• Developing a benchmark for secure and privacy-preserving MLLMs

Innovation

• PEBench benchmark for machine unlearning evaluation
• Dataset with personal entities and event scenes
• Standardized framework for privacy-preserving MLLMs