PPE-Bench: A Benchmark for Evaluating MLLM Unlearning under Private-Public Entanglement

2026-07-02

AI Summary

Existing machine unlearning benchmarks for multimodal large language models (MLLMs) overlook the visual entanglement between private information and public knowledge—such as celebrities or landmarks—in images, leading to inadequate evaluation. This work introduces the first multimodal unlearning benchmark that explicitly incorporates such visually entangled scenarios and proposes two simple yet effective methods to unlearn private targets while better preserving associated public knowledge. Experimental results demonstrate that although current unlearning approaches can suppress leakage of private information, they often severely degrade co-occurring public knowledge, thereby underscoring the necessity and challenge of the proposed benchmark. This study provides a more realistic and rigorous evaluation framework for multimodal machine unlearning research.

Abstract

Multimodal Large Language Models (MLLMs) have shown strong capabilities, but they may memorize private information from web data, raising privacy concerns. Machine unlearning offers a way to remove such private knowledge without retraining from scratch. However, existing MLLM unlearning benchmarks have two major limitations. First, they rely on simplified images that contain only the single target individual, failing to reflect the visual complexity of real-world photos. Second, they typically assume that the forget set and retain set are fully separated, ignoring the fact that private information is often visually entangled with benign public information. For example, a private individual may appear with a public figure or in front of a well-known landmark, where unlearning the private target should not damage the public context. To address these limitations, we propose PPE-Bench, a new benchmark for evaluating MLLM unlearning under private-public entanglement. Each image contains a target individual to be forgotten and public information to be preserved, including public figure and landmark. We further introduce two simple but effective methods to better preserve public information during unlearning. Through experiments, we find that existing unlearning methods can reduce private information leakage, but often substantially harm adjacent public information.

Problem

• machine unlearning
• multimodal large language models
• privacy
• private-public entanglement
• unlearning benchmark

Innovation

• PPE-Bench
• MLLM unlearning
• private-public entanglement
• multimodal forgetting
• privacy-preserving