Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench

2024-10-29

arXiv.org

AI Summary

Multimodal large language models (MLLMs) pose significant privacy risks due to unintended memorization and leakage of sensitive multimodal training data. Method: This paper introduces the first machine unlearning framework tailored for MLLMs, featuring (i) MLLMU-Bench—a novel benchmark comprising 653 synthetic/celebrity profiles and over 1,000 image-text QA pairs; (ii) the first systematic formalization of multimodal unlearning tasks; and (iii) a four-dimensional evaluation metric covering effectiveness, generalization, utility preservation, and modality adaptability. We adapt generative unlearning algorithms (GA, RMU, Scrub) with multimodal input modeling, cross-modal consistency constraints, and utility retention mechanisms. Results: Experiments reveal that unimodal unlearning excels in generation and cloze tasks, whereas multimodal collaborative unlearning significantly improves performance on image-text classification—demonstrating the necessity and efficacy of modality-aware unlearning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

Generative models such as Large Language Models (LLM) and Multimodal Large Language models (MLLMs) trained on massive web corpora can memorize and disclose individuals' confidential and private data, raising legal and ethical concerns. While many previous works have addressed this issue in LLM via machine unlearning, it remains largely unexplored for MLLMs. To tackle this challenge, we introduce Multimodal Large Language Model Unlearning Benchmark (MLLMU-Bench), a novel benchmark aimed at advancing the understanding of multimodal machine unlearning. MLLMU-Bench consists of 500 fictitious profiles and 153 profiles for public celebrities, each profile feature over 14 customized question-answer pairs, evaluated from both multimodal (image+text) and unimodal (text) perspectives. The benchmark is divided into four sets to assess unlearning algorithms in terms of efficacy, generalizability, and model utility. Finally, we provide baseline results using existing generative model unlearning algorithms. Surprisingly, our experiments show that unimodal unlearning algorithms excel in generation and cloze tasks, while multimodal unlearning approaches perform better in classification tasks with multimodal inputs.

Problem

• Address privacy risks in MLLMs
• Introduce MLLMU-Bench for unlearning
• Evaluate unlearning efficacy and utility

Innovation

• Introduces MLLMU-Bench for unlearning
• Evaluates efficacy and generalizability
• Compares unimodal and multimodal approaches