SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearning

2026-03-27

AI Summary

This work addresses the lack of fine-grained evaluation of unlearning capabilities in multimodal contrastive learning models at the level of sensitive attribute associations. To this end, we propose SALMUBench—the first benchmark framework specifically designed for evaluating unlearning of sensitive associations in multimodal settings. Built upon a synthetic dataset of 60K person–attribute pairs, our framework constructs both contaminated and clean model variants and introduces two structured retention sets—identity-preserving and association-preserving—to enable precise quantification of unlearning efficacy and collateral damage. Experiments reveal failure modes of existing methods, which either insufficiently forget sensitive links or over-generalize, while demonstrating the feasibility of effectively removing targeted associations. We release the dataset, models, evaluation scripts, and a leaderboard to foster standardized research in multimodal unlearning.

Abstract

As multimodal models like CLIP become integral to downstream systems, the need to remove sensitive information is critical. However, machine unlearning for contrastively-trained encoders remains underexplored, and existing evaluations fail to diagnose fine-grained, association-level forgetting. We introduce SALMUBench (Sensitive Association-Level Multimodal Unlearning), a benchmark built upon a synthetic dataset of 60K persona-attribute associations and two foundational models: a Compromised model polluted with this data, and a Clean model without it. To isolate unlearning effects, both are trained from scratch on the same 400M-pair retain base, with the Compromised model additionally trained on the sensitive set. We propose a novel evaluation protocol with structured holdout sets (holdout identity, holdout association) to precisely measure unlearning efficacy and collateral damage. Our benchmark reveals that while utility-efficient deletion is feasible, current methods exhibit distinct failure modes: they either fail to forget effectively or over-generalize by erasing more than intended. SALMUBench sets a new standard for comprehensive unlearning evaluation, and we publicly release our dataset, models, evaluation scripts, and leaderboards to foster future research.

Problem

• machine unlearning
• multimodal models
• sensitive data
• association-level forgetting
• contrastive learning

Innovation

• machine unlearning
• multimodal models
• association-level forgetting
• contrastive learning
• benchmark