MemLeak: Diagnosing Information Leaks in Multimodal Agent Memory

2026-06-29

AI Summary

This study addresses a critical privacy vulnerability in multimodal agents: even after explicit memory deletion, sensitive facts may still be recoverable through implicit visual cues retained in user-uploaded images, leading to unintended information leakage. The work is the first to identify and systematically analyze this risk, introducing an Information Provenance Graph (IPG) to categorize memory representations and establishing the MemLeak benchmark to evaluate the recoverability of residual information from both textual and visual modalities. Building on these insights, the authors propose a content-aware semantic erasure strategy that significantly reduces image-induced information recovery rates from 12.0% (compared to a blind baseline of 0.0%) to 2.0% across multiple multimodal large language models and real-world memory systems. Notably, 47% of such leaks are undetectable via text alone. Human evaluation with dual annotation (Cohen’s κ = 0.88) confirms both the prevalence of visual residual risks and the effectiveness of the proposed mitigation approach.

Abstract

When a multimodal AI agent is asked to forget a fact, current memory systems usually delete the text entry and report success. We find that the fact can remain recoverable from retained user images, including images tagged to entirely different facts, because VLMs use implicit visual cues at inference time. We introduce the Information Provenance Graph (IPG), a taxonomy that classifies memory representations by deletion affordance. The IPG reveals that deletion fails through multiple channels. Our benchmark, MemLeak, measures this across a deletion cascade: direct probing of deletion-capable systems yields <1%, but retained correlated text enables 18.3% recovery, and retained images enable 12.0% recovery (0.0% blind baseline, 0.3% FPR) -- with 47% of image leaks not text-recoverable. Content-aware semantic deletion reduces the image residual to 2.0%. The residual appears across multiple VLMs, a production memory system, and real Unsplash-licensed photographs. Dual-annotator human validation (kappa = 0.88) confirms judge reliability.

Problem

• information leak
• multimodal agent memory
• visual language models
• data deletion
• memory systems

Innovation

• Information Provenance Graph
• MemLeak
• multimodal memory deletion
• visual language models
• semantic forgetting