Every Picture Tells a Dangerous Story: Memory-Augmented Multi-Agent Jailbreak Attacks on VLMs

2026-04-14

AI Summary

This work proposes MemJack, a novel framework that systematically explores the deep semantic attack surface of natural images against vision-language models (VLMs), addressing the limitation of existing jailbreaking methods that rely on pixel perturbations or explicitly harmful imagery. MemJack leverages a memory-augmented multi-agent collaboration mechanism to dynamically map visual entities to malicious intents, generating multi-perspective, vision-semantic-aligned adversarial prompts. It further integrates iterative nullspace projection (INLP) to circumvent early rejection mechanisms. By introducing a multimodal episodic memory bank, MemJack enables highly coherent, cross-image, multi-turn attacks, significantly enhancing transferability and success rates. Evaluated on the unmodified COCO val2017 dataset, it achieves a 71.48% attack success rate against Qwen3-VL-Plus (reaching 90% under extended budgets) and introduces MemJack-Bench, a new benchmark comprising 113,000 attack trajectories.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

The rapid evolution of Vision-Language Models (VLMs) has catalyzed unprecedented capabilities in artificial intelligence; however, this continuous modal expansion has inadvertently exposed a vastly broadened and unconstrained adversarial attack surface. Current multimodal jailbreak strategies primarily focus on surface-level pixel perturbations and typographic attacks or harmful images; however, they fail to engage with the complex semantic structures intrinsic to visual data. This leaves the vast semantic attack surface of original, natural images largely unscrutinized. Driven by the need to expose these deep-seated semantic vulnerabilities, we introduce \textbf{MemJack}, a \textbf{MEM}ory-augmented multi-agent \textbf{JA}ilbreak atta\textbf{CK} framework that explicitly leverages visual semantics to orchestrate automated jailbreak attacks. MemJack employs coordinated multi-agent cooperation to dynamically map visual entities to malicious intents, generate adversarial prompts via multi-angle visual-semantic camouflage, and utilize an Iterative Nullspace Projection (INLP) geometric filter to bypass premature latent space refusals. By accumulating and transferring successful strategies through a persistent Multimodal Experience Memory, MemJack maintains highly coherent extended multi-turn jailbreak attack interactions across different images, thereby improving the attack success rate (ASR) on new images. Extensive empirical evaluations across full, unmodified COCO val2017 images demonstrate that MemJack achieves a 71.48\% ASR against Qwen3-VL-Plus, scaling to 90\% under extended budgets. Furthermore, to catalyze future defensive alignment research, we will release \textbf{MemJack-Bench}, a comprehensive dataset comprising over 113,000 interactive multimodal jailbreak attack trajectories, establishing a vital foundation for developing inherently robust VLMs.

Problem

• Vision-Language Models
• multimodal jailbreak
• semantic vulnerabilities
• adversarial attacks
• visual semantics

Innovation

• Memory-Augmented
• Multi-Agent Jailbreak
• Visual-Semantic Camouflage
• Iterative Nullspace Projection
• Multimodal Experience Memory