Multimodal Safety Is Asymmetric: Cross-Modal Exploits Unlock Black-Box MLLMs Jailbreaks

2025-10-20

AI Summary

This work identifies a novel **cross-modal security asymmetry** in text–vision multimodal large language models (MLLMs), wherein visual alignment disproportionately weakens safety constraints compared to textual inputs, enabling previously unrecognized jailbreaking vulnerabilities. We formally define this phenomenon for the first time and propose a reusable, black-box jailbreaking framework grounded in atomic strategy primitives. Our method integrates attention analysis, latent-space probing, and multi-agent reinforcement learning to automatically synthesize cross-modal adversarial inputs. Evaluated on 12 state-of-the-art open- and closed-source MLLMs, our approach achieves an average jailbreaking success rate improvement of 27.6% over existing SOTA methods. This work establishes a new paradigm for multimodal safety evaluation and defense, highlighting the critical need to account for modality-specific security dynamics in MLLM alignment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Safety and Robustness

Application Category

• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Multimodal large language models (MLLMs) have demonstrated significant utility across diverse real-world applications. But MLLMs remain vulnerable to jailbreaks, where adversarial inputs can collapse their safety constraints and trigger unethical responses. In this work, we investigate jailbreaks in the text-vision multimodal setting and pioneer the observation that visual alignment imposes uneven safety constraints across modalities in MLLMs, thereby giving rise to multimodal safety asymmetry. We then develop PolyJailbreak, a black-box jailbreak method grounded in reinforcement learning. Initially, we probe the model's attention dynamics and latent representation space, assessing how visual inputs reshape cross-modal information flow and diminish the model's ability to separate harmful from benign inputs, thereby exposing exploitable vulnerabilities. On this basis, we systematize them into generalizable and reusable operational rules that constitute a structured library of Atomic Strategy Primitives, which translate harmful intents into jailbreak inputs through step-wise transformations. Guided by the primitives, PolyJailbreak employs a multi-agent optimization process that automatically adapts inputs against the target models. We conduct comprehensive evaluations on a variety of open-source and closed-source MLLMs, demonstrating that PolyJailbreak outperforms state-of-the-art baselines.

Problem

• Investigates multimodal safety asymmetry in MLLMs causing uneven security constraints
• Develops black-box jailbreak method exploiting cross-modal vulnerabilities through reinforcement learning
• Systematizes attack strategies into reusable primitives for automated adversarial input generation

Innovation

• Reinforcement learning-based black-box jailbreak method
• Atomic Strategy Primitives translate harmful intents
• Multi-agent optimization adapts inputs automatically