Implicit Jailbreak Attacks via Cross-Modal Information Concealment on Vision-Language Models

2025-05-22

AI Summary

To address the vulnerability of multimodal large language models (MLLMs) to explicit jailbreak attacks—whose detectability increases due to cross-modal alignment enhancement—this paper proposes Implicit Jailbreak Attack (IJA), a novel implicit adversarial framework. IJA encodes malicious instructions into the least significant bits (LSBs) of benign images and leverages synergistic text-image prompting to trigger harmful responses, thereby circumventing explicit textual injection in the language modality. Methodologically, IJA introduces a cross-modal implicit attack paradigm integrating proxy-model-guided adversarial suffix generation, cross-modal prompt coupling, and iterative template-based feedback fine-tuning. Evaluated on commercial MLLMs—including GPT-4o and Gemini-1.5 Pro—IJA achieves over 90% attack success rate with an average of only three queries, significantly outperforming state-of-the-art approaches in both efficacy and query efficiency.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Multimodal large language models (MLLMs) enable powerful cross-modal reasoning capabilities. However, the expanded input space introduces new attack surfaces. Previous jailbreak attacks often inject malicious instructions from text into less aligned modalities, such as vision. As MLLMs increasingly incorporate cross-modal consistency and alignment mechanisms, such explicit attacks become easier to detect and block. In this work, we propose a novel implicit jailbreak framework termed IJA that stealthily embeds malicious instructions into images via least significant bit steganography and couples them with seemingly benign, image-related textual prompts. To further enhance attack effectiveness across diverse MLLMs, we incorporate adversarial suffixes generated by a surrogate model and introduce a template optimization module that iteratively refines both the prompt and embedding based on model feedback. On commercial models like GPT-4o and Gemini-1.5 Pro, our method achieves attack success rates of over 90% using an average of only 3 queries.

Problem

• Stealthy jailbreak attacks via hidden image instructions
• Bypassing cross-modal alignment in vision-language models
• High success rates on commercial MLLMs with minimal queries

Innovation

• Implicit jailbreak via steganography in images
• Adversarial suffixes from surrogate models
• Template optimization with iterative feedback