Understanding and Enhancing Encoder-based Adversarial Transferability against Large Vision-Language Models

2026-02-10

AI Summary

Existing encoder-based adversarial attacks exhibit poor transferability across large vision-language models (LVLMs), particularly under black-box settings. This work systematically identifies the root causes for the first time: inconsistent visual focus across models and redundant semantic alignment within individual models. To address these issues, we propose the Semantic-Guided Multimodal Attack (SGMA) framework, which leverages attention guidance to concentrate perturbations on visually critical regions while simultaneously disrupting cross-modal alignment. Extensive experiments across multiple mainstream LVLMs and tasks demonstrate that SGMA significantly improves the transfer success rate of adversarial examples, thereby exposing critical security vulnerabilities in real-world deployments of LVLMs.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Adversarial Agents

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Large vision-language models (LVLMs) have achieved impressive success across multimodal tasks, but their reliance on visual inputs exposes them to significant adversarial threats. Existing encoder-based attacks perturb the input image by optimizing solely on the vision encoder, rather than the entire LVLM, offering a computationally efficient alternative to end-to-end optimization. However, their transferability across different LVLM architectures in realistic black-box scenarios remains poorly understood. To address this gap, we present the first systematic study towards encoder-based adversarial transferability in LVLMs. Our contributions are threefold. First, through large-scale benchmarking over eight diverse LVLMs, we reveal that existing attacks exhibit severely limited transferability. Second, we perform in-depth analysis, disclosing two root causes that hinder the transferability: (1) inconsistent visual grounding across models, where different models focus their attention on distinct regions; (2) redundant semantic alignment within models, where a single object is dispersed across multiple overlapping token representations. Third, we propose Semantic-Guided Multimodal Attack (SGMA), a novel framework to enhance the transferability. Inspired by the discovered causes in our analysis, SGMA directs perturbations toward semantically critical regions and disrupts cross-modal grounding at both global and local levels. Extensive experiments across different victim models and tasks show that SGMA achieves higher transferability than existing attacks. These results expose critical security risks in LVLM deployment and underscore the urgent need for robust multimodal defenses.

Problem

• adversarial transferability
• large vision-language models
• encoder-based attacks
• black-box scenarios
• visual grounding

Innovation

• adversarial transferability
• vision-language models
• encoder-based attack
• semantic grounding
• multimodal alignment