FORCE: Transferable Visual Jailbreaking Attacks via Feature Over-Reliance CorrEction

2025-09-25

AI Summary

Visual jailbreaking attacks succeed readily on open-source multimodal large language models (MLLMs) but exhibit poor cross-model transferability, hindering effective vulnerability assessment of closed-source MLLMs. We identify that this limitation stems from excessive reliance on shallow, low-semantic-frequency features and high-curvature loss regions, impairing generalization. Method: We propose a feature-calibration-based transfer enhancement framework. It leverages intermediate-layer feature analysis and frequency-domain representation modeling to design a feature rescaling mechanism and a multi-layer feasible-region exploration strategy, steering attacks toward more robust, deeper, and semantically rich universal features. Results: Experiments demonstrate substantial improvements in cross-model transfer success rates of visual jailbreaking attacks on closed-source MLLMs. Our approach establishes a scalable, highly generalizable automated testing paradigm for visual red-teaming of closed-source multimodal systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The integration of new modalities enhances the capabilities of multimodal large language models (MLLMs) but also introduces additional vulnerabilities. In particular, simple visual jailbreaking attacks can manipulate open-source MLLMs more readily than sophisticated textual attacks. However, these underdeveloped attacks exhibit extremely limited cross-model transferability, failing to reliably identify vulnerabilities in closed-source MLLMs. In this work, we analyse the loss landscape of these jailbreaking attacks and find that the generated attacks tend to reside in high-sharpness regions, whose effectiveness is highly sensitive to even minor parameter changes during transfer. To further explain the high-sharpness localisations, we analyse their feature representations in both the intermediate layers and the spectral domain, revealing an improper reliance on narrow layer representations and semantically poor frequency components. Building on this, we propose a Feature Over-Reliance CorrEction (FORCE) method, which guides the attack to explore broader feasible regions across layer features and rescales the influence of frequency features according to their semantic content. By eliminating non-generalizable reliance on both layer and spectral features, our method discovers flattened feasible regions for visual jailbreaking attacks, thereby improving cross-model transferability. Extensive experiments demonstrate that our approach effectively facilitates visual red-teaming evaluations against closed-source MLLMs.

Problem

• Visual jailbreaking attacks lack cross-model transferability to closed-source MLLMs
• Generated attacks reside in high-sharpness regions sensitive to parameter changes
• Attacks exhibit improper reliance on narrow layer and poor frequency features

Innovation

• Corrects over-reliance on narrow layer features
• Rescales frequency features by semantic content
• Discovers flattened regions to improve transferability